Cosmos 3 Edge, Part 1: Understanding Physical AI on NVIDIA Jetson
NVIDIA Cosmos 3 Edge is an edge-optimized world model with approximately 4 billion parameters, combining multimodal understanding, image and video generation, and robot-action capabilities. This first article in CAVEDU's
NVIDIA Cosmos 3 Edge is an edge-optimized world model with approximately 4 billion parameters, combining multimodal understanding, image and video generation, and robot-action capabilities. This first article in CAVEDU's Cosmos series introduces the model, its Jetson deployment options, reported inference performance, and the path from visual input to decisions.
Read the series
- Model capabilities and a first look at Physical AI
- Workflows, action reasoning, and action chunks
- SO-101 sim-to-real and on-device robot control
These links open the original CAVEDU articles in Traditional Chinese.
From chat windows to Physical AI
AI is moving beyond conversations on a screen toward systems that perceive, understand, and interact with the physical world. Autonomous mobile robots, robot arms, and embodied agents need an onboard computer that can process observations with low latency.
NVIDIA Jetson provides a platform for this work, from Orin devices to Jetson Thor. The original article also discusses NVIDIA's announced Orin Nano 2 roadmap for 2027. See the announcement referenced in the original for its context rather than treating a future product as currently available.
Why put Physical AI on Jetson?
A robot working in the real world cannot always wait for a cloud round trip. Manipulating a moving object or responding to a pedestrian makes latency and connection reliability important system requirements.
Jetson brings inference closer to the sensors and actuators. Its software stack supports deploying open models on the device, while different hardware tiers serve different memory and compute requirements. The source article references JetPack 7.2 and 7.2.1; select software and containers that actually match your target module and installed release.
Local inference alone does not establish a safety guarantee. The model, controller, sensors, and timing must work together.
What does Cosmos 3 Edge add?
Cosmos is NVIDIA's world foundation model platform for understanding, simulating, and interacting with physical environments. Its applications include robotics, autonomous driving, and intelligent spaces.
Rather than only naming objects in an image, the goal is to model relationships involving time, space, motion, and actions. Cosmos 3 Edge brings these capabilities toward embedded deployments.
Architecture and serving options
Cosmos 3 Edge is the edge-oriented member of NVIDIA's omnimodal Cosmos 3 family. The source describes a 2.4B multimodal reasoner plus a diffusion-based generation module, totaling approximately 4B parameters. Its supported workflows include text, image, and video understanding; image and video generation; and chunked robot-action trajectories.
A single checkpoint supports two serving paths:
- vLLM: Text, image, and video reasoning through an OpenAI-compatible chat API. This path loads the 2.4B reasoner from the full checkpoint. Jetson AI Lab also describes the separately released Cosmos3-Edge-Reasoner.
- vLLM-Omni: Generation and robot-action workflows through its video API. Deploy this separately or alongside the reasoner as the workload requires.
The full checkpoint and the reasoner-only deployment do not have identical resource requirements. Consult the Jetson AI Lab model guide before selecting a configuration.
Main capabilities
- Multimodal reasoning: Text, image, and video understanding. The source reports vLLM serving on Jetson Thor and AGX Orin, plus a validated Hugging Face Transformers BF16 reasoner configuration on an 8 GB Orin Nano.
- Robot policies: Chunked action inference using vLLM-Omni, with a DROID-post-trained checkpoint.
- World generation: Text-to-video, image-to-video, and text-to-image generation through the diffusion component.
- Forward and inverse dynamics: Action-conditioned world modeling for Physical AI pipelines.
Reported reasoning performance
The original article reproduces these Jetson AI Lab figures:
| Module | Text (tokens/s) | Image (tokens/s) | Video (tokens/s) |
|---|---|---|---|
| Thor T5000 | 68.2 | 67.2 | 67.1 |
| Thor T4000 | 63.9 | 65.6 | 65.5 |
| AGX Orin 64 GB | 44.1 | 45.2 | 45.1 |
Reported conditions: vLLM streaming chat API, BF16, batch size 1, a maximum of 128 output tokens, and greedy decoding. The source notes that single-stream decoding is memory-bandwidth-limited, helping explain the similar Thor results. At eight concurrent streams, it reports aggregate throughput of approximately 293 tokens/s on Thor T5000 and 171 tokens/s on AGX Orin.
These are benchmark configurations, not guaranteed application performance or end-to-end robot-control rates.
Inputs, outputs, and intended uses
Inputs can include text prompts, images, short videos, and robot-view observations, optionally conditioned on actions. The vLLM path returns text; vLLM-Omni supports generated images, video, and action chunks. Chunk structure depends on the selected policy and configuration; consult its documentation rather than assuming that all endpoints share the same action dimensions or horizon.
Intended uses include edge multimodal reasoning for robots and intelligent spaces, robot policies on Jetson Thor, world simulation and prediction, and dynamics modeling.
From visual input to a decision
The Jetson AI Lab tutorial provides the deployment reference. Conceptually, a Physical AI inference pipeline involves four stages.
Step 1: Prepare the environment and load the model
Prepare a compatible Jetson and software stack. Prebuilt Docker containers can reduce dependency setup work, but the container and model still need to match the hardware, memory, and software version.
Resources: Jetson Thor developer kit and CAVEDU's JetPack installation tutorial.
Step 2: Provide multimodal observations
A camera supplies the robot's visual input. Frames or video observations can be fed to the model, with tools such as DeepStream supporting sensor-processing pipelines. A Live VLM WebUI provides another way to experiment with visual input.
Step 3: Reason about the scene
Consider a video in which someone pushes a coffee cup toward a table's edge. An object detector identifies a person, a table, and a cup. A world-understanding model aims to reason about the relationship and predict that the cup may fall. This is an illustrative scenario, not a measured successful interception demonstration.
Step 4: Connect the result to downstream decisions
An application can use tools such as the NVIDIA Agent Toolkit to organize decisions and downstream tool calls. Jetson Device Skills provides platform-oriented development workflows; it is not a replacement for the robot controller or a ready-made Cosmos-to-arm adapter.
A hypothetical decision might request moving an arm toward the cup. Real execution still requires robot-specific control, timing, and validation. As the original author notes, a small SO-ARM is slow enough that seeing, understanding, and actually catching a falling cup remains difficult. A Reachy Mini interaction is another possible direction for experimentation.
FAQ
How does Cosmos 3 Edge differ from Cosmos 3 Nano?
The source describes Edge as an approximately 4B model focused on edge deployment, multimodal reasoning, and robot policies, while the 16B Nano serves larger-resource reasoning and post-training workflows. Check the Cosmos model matrix for the current scope of each model.
Can an 8 GB Orin Nano run Cosmos 3 Edge?
Jetson AI Lab reports a Transformers BF16 multimodal reasoner configuration on Orin Nano 8 GB. That does not mean the entire 4B model's generation and robot-action features run on the same hardware in the same way.
Model-family reference from the original article
| Model | Parameters | Listed memory | Intended focus |
|---|---|---|---|
| Cosmos3 Edge | 4B | 32 GB | Omnimodal reasoning, generation, and actions |
| Cosmos3 Nano | 16B | 16 GB | NVFP4 reasoning on Thor |
| Cosmos Reason 2 2B | 2B | 8 GB | Lightweight visual-language reasoning |
| Cosmos Reason 2 8B | 8B | 18 GB | Higher-capacity visual-language reasoning |
| Cosmos Reason1 7B | 7B | 16 GB | Previous-generation physical-scene reasoning |
These figures refer to particular deployments in the source, not a universal minimum for every precision, context length, or serving mode. Refer to the corresponding Jetson AI Lab model guides for implementation details.
What comes next?
Getting the model running is the beginning. Edge devices can increasingly host the reasoning needed to understand observations and anticipate changes. The next article connects these capabilities to a Physical AI workflow: how do observations become action decisions, and how are those decisions updated as the world changes?
English translation and editorial clarification of CAVEDU's original article. Original author: CAVEDU 阿吉. Figures retain their source labels; some embedded diagrams contain Traditional Chinese. Product versions and benchmarks reflect the source article and its cited configurations.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.


