Cosmos 3
thinks, imagines
and acts.
An omnimodal world model for Physical AI: language, vision, audio and action in one shared framework.
3
The short version
Robots need a pile of models.
One model sees, another predicts video, another chooses actions. The hand-offs are slow, brittle and expensive to train.
Use one omnimodal backbone.
Cosmos 3 jointly models language, images, video, audio and action, then switches behaviour through its input-output token arrangement.
A shared world representation.
The same foundation can support reasoning, synthetic data generation, future simulation and robot policy post-training.
Why this matters for Physical AI
The key shift is from composing specialist models to training a shared model of world state, time and intervention.
From model stack
to world backbone
see + plan
choose action
simulate future
Useful components, but every boundary becomes a coordination problem.
language · image · video · audio
future visual / audio state
embodiment-aware action
Shared tokens let the model keep context across modalities and tasks.
unified sequence format
core modalities named in the paper
model scales: Edge, Nano, Super
downstream task configurations
One backbone,
many jobs
Multimodal reasoning
Interpret images and video, answer questions, infer scene semantics and reason about embodied tasks.
Synthetic training data
Generate diverse image, video and audio-visual scenes to increase coverage for Physical AI training.
Forward dynamics
Given an observed state and action, predict plausible future visual states for evaluation or planning.
Robot policy
Post-train on target data to jointly predict action and its expected visual consequence.
Control-guided video
Use edge or depth control video as conditioning and generate corresponding RGB video.
Audio-visual worlds
Generate video with synchronised audio, using a shared temporal axis across modalities.
The architecture
The trick is not simply “put all modalities in one transformer”. Cosmos 3 separates reasoning and generation while keeping them jointly conditioned.
Two token worlds
meet in attention
Language and understanding tokens use causal attention, preserving autoregressive integrity.
Continuous modality tokens use full attention and are denoised, conditioned by the reasoning context.
Action is a first-class modality
Cosmos 3 treats action as a causal variable: it explains how the world changes, not just what the world looks like.
Action → future
Given current video and clean actions, denoise the next visual state.
Future → action
Given observed visual transitions, infer the action tokens that explain them.
Action ↔ future
Jointly predict the intervention and its expected visual consequence.
camera / vehicle motion
hands / end effectors
fingertips / open-close
Time must mean
the same thing
Tokens run at different clocks.
Video, audio and action streams have different sampling rates. A one-step token shift cannot automatically mean the same physical duration.
video FPS base
audio tokens / sec
action FPS envelope
Absolute temporal modulation.
Cosmos 3 scales temporal position increments by the stream’s temporal resolution, aligning modalities on a shared physical axis.
Data, training
and evidence
The architecture is only half the story. The paper’s strategy is to build a data engine, then specialise the shared model without changing its bones.
Pre-train broadly.
Specialise late.
Reasoner pre-training
language + visual understanding
Generator pre-training
multimodal denoising
Mid-training
joint physical-action data
Post-training
text-to-image, video, policy
Five named families
PhyxSim, RobotSim, DriveSim, SynHuman and Warehouse datasets broaden physical interactions and environments.
JSON as control surface
Captions encode layout, lighting, actions, camera motion, state changes and audio cues.
Intent → scene programme
A reasoner expands a short request into a schema-constrained multimodal specification.
What the paper reports
The table below reproduces selected values from the paper’s overview. These are the authors’ benchmark results, not an independent reproduction.
Text-to-image*
Image-to-video
Robot policy*
* denotes a post-trained Cosmos 3 variant in the paper’s table.
In robot forward dynamics, Cosmos3-Nano and Super post-trained from the mid-trained checkpoint reached 25.52 dB and 26.04 dB PSNR, compared with 22.99 dB for Ctrl-World.
The result supports the value of unified action mid-training for downstream adaptation. It doesn’t establish universal superiority across all environments.
What it proves —
and what it doesn’t
Broad capability coverage
Cosmos 3 supports a wide range of reasoning, generation, simulation and action modes under one framework.
Shared representations may reduce hand-off friction
This follows from the architecture, but the paper doesn’t provide a full systems-level cost or reliability comparison against every modular stack.
Real-world closed-loop robustness
The report shows benchmark and post-training results. Safe deployment still needs task-specific evaluation, calibration and failure handling in the target environment.
Implementation guide
The practical lesson is to treat Cosmos 3 as a foundation plus an evaluation and data programme, not as a drop-in robot brain.
A sensible adoption path
State → action → outcome
Choose one bounded workflow. Specify the observation stream, action interface, prediction horizon and success metric.
Build a replayable testbed
Use logged trajectories or synthetic scenes. Keep seeds, prompts, model versions and environment settings inspectable.
Align clocks and coordinate frames
Record FPS, action frequency, temporal compression, sensor latency and coordinate conventions. Don’t hide these in preprocessing.
Specialise without forking the world model
Fine-tune on the target embodiment and task, while preserving a shared representation and a held-out generalisation set.
Ask “what happens if?”
Test action consistency, temporal coherence, state changes, object identity, safety constraints and recovery after mistakes.
Close the loop slowly
Move from offline replay to simulation, shadow mode, low-risk interventions and supervised deployment with rollback.
The useful mental model
Cosmos 3 is a bridge between synthetic worlds and real-world agents.
Its distinctive bet is that reasoning, generation and action become more useful when trained as related views of the same evolving world.
If you’re building with it:
Deck note: all benchmark figures and technical descriptions are attributed to the supplied paper. Inference and implementation advice are labelled as such.
Built as a self-contained HTML deck. Use ← / →, the buttons, or scroll. Press Home / End to jump.