Renderers, Simulators, Planners — and the loop that connects them all.
Computer vision, robotics, reinforcement learning, and generative AI all claim to be building world models. Each means something quite different.
The POMDP loop — formalised by Sutton & Barto — is where the term "world model" got its technical meaning. Traced to Kenneth Craik's 1943 proposal that minds reason by running "small-scale models" of reality.
Outputs observations — pixels for human eyes. A video model turning a prompt into a drone shot is a renderer. Quality that matters: visual fidelity. Constraint: no explicit 3D understanding; beautiful output can be physically impossible.
Examples: text-to-video, Genie 3, World Labs RTFM
Outputs state — geometrically, physically faithful representations that both humans and computer programs can compute on. Serves architects, filmmakers, RL agents, robot controllers, and AV testing. The linchpin.
Examples: World Labs Marble, NVIDIA Omniverse
Outputs actions. Given an observation and a goal, a planner answers "what should the agent do next?" — the inverse of the renderer. VLA models and World Action Models are planners closing the perception-action loop.
Examples: Vision-Language-Action models, model-based RL systems
All three draw on the same underlying knowledge: geometry, physics, dynamics. The categories describe what they output, not what they know.
The three functions are at wildly different stages of commercial and technical maturity.
Image- and text-to-video products are expanding rapidly in consumer and enterprise markets. Google's Nano Banana model has put renderer-quality generation in the hands of hundreds of millions. The ceiling: visual plausibility, not physical accuracy.
Gets the least public attention but is the most consequential. Geometry, physics, and dynamics are the world itself. NVIDIA Omniverse alone targets an estimated trillion-dollar addressable market. The hardest open problems live here.
The most intriguing and most nascent. Robotic demos in the last two years look impressive — but almost all are confined to heavily constrained lab setups. The gap between a demo reel and a robot that reliably works in a kitchen remains vast.
A model mastering simulation can project into pixels (for renderers) and into action predictions (for planners). A model mastering only rendering, or only planning, cannot do either. The bridge is the point of leverage.
3D data with explicit geometry, material properties, and physical annotations is orders of magnitude scarcer than internet video. Renderers are data-rich; simulators and planners are not.
The difference between how things behave in simulation versus reality remains a core, unsolved problem. AI-generated geometry can look correct while containing self-intersections or wrong scale that produce nonsensical physics.
Rigid bodies, deformable objects, fluids, and cloth interacting simultaneously remains orders of magnitude more expensive than single-domain simulation.
Robotics training, AV testing, architectural visualisation, engineering simulation, and drug discovery all depend on something simulation-shaped. NVIDIA Omniverse alone targets >$1T addressable market.
Takes multimodal prompts — text, image, video, or spatial sketch — and generates explorable 3D environments. Outputs Gaussian splats for visual exploration alongside collision meshes a physics engine can operate on.
A single model outputting both Gaussian splats (renderer territory) and collision meshes (simulator territory) is a proof of concept that the categories can be unified. Marble is only the first chapter of a much longer arc.
The lines between rendering, simulation, and planning are beginning to collapse — Marble is an early signal of that convergence.
The shared insight: the knowledge required to render a world, simulate it, and act in it is largely the same. A model that truly understands a cup on a table should render it, simulate a push, and plan a hand to pick it up.
A sufficiently rich model of the world is all that any agent needs — to see worlds, build them, and act in them. What gives this "big bet" weight now is that three threads, each already driving multi-billion-dollar industries on its own, are starting to behave like one.
The defining open problem: unifying rendering, simulation, and planning inside one architecture.
The data picture is uneven — renderers are awash in internet video while simulators face 3D asset shortages. Optimising for visual beauty can sacrifice the precision a robot needs. Reconciling these tensions is the central research challenge.
Source: A Functional Taxonomy of World Models, Fei-Fei Li & World Labs, June 2026