Dev.to AI 🤖 Ai 👁 0 📖 6 min read

World models: what they are and the video they learn from

A world model is an AI system that predicts how a scene will change, including how it responds to an action such as a step, a steering input or a robot's grip. Labs train them first on very large amounts of ordinary vide

A world model is an AI system that predicts how a scene will change, including how it responds to an action such as a step, a steering input or a robot's grip. Labs train them first on very large amounts of ordinary video, then on much smaller sets of video paired with actions: robot teleoperation logs, first-person footage with hand tracking, or driving video with vehicle data. Plain video is plentiful. Video of skilled, hands-on work from many real sites, ideally with actions recorded, is the scarce part.

What is a world model?

It is a model that simulates part of the world well enough for an agent to plan or practice in it. Google DeepMind describes world models as systems that "use their understanding of the world to simulate aspects of it" (Genie 3 announcement), so an agent can predict both how an environment evolves and how its own actions change it.

The difference from a video generator is interaction. A video generator renders a whole clip from a prompt. A world model produces the next frame given what has happened so far and what the agent just did. Genie 3, announced in August 2025, generates navigable worlds at 24 frames per second in 720p and stays consistent for a few minutes. DeepMind lists its limits plainly: a narrow range of actions the agent can take directly, and interaction measured in minutes rather than hours.

NVIDIA frames the same idea for robots and vehicles. Its Cosmos paper argues that physical AI needs a digital twin of the world to train in before it acts in the real one (Cosmos, arXiv).

What video do world models learn from?

Mostly unlabeled video at enormous scale, then a much smaller amount of footage where the actions are known. The labs' own papers show the pattern:

Model or dataset Who published it What it trained on or contains Actions recorded?
Genie (2024) Google DeepMind About 244,000 hours of 2D platformer gameplay video, filtered to 30,000 hours No; it infers "latent actions"
V-JEPA 2 (2025) Meta Over 1 million hours of internet video, then under 62 hours of robot video from DROID Only in the robot stage
Cosmos world foundation models (2025) NVIDIA 9,000 trillion tokens, including 20 million hours of data from driving, robotics and synthetic environments Varies by source
DROID (2024) Academic robotics team 350 hours of teleoperated robot manipulation, 76,000 trajectories, 564 scenes Yes, robot commands
EgoDex (2025) Apple 829 hours of first-person video with 3D hand and finger tracking, 194 tabletop tasks Hand and finger poses
Ego4D (2021) Research consortium 3,670 hours of daily-life video from 931 camera wearers in 74 locations No
Egocentric-10K (2025) Build AI 10,000 hours of head-mounted video from 85 real factories, no audio No; clip metadata only

Sources: Genie, V-JEPA 2, NVIDIA's Cosmos developer post, DROID, EgoDex, Ego4D, Egocentric-10K.

Why do action labels matter so much?

Because they connect what the camera sees to what caused it. Without them a model learns how scenes tend to change; with them it learns what happens when you do a particular thing, which is what a robot needs in order to plan.

The V-JEPA 2 paper is the clearest example. Meta pre-trained on over a million hours of internet video with no actions at all, then post-trained an action-conditioned version on under 62 hours of robot video. That version picked and placed objects on robot arms in two labs it had never seen, with no data collected in those environments. A huge amount of passive video did most of the work, and a small amount of action data made it usable for control.

Genie took the other route: it learned a set of latent actions from gameplay video alone, with no labels. That works well for games, where a small set of controls drives most changes. Physical work has far more ways to act, which is why labs still pay for recorded actions.

Price reports reflect the gap. One industry estimate cited by Forbes puts human video at $2 to $5 an hour and teleoperation data at $50 to $200 an hour. Maeil Business Newspaper reported in August 2026 that first-person work video traded at more than $7 an hour unprocessed and $15 to $30 an hour with each action labeled.

What are teleoperation and first-person video?

They are the two main ways labs capture action-rich footage. In teleoperation, a person drives a robot (often with a VR headset or a matching set of arms) while the robot records its cameras and every command. In first-person (egocentric) video, a worker wears a head or chest camera that keeps the hands in frame.

Teleoperation gives exact robot actions but needs the robot on site and an operator for every hour. First-person video is cheap and varied, and with tracking hardware like the Apple Vision Pro used for EgoDex it can carry precise hand poses too. Our explainer on how robots learn from video of people working compares these methods with motion capture and simulation.

What does this mean for companies holding work footage?

Footage of real work from many sites is the kind labs cannot film at scale themselves, and world-model developers have started paying for it. NATIX, a driving-camera network, says it signed a six-figure agreement to supply 3,000 hours of real-world driving video to a lab building general world models (NATIX, October 2026). A Machine Economy briefing noted that the claim is the seller's own and that no independent check of the price or delivery has been published.

If your company runs helmet cameras, body cameras, line cameras or fleet dashcams, three things decide whether that footage fits:

  1. Point of view. First-person footage with hands and tools in frame is closest to what manipulation research uses. Fixed and road-facing cameras suit activity and driving models.
  2. Variety. Many sites, seasons and tasks beat many hours of one bay. The DROID paper leads with its 564 scenes and 50 collectors on three continents, not only its hours.
  3. Anything recorded alongside the video. Timestamps, vehicle speed and steering, or machine states logged at the same time are, in our view, the closest thing most companies have to action labels.

For format, Build AI's public card for Egocentric-10K is a useful reference: 1080p at 30 frames per second, a head-mounted fisheye camera, calibrated camera parameters for each worker, and no audio. Then the legal side: blur faces, strip audio, and check what staff and customers were told about the cameras. Our guide to selling video footage to physical AI companies covers the privacy checks and the prices reported so far.

Common questions

Is a world model the same as a video generation model?
They share architecture but differ in use. A video generator produces a finished clip. A world model generates frame by frame in response to actions, so an agent can act inside it. DeepMind notes that generating this way is harder than generating a whole video, because small errors pile up over time.

Can security camera footage train a world model?
Some of it can, for activity and safety models, but fixed cameras rarely show hands at the working distance manipulation research needs. Security footage is also the riskiest to license, because it shows identifiable customers and staff, and biometric and recording laws apply to it.

How many hours does a buyer need?
No buyer publishes a minimum. DROID's 350 hours of robot data was described as large-scale, and NATIX's reported deal covered 3,000 hours. For a business, variety and labels usually matter more than a raw total, so describe sites and task types alongside hours.

Do I need to label my footage before talking to a buyer?
No. A description at company level, with a few sample clips, is enough for a buyer to say whether it fits. Labels raise the per-hour price in the reports above, but labeling is costly, so agree the labeling scheme with a buyer before you pay for it.

If you hold footage of real work and want a rough sense of its value, our free data value estimate takes six questions and needs no files.

Origin Data Partners is an independent data partner that helps companies get their data bought. We are independent of the companies named here, and a buyer may pay Origin when a deal closes.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.