World Models: The Simulation Strikes Back

What even is a “World Model”?

You’ve probably heard of world models in release articles from labs. Perhaps even overheard a conversation in a cafe saying, “Yeah, I’m training world models now at insert-large-lab”. But what is a world model, and why should you know about them?

The answer is, no one knows. You’re not alone.

There have been many differing meanings associated with it. Some of them include:

• a video generation model with memory
• a state-machine that predicts next states in unknown environments and provide action predictions
• Large-enough Gaussian Splat generation models
• a general model that simulates any possible scenario given some past states, and output any modality the user requests

None of them is wrong.

Based on them, let’s try to make a set of fundamental criteria, based on which these definitions seem to work. A system that

• uses coherent past state and action information
• works for a given possible environment
• can predict any number of plausible representations
• can predict in the future, continually

Consider this a humble attempt to simplify things and reach a consensus. Complex? Yes. Gibberish? No. Let’s break it down.

Uses coherent past state and action information

This refers to using one or more past representations of our environment. This could be any modality, be it image(s), video(s), or even text descriptions, along with any actions taken in the environment until that point. The model depends on us to ensure it is coherent in nature, because it will interpret physics based on that. If we provide 5 images of a ball falling from a table, but pass them in random order, the model will be confused. On the other hand, if we provide it in the correct order, the model should be able to simulate what happens next.

Works for a given possible environment

Impossible environments are fun to think about, but they are inevitably outside what models are generally trained on. As long as we don’t have a general model of the universe, we probably have to stick to plausible worlds only. This doesn’t mean you can’t have creative worlds, but we can only expect them to be accurate if the physics in such worlds works somehow relatively to the real world, at least.

Can predict any number of plausible representations

Once the model has information about past states, and assuming that it follows physical laws, it can now predict what comes next. This would be independent of modality, i.e. the outputs could be images, videos, actions (for robotics), rewards, or any other possible representation (“arbitrary”). A model can learn real-world physics or custom rules, such as physics off of Star Trek. It is still a good 3D world model if it applies those rules consistently across environments.

Can predict in the future, continually

We don’t want to stop with just a single time-step’s prediction, we want to go deeper! To do so, the model must keep generating (either autoregressive or diffusion, or some other paradigm!) future states that do not break continuity. This would also look like continually predicting the transition dynamics from one state to the next.

But even my video model can do that, no?

Perhaps, yes. A sufficiently good video model can do some of what we just described. But most models will use external tools or additions (think LoRAs) to perform those tasks. It will not be a true world model by our definition, since the model by itself does not “think about” those inherent representations. If we can understand how world models reason through mechanistic interpretability, we should first determine whether they internally represent physical rules or only imitate patterns learned from data.

The best example is the famous Chinese Room test. Assume a person sitting inside a room who speaks English. This person has no idea how to speak Mandarin Chinese but has a rule set that tells them how to translate and reply. Now, people outside the room can send messages in Mandarin to the person inside the room, and get replies back in Mandarin too. So, people outside the room can assume that the person inside the room speaks fluent Chinese, even though the person cannot.

This directly relates to what distinguishes World Models and current models (be it Video models, sufficiently strong Omni models, and so on). How can we tell whether a 3D world model only mimics physics, or actually learns physical laws and applies them to new, unseen environments?

Children can do this. They learn that a ball will fall when dropped from a height, and understand that gravity (a word they learn later) will always pull things towards the ground. So if they drop another object, it would fall too. If a model understands this law, we can sufficiently assume that it can apply such laws independently of the environment it is dealing with. This would be a good example of a world model!

Causality as a requirement

This brings us to discuss causality as a key requirement for world models. A strong ability to predict consequences can also support memory. For example, if I make a painting with 20 strokes of a brush, I don’t need to remember what that painting looks like as long as I can recreate those exact same strokes in the right order. That is why causality becomes a defining feature of world models. Let’s go deeper.

Defining Causality

What is causality for us? One event drives another consequentially. That’s all. The outcome of the first event will influence what the second event looks like. Using the painting example again, painting a straight line on a canvas should leave a straight line mark on the canvas.

Memory is still mathematically necessary

If the model can easily reconstruct earlier world states, it can track changes and predict next states during inference without relying on memory. This lets a standalone world model understand time and change without external modules. But this would only hold in a deterministic world where every action has a single outcome. For every other environment, memory is still important.

Memory is essential when the environment is only partly observable or when the same action can lead to different outcomes. The model needs an accurate recall of past states and actions to predict what may happen next. When outcomes are uncertain, it should predict a probability distribution over possible next states.

In deterministic worlds, memory can speed up inference by caching actions and their outcomes. However, the model can still reconstruct the world from past states and actions alone, using causal rules, without stored memory.

Current landscape of World Models

There are many models out there today that come close to fitting our definition. Let’s take a quick look at some of them and see where they shine.

Lightricks LTX-2.5

The latest model from Lightricks, it is a world model that can work with images, audio, and video to simulate worlds on local hardware. It is highly customizable, supports the generation of 4K HDR visuals, RAW workflows, and also supports multi-shot generation within its videos.

import torch
from diffusers import LTX2Pipeline
from diffusers.pipelines.ltx2.utils import DEFAULT_NEGATIVE_PROMPT, DISTILLED_SIGMA_VALUES
from diffusers.utils import encode_video

MODEL_ID = "Lightricks/LTX-2.5-Diffusers"

pipe = LTX2Pipeline.from_pretrained(MODEL_ID, dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

video, audio = pipe(
    prompt="A cinematic shot of a red fox walking through a snowy forest at dawn, "
           "the camera tracking alongside, snow crunching underfoot.",
    negative_prompt=DEFAULT_NEGATIVE_PROMPT,
    width=960,
    height=544,
    num_frames=121,
    frame_rate=24.0,
    sigmas=DISTILLED_SIGMA_VALUES,
    guidance_scale=1.0,
    audio_guidance_scale=1.0,
    stg_scale=0.0,
    audio_stg_scale=0.0,
    modality_scale=1.0,
    audio_modality_scale=1.0,
    generator=torch.Generator("cuda").manual_seed(42),
    output_type="np",
    return_dict=False,
)

encode_video(
    video[0],
    fps=24,
    output_path="ltx25.mp4",
    audio=audio[0].float().cpu(),
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
)

NVIDIA Cosmos 3

Built by NVIDIA, it is an “omni” model that supports various output modalities and is especially tuned for generating data for robotics training (through action prediction and video generation). The model is especially suited for generating long-tail scenarios (which don’t always occur in training data) and can be useful for synthetic data generation tasks for training downstream models as well.

import json
import torch
from diffusers import Cosmos3OmniPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
from diffusers.utils import export_to_video

# JSON-upsampled positive and negative prompts (see "Prompt upsampling" above).
json_prompt = json.load(open("assets/example_t2v_prompt.json"))
negative_prompt = json.load(open("assets/negative_prompt.json"))

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano", dtype=torch.bfloat16, device_map="cuda"
)
pipe.scheduler = UniPCMultistepScheduler.from_config(
    pipe.scheduler.config, flow_shift=10.0, use_karras_sigmas=False
)

result = pipe(
    prompt=json.dumps(json_prompt),
    negative_prompt=json.dumps(negative_prompt),
    num_frames=189,
    height=720,
    width=1280,
    num_inference_steps=35,
    guidance_scale=6.0,
    fps=24.0,
)
# macro_block_size=1 allows arbitrary frame sizes (Cosmos3 outputs are not always divisible by 16).
export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)

HunyuanWorld 2.0

This model builds on earlier works from the same team. It takes multimodal input, generates an initial panoramic image for conditioning, plans the scene, produces videos along the planned camera trajectories, and synthesizes the final 3D world as a mesh or 3D Gaussian Splatting (3DGS) representation. It is a powerful but computationally heavy model that can generate large, explorable 3D worlds for users to navigate and interact with.

Black Forest Labs Flux 3 Action

This model is the latest in the Flux 3 series, bringing action-prediction to the table. It is a 7B World Model meant to be used for various robotics tasks. It is pretrained jointly on video, audio, and image data together. It takes camera observations, the robot’s current state, and a natural-language task instruction, then predicts the next chunk of robot actions while jointly denoising/predicting future video frames. It ends up punching above its weight, getting state-of-the-art results on robotics benchmarks against bigger models.

World Labs Marble

Marble is a multimodal input model that can generate 3D worlds in a Gaussian Splat, Video, or Mesh format. It can also combine multiple 3D worlds, mixing and matching them into something unique. It is one of the best models out there for full-scale world generation, especially with strong creative direction.

import os, time, requests

API = "https://api.worldlabs.ai/marble/v1"
HEADERS = {"WLT-Api-Key": os.environ["WLT_API_KEY"]}

op = requests.post(
    f"{API}/worlds:generate",
    headers=HEADERS,
    json={
        "model": "Marble 0.1-mini",
        "world_prompt": {
            "type": "text",
            "text_prompt": "A futuristic city at night"
        },
    },
).json()

while not op["done"]:
    time.sleep(5)
    op = requests.get(f"{API}/operations/{op['operation_id']}", headers=HEADERS).json()

world = requests.get(
    f"{API}/worlds/{op['response']['world_id']}",
    headers=HEADERS,
).json()

print(world["world_marble_url"])

World Labs Atlas

A successor of Marble, Atlas is a world model trained with spatial intelligence as a first-class goal. It can take camera trajectories and an initial-condition frame to generate video or Gaussian Splat-based worlds. It is objectively better than Marble at 3D reconstruction, among other tasks.

How to evaluate a World Model?

We took a look at some of the most popular world models out there, but how do you understand their nuances, and decide what works best for you? Let’s now discuss some axes of evals we can compare these models against, and what they mean in the context of world models.

Action

Some world models can emit actions as outputs. This generally includes actions of how an embodied object should manipulate a given environment to maximize reward. Evaluating actions can be tricky, because there may be no single answer to the right action as long as the requested outcome is the same. I could paint a circle clockwise or counterclockwise, yet the painting will be identical with no difference at all. The environment can also use a reward function to evaluate the model’s actions and their effects on the 3D world.

Memory

As discussed above, memory is a must-have feature of any world model architecture. This can be built into the model, handled with external tools, or approximated by using causal simulation as a form of memory. The goal here is to understand whether the model can remember the changes the user makes in the environment over time. For example, if I point my world model’s camera at a wall, paint a circle on it, move my camera away, and come back to the wall, I should still be able to see my painted circle on the wall (example taken from Genie 3’s official blog). Here in our example, the action is to paint the wall and the ability of the model to remember the painting is what we would call memory.

Physical Coherence

Coherence is a core evaluation metric for world models. It requires generated results to follow real-world physics or the defined physics of the target world. Following custom physics also tests the model’s level of control. This could be as simple as making sure you do not place a car upside down on the road when simulating an autonomous-driving situation, or depicting motion of human joints in unnatural ways when trying to simulate pose.

Control

Worlds are never simple. They are diverse, they can have a ton of objects in them or no objects at all. The variance in their complexity is challenging, and that is a way we can evaluate what the breaking point for models are as of now. If a model can create a 3D scene from detailed instructions, build object hierarchies, and make objects interact correctly, it demonstrates spatial and physical understanding.

Similarly, if the model can simulate object movement and adjust it based on user input, the model demonstrates temporal control. These are not the only forms of control, though. A strong world model should also adapt its physical rules based on user input. For example, a good model would be able to simulate the motion of a ball falling on Earth but also on Mars, given it knows the proportional difference between gravity on Earth and on Mars. This becomes an integral control parameter when making ultra-creative worlds that diverge from reality (perhaps something like Star Trek? Or as Captain Kirk would say, “To boldly go where no model has gone before”).

Continuity

Lastly, a world model should also be able to continually generate its outputs given the available compute. If it’s a video model for example, it should be able to keep generating endless long-form videos without breaking consistency for the world it is generating. This is important, because most models traditionally fail on this axis with respect to quality degradation and/or issues with memory for long time horizons.

Where to go from here?

This space is readily evolving, and there are multiple ways this could go. At this point, all of us would benefit from reading more and informing our opinions as they evolve. To begin with, some of the best resources for doing so would include

• The Functional Taxonomy of World Models by Fei-Fei Li talks about the 3 distinct categories of World Models out there right now
• The World Model and Spatial Intelligence Era is a well-written policy brief from Stanford’s Human-Centered Artificial Intelligence Lab that talks about how World Models would benefit the economy, existing policy around them, and how they benefit the world around us from a human perspective
• What is a World Model? is a glossary piece from NVIDIA that talks at 10,000 feet about the technical details that go into making a World Model, including how they are trained, what are the pieces of such a model, and how one can use them
• Stanford CS248A Lecture 18 is a lecture from Stanford’s Computer Graphics course that dives specifically into World Models. The associated homework for this lecture also instructs how to train a world model of our own.

This blog is only the beginning of a journey that the community needs to take together in defining a single goalpost to say what constitutes a world model, and tries to kickstart that conversation. I sincerely thank you for following along, and hope to see you in the next one! 👋

Bonus: Are pixels even necessary in World Models?

Pixels are easy for humans to understand and make sense of, but a model that can work with enough information to represent an environment may not actually need any visual representation or pixels. For example, if my world model is trying to simulate a game of chess and is trying to predict board states as the opponent makes moves, it may not need anything more than the game state which can be represented in concise forms (like FEN, or Algebraic notations). Such a representation reduces the “cognitive load” of the model in trying to extract information from the visual and then apply its logic to it, and instead simplifies the task for the model to play with. So perhaps pixels are not always necessary, but they sure are convenient! :-)