Simulating consequences before acting: the World Model PAN 2

Actions described in text become video simulations with PAN 2. The World Model’s weights and code are released under Apache 2.0.

Given an observation, which may include an image, and actions described in text, the World Model PAN 2 generates a video up to 20 seconds long. Developed by the Institute of Foundation Models (IFM) at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), it is intended to explore the consequences of a decision before it is carried out. An agent could compare several simulated scenarios, revise its plan, and choose which action to take.

The demonstrations cover when a surfer stands up, how liquids behave in water, and whether objects persist after a camera movement. They illustrate the properties the team is pursuing: preserving the order of events, reproducing physical interactions, and maintaining a scene’s state when it moves out of view. The fidelity of these simulations is precisely what the published evaluations examine. Language prepares the next state, diffusion turns it into images

In the technical blog post on PAN 2, the Generative Latent Prediction (GLP) architecture divides the work between two components. A pretrained multimodal language model describes the next state of the world in a structured format. It specifies the actors, events, and their timing, separating camera movements from actions within the scene.

A 29-billion-parameter latent diffusion model, trained from scratch, then generates the visual details and movements left unspecified by that description. A decoder converts its internal representations into images. The 29 billion parameters therefore refer to the perceptual component, excluding the other models involved.

The contribution of textual reasoning is tested while keeping the diffusion component unchanged. According to the team, simply rewriting instructions into a structured format yields 27.8% full-plan completion, compared with 34% using full reasoning. This internal result measures the language model’s contribution in that configuration; it does not establish a general understanding of physical laws. What the Apache 2.0 release covers

The open weights and code announced by the lab are distributed under Apache 2.0 for PAN 2’s own components. Third-party components, including the Qwen3.5-9B text encoder and Wan 2.2 decoder, retain their respective licenses. The repository includes pretraining and supervised fine-tuning, but specifies that reward-based learning and the checkpoint combination process used to produce the final weights are not included in the published training code.

Running the model is documented across several environments, including Hugging Face Diffusers and vLLM-Omni. In the latter guide, an example reports approximately 119 seconds to generate 13 seconds of video on four NVIDIA B300 GPUs. This configuration produces a sequence computed in advance, with substantial hardware costs. Completing individual steps does not guarantee a coherent plan

The internal study comprises 100 everyday tasks, presented both with and without an initial image, each involving three to six steps to simulate over 20 seconds. Reported results reach 75.1% step completion, but only 34% of plans have every step completed. For MiniMax-H3, the corresponding figures are 75.1% and 32.5% in the same table. Verification uses automated visual and textual judges, supplemented by geometric estimation for camera movements.

The model card specifies that these plans were designed to succeed: the fidelity of scenarios intended to fail has not yet been measured. It also describes shortcuts that violate physical constraints and appearance drift across sequences. Actions expressed in text remain imprecise for fine movements, such as positioning each finger on an object.

A further experiment pairs a separate agent, based on Kimi K3, with a simulation of a tube containing a ball. One cap is glued shut, a fact provided to the World Model but hidden from the agent. After several attempts, including one misinterpreted simulation, the agent tries the other cap and retrieves the ball on its sixth attempt. Each attempt starts from the same initial image, while the agent retains the history of its plans and observations. Training agents at scale using such episodes remains a proposed next step.