Source-linked AI summary
World-in-World: World Models in a Closed-Loop World
Jiahan Zhang, Muqing Jiang, Nanru Dai, Taiming Lu, Arda Uzunoglu, Shunchi Zhang, Yana Wei, Jiahao Wang, Vishal M. Patel, Paul Pu Liang, Daniel Khashabi, Cheng Peng, Rama Chellappa, Tianmin Shu, Alan Yuille, Yilun Du, Jieneng Chen
TL;DR
Embodied world models lack unified evaluation of whether visually plausible predictions actually help agents complete tasks. World-in-World addresses this gap with a closed-loop benchmark, unified online planning, and a standardized action API, finding that task success depends more on controllability than visual quality and improves with post-training data and inference-time compute. The paper also identifies limited generalization and long-horizon planning as important remaining challenges.
Problem
Existing benchmarks largely emphasize open-loop visual quality, leaving whether world models improve embodied reasoning and task performance in closed-loop interaction unresolved.
Method
World-in-World provides a closed-loop benchmark with unified online planning, a standardized action API, four embodied tasks, and post-training using action-observation data.
Results
Visual quality alone does not guarantee task success; scaling action-observation post-training and inference-time compute improves closed-loop performance.
Takeaways & Limitations
World models should be judged by embodied task success and controllable, reliable interaction rather than visual realism alone.
Takeaways & Limitations
World models remain limited by poor generalization to unseen embodied environments and difficulty maintaining spatiotemporal history over long horizons.
Abstract
from arXiv · showhide
Generative world models (WMs) can now simulate worlds with striking visual realism, which naturally raises the question of whether they can endow embodied agents with predictive perception for decision making. Progress on this question has been limited by fragmented evaluation: most existing benchmarks adopt open-loop protocols that emphasize visual quality in isolation, leaving the core issue of embodied utility unresolved, i.e., do WMs actually help agents succeed at embodied tasks? To address this gap, we introduce World-in-World, the first open platform that benchmarks WMs in a closed-loop world that mirrors real agent-environment interactions. World-in-World provides a unified online planning strategy and a standardized action API, enabling heterogeneous WMs for decision making. We curate four closed-loop environments that rigorously evaluate diverse WMs, prioritize task success as the primary metric, and move beyond the common focus on visual quality; we also present the first data scaling law for world models in embodied settings. Our study uncovers three surprises: (1) visual quality alone does not guarantee task success, controllability matters more; (2) scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and (3) allocating more inference-time compute allows WMs to substantially improve closed-loop performance.
1. Introduction
World-in-World addresses the lack of closed-loop evaluation for embodied world models by combining standardized planning and action interfaces with task-focused benchmarking. Its findings show that visual quality alone does not ensure task success, while post-training and inference-time scaling improve practical performance.
- Motivation and contribution: World-in-World introduces a closed-loop benchmark that evaluates world models through embodied interaction rather than visual quality alone.The framework is designed to test whether models support perception, planning, execution, and replanning in embodied tasks.
- Motivation and contribution: A unified online planning strategy and standardized action API integrate heterogeneous world models into the same decision-making protocol.The planning strategy anticipates environmental changes and rewards before action commitment, while the API harmonizes model-specific control inputs.
- Findings: Visual quality does not necessarily translate into task success, making closed-loop performance the primary measure of practical utility.Figure 2 directly contrasts task success with generation quality, and the benchmark emphasizes embodied outcomes over isolated visual metrics.
- Findings: Scaling post-training with action-observation data is more effective than upgrading pretrained video generators, while additional inference-time compute improves closed-loop performance.The study reports both training-time data scaling and inference-time scaling as consistent performance trends.
2. World-in-World: a Closed-Loop Interface for Visual World Models
World-in-World evaluates visual world models through closed-loop embodied decision-making rather than visual generation alone. It provides a unified planning loop and action interface across diverse models and tasks.
- World-in-World benchmarks visual world models by coupling prediction, control, and reward during closed-loop embodied operation.The evaluation emphasizes practical utility for agents rather than generative quality in isolation.
- The unified strategy repeatedly proposes candidate plans, simulates counterfactual futures, revises them by predicted utility, and executes the selected decision.After execution, the resulting observation initiates the next proposal-simulation-revision cycle.
- The planning framework samples M candidate action sequences, transforms them through the API, and uses visual world-model rollouts to predict future observations.The proposal policy can be viewed as a beam search whose width is the number of candidate plans.
- A standardized action API converts action sequences into text prompts, camera trajectories, or low-level controls expected by different world models.This interface supports heterogeneous model input formats while preserving correspondence between agent actions and model controls.
- The benchmark covers Active Recognition, Active Embodied Question Answering, Image-Goal Navigation, and Robotic Manipulation across perception, navigation, and object-level control.Active Recognition includes targets under occlusion or extreme viewpoints and uses 551 episodes across 29 Matterport3D scenes in Habitat-Sim.
3. Evaluation Results and Analysis
World-in-World shows that closed-loop world models can improve embodied decision-making across perception and navigation, while manipulation remains challenging. Its analyses find that controllability, action-observation post-training, and inference-time computation are more consequential for task success than visual quality alone.
- Benchmark Results: Manipulation gains are modest: SVD† reaches 46.5% SR versus 44.5% for the VLM baseline, reflecting difficulty modeling contact-rich dynamics and action-conditioned object motion.The reported gap is much smaller than the improvements observed in perception and navigation.
- Ablation and Findings: Post-training improves embodied utility: Wan2.1† raises AR accuracy from 58.26% to 62.61% and ImageNav SR from 38.19% to 45.14%.SVD† also improves AR accuracy from 57.71% to 60.98% and ImageNav SR from 40.28% to 43.05%.
- Ablation and Findings: Controllability correlates positively with AR success rate, indicating that precise action-response alignment matters more for task success than visual quality alone.Controllability is measured as 1 − LPIPS between ground-truth and predicted observations, while generation quality combines aesthetic and image-quality predictors.
- Ablation and Findings: More action-observation data consistently improves post-trained models: Wan2.1† rises from 60.25% to 63.34% AR performance across the tested data range.Wan2.2† reaches nearly the same performance as Wan2.1† after 40K instances despite substantially larger web-video pretraining, while larger models benefit more and saturate less.
- Ablation and Findings: Increasing inference-time computation improves closed-loop performance: SVD† rises from 53.36% to 60.98% SR when average inferences per episode increase from 3 to 11.The scaling trend reflects the planner simulating more potential futures before choosing actions.
- Ablation and Findings: Panoramic input does not consistently outperform front-view input, likely because panorama-to-perspective conversion introduces resolution loss.The comparison evaluates post-trained models under both context formats.
4. Discussion and Future Directions
The paper identifies four practical constraints on world-model utility: generalization, long-horizon planning, interaction dynamics, and the strength of decision policies. It proposes corresponding directions including better action representations, memory, physics-aware modeling, and stronger policies.
- Generalization: World models may produce plausible but physically or semantically inconsistent rollouts in unseen environments, misleading planning and reducing success.Larger models or more pretraining data can partly help, but robust generalization remains unresolved.
- Generalization: Future work should improve transfer to novel environments through unified action representations and curriculum or domain-specific data collection.
- Long-horizon planning: Long-horizon planning remains difficult because visual world models have limited mechanisms for accumulating spatiotemporal history.Panoramic inputs produced inconsistent gains, motivating spatial and episode-level memory.
- Interaction dynamics: Current visual world models often miss contact, compliance, friction, and articulated-object state changes, causing physically invalid manipulation rollouts.Physics-guided generation and physical-property conditioning are proposed as remedies.
- Decision policies: Overall performance depends on both world-model fidelity and proposal and revision policies, so stronger base policies can raise the performance ceiling.Future work should improve policies and integration with decision-making.
5. Conclusion
World-in-World evaluates generative world models through embodied interaction rather than isolated visual metrics. Its unified controls and closed-loop planning expose gaps between visual realism and task success while showing benefits from post-training and inference-time scaling.
- Conclusion: World-in-World evaluates generative world models through embodied interaction across four demanding tasks using unified controls and closed-loop planning.The benchmark lets heterogeneous models serve as perception and planning utilities for embodied agents.
- Conclusion: The experiments reveal large gaps between visual metrics and task success, underscoring the need for closed-loop evaluation.The benchmark moves beyond isolated generation quality.
- Conclusion: Pretrained video generators improve with post-training data scaling and inference-time scaling.
- Conclusion: The benchmark aims to guide world models toward reliable perception, planning, and action in embodied scenarios alongside visual realism.
B. Embodied Task Details
World-in-World evaluates four embodied tasks and documents the policies and world-model configurations used across them. The task suite spans recognition, navigation, question answering, and manipulation.
- Task suite: The benchmark evaluates Active Recognition, Image-Goal Navigation, Active Embodied Question Answering, and Robotic Manipulation.
- Evaluation setup: The task-details section also describes the shared policies and summarizes the world-model configurations.
B.1. Active Recognition (AR)
The benchmark details closed-loop task settings in which world models augment recognition and navigation by predicting future observations for perception and action selection. It specifies episode budgets, candidate-plan evaluation, and task metrics across AR and ImageNav.
- Active Recognition: In Active Recognition, agents face 551 challenging episodes across 29 Matterport3D scenes, with extreme viewpoints or heavily occluded targets requiring active exploration.
- Active Recognition: AR allows at most K=10 decision steps and asks the agent to identify the target while selecting the next navigation primitive from panoramic and front-view observations.
- Active Recognition: The AR world model supplies synthetic future views for recognition and compares simulated candidate outcomes to choose informative actions.This supports both perception under occlusion and planning beyond greedy action selection.
- Active Recognition: AR reports Success Rate as the fraction of correct final labels and Mean Trajectory Length as the average number of executed actions.
- Image-Goal Navigation: ImageNav requires reaching a location shown in a reference image, using 144 episodes from 87 HM3D validation scenes.
- Image-Goal Navigation: In ImageNav, the world model predicts future observations for candidate action sequences and scores their informativeness for locating the goal.
- Image-Goal Navigation: ImageNav reports Success Rate, Mean Trajectory Length, and Success weighted by Path Length to capture success and path efficiency.
B.3. Active Embodied Question Answering (A-EQA)
A-EQA requires agents to explore actively before answering open-ended questions, using a two-level planner-controller policy and world-model rollouts to select rewarding exploration plans.
- Task setup: A-EQA evaluates open-ended question answering after active exploration across object recognition, attributes, states, localization, spatial reasoning, and functional reasoning.The evaluation set contains 184 questions across 54 indoor scenes.
- Task setup: The exploration budget is limited to 250 low-level actions, after which the episode ends unless the agent has already produced a final answer.A single decision step may contain multiple low-level actions.
- Policy: The two-level policy separates deliberation from control: a high-level planner issues textual instructions or landmark commands, while a low-level controller executes them.Landmark commands use depth and a custom pathfinder; textual instructions generate low-level action sequences.
- World-model integration: At each decision point, the planner proposes multiple action plans, queries the world model for predicted observations, and selects the plan with maximum estimated reward.This unified query supports both long-horizon exploration and deciding when to stop and answer.
- Metrics: A-EQA performance combines answer quality and exploration efficiency through Answering Score, Mean Trajectory Length, and SPL.SPL rewards accurate answers and efficient navigation; Mean Trajectory Length is lower-is-better.
B.5. Policies in Embodied Tasks
The framework distinguishes base, proposal, and revision policies, using candidate generation and rollout-based selection when a world model is enabled.
- Policy roles: A base policy acts without a world model, while world-model integration uses the same policy for proposal and generates a separate revision step to select among predicted rollouts.The revision policy evaluates candidate action sequences and chooses the one with the highest expected task reward.
- Perception and navigation policies: The framework uses VLM and heuristic base policies for AR and ImageNav, with Qwen2.5-VL-72B-Instruct-AWQ as the default VLM.The heuristic policy samples primitive actions using previous actions and handcrafted rules.
- Manipulation policies: Manipulation experiments compare a VLM policy with a 3D diffusion policy based on 3D Diffuser Actor.The diffusion policy is trained with task variants occurring at equal frequency.
- Policy robustness: Replacing the default VLM with InternVL3-78B-AWQ still leaves world-model integration consistently beneficial.The comparison is reported in Table 5 for InternVL3 variants with and without a world model.
B.6. World Models in Embodied Tasks
World models are integrated across embodied tasks through heterogeneous input and output formats, including perspective frames, panoramas, depth, and semantic maps.
- Output formats: Evaluated world models fall into perspective models that generate perspective frames and panoramic models that generate equirectangular panoramas.Examples include NWM, LTX-Video, and Wan2.1 among perspective models, and PathDreamer and SE3DS among panoramic models.
- Input formats: Panoramic models receive 576 × 1024 equirectangular panoramas, while perspective models generally use 480 × 480 front-view observations.Some models additionally require depth or per-pixel semantic labels.
- Input formats: Depth-aware models receive ground-truth Habitat depth, and PathDreamer additionally uses semantic labels predicted from the initial RGB-D frame.RedNet produces the per-pixel labels required by PathDreamer.
C. Post-Training Recipe for Embodied World Models
The post-training recipe adapts pretrained video generators into action-controllable world models by aligning actions with future observations across navigation and manipulation.
- Recipe overview: Post-training adapts an off-the-shelf video generator into an action-controllable world model for embodied tasks.The recipe formalizes the learning objective and action-observation alignment before specifying task-specific training setups.
- Action spaces: Navigation uses discrete primitives such as Forward, Turn-Left, Turn-Right, and Stop, whereas manipulation uses continuous 7-DoF end-effector poses.Navigation actions specify relative transformations between consecutive observations.
- Action-observation alignment: A special null action is prepended for navigation to preserve one-to-one frame-action alignment, while manipulation actions naturally align one-to-one with frames.The null action is used because no action precedes the first frame.
- Action conditioning: The unified API converts native action sequences into control signals expected by each pretrained generator, producing action-conditioned future-observation rollouts.The control interface may be a text prompt, camera trajectory, or low-level control sequence.
C.2. Post-Training Configuration
The study post-trains diverse world models on task-specific action-observation data, using standardized configurations and lightweight adaptation procedures.
- Post-Training Configuration: Habitat-Sim tasks fine-tune pretrained video generators on panoramic action-observation data to predict future frames from discrete navigation actions.The action space includes Forward, Turn_Left, Turn_Right, and Stop primitives.
- Post-Training Configuration: Post-training uses official pretrained weights and recommended implementations, with models adapted on corresponding datasets for one epoch.The standard configuration uses approximately 40K sampled instances for Habitat-Sim and manipulation tasks unless otherwise stated.
- Post-Training Configuration: The benchmark compares pretrained and post-trained world models across image-generation and video-generation families, including action-conditioned variants.The model inventory spans task-focused image generators, video generators, and post-trained action-conditioned models.
- Post-Training Configuration: The post-training stage is lightweight relative to full pretraining, using LoRA for 14B-parameter variants and full-weight fine-tuning for other models.The resource summary covers approximately 40K domain-specific clips per model.
D. Post-Training Dataset Construction
The post-training dataset is designed to provide diverse, physically plausible trajectories and aligned observations or actions for navigation and manipulation. Its sampling procedure favors spatially balanced coverage while the evaluation examples emphasize precise action control and task-oriented planning.
- D. Post-Training Dataset Construction: The dataset records manipulation demonstrations with front-view RGB observations aligned to corresponding 7-DoF end-effector pose sequences.The RLBench-based procedure produces 200 demonstrations per manipulation task, with approximately 150 observations per demonstration.
- D.1. Trajectory Sampling: The panoramic trajectory corpus targets diversity, physical plausibility, and manageability while avoiding paths through walls, redundant sampling, and excessive data.The construction uses a sampling procedure over scene meshes and navigable points.
- D.1. Trajectory Sampling: Waypoints are sampled across each scene, ranked by peripheral coverage, and filtered with a minimum spacing before shortest collision-free paths are generated between selected points.The procedure uses geodesic distances and Habitat path-finding to record panoramic RGB-D frames along each path.
- D.1. Trajectory Sampling: Dynamic waypoint reselection covers peripheral regions while avoiding redundant interior corridors, producing balanced and diverse viewpoints compared with random waypoint sampling.Figure 12 visualizes the final waypoints after radius-based pruning.
- D.2. World Model Rollouts: Forward-only rollout examples distinguish models that obey commands from those that drift or hallucinate, showing why precise action control matters for downstream planning.The prompts use imagined observations to evaluate action outcomes, choose navigation actions, and avoid states where the target is not visible.
- D.3. Task Prompts: The four task prompts cover Active Recognition, Image-Goal Navigation, Active Embedded Question Answering, and Robotic Manipulation, with planning objectives tailored to visibility, alignment, and recognition.The recognition and navigation prompts specify viewpoint selection and sequential action planning around target objects or goal images.