Source-linked AI summary
World in World: Explore the World with World Models
Chenxi Song, Yanming Yang, Chi Zhang
TL;DR
Flexible control of autoregressive video world models is difficult because source-video exploration must preserve event timing, target-view placement, newly exposed regions, and revisited appearance. World in World provides a training-free interface that presents heterogeneous evidence as clean states to a frozen model’s native self-attention, achieving the highest average score across seven VBench dimensions and the lowest camera trajectory errors among compared methods.
Problem
Flexible source-video exploration requires synchronized, spatially aligned, complete, and revisitable rollouts, while existing approaches often rely on task-specific modules or additional training.
Method
World in World converts source observations, target-view projections, rendered geometry, and generated history into camera- and time-labelled clean states, then localizes and regulates them through native self-attention.
Results
WiW achieves the highest average score across seven VBench dimensions and the lowest camera trajectory errors among compared methods on DAVIS and OpenVid-1M evaluations.
Takeaways & Limitations
Constructing, selecting, and regulating visual evidence can extend a pretrained world model to explore the dynamic world depicted in a given video.
Abstract
from arXiv · showhide
Autoregressive video world models enable interactive, long-horizon exploration, but flexible control remains challenging. Exploring a source video from new viewpoints requires the generated rollout to remain synchronised with the recorded event, place observed content in the requested view, plausibly complete newly exposed regions, and recover previously generated appearance on revisits. Existing methods typically address these requirements through task-specific modules or additional training. We present World in World, a training-free inference-time interface that converts heterogeneous control evidence into camera- and time-labelled clean visual states, which are read through the native self attention of a frozen causal video model. The evidence comprises source-video observations, target-view scene projections, geometry renderings that guide completion of newly exposed subject regions, and retrieved generated states beyond the rolling cache. Each evidence source carries token-level support and its own availability schedule. A correspondence router combines persistent point identities with geometry to establish token correspondences, guiding supported queries towards matching source-video tokens. Evidence-wise attention CFG (EWA) then independently regulates each auxiliary channel's additional contribution using attention responses from the same denoising forward pass. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer with the same frozen backbone. We evaluate World in World on camera-controlled video rerendering under diverse viewpoint changes, assessing perceptual quality, temporal consistency, and camera-following accuracy.
1. Introduction
World in World addresses flexible control of interactive video world models by converting heterogeneous visual evidence into clean states that a frozen model reads through native self-attention. Its training-free interface combines evidence construction, correspondence-guided localization, and independently regulated attention to support camera-controlled exploration of recorded dynamic worlds.
- 1. Introduction: Flexible exploration must preserve event synchronization, target-view placement, newly exposed content, and previously generated appearance during long-horizon revisits.These requirements arise because evidence differs in representation, spatial coverage, and temporal relevance as viewpoints and scene states change.
- 1. Introduction: Native self-attention provides a shared entry point because causal world models already process clean visual states associated with camera poses and temporal positions.WiW therefore frames control extension as visual evidence construction and orchestration rather than model adaptation.
- 1. Introduction: World in World introduces a training-free visual-evidence interface that extends frozen causal video world models without control-specific training or adaptation.The interface converts source observations, target-view projections, rendered geometry, and generated history into camera-, temporal-, and spatially annotated clean states.
- 1. Introduction: The method constructs complementary evidence so source observations supply recorded appearance, while additional representations address spatial alignment, completion, and history.Each evidence source contributes the information it can provide most reliably through the same clean-state interface.
- 1. Introduction: Correspondence-guided routing localizes relevant evidence, while evidence-wise attention control regulates how strongly each source influences generation.These mechanisms separately address where to read evidence and how much to use it under changing reliability.
- 1. Introduction: WiW supports navigation through a recorded dynamic world along new camera trajectories while preserving the original event’s appearance and temporal progression.The system is instantiated with all pretrained parameters frozen and evaluated for camera-controlled rerendering across diverse viewpoint changes.
2. Related work
Video world models support ongoing interactive exploration by predicting observations under changing camera positions, viewing directions, or actions. Related approaches use explicit scene representations and dedicated conditioning pathways, whereas World in World’s pipeline organizes multiple visual evidence sources for control.
- 2. Related work: Video world models predict observations while an observer moves through an environment under requested camera or action changes.Interactive systems continually predict subsequent observations from observed or generated visual states rather than producing only a finite preconditioned clip.
- 2. Related work: Prior camera-control methods encode trajectories through rays, poses, positional encodings, projected features, or rendered geometric proxies with corresponding conditioning modules.These approaches provide task-specific mechanisms for introducing camera control into generation.
- 2. Related work: Related systems can jointly train rendered control fields, sparse memories, and distilled causal students to combine multiple conditions.Different control types commonly use different input interfaces and training procedures.
- 2. Related work: The World in World pipeline combines target-view projections, rendered geometry, retrieved historical states, correspondence-guided routing, and evidence-wise attention regulation.Temporary evidence is removed after each chunk, while finalized outputs enter a rolling cache and archived features support later retrieval.
3. Method
World in World provides a unified, training-free interface that converts heterogeneous visual evidence into clean states readable through a frozen model’s native self-attention. It combines target-view projections, geometry renderings, generated-history retrieval, correspondence routing, and evidence-wise guidance for controllable rerendering.
- 3.1. Control through Visual Evidence: WiW converts source observations, target-view projections, rendered geometry, and generated history into clean visual states with camera, temporal, and spatial-validity information.These states are accessed through the frozen backbone’s native self-attention.
- 3.2. Target-View Scene Evidence: Target-view projections provide layout references by reorganizing source appearance according to the requested camera, visibility, and geometric reliability.Token-level support weights reduce the influence of uncertain projected regions.
- 3.3. Rendered Geometry Evidence: Rendered subject geometry supplies shape and appearance proposals for newly exposed surfaces that source-view projections cannot cover.The rendering is encoded through the shared interface with support masks and target-camera depth.
- 3.4. Generated History Evidence: Retrieved generated history restores earlier appearance and layout references when the native rolling cache has evicted states needed for long-horizon revisits.This extends the evidence interface beyond recently finalized states.
- 3.5. Attention Routing and Guidance: Correspondence-guided routing uses persistent point identities and camera geometry to direct queries toward matching source-video tokens.A positive correspondence weight is added to the routing attention score before joint normalization.
- 3.5. Attention Routing and Guidance: Evidence-wise attention CFG independently adjusts auxiliary evidence strength by amplifying complementary attention information without repeatedly amplifying the native response.EWA removes the projection along the native response direction before applying evidence-specific guidance.
4. Experiments
The experiments evaluate World in World on camera-controlled video rerendering and examine its shared visual-evidence interface across additional control applications and ablations.
- Experimental Setup: WiW is evaluated on camera-controlled video rerendering using DAVIS and OpenVid-1M, with VBench, camera trajectory errors, and image-quality metrics.Camera control is measured by TransError and RotError, averaged across trajectories estimated with Depth Anything 3 and ViPE.
- Additional Applications: The same frozen-model interface supports bullet-time rendering, stabilization, editing, K/V sharing between generation cases, and human-motion transfer.Model instances communicate by passing cached K/V as visual evidence.
- Camera-Controlled Rerendering: Qualitative comparisons use identical source videos and target camera paths to assess subject and scene consistency, viewpoint control, and newly exposed-region completion.Representative frames compare outputs under diverse camera motion.
- Ablation Studies: The full method achieves the lowest camera errors and best or tied-best results across reported VBench dimensions.Removing target-view warping raises rotation and translation errors to approximately 3.4× and 10.9× the full-method values.
5. Conclusion
World in World extends frozen causal video world models through a training-free visual-evidence interface that uses native self-attention. Evaluations report strong VBench and camera-control results, while qualitative applications and ablations illustrate the interface’s broader uses.
- Method: WiW represents source observations, target-view projections, rendered geometry, and generated history as camera- and time-labelled clean visual states.Correspondence-guided attention routing localizes source-video evidence, while evidence-wise attention CFG regulates auxiliary contributions without extra denoising-network evaluations.
- Applications: The same visual-evidence interface covers bullet-time rendering, video stabilization, video editing, K/V sharing, and human-motion transfer.These applications use cached visual information and the frozen backbone rather than task-specific training.
- Ablations: Ablations show that target-view warping and source-camera conditioning support cross-view structure, while CGAR, EWA, geometry, and historical retrieval address distinct evidence gaps.Historical retrieval preserves appearance and layout when earlier states are no longer in the rolling cache.
- Results: WiW achieves the highest average score across seven VBench dimensions and the lowest camera trajectory errors among compared methods on DAVIS and OpenVid-1M.Qualitative results also demonstrate downstream applications through the same interface.