Source-linked AI summary
Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory
Ruiqi Wu, Xuanhua He, Meng Cheng, Tianyu Yang, Yong Zhang, Zhuoliang Kang, Xunliang Cai, Xiaoming Wei, Chunle Guo, Chongyi Li, Ming-Ming Cheng
TL;DR
Real-world world-model training is limited by noisy pose estimates and scarce viewpoint revisits, while long-context memory must remain computationally manageable. Infinite-World addresses these issues with hierarchical pose-free compression, uncertainty-aware action labeling, and revisit-dense finetuning. The paper reports coherent 1000-frame simulation and superior visual quality, action controllability, and memory consistency.
Problem
Real-world world-model training lacks an effective paradigm because pose estimations are noisy and viewpoint revisits are scarce.
Method
Infinite-World combines HPMC for fixed-budget pose-free history compression with uncertainty-aware action labeling and revisit-dense finetuning.
Results
Infinite-World achieves coherent simulation over a 1000-frame horizon with superior visual quality, action controllability, and memory consistency.
Takeaways & Limitations
The framework provides an interactive world model that learns long-range memory and action response from noisy real-world data.
Takeaways & Limitations
Further work is needed to mitigate cumulative drift and visual degradation and to improve inference speed and visual fidelity.
Abstract
from arXiv · showhide
We propose Infinite-World, a robust interactive world model capable of maintaining coherent visual memory over 1000+ frames in complex real-world environments. While existing world models can be efficiently optimized on synthetic data with perfect ground-truth, they lack an effective training paradigm for real-world videos due to noisy pose estimations and the scarcity of viewpoint revisits. To bridge this gap, we first introduce a Hierarchical Pose-free Memory Compressor (HPMC) that recursively distills historical latents into a fixed-budget representation. By jointly optimizing the compressor with the generative backbone, HPMC enables the model to autonomously anchor generations in the distant past with bounded computational cost, eliminating the need for explicit geometric priors. Second, we propose an Uncertainty-aware Action Labeling module that discretizes continuous motion into a tri-state logic. This strategy maximizes the utilization of raw video data while shielding the deterministic action space from being corrupted by noisy trajectories, ensuring robust action-response learning. Furthermore, guided by insights from a pilot toy study, we employ a Revisit-Dense Finetuning Strategy using a compact, 30-minute dataset to efficiently activate the model's long-range loop-closure capabilities. Extensive experiments, including objective metrics and user studies, demonstrate that Infinite-World achieves superior performance in visual quality, action controllability, and spatial consistency.
1. Introduction
Infinite-World targets the reality gap in interactive world modeling by addressing noisy real-world poses, sparse viewpoint revisits, and inefficient long-context memory. It combines pose-free hierarchical compression with uncertainty-aware labeling and revisit-dense finetuning to support 1000+ frame simulation and improved controllability and memory consistency.
- Challenges: Real-world videos challenge world-model training because pose estimates are noisy and viewpoint revisits are scarce.These issues hinder precise action-response mapping and learning global spatial structure.
- Challenges: Standard attention has O(L^2) complexity, while pose-dependent retrieval and temporal downsampling respectively depend on inaccurate poses or lose information.The paper identifies a need for a pose-free, computationally efficient memory mechanism.
- Approach: Infinite-World introduces HPMC, which compresses historical latents into a fixed memory budget for long-context generation.The compressor is jointly optimized with the DiT backbone to anchor generations in the distant past at bounded computational cost.
- Approach: Uncertainty-aware action labeling uses tri-state motion categories to reduce the impact of noisy pose estimations while retaining raw-video training data.The strategy preserves uncertain samples without allowing ambiguous motion to corrupt deterministic action labels.
- Approach: A revisit-dense finetuning strategy uses a compact dataset to activate long-range loop-closure capabilities.The contribution list pairs this training framework with uncertainty-aware labeling for real-world raw videos.
- Results: Extensive experiments, user studies, and visual comparisons report superior visual quality, action controllability, and memory consistency.The paper presents these evaluations as evidence for Infinite-World’s overall performance.
2. Related Work
Related work establishes progress in interactive and long-horizon video generation while highlighting a persistent gap between synthetic environments and real-world complexity. Existing approaches trade off pose sensitivity, information loss, computational overhead, or limited accessibility.
- Interactive world models: Recent video-generation models have expanded toward physically consistent, action-responsive, and long-horizon world modeling.The literature spans interactive video models and real-world simulation systems.
- Real-world gap: Real-world modeling remains difficult because accurate camera poses are hard to obtain and viewpoint revisits are scarce.These constraints limit action controllability and long-context memory acquisition.
- Real-world gap: Genie-3 demonstrates strong real-world simulation performance, but its closed-source design leaves its training paradigm and architecture inaccessible.Infinite-World is positioned as an open and efficient framework addressing this gap.
- Memory mechanisms: Geometry-dependent methods reduce context using camera poses or scene reconstruction but remain sensitive to real-world pose-estimation errors.Examples include 3D reconstruction and field-of-view overlap retrieval.
- Memory mechanisms: Compression-based methods slow latent growth through downsampling, implicit finetuning, or memory compressors, but may lose information or incur computational overhead.The related methods represent distinct trade-offs in efficiency and memory preservation.
3. Methodology
Infinite-World combines hierarchical pose-free memory compression, uncertainty-aware action encoding, and revisit-dense data strategy for coherent long-horizon generation from noisy real-world data. The method compresses history under a fixed budget, aligns discrete actions with latent tokens, and uses revisit-rich training to improve spatial memory.
- Framework: The methodology consists of a hierarchical pose-free memory compressor, uncertainty-aware action labeling, and a revisit-dense training objective.Together, these components target constant-complexity context modeling, robust motion control, and long-range memory.
- Hierarchical Pose-free Memory Compressor: HPMC transforms historical latents into a fixed memory budget through direct compression for short contexts and hierarchical compression for long horizons.The overview describes local and global compression stages jointly optimized with the DiT backbone.
- Hierarchical Pose-free Memory Compressor: For short contexts, a temporal encoder directly compresses raw latents and reduces temporal resolution by a factor of k = 4.The resulting compressed tokens condition the DiT backbone.
- Hierarchical Pose-free Memory Compressor: For long horizons, overlapping chunks receive local compression before concatenated representations undergo global compression into a bounded unified memory.This recursive structure preserves a strictly bounded footprint while covering wide-span history.
- Context injection: The DiT input concatenates compressed history, the last-frame latent, and noisy target latents, with a binary mask separating context from denoising targets.This supplies both long-range and local visual context during generation.
- Pose-free anchoring: HPMC is jointly optimized with DiT so future-frame generation loss trains the compressor to preserve historical cues for loop-closure without external pose metadata.The compressor learns relevant history end-to-end rather than relying on explicit geometric priors.
- Uncertainty-aware Action Labeling: Continuous camera motion is decoupled into translation and rotation, then categorized as No-operation, Discrete Action, or Uncertain using two thresholds.The Uncertain state retains low-signal motion while shielding action labels from noisy trajectories.
- Action encoding: Two stride-2 convolutional layers downsample action embeddings 4× so active action segments match the compressed target-latent resolution.The aligned embedding is added element-wise to video tokens to modulate the noisy latent space.
2. Context-Bound Extrapolation: We observe a strict cou-
The pilot study finds that spatial memory activates with limited data but fails when inference extends beyond the temporal context seen during training. This motivates revisit-dense finetuning with long-duration trajectories to support stable 1000-frame loop closure.
- Inference beyond the 4-chunk training context causes catastrophic memory collapse, visual drift, and hallucinations.The collapse is observed when a model trained on four chunks is evaluated over longer sequences such as six chunks.
- Long-horizon modeling is bottlenecked by trajectory duration and topological density rather than total data quantity.
- Open-domain pre-training supplies diverse visual priors and local dynamics but typically lacks long-term viewpoint revisits and complex loop closure.
- A compact Revisit-Dense Dataset activates spatial memory and enables stable 1000-frame loop closure at practical cost.
4. Experiments
Experiments compare Infinite-World with multiple interactive world-model baselines using automated metrics, user rankings, visual comparisons, and memory-efficiency tests. Across these evaluations, the model preserves long-range scene structure, supports responsive control, and maintains bounded memory growth.
- Benchmark and evaluation: The benchmark covers 100 scenarios across Indoor, Street, Nature, and Fantasy domains with 10 manually designed long action trajectories.
- Benchmark and evaluation: The evaluation combines VBench automated metrics with human preference rankings over memory consistency, visual fidelity, and action responsiveness.
- Quantitative and user results: Infinite-World achieves the best or second-best performance across all reported VBench dimensions, despite Yume 1.5’s slightly higher average score of 0.8141 versus 0.8119.The reported average-score lead for Yume 1.5 is attributed primarily to aesthetic quality and its larger parameter scale.
- Quantitative and user results: 1719 ELO gives Infinite-World a 177-point lead over HY-World-1.5’s 1542 in the user study.
- Quantitative and user results: Infinite-World ranks first in Memory Consistency at 1.92 and Visual Fidelity at 1.67, while its Action Responsiveness rank of 1.54 is comparable to HY-World-1.5’s 1.50.
- Efficiency and qualitative comparisons: Visual comparisons show Infinite-World preserving global landmarks and performing long-range loop closures where baselines exhibit drift, distortions, or forward-motion bias.
- Efficiency and qualitative comparisons: Hierarchical compression stabilizes memory near 45GB as the horizon extends beyond 1300 frames, unlike configurations with linear growth or memory exhaustion.
- Ablations: RDD finetuning improves Memory Consistency from 2.40 to 1.83 and action control from 2.95 to 1.61.
5. Conclusion and Future Works
Infinite-World achieves coherent interactive simulation over a 1000-frame horizon by combining pose-free memory compression with uncertainty-aware action labeling. The authors identify cumulative drift, visual degradation, inference speed, and visual fidelity as areas for future improvement.
- Infinite-World achieves coherent simulation over a 1000-frame horizon.
- The model integrates a Hierarchical Pose-free Memory Compressor with Uncertainty-aware Action Labeling to address computational complexity and noisy real-world pose estimations.
- Revisit-Dense Finetuning activates long-range loop-closure with minimal data.
- Future work targets cumulative drift and visual degradation through self-forcing or refined noise schedules.
- Model distillation and larger backbones are proposed to improve inference speed and visual fidelity.
Impact Statement
The paper frames long-horizon interactive world modeling as relevant to autonomous-system reliability and large-scale generative-model training. It also acknowledges ethical responsibility concerning misuse of high-fidelity video generation and evaluates interactive performance through human subjective assessment.
- Long-horizon interactive world modeling may affect autonomous-system reliability and the efficiency of large-scale generative-model training.
- The paper highlights simulation and embodied AI as application areas while acknowledging responsibility to prevent misuse of high-fidelity video generation.
- Human subjective evaluation was conducted to compare Infinite-World’s interactive performance and long-term consistency with state-of-the-art baselines.
A.1. Evaluation Interface and Procedure
The evaluation uses anonymized, side-by-side comparisons of synchronized videos generated from identical initial conditions and action sequences. Thirty domain-relevant volunteers each complete ten trials, producing 300 pairwise comparisons.
- Each session randomly compares Infinite-World with one of four baseline models through a side-by-side web interface.
- Participants view synchronized videos generated from identical initial frames and interactive action sequences.
- The double-blind procedure anonymizes methods as Method A and Method B and randomizes their horizontal positions.
- Thirty volunteers with computer-vision or interactive-media backgrounds participate, with each trial taking approximately three minutes.
- Each participant completes ten trials, yielding 300 pairwise comparisons while limiting fatigue.
A.3. Evaluation Dimensions
Participants rate methods on three fine-grained dimensions using a 5-point scale: visual quality, memory ability during loop-closure, and action response ability.
- The evaluation uses a 5-point scale across three fine-grained dimensions.
- Visual Quality: Visual Quality evaluates temporal stability and aesthetic fidelity.
- Memory Ability: Memory Ability assesses scene consistency when returning to previously visited viewpoints.
- Action Response Ability: Action Response Ability measures the accuracy and immediacy of world-state evolution after action inputs.
A.4. Statistical Analysis
The user study synthesizes sparse pairwise comparisons into an ELO-based global ranking and reports mean numerical ranks for three fine-grained dimensions.
- ELO scores aggregate randomly paired model comparisons into a global measure of human preference.The system randomly selects two methods from five candidates for each evaluation session.
- Mean numerical ranks summarize volunteer judgments across three fine-grained dimensions, with lower values indicating better performance.
B. More Visual Comparisons
The paper presents additional long-horizon visual comparisons, focusing on Infinite-World against HY-World-1.5. These comparisons show stronger memory consistency and preservation of scene structure during loop closures.
- Figures 7 and 8 directly compare Infinite-World with HY-World-1.5 for long-horizon visual analysis.HY-World-1.5 is identified as the second-best baseline in the user study.
- Infinite-World maintains long-term stability and memory consistency while preserving key scene structures during loop closures.
- The visual comparison materials include a user-study webpage and long-horizon comparison figures.