Source-linked AI summary
ReWorld: An Interactive World Model with Long-Horizon Memory
Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang, Tianyi Gui, Lianghua Huang, Yingcong Chen
TL;DR
Interactive world models must jointly follow actions, remember distant places, and stream in real time despite conflicting horizon requirements. ReWorld separates control and memory during training, bounds memory at inference with a pose-indexed landmark bank, and uses four-step LoRA distillation. It leads recent interactive world models on control fidelity and generation quality while retaining revisit fidelity at long rollout lengths.
Problem
Interactive world models need action following, long-horizon spatial recall, and real-time streaming, although control favors short context while memory requires unbounded history.
Method
ReWorld uses routed mixed-window attention for control and memory, chunk-drop training with a bounded pose-indexed landmark bank, and four-step LoRA distillation.
Results
ReWorld leads six recent interactive world models on control fidelity and generation quality, while its fixed cache preserves revisit fidelity at rollout lengths where sliding-window evidence is evicted.
Takeaways & Limitations
A single backbone supports both high-fidelity multi-step generation and real-time interactive streaming with bounded long-horizon memory.
Takeaways & Limitations
Memory remains keyed on camera pose alone, and the evaluation includes a low-motion baseline whose short paths are easier to reproduce from memory.
Abstract
from arXiv · showhide
An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ($11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ($64$\,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.
1 Introduction
ReWorld addresses interactive world modeling by separating control from long-range memory during training and consolidating memory under a bounded inference cache. Its design combines mixed attention windows, pose-based landmark retrieval, aligned multi-source data, and real-time distillation.
- Motivation: Interactive world models must follow actions, preserve revisited places, and stream video at interactive rates over unbounded rollouts.These requirements create a tension between short-horizon control and long-horizon memory.
- Control–Memory Decoupling: Control and memory are trained under different attention windows: control uses short windows, while memory requires long-range access.When trained together, control improves but revisit fidelity can drop; mixed windows and routed heads separate the learning conditions.
- Control–Memory Decoupling: Mixed per-head windows and random head routing train most heads locally while exposing every head to global context across iterations.Routing prevents heads from specializing in windows unavailable during bounded-cache deployment.
- Bounded Memory: Chunk-drop training and consolidation-based inference make sparse caches usable while preserving spatial memory under a fixed KV budget.A bounded landmark bank stores selected aged chunks, evicts spatially redundant entries, and retrieves landmarks near the current pose.
- Data and Deployment: Metric-aligned synthetic, real, and game footage share one physical action scale, with palindrome trajectories providing revisit supervision.The same key press therefore corresponds to the same physical camera movement across sources.
- Data and Deployment: LoRA-confined few-step distillation gives one backbone both a high-fidelity multi-step mode and a real-time interactive operating point.The evaluation spans action following, long-horizon recall, and general video quality.
2 Method
ReWorld separates control and memory during training, then serves both through a bounded, pose-indexed cache for streaming generation. Mixed attention windows, random routing, pose conditioning, landmark retrieval, and LoRA distillation support action-controllable real-time video.
- Streaming generation: ReWorld converts a bidirectional diffusion transformer into a chunkwise-causal streaming generator that emits one latent chunk per denoising pass.The model appends generated chunks' keys and values to the cache.
- Pose and action conditioning: MRoPE uses camera-to-world pose descriptors to make attention depend on pose difference, allowing similar viewpoints to be retrieved despite temporal separation.Direct action injection separately supplies the commanded camera trajectory through Plücker ray maps added to patch embeddings.
- Mixed attention windows: Mixed attention trains control on 18 local heads attending to the last 12 latent frames and memory on 6 global heads attending to the full causal history.The 24 heads use a fixed 1:3 global-to-local ratio.
- Random head routing: Randomly switching among 12 six-head global partitions prevents control or memory from binding to particular attention heads.Each head therefore trains under both local and global roles while preserving the 1:3 ratio.
- Bounded memory: At deployment, a bounded landmark bank retrieves pose-nearest chunks into the cache, while sparse chunk-drop training matches the non-contiguous histories encountered at inference.The bank retains full-resolution chunks and manages diversity through pose-based admission and redundancy eviction.
- Real-time distillation: Four-step distribution-matching distillation trains only a LoRA adapter on a frozen backbone, preserving a shared high-fidelity and real-time operating point.The action and pose injection path carries no LoRA.
3 Data Pipeline
ReWorld builds an eight-source corpus around a shared physical action scale, using UE data as a metric anchor and palindrome trajectories to supervise revisits. Filtering, balanced trajectory generation, and source-specific pose rescaling make camera actions comparable across domains.
- 220,724 pose-annotated clips form the eight-source corpus, spanning UE-rendered, real-world, and game footage.
- The curation funnel removes clips that move too fast or too slowly, while the training mixture samples the eight sources uniformly at clip level.
- 337 environments are processed by an automated UE pipeline that plans direction-balanced camera trajectories and exports exact poses and metadata.
- A per-source translation rescaling aligns estimated and engine poses to one metric action scale, using the UE median increment of 0.3667 as the anchor.
- Palindrome windows are sampled with probability 0.2 on the two UE sources, providing explicit out-and-back revisit supervision.
4 Experiments
Experiments evaluate controllability, long-horizon recall, video quality, and ablations under standardized protocols. ReWorld leads overall control and quality comparisons, while routed attention preserves control performance and improves recall relative to unrouted action fusion.
- Camera Controllability: 240 clips per method test six canonical trajectories from 40 start images, with Sim(3) and orientation alignment isolating trajectory-following fidelity.
- Camera Controllability: ReWorld attains the best overall RotErr and CamMC, remains first-tier on translation-dominant trajectories, and is competitive on rotation-heavy arcs.
- Long-Horizon Memory: At k=96, the start view lies beyond every bounded baseline window, while ReWorld returns to the starting view with landmark layout intact on long out-and-back rollouts.
- Generation Quality: ReWorld achieves the best mean across seven prompt-free VBench dimensions, with first-tier temporal scores while sustaining high Dynamic Degree.
- Ablations: At k ∈ {288, 384}, sliding-window retrieval has necessarily evicted the needle, whereas the landmark-bank policy remains the discriminative bounded-cache comparison against full-KV.
- Ablations: Adding action injection lowers RotErr from 17.66° to 13.21° but reduces revisit SSIM from 0.3898 to 0.3376; routing restores SSIM to 0.3752 without sacrificing control.
5 Related Works
Prior interactive world models address streaming, control, and memory through autoregressive generation, pose or action conditioning, and varied history representations. ReWorld combines bounded caching with pose-indexed landmark consolidation to retain spatial recall without storing the entire rollout.
- Streaming Video Generation: Streaming approaches use autoregressive reformulations, causal KV caching, context packing, streaming pipelines, and few-step distillation to support interactive generation.
- Interactive World Models: Interactive world models differ in whether control is supplied through relative pose folded into attention or an injected action signal.
- Memory in World Models: Memory systems store growing frame archives, external 3D reconstructions, or pose-dependent attention histories, while sinks and sliding windows bound cache size by forgetting older context.
- Memory in World Models: ReWorld consolidates aged chunks into a fixed-capacity landmark bank, evicts spatially redundant members, and retrieves pose-nearest landmarks into a bounded working cache.
6 Conclusion
ReWorld combines window-separated training for control and memory with fixed-budget inference consolidation. It streams high-resolution video in real time, leads recent interactive world models on control fidelity and visual quality, and preserves revisit fidelity after sliding-window eviction.
- ReWorld separates control and memory training with mixed per-head attention windows and random head routing.Inference then consolidates memory using a bounded cache and pose-indexed landmark bank.
- A fixed KV budget holds the entire past through a bounded cache backed by a pose-indexed landmark bank.Chunk-drop training makes sparse caches in-distribution.
- ReWorld uses metrically aligned multi-source data and few-step LoRA distillation to stream high-resolution video in real time.
- ReWorld leads recent interactive world models on control fidelity and visual quality while revisit fidelity persists after sliding-window eviction.
A Data Pipeline Details
The appendix expands the data-pipeline description with source-specific construction details, scale divisors and audits, and loader-level sampling and conditioning specifications.
- The appendix provides per-source construction details for the data pipeline.
- It documents per-source scale divisors and their audits.
- It specifies loader-level sampling and conditioning details.
A.1 Per-Source Construction
The per-source construction combines Unreal-rendered, real-world, game-roaming, simulator/game, and walking-video data with pose and video processing tailored to each source.
- The Unreal rendering pipeline produces fly-through clips with ground-truth trajectories, camera matrices, intrinsics, and scene metadata.The full run contains 130,986 raw clips across 337 environments.
- Unreal clips pass luminance and motion filters that remove dark or nearly frozen footage.Clips are probed using 16 uniformly sampled grayscale frames.
- DL3DV and RealEstate10K provide real-scene or real-estate footage with pose estimates and captions.DL3DV uses frame-accurate 15 s slices, while RealEstate10K retains clips of at least 8 s.
- Game-roaming footage is curated from gameplay recordings, sliced into 15 s clips, and cleaned using pose, darkness, and content filters.The cleaned set contains 18,387 clips across 79 games.
- OmniWorld-Game reconstruction runs are kept frame-contiguous, cut into overlapping windows, and re-anchored to the identity pose.This yields 5,629 clips.
- Sekai contributes real-walking clips with per-frame pose estimates and a companion game-walking split rendered in UE5.
A.2 Alignment Details and Audits
Alignment details normalize motion scale and coordinate conventions across sources, then audit the resulting per-source increments and residual spread.
- Per-source divisors are measured from training-time latent translation increments and applied to every pose's translation column.The training-time window is resampled to 24 fps with 0.1640625 s per latent step.
- The game-roaming source is already metric under VIPE, while the UE expansion set is deliberately not renormalized separately despite moving more slowly.Its raw per-latent-step translation median is approximately 20 cm versus approximately 37 cm for the control set.
- A single global scalar per source is sufficient because measured per-clip scale spread remains under one order of magnitude for the audited sources.The reported p90/p10 ratios are 3.65, 2.66, and 2.86/2.67.
- Coordinate conventions are normalized jointly with scale by converting sources to OpenCV camera-to-world matrices.The UE basis requires a non-trivial 90° vertical rotation rather than a naive sign flip.
A.3 Sampling and Conditioning Details
The sampling pipeline extracts randomized windows and converts camera trajectories into chunk-level action and pose conditioning for training.
- Window extraction: 189-frame windows are randomly cropped from long clips and resampled to 24 fps, yielding diverse ≈8 s excerpts during training.A 60 s clip contributes a different 7.88 s excerpt each epoch.
- Palindrome augmentation: Palindrome augmentation makes the camera retrace its path within a training window and adds reverse-motion coverage.
- Palindrome augmentation: Pose downsampling uses SE(3)-aware rotation slerp to avoid singular matrices at the palindrome mirror point.
- Emitted conditioning: The loader emits RGB frames, per-chunk 6-DoF actions, and normalized relative camera-to-world poses for pose-indexed attention.The 189 poses become 48 latent steps, with translations normalized by the window’s maximum radius.