Source-linked AI summary
RELIC: Interactive Video World Model with Long-Horizon Memory
Yicong Hong, Yiqun Mei, Chongjian Ge, Yiran Xu, Yang Zhou, Sai Bi, Yannick Hold-Geoffroy, Mike Roberts, Matthew Fisher, Eli Shechtman, Kalyan Sunkavalli, Feng Liu, Zhengqi Li, Hao Tan
TL;DR
Interactive world models need real-time streaming, spatial memory, and precise control, but existing approaches struggle to provide them together. RELIC combines camera-aware compressed memory with long-context teacher distillation and curated action-control data, achieving 16 FPS generation and stronger reported quality and memory behavior against baselines. Its main scope limits are static-scene diversity, minute-scale generation, and resource-heavy inference.
Problem
Practical interactive world models require real-time long-horizon streaming, consistent spatial memory, and precise user control simultaneously.
Method
RELIC uses compressed historical latents with relative actions and absolute camera poses in the KV cache, plus a 20-second teacher and memory-efficient self-forcing distillation.
Results
16 FPS generation throughput on 4 H100 GPUs accompanies strong action and text following, long-horizon spatial consistency, and strongest overall visual-quality performance among compared baselines.
Takeaways & Limitations
RELIC provides a real-time interactive video world model that maintains spatial memory without explicit 3D scene representations.
Takeaways & Limitations
The system has limited diversity and scene dynamics, struggles with minute-scale videos, and incurs substantial latency under resource-constrained settings.
Abstract
from arXiv · showhide
A truly interactive world model requires three key ingredients: real-time long-horizon streaming, consistent spatial memory, and precise user control. However, most existing approaches address only one of these aspects in isolation, as achieving all three simultaneously is highly challenging-for example, long-term memory mechanisms often degrade real-time performance. In this work, we present RELIC, a unified framework that tackles these three challenges altogether. Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time. Built upon recent autoregressive video-diffusion distillation techniques, our model represents long-horizon memory using highly compressed historical latent tokens encoded with both relative actions and absolute camera poses within the KV cache. This compact, camera-aware memory structure supports implicit 3D-consistent content retrieval and enforces long-term coherence with minimal computational overhead. In parallel, we fine-tune a bidirectional teacher video model to generate sequences beyond its original 5-second training horizon, and transform it into a causal student generator using a new memory-efficient self-forcing paradigm that enables full-context distillation over long-duration teacher as well as long student self-rollouts. Implemented as a 14B-parameter model and trained on a curated Unreal Engine-rendered dataset, RELIC achieves real-time generation at 16 FPS while demonstrating more accurate action following, more stable long-horizon streaming, and more robust spatial-memory retrieval compared with prior work. These capabilities establish RELIC as a strong foundation for the next generation of interactive world modeling.
1 Introduction
Interactive video world models must combine real-time long-horizon streaming with consistent spatial memory, but these requirements impose conflicting computational demands. RELIC addresses this tension with compressed camera-aware memory, long-context distillation, and curated action-controlled data.
- Motivation: Real-time streaming responds continuously to user controls, while spatial memory preserves previously explored content for persistent 3D-consistent worlds.Autoregressive models must maintain stable rollouts while storing and retrieving long histories.
- Approach: RELIC represents memory as highly compressed historical latents encoded with relative actions and absolute camera poses in the KV cache.The design supports implicit viewpoint-aware 3D scene-content retrieval while retaining memory efficiently.
- Approach: RELIC fine-tunes a bidirectional teacher for 20-second sequences and introduces memory-efficient replayed back-propagation for long-context student distillation.The extended teacher horizon supplies longer-range spatial and temporal supervision than the typical 5-second window.
- Data and efficiency: 350 licensed Unreal Engine scenes provide approximately 1600 minutes of annotated training video, while RELIC generates at 16 FPS on 4 H100 GPUs.The dataset includes single and mixed actions plus viewpoint revisitations for control and memory learning.
2 Related Works
Related work addresses autoregressive video generation, few-step distillation, long-context memory, spatial consistency, and world-exploration data, but each area has distinct limitations.
- Video World Models: Interactive world modeling integrates autoregressive generation and few-step distillation with long-context memory and spatial-consistency techniques.
- Long Video Generation: Long-video methods target autoregressive stability and mitigate drifting through inference or training noise-scheduling strategies.Drift can appear as quality degradation, color oversaturation, or static outputs during long rollouts.
- Data for World Exploration: World-exploration datasets combine video with action annotations, but web data are rare, expensive to obtain, action-imbalanced, and often compositionally coupled.
3 Data Curation for Interactive Video World Model
RELIC’s data pipeline uses Unreal Engine scenes, curated camera trajectories, filtering, pose-derived action labels, and captions designed for precise joint control.
- 3.1 Data Overview: The dataset targets diverse 3D environments and precise camera control through Unreal Engine-rendered trajectories from 350 licensed scenes.It contains over 1400 human-controlled trajectories and more than 1600 minutes of 720p video, with clips averaging approximately 75 seconds and reaching 9 minutes.
- 3.1 Data Overview: The pipeline visualizes video-duration and action distributions while producing high-quality video-action-text triplets through filtering, captioning, and balancing.
- 3.2 Data Processing and Filtering: UE trajectories are balanced in action space and frequently revisit locations and scene content to support long-horizon retrieval.This design addresses imbalanced and tightly coupled actions common in real-world navigation videos.
- 3.2 Data Processing and Filtering: Filtering removes unnatural motion, micro-jitters, unstable paths, exposure problems, and rendering defects before training.These steps preserve smooth camera dynamics, stable viewpoints, lighting quality, and high-fidelity rendering.
- 3.3 Data Annotations: Relative translation is computed in camera coordinates, normalized by clip motion scale, and paired with rotational differences decomposed into yaw, pitch, and roll.Inference can adjust the average displacement coefficient to control overall generated motion scale.
- 3.3 Data Annotations: Captions suppress camera and fine-grained object motion because common vision-language models overemphasize those dynamics, improving joint action and text control.
4 RELIC World Model
RELIC combines long-duration teacher generation, compressed spatial memory, emergent retrieval training, memory-efficient self-forcing distillation, and runtime optimization. The resulting model targets real-time interactive streaming with long-horizon memory.
- Requirements: RELIC must simultaneously generate responsive video from image, text, and action streams while preserving spatial and temporal consistency over long horizons.
- Challenges: Prior approaches often prioritize real-time performance or long-term generation, whereas long-horizon memory adds computation and GPU-memory demands.
- Framework: RELIC extends the teacher to 20-second generation, compresses historical tokens in the KV cache, trains emergent retrieval, and distills a few-step autoregressive model.Runtime optimizations further target real-time inference.
4.1 Action-Conditioned Teacher for Long Video Generation
RELIC extends a Wan-2.1-based bidirectional video diffusion model for 20-second, action-conditioned generation with precise camera control. It combines relative actions, absolute poses, and continuous-valued inputs to support long-duration, spatially consistent video.
- Base Architecture: RELIC builds on Wan-2.1’s spatio-temporal latent representation and diffusion-transformer architecture for action-conditioned video generation.The base model uses an ST-VAE and DiT blocks, with text integrated through cross-attention.
- Base Architecture: The teacher generates 20-second videos conditioned on text, action labels, and camera poses.Conditioning is injected throughout denoising to produce spatially consistent and action-aligned frames.
- Action Control: RELIC uses a 13-degree action space covering six translational motions, six rotational motions, and a static action.Together, these controls provide full 6-DoF camera viewpoint control between consecutive frames.
- Action Control: Relative actions and derived absolute camera poses are embedded through dedicated encoders and injected into transformer blocks through distinct pathways.Relative-action embeddings modify latents, while pose embeddings modify query and key projections before attention.
- Action Control: Continuous-valued action encoding represents motion strength and supports streamed keyboard inputs scaled by a coefficient γ.This allows motion magnitudes to match observed frame transitions and vary across unit velocities.
- Long-Horizon Teacher: A 5-second training context is insufficient for long-term memory retrieval and robustness to significant viewpoint changes.RELIC therefore trains the teacher directly in a long-horizon setting beyond Wan-2.1’s original 5-second generation horizon.
4.2 Autoregressive Student for Real-Time Streaming
RELIC distills a 20-second bidirectional teacher into a few-step, memory-aware causal student for long-horizon interactive streaming. The student uses block-wise causal attention and block-causal latent generation.
- Design Goals: The design targets three core requirements: recalling revisited scenes, enabling low-latency exploration, and providing extended temporal context.These requirements correspond to memory, streaming inference, and long-horizon generation.
- Autoregressive Student: RELIC distills its 20-second bidirectional teacher into a few-step, memory-aware causal student video diffusion model.The student is designed to support memory, streaming inference, and long-horizon generation.
- Autoregressive Student: The student replaces bidirectional attention with block-wise causal attention and generates latent frames in a block-causal manner.This architecture follows recent autoregressive video-diffusion principles for causal generation.
4.3 Memory
RELIC addresses the tension between efficient streaming and long-range retrieval with a two-branch KV-cache memory. Recent content remains uncompressed, while older spatial information is compressed on a predefined schedule.
- Memory Challenge: Short sliding-window attention reduces latency but limits retrieval of long-range information needed for consistent world modeling.Retaining all past tokens would restore history but creates increasing computational and memory costs.
- Memory Design: RELIC’s KV cache combines a rolling-window cache with a compressed long-horizon spatial-memory cache.Recent latents are stored as uncompressed KV tokens, while earlier latents are stored using spatially downsampled KV tokens.
- Memory Design: The rolling window stores recent video latents from indices i −w through i to preserve short-term context.Its limited size encourages the model to use compressed long-range memory rather than relying only on recent patterns.
- Memory Design: The long-horizon cache compresses earlier latents using an interleaved schedule of 1×, 2×, and 4× spatial downsampling.This configuration balances memory compression with access to historical spatial information.
4.4 Distillation Framework
RELIC uses self-forcing and hybrid ODE initialization to distill a long-context bidirectional teacher into a causal student. Replayed back-propagation makes full-rollout distillation memory-efficient by differentiating one generation block at a time.
- Self-Forcing: Self-forcing trains the student on its own previously generated history, reducing exposure bias during autoregressive rollout.The student predicts new chunks conditioned on generated rather than ground-truth context.
- Long-Horizon Distillation: Unlike short-segment distillation, RELIC distills from a bidirectional teacher trained on much longer sequences to preserve long-range spatial memory.Short 5-second teacher windows produce loosely coupled segments that limit long-range retrieval and reasoning.
- ODE Initialization: ODE initialization adapts the bidirectional teacher into a causal model by initializing the student with teacher weights and precomputed ODE trajectories.The student regresses trajectories at the four denoising time steps used during distillation and inference.
- ODE Initialization: Hybrid forcing combines clean compressed context with noisy causal blocks, improving initialization for long-horizon memory retrieval over either forcing method alone.The first B −K blocks are clean and compressed, while the remaining K blocks are uncompressed and noise-perturbed.
- Replayed Back-Propagation: Replayed back-propagation first generates the full student rollout without gradients, caches DMD score differences, then replays blocks with autograd enabled.This preserves full-sequence teacher-distribution gradients while avoiding an autograd graph spanning the entire rollout.
- Replayed Back-Propagation: Block-wise differentiation reduces peak GPU memory to that of a single video-latent block while retaining gradients reflecting the full-length video distribution.Each block’s graph is freed after processing, and parameters are updated once after the full replay.
4.5 Runtime Efficiency Optimization
RELIC optimizes inference by reducing kernel, memory, and recomputation overhead, then distributing computation and memory loads across multiple GPUs.
- Inference optimizations: torch.compile reduces kernel-launch overhead and memory costs in RMSNorm, RoPE embeddings, and modulation layers.The implementation also uses self-attention KV caching and FP8 E4M3 storage for the cache.
- Multi-GPU parallelization: Sequence/context parallelism shards linear and cross-attention computation over the sequence dimension, while tensor parallelism shards self-attention over attention heads.NCCL All-to-All operations switch between the parallelization layouts.
5 Experiments
Experiments evaluate RELIC’s efficiency, visual quality, controllability, diversity, spatial memory, and qualitative robustness against existing world models on 20-second videos and varied scenes.
- Capabilities showcase: RELIC generates high-quality, diverse, controllable videos with strong spatial consistency over long horizons.The capabilities are demonstrated across varied artistic styles, adjustable camera velocities, and complex multi-key controls.
- Diversity: RELIC generalizes from a single initial frame across oil paintings, comic illustrations, vector art, low-poly renders, and other stylized domains.It also exhibits distance awareness and 3D shape understanding during camera motion.
- Long-horizon memory: RELIC retrieves previously generated scene content after large camera movements and extended periods outside the field of view.This uses compressed historical video tokens and absolute camera pose without explicit 3D representations or handcrafted memory heuristics.
- Control: RELIC supports a wide spectrum of translational and rotational velocities while producing temporally stable outputs.Users adjust exploration speed through the displacement coefficient λ, and combined translation-rotation inputs are supported.
- Visual quality: RELIC achieves the strongest overall visual-quality performance among compared baselines and higher Aesthetics scores than Hunyuan-GameCraft and Matrix-Game 2.0.Despite training at 480p, it performs comparably to Hunyuan-GameCraft trained on 720P videos for Image Quality.
- Action accuracy: RELIC follows commanded camera actions more faithfully than Hunyuan-GameCraft and Matrix-Game-2.0 in qualitative comparisons.The baselines exhibit incorrect motion, black voids, or static outputs, whereas RELIC follows the intended trajectory without artifacts.
- Memory comparison: RELIC consistently regenerates previously observed content, while Hunyuan-GameCraft and Matrix-Game-2.0 lose object persistence or input-image context.The comparison is qualitative because aligning all baselines with identical actions is difficult.
6 Discussion and Conclusion
RELIC’s main limitations are restricted diversity and scene dynamics, difficulty generating minute-scale videos, and increased latency under resource constraints. The conclusion presents its compressed-memory and long-context distillation design as a foundation for interactive world simulation.
- Limitations: Training primarily on static Unreal Engine scenes limits generated-video diversity and scene dynamics.The authors identify the dataset composition as the main source of this limitation.
- Limitations: RELIC struggles to generate extremely long videos on the scale of minutes.The limitation is stated as a scope boundary for the current system.
- Limitations: Large model size, long-horizon KV-cache requirements, and iterative denoising steps increase inference latency in resource-constrained settings.The authors suggest targeted pipeline refinements and scaling data and training as possible mitigations.
- Conclusion: RELIC combines spatial-aware compressed historical latents with full-horizon supervision from a long-context teacher to address drifting and memory forgetting.The conclusion describes this architecture as supporting real-time inference and long-horizon spatial memory without explicit geometric representations.
- Conclusion: The architecture is presented as a scalable foundation for general-purpose world simulators, with potential applications in embodied AI and immersive virtual content creation.This stated potential extends beyond the demonstrated virtual scene exploration setting.