PaperlayerView exact revision on arXiv

Source-linked AI summary

World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

Junyi Wu, Zekun Li, Chenhongyi Yang, Chen Guo, Lingchen Yang, Yongan Zhang, Xinyi Zhang, Hongbing Hu, Bartek Rymkowski, Srinath Sridhar, Yan Yan, Xijie Huang

arXiv:2610.17434v1cs.CVcs.AIcs.MM

TL;DR

Long-horizon video world models can lose earlier scene information beyond their active context, while existing memory approaches can incur increasing storage or attention costs. World-T3 stores pose-keyed observations in test-time-trained model parameters and distills its autoregressive model into a few-step generator. It ranks first on six revisit-consistency measures and sustains 20.6 FPS at 1280 × 704 on one NVIDIA B200 GPU.

  • Problem

    Autoregressive video world models can lose earlier scene information beyond active context, while existing approaches may incur unbounded storage or escalating attention costs as rollouts grow.

    Source: §1 · p. 2

  • Method

    World-T3 consolidates evicted observations into pose-indexed parametric TTT memory with recursive least-squares updates, then uses dual-teacher distillation for few-step generation.

    Source: §1 · p. 2

  • Results

    World-T3 ranks first on all six evaluated revisit-consistency measures and is the only evaluated model above 16 FPS, sustaining 20.6 FPS at 1280 × 704 on one NVIDIA B200 GPU.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    World-T3 provides a viable and efficient path toward persistent memory in interactive video world models.

    Source: §5 · p. 10, F Limitations · p. 26

    The authors acknowledge that limitations remain to be addressed in future work.

    Source: F Limitations · p. 25

Abstract

from arXiv · show

Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context. We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. On a long-horizon revisit benchmark, World-T3 raises revisit PSNR from 13.52 to 18.31 dB and reduces absolute rotation error from 3.344 to 1.075 relative to the strongest baseline on both metrics, while sustaining 20.6 FPS at 1280 x 704 on a single NVIDIA B200 GPU.

1 Introduction

Long-horizon video world models can lose scene information once it leaves active context, while existing memory approaches may grow in storage or attention cost. World-T3 consolidates observations into pose-indexed parametric memory and combines this with a three-stage training pipeline for persistent, interactive generation.

1 Introduction

  • Existing autoregressive video diffusion models can remain locally coherent yet resynthesize earlier scene content after it falls outside the active history context.This motivates preserving observed content as persistent world state during sustained interaction.Source: §1 · p. 2
  • Explicit geometry, historical-frame retrieval, and memory-token approaches can improve consistency, but may incur unbounded storage or escalating attention complexity as rollouts grow.The paper asks whether online parametric memory can recover previous scene content with storage and access costs independent of rollout length.Source: §1 · p. 2
  • World-T3 writes pose-content associations from evicted video blocks into fixed-size, pose-indexed fast-weight memory and retrieves them using incoming camera poses.Its recursive least-squares update incorporates accumulated pose-key covariance to reduce redundant writes along smooth camera trajectories.Source: §1 · p. 2
  • A three-stage pipeline adapts a pretrained video diffusion model for camera control, autoregressive TTT memory, and four-step generation through dual-teacher distillation.Training uses 908K camera-annotated clips, and deployment sustains 20.6 FPS at 1280 × 704 on one NVIDIA B200 GPU.Source: §1 · p. 2
  • Against five recent video world models, four-step World-T3 ranks first on all six revisit-consistency measures and is the only evaluated model above 16 FPS on one NVIDIA B200 GPU.The reported playback threshold is for real-time generation.Source: §1 · p. 2

2 Related Work

Prior work addresses camera-controlled video generation, persistent scene state, and test-time training memory. World-T3 positions its approach against methods whose memory costs grow with rollout length or whose fixed capacity can dilute earlier observations.

2 Related Work

  • Interactive video world models condition future observations on camera trajectories or actions, but extended generation can bring camera drift and temporal incoherence.Prior camera-conditioning representations include pose vectors, Plücker ray maps, and relative positional encodings.Source: §2 · pp. 2–3
  • Prior persistence methods reconstruct geometry, retrieve earlier frames, or compress history into tokens, trading growing storage costs against dilution of earlier observations in fixed-capacity buffers.World-T3 instead absorbs history into network weights with constant memory cost regardless of sequence length.Source: §2 · p. 3
  • Test-time training stores observations in a fixed-size parametric state, but smooth camera trajectories can produce near-collinear keys that make local gradient writes redundant.World-T3 addresses this redundancy by solving the accumulated objective for memory consolidation.Source: §2 · p. 3

3 World-T3

World-T3 progresses from a camera-conditioned bidirectional teacher to an autoregressive model with pose-keyed TTT memory, then distills that model for few-step generation. Its memory consolidates evicted observations with recursive least squares and retrieves them by camera pose.

3 World-T3

  • World-T3 transforms a pretrained text-image-conditioned video diffusion model through camera-control training, autoregressive TTT memory training, and four-step distillation.The three stages target high-fidelity priors, interactive generation with persistent scene memory, and real-time streaming.Source: §3 · p. 3

3.1 Stage 1: Bidirectional Camera-Conditioned Teacher

  • The Stage 1 bidirectional teacher uses camera conditioning and attends across the full temporal horizon to construct globally coherent frames.It initializes the Stage 2 autoregressive model and supplies the reference score function for Stage 3 distribution matching.Source: §3.1 · pp. 3–4, Figure 2 · p. 3

3.2 Stage 2: Autoregressive Teacher with TTT Memory

  • Stage 2 adds block-causal attention and TTT memory to address context decay and hallucination when autoregressive generation revisits earlier viewpoints.The block-causal model uses bounded visual context and an evolving memory state to keep per-block computation O(1) with rollout length.Source: §3.2 · p. 4

3.2.1 Block-Causal Adaptation

  • The bounded context combines a persistent first-block sink with K recent clean blocks, while masked teacher forcing excludes future blocks during training.This design anchors long-term visual appearance while supporting local continuity across block boundaries.Source: §3.2.1 · p. 4

3.2.2 Test-Time Memory Consolidation

  • When a block leaves context, its camera pose and clean visual state form a key-value association written to fast-weight memory and later queried by pose.Recursive least squares tracks inverse regularized feature correlations so updates attenuate directions already spanned by prior memory, preserving novel residuals.Source: §3.2.2 · pp. 4–6

3.3 Stage 3: Dual-Teacher Few-Step Distillation

  • Stage 3 uses bidirectional-teacher guidance for short-range realism and motion coherence alongside memory-augmented autoregressive guidance for long-horizon spatial consistency.The memory states for the teacher and critic are initialized from the student's own rollout, so the guidance penalizes cumulative drift under those rollout conditions.Source: §3.3 · p. 6

4 Experiments

The experiments assess persistent scene memory, camera control, and efficient generation. World-T3 improves revisit fidelity and camera following, including under larger sweeps, while its distilled student supports real-time generation.

4.1 Evaluation Protocol

  • The benchmark uses 120 ten-second, 1280×704 out-and-back clips and evaluates revisit fidelity beyond the 21-latent-frame context window.Revisit evaluation pairs identical requested camera poses; camera-following metrics include absolute rotation error and relative pose error.Source: §4.1 · p. 8

4.2 Revisit Fidelity and Camera Control

  • 0.0800 to 0.0390 DINOv2 feature distance and 4.79 dB higher PSNR, alongside RotErr falling from 3.344° to 1.075° versus SANA-WM.The DINOv2 distance and PSNR gains compare the Stage 3 model with SANA-WM; the RotErr values also compare Stage 3 with SANA-WM.Source: Table 1 · p. 8, §4.2 · p. 9
  • In indoor and outdoor out-and-back examples, World-T3 retains object identities and structural alignment while baselines often change scene layout or appearance.Figure 4 compares shared timestamps, with the first and last columns showing initial and return viewpoints.Source: Figure 4 · p. 7, §4.2 · p. 9
  • World-T3 maintains high revisit semantic similarity across sweep tiers and has its largest advantage over SANA-WM in the highest-sweep tier.The tiers stratify clips by yaw sweep normalized by horizontal field of view.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Step Distillation and Efficiency

  • Distillation reduces sampling from 50 to 4 steps; the student raises revisit PSNR from 16.69 to 18.31 dB but lowers VBench from 75.15 to 74.52.The student also has lower Local Memory scores, while paired tests find no significant teacher–student discrepancy across the four camera metrics.Source: §4.3 · p. 9
  • With online memory updates active, the 4-step student sustains 20.6 FPS at 1280×704 on one NVIDIA B200 GPU.Fixed-size parametric memory and bounded recent context keep storage and per-step costs independent of rollout duration.Source: §4.3 · p. 9

5 Conclusion

World-T3 uses test-time training to consolidate historical observations into parametric memory. The paper reports reliable persistent-world-state preservation, superior revisit consistency, and 20.6 FPS generation at 1280 × 704 on one NVIDIA B200 GPU.

5 Conclusion

  • World-T3 consolidates historical observations into a parametric memory through test-time training.It is a camera-controlled world model.Source: §5 · p. 10
  • An online recursive least-squares rule selectively writes novel visual contexts to address spatial redundancy and prevent stored scene memory from degradation.The model is trained progressively and then distilled for few-step generation.Source: §5 · p. 10
  • Evaluations report superior revisit consistency and 20.6 FPS at 1280 × 704 on one NVIDIA B200 GPU.The paper concludes that World-T3 reliably preserves persistent world state.Source: §5 · p. 10

A Additional Qualitative Revisit Results

The additional material describes qualitative approximate revisits and the data and camera annotations used to train the model. The corpus combines real-world and synthetic clips, with camera trajectories recovered in a unified coordinate system.

A Additional Qualitative Revisit Results

  • Across approximate revisits in a 30-second real-camera sequence, World-T3 more consistently retains scene structures and appearance cues than DreamX-AR and SANA-WM.Examples include lamp-post, backboard, pavilion-roof, and pool-surface details.Source: A Additional Qualitative Revisit Results · p. 14

B Training Data

  • The corpus includes 601K base clips and 307K augmented clips designed for viewpoint recurrence.The collection targets visual diversity, precise camera annotations, long sequences, and revisits.Source: B Training Data · p. 14

B.1 Corpus Composition

  • The training corpus combines natural real-world footage with synthetic renderings that provide noise-free camera poses and additional trajectory patterns.Real-world sources include indoor and outdoor scenes, urban traversal, and architectural walkthroughs.Source: §B.1 · p. 14

B.2 Camera Annotation

  • VGGT-Ω recovers per-frame intrinsics and extrinsics in a unified scale, anchoring each trajectory to its initial frame as the world origin.The trajectories condition video generation and induce a co-visibility matrix measuring pairwise surface overlap.Source: §B.2 · p. 14

B.3 Data Filtering

The appendix describes filtering clips using visual and geometric criteria alongside semantic assessment. It also specifies camera-invariant scene captions and procedures for constructing recurring viewpoints in training data.

B.3 Data Filtering

  • A two-stage filtering pipeline combines objective imagery and geometry criteria with vision-language-model semantic assessment.Objective checks constrain perceptual quality, motion, luminance, and camera geometry; semantic screening checks entity counts and usability.Source: §B.3 · pp. 14, 16

B.4 Text Captioning

  • Scene-static captions describe objects, arrangement, materials, colors, illumination, and intrinsic dynamics while excluding camera-motion language.This keeps camera motion communicated through pose conditioning rather than text.Source: §B.4 · p. 16

B.5 Revisit Construction

  • The corpus creates revisits by replaying camera paths in static real-world scenes and designing closed-loop synthetic trajectories, yielding 307K recurring-viewpoint clips.Dynamic entities that change between visits are excluded from replayed clips.Source: §B.5 · p. 16

C Training Details

Training proceeds through five runs grouped into three stages, moving from camera-conditioned generation to persistent memory and few-step distillation. The runs share a fixed video configuration, with stage-specific training and memory settings.

C Training Details

  • Five consecutive runs implement three stages: camera-conditioned SFT, block-causal adaptation, memory-augmented fine-tuning, consistency distillation, and distribution matching distillation.The first run is Stage 1, the next two are Stage 2, and the final two are Stage 3.Source: C Training Details · p. 16

C.1 Shared Configuration

  • Training uses 1280 × 704 video at 16 FPS, with a frozen VAE producing 48-channel latents on a 44 × 80 grid.Unless specified otherwise, temporal settings such as context windows and rollout lengths are measured in latent frames.Source: §C.1 · p. 17

C.2 Stage 1: Bidirectional Camera-Conditioned SFT

  • Stage 1 trains LoRA adapters and camera modules while freezing the remaining backbone, using bidirectional attention over 21 latent frames.Camera conditioning is randomly dropped with probability 0.05; the learning rate peaks at 5 × 10^-5 after 500 warm-up iterations.Source: §C.2 · p. 17

C.3 Stage 2: Autoregressive Adaptation and Memory Training

  • Stage 2 first adapts the model to block-causal generation, then jointly fine-tunes the backbone, camera modules, and TTT memory.The autoregressive stage uses three-latent-frame blocks and a 21-latent window; memory training uses 63-latent episodes and 14 online consolidation updates per episode.Source: §C.3 · pp. 17–18

C.4 Stage 3: Few-Step Distillation

  • Stage 3 applies four-step consistency distillation followed by dual-teacher distribution matching, using the Stage 2 memory teacher and then bidirectional and memory-augmented teacher guidance.The DMD phase runs for 800 iterations, with generator updates every fifth iteration; the final model is the iteration-800 EMA checkpoint.Source: §C.4 · p. 18

D.1 Evaluation Set and Shared Preprocessing

The revisit benchmark uses held-out clips selected and stratified by trajectory geometry and source domain. Evaluation standardizes generation conditions and sequence length, with a small encoding asymmetry between reference and generated videos.

D.1 Evaluation Set and Shared Preprocessing

  • The revisit benchmark contains 120 clips from an unseen, held-out test split, selected using a minimum normalized yaw-sweep constraint.The resulting set has a median angular sweep of 38° and median sweep/FOV ratio of 0.46, with source domains represented according to their proportions in the dataset pool.Source: §D.1 · pp. 18–19
  • All methods use identical seeds, prompts, and prescribed 6-DoF camera trajectories, and are evaluated for 160 frames at 16 FPS and 1280×704 resolution.The ground-truth reference is decoded from cached VAE latents and truncated to 160 frames.Source: §D.1 · p. 19
  • Ground-truth videos pass through one encoding stage, while generated videos are re-encoded and pass through two, an asymmetry that can slightly favor ground truth.Evaluation truncates by frame count rather than elapsed time to keep sequence lengths uniform.Source: §D.1 · p. 19

D.2 Metric Definitions

The section defines revisit, pixel-fidelity, camera-following, visual-quality, and local-temporal metrics. Together, these metrics assess return-view similarity and reference fidelity alongside camera errors and short-term continuity.

D.2 Metric Definitions

  • Revisit similarity compares DINOv2 features for four fixed outbound-return frame pairs, whose temporal separations all exceed the 21-latent-frame active context window.The fixed-index pairing retains camera-following errors rather than matching frames by optical flow.Source: §D.2 · p. 19
  • The reference-based revisit delta is the mean increase in distance to ground truth from first to return visits, distinguishing faithful retention from self-consistent but incorrect content.The comparison averages outbound and return views against their corresponding VAE-decoded references.Source: §D.2 · p. 19
  • 19.99 dB first-visit PSNR and 18.31 dB return-visit PSNR are reported for the 4-step student model, a 1.68 dB degradation at full 1280 × 704 resolution.Pixel-level metrics also penalize camera-control drift alongside appearance degradation.Source: §D.2 · p. 19
  • Absolute camera errors compare reconstructed and reference poses, while relative pose errors measure consecutive-frame rotation and translation; absolute errors reveal accumulated pose drift.Translation errors use a scalar alignment, while rotation metrics use no alignment.Source: §D.2 · pp. 19–20
  • VBench reports seven dimensions on a 0–100 scale, with its overall score defined as their unweighted average.The dimensions include imaging and aesthetic quality, subject and background consistency, motion smoothness, temporal flickering, and overall consistency.Source: §D.2 · p. 20
  • The Local Memory score combines detected scene cuts and visual mutations to assess localized temporal smoothness and short-term frame transitions.Frames are sampled every 0.5 seconds, and visual mutations are adjacent sampled frames with DINOv2 feature distance above 0.10.Source: §D.2 · p. 20

D.3 Additional Results and Trade-offs

World-T3’s student retains revisited scene content better across complementary metrics and paired comparisons, while trading off some visual-quality scores. The section also reports real-time inference and describes the evaluation settings and optimizations.

D.3 Additional Results and Trade-offs

  • Student return-visit error rises to 0.0499 versus HY-World’s 0.1867, with error accumulation of 0.0218 versus 0.1593.The comparison assesses scene retention across evaluated revisit intervals, separately from aggregate visual quality.Source: §D.3 · p. 21

D.4 Complementary Metrics for Revisit Fidelity

  • Across all six revisit measures, World-T3 Stage 3 and Stage 2 rank first and second among evaluated methods.The measures cover outbound-return consistency and GT-referenced fidelity.Source: §D.4 · p. 21, Table 6 · p. 21

D.5 Paired Statistical Comparisons

  • The student beats SANA-WM on DINOv2 revisit distance on 111/120 clips and revisit PSNR on 112/120; paired VBench tests favor Matrix-Game 3.0 and HY-World 1.5.On 96/120 clips, the student simultaneously surpasses every external baseline in memory similarity and revisit PSNR, and closes about 52% of the DINOv2 feature-distance gap to the ground-truth limit.Source: §D.5 · pp. 21–22

D.6 Camera-Sweep Analysis

  • In the highest camera-sweep tier, the student’s rotational camera error is 1.288° versus 2.241° for the teacher and 4.316° for SANA-WM.Across tiers T1, T2, and T3, it has lower DINOv2 distances than SANA-WM on 37/40, 36/40, and 38/40 clips, respectively.Source: §D.6 · p. 22

D.7 Teacher-Student Comparisons

  • Against the teacher, the student improves revisit PSNR on 91/120 clips and revisit LPIPS on 86/120, while first-visit PSNR and camera metrics remain comparable.Revisit SSIM rises from 0.5313 to 0.5795, while Local Memory decreases from 0.9442 to 0.9258.Source: §D.7 · p. 22

E Inference Efficiency

  • The 4-step student generates 20.6 FPS at 1280 × 704 on one NVIDIA B200 GPU with online memory updates enabled.The teacher uses 50 denoising steps per block; the student uses four, a 12.5× reduction in step count.Source: E Inference Efficiency · p. 22

E.1 Workload and Measurement Settings

  • The efficiency evaluation measures 30-second rollouts on B200 GPUs, where the context window is full for most of the rollout.Settings include one discarded warm-up and the median of three timed rollouts.Source: §E.1 · p. 23

E.2 Optimizations

  • Optimizations reuse memory features and unchanged context keys and values, compile model passes, accelerate decoding, and remove host-side stalls.The cumulative optimization table measures a 30-second rollout on one B200.Source: §E.2 · p. 23, Table 7 · p. 24

E.3 Results

Cumulative optimizations reduce single-GPU rollout latency, and fidelity tests compare the accelerated implementation with reference implementations. The student is the only compared model reported to generate a 30-second rollout faster than real time on one GPU.

E.3 Results

  • Cumulative optimizations cut 30-second rollout latency from 70.48 to 22.59 seconds, a 3.12× reduction.Sampling time per block falls from 943 to 391 ms, and decoding time from 890 to 182 ms.Source: §E.3 · p. 24, Table 9 · p. 25

E.4 Numerical Fidelity

  • The accelerated rollout is closer to the FP32 rollout than the reference bf16 rollout on 7 of 8 clips.At operator level, the accelerated implementation is closer to FP32 than the reference in all three tested cases; replay-benchmark videos were generated with the evaluation implementation, and the tests address their equivalence directly.Source: §E.4 · pp. 24–25

E.5 Efficiency Comparison with Other Video World Models

  • On one GPU, World-T3’s student is the only compared model that generates a 30-second rollout faster than real time.With multiple GPUs, HY-World 1.5 also becomes faster than real time, while World-T3 remains the most efficient.Source: §E.5 · p. 25

F Limitations

World-T3 still faces pose-estimation and long-horizon visual-quality limits, although the authors describe it as a viable, efficient path toward persistent memory.

F Limitations

  • Pose-estimation noise or drift can misalign pose-keyed retrieval and accumulate over extended horizons, reducing memory recall on revisits.The issue is especially noted in weak-texture environments or during rapid camera maneuvers.Source: F Limitations · p. 25
  • Higher-quality camera-trajectory datasets or more robust camera-estimation pipelines are needed to address pose inaccuracies.Source: F Limitations · p. 26
  • Beyond the training context length, frames can gradually lose visual fidelity or fine-grained texture; photorealistic high-resolution generation and unbounded extrapolation remain open challenges.Source: F Limitations · p. 26
  • The authors characterize World-T3 as a viable and efficient path toward persistent memory in interactive video world models.Source: F Limitations · p. 26
Loading 2610.17434v1…