Source-linked AI summary
Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering
Wenchao Ma, Changran Liu, Sharon X. Huang, Haomiao Jiang
TL;DR
Long-horizon autoregressive renderers can lose revisit consistency when bounded KV caches evict earlier views. This paper introduces inference-time correspondence-guided memory, which outperforms training-free cache-management schemes on consistency metrics without degrading overall video quality.
Problem
Bounded KV caches make pretrained autoregressive renderers regenerate inconsistent appearances when cameras revisit locations after earlier context is evicted.
Method
The method retrieves pose-matched historical latent chunks and uses depth-reprojection correspondences to bias attention toward geometrically corresponding tokens.
Results
The method consistently outperforms existing training-free cache-management schemes on pixel, semantic, and geometric consistency metrics without degrading VBench-Long video quality.
Takeaways & Limitations
Training-free correspondence-guided memory improves revisit consistency while preserving overall visual fidelity in streaming generative rendering.
Takeaways & Limitations
The method relies on engine-supplied camera poses and metric depth, limiting direct applicability when those annotations are unavailable.
Abstract
from arXiv · showhide
Recent conditional video generation models have shown promising potentials to transform 3D engine renderings, such as depth maps and untextured geometry, into photorealistic videos for gaming and immersive content creation. These applications require long-horizon auto-regressive generation that continuously synthesizes new frames while preserving a persistent 3D world. Auto-regressive generators synthesize video chunk by chunk with a bounded KV cache, so when the camera revisits a location after its context has been evicted, the model often regenerates inconsistent appearance, even though the conditioning renderings (e.g., depth) remain perfectly aligned with the underlying geometry. We address this revisit inconsistency without any post-training by exploiting correspondences the 3D engine already provides: temporal correspondence retrieves pose-matched historical latent chunks into the KV cache as loop-closure memory, while spatial correspondence from camera pose and depth reprojection biases token-level attention toward geometrically corresponding regions of the retrieved chunks. We demonstrate our method on loop-closure trajectories mined from TartanAir and TartanGround dataset to mirror complicate real-world application scenarios, where it outperforms existing training-free baselines on revisit consistency without losing overall video quality. Project Page: https://wenchao-m.github.io/ClosetheLoop.github.io/
1 INTRODUCTION
Long-horizon autoregressive rendering can lose previously visited views when bounded KV-cache context is evicted, causing revisit inconsistency. The proposed training-free correspondence-guided memory retrieves pose-matched latent chunks and uses depth-based spatial attention to improve consistency without degrading overall video quality.
- Problem: Long-horizon streaming generation must preserve a persistent world as cameras leave and later revisit regions beyond the video model’s native context window.The generator synthesizes frames chunk by chunk while maintaining consistency over horizons exceeding its bounded context.
- Problem: Revisit inconsistency occurs when generators forget previously visited content, motivating memory mechanisms that extend beyond local history.Prior approaches use explicit 3D state, geometric memories, hierarchical latents, or learned context-querying modules.
- Method: The method retrieves pose-matched historical latent chunks into the clean KV cache as loop-closure memory, exposing previously generated views beyond the default cache window.Temporal correspondence selects historical chunks whose camera poses match the current chunk.
- Method: Camera pose and metric depth reproject current tokens onto corresponding regions in retrieved frames, guiding spatial attention during generation.The spatial correspondence mechanism links current-frame tokens to geometrically corresponding regions in the retrieved chunk.
- Results: The method consistently outperforms existing training-free cache-management schemes on pixel, semantic, and geometric-correspondence metrics without degrading VBench-Long video quality.Evaluation uses loop-closure trajectories constructed from TartanGround and TartanAir, and also demonstrates integration with a live game engine.
2 RELATED WORK
Related work spans generative rendering from structured engine or simulator signals and methods for maintaining long-term consistency in generated video. Existing approaches use explicit memory, retrieval, compact historical state, post-training, distillation, or inference-time memory design.
- Generative Rendering: Generative rendering combines graphics-engine controllability with generative-model realism, evolving from conditional GANs to diffusion models conditioned on 3D or simulator-derived signals.Early video-to-video methods translate semantic layouts or rendered guidance into photorealistic videos, while recent diffusion methods condition image or video models on structured 3D signals.
- Long-Term Consistency: Long-term consistency methods use explicit memory, retrieval, or compact historical state to preserve content beyond the visible field.Examples include street-view retrieval, surfel-indexed view memory, separate static-background and dynamic-object memories, compact long-horizon memory, adaptive context querying, hierarchical latents, and persistent 3D state.
- Long-Video Generation: Another line improves long-video generation through post-training, distillation, or inference-time memory design.Representative methods train memory blocks, combine self-forcing with sliding-window causal attention and attention sinks, or augment autoregressive generation with hybrid state-space memory.
3 PRELIMINARIES
Autoregressive video diffusion models generate long videos in latent chunks using a fixed-size sliding-window KV cache, keeping memory and per-chunk computation constant. The method is instantiated as Causal Wan-VACE by causalizing both the Wan-VACE main and VACE control branches, each with its own KV cache.
- Autoregressive Video Diffusion: Autoregressive video diffusion models denoise each current latent chunk while attending to a fixed-size sliding window of previously generated chunks through a KV cache.This design keeps memory and per-chunk computation constant as video length grows.
- Autoregressive Video Diffusion: Self-Forcing distills a bidirectional video diffusion teacher into a causal, few-step student by matching training to the inference process.
- Causal Wan-VACE: Causal Wan-VACE conditions Wan’s text-to-video DiT on depth maps encoded by the Wan VAE and processed through VACE control blocks.The control-block outputs are injected as residuals into selected backbone layers.
- Causal Wan-VACE: Because the pretrained model is bidirectional, Causal Wan-VACE causalizes both the main and VACE control branches, each with its own KV cache.
4 METHOD
The method improves long-horizon revisit consistency using two complementary, inference-time mechanisms on a frozen autoregressive generator: pose-retrieved loop-closure memory and geometry-guided attention bias. Together, they restore earlier appearance and steer attention toward geometrically corresponding regions without warping cached features.
- Loop-closure memory: Pose-retrieved loop-closure memory reinstates a historical latent chunk when the current camera pose revisits a previously generated viewpoint.Retrieval uses distance, viewing-direction, and minimum temporal-gap gates; if no candidate is admissible, the retrieved slot extends the recent window.
- Geometry-guided attention: Geometry-guided attention bias reprojects each current query token into the retrieved view and applies a Gaussian suppression prior to corresponding retrieved-key columns.The reprojection lifts tokens using metric depth and relative camera poses, while visibility testing rejects occluded correspondences.
- Loop-closure memory: The clean-cache budget is partitioned into a persistent anchor, one retrieved chunk, and a recent window, with M=4F=12 and F=3 latent frames per chunk.The anchor supplies a global reference, the recent window preserves local continuity, and the retrieved slot provides view-specific long-term memory.
- Cache implementation: Retrieved chunks are packed adjacent to the recent window in RoPE position space, and the corresponding VACE control-cache slice is copied to the same location.This keeps cache positions within a fixed range during unbounded streaming and aligns the main and control branches.
- Geometry-guided attention: The attention prior narrows where the model looks without forcing content placement, leaving the learned attention to determine what content is copied.Only logits are biased; cached keys, values, latents, and hidden states are never warped or blended, preserving sharp cached content.
5 EXPERIMENT
Experiments on TartanGround-Revisit and TartanAir-Revisit show that training-free cache management improves revisit consistency, with geometry-based memory selection performing best across consistency metrics. Qualitative results and ablations attribute gains to pose-retrieved memory and geometry-guided attention.
- Evaluation Setup: TartanGround provides challenging revisit trajectories with long gaps beyond cache horizons, translation-dominant motion, parallax, camera poses, and planar metric depth.The dataset is repurposed because its navigation trajectories naturally leave and return to places while supplying the annotations required by the method.
- Main Results: Every training-free cache-management scheme improves revisit consistency over the base model, while geometry-based memory selection achieves the best score on every consistency metric.The same ordering holds on TartanAir-Revisit, and the proposed method roughly doubles the base model’s high-confidence keypoint matches on TartanGround-Revisit.
- Qualitative Results: Qualitatively, baselines regenerate altered facades and inserted structures on return, whereas the proposed method re-renders boxed scene regions consistently across the loop.Figure 4 compares first visits with returns using reference and depth conditioning columns.
- Ablation: The attention sink stabilizes global appearance, pose retrieval supplies the earlier view required for loop closure, and geometric bias further improves keypoint matches.The geometric bias refines where each token reads within the retrieved chunk, matching the placement measured by the matcher-based metric.
- Engine Integration: The approach integrates with an in-house game engine that streams camera poses and metric depth directly to the generator for loop-closure rendering.Representative cases include a complex castle and scenes built from untextured geometric primitives, where baselines alter structures and the method preserves them.
6 CONCLUSION
The paper presents a training-free approach to revisit-consistent streaming generative rendering that uses 3D-engine correspondences for loop-closure memory and geometry-guided attention. Its main limitation is reliance on engine-supplied camera poses and metric depth.
- 6 CONCLUSION: The method retrieves pose-matched historical chunks into the bounded KV cache as loop-closure memory for revisit consistency.It exploits temporal correspondences already provided by the 3D engine.
- 6 CONCLUSION: A depth-reprojection Gaussian bias steers attention toward geometrically corresponding tokens in retrieved chunks.This uses spatial correspondence from the engine’s geometry-related inputs.
- 6 CONCLUSION: The main limitation is reliance on engine-supplied camera poses and metric depth.The passage notes that estimated geometry from online reconstruction methods such as VG... could address this dependency.