Source-linked AI summary
AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
Zun Wang, Han Lin, Jaehong Yoon, Jaemin Cho, Yue Zhang, Mohit Bansal
TL;DR
Long-horizon camera-controlled video generation is hindered by cross-view misalignment in globally reconstructed 3D memories. AnchorWeave replaces that global memory with retrieved local geometric memories and a pose-aware multi-anchor controller, improving visual quality and long-term scene consistency across evaluated settings.
Problem
Global 3D memories accumulate pose- and depth-induced cross-view misalignment, degrading the conditioning used for long-horizon world-consistent video generation.
Method
AnchorWeave stores per-frame local point clouds, retrieves them by trajectory coverage, and jointly fuses rendered anchors using shared attention and pose-guided control.
Results
AnchorWeave significantly improves visual quality and long-horizon scene consistency, with ablations supporting local memory, coverage-driven retrieval, and multi-anchor control.
Takeaways & Limitations
Local geometric conditioning with learned reconciliation offers a reliable alternative to global geometric fusion for scalable long-horizon video generation.
Abstract
from arXiv · showhide
Maintaining spatial world consistency over long horizons remains a central challenge for camera-controllable video generation. Existing memory-based approaches often condition generation on globally reconstructed 3D scenes by rendering anchor videos from the reconstructed geometry in the history. However, reconstructing a global 3D scene from multiple views inevitably introduces cross-view misalignment, as pose and depth estimation errors cause the same surfaces to be reconstructed at slightly different 3D locations across views. When fused, these inconsistencies accumulate into noisy geometry that contaminates the conditioning signals and degrades generation quality. We introduce AnchorWeave, a memory-augmented video generation framework that replaces a single misaligned global memory with multiple clean local geometric memories and learns to reconcile their cross-view inconsistencies. To this end, AnchorWeave performs coverage-driven local memory retrieval aligned with the target trajectory and integrates the selected local memories through a multi-anchor weaving controller during generation. Extensive experiments demonstrate that AnchorWeave significantly improves long-term scene consistency while maintaining strong visual quality, with ablation and analysis studies further validating the effectiveness of local geometric conditioning, multi-anchor control, and coverage-driven retrieval.
1. Introduction
Long-horizon, camera-controlled video generation remains difficult because globally fused 3D memories accumulate cross-view errors that contaminate conditioning. AnchorWeave replaces this memory with local geometric memories and learns to combine them for more consistent generation.
- Long-horizon video world models must generate and revisit environments consistently under user-controlled camera viewpoints.
- Global-memory methods reconstruct unified 3D scenes from historical views and render anchor videos along target camera trajectories.
- Pose and depth errors place the same surfaces at different 3D locations, producing fused geometry whose artifacts can cause ghosting, drift, and hallucinated content.
- AnchorWeave uses per-frame local point-cloud memories, which avoid multi-view fusion artifacts and provide cleaner geometric conditioning.
- AnchorWeave retrieves and weaves multiple local anchors during an update–retrieve–generate loop, significantly improving visual quality and long-horizon scene consistency.Experiments compare against global point-cloud and implicit cross-view conditioning baselines on RealEstate10K and DL3DV.
2. Related Work
Prior camera-controllable video work uses explicit camera conditioning, structured 3D guidance, trajectory or motion encoding, optimization, and simulator-generated data. Memory-augmented methods extend context through retrieved history or explicit 3D representations, while AnchorWeave uses local geometric anchors instead of noisy global reconstruction.
- Camera-controllable generation spans text-to-video, image-to-video, and video-to-video settings, with early methods injecting camera parameters into diffusion models.
- Later approaches add structured 3D guidance through rendered point clouds or anchor videos to improve geometric faithfulness.
- Other methods formulate camera control through trajectory tracking or motion encoding, use test-time optimization, or train on simulator-generated paired data.
- Memory-augmented video generation retrieves past frames or constructs explicit 3D memory to extend consistency beyond limited generator context.
- Unlike global-point-cloud approaches such as SPMem, Spatia, and MagicWorld, AnchorWeave uses local geometric anchors with multi-anchor joint conditioning.
3. Methodology: AnchorWeave
AnchorWeave maintains per-frame local point-cloud memories, retrieves those covering a target trajectory, and combines their rendered anchors with pose-aware multi-anchor control. The resulting fused signal guides a video diffusion backbone through an iterative long-horizon generation loop.
- 3.2. Local Geometric Memory: AnchorWeave stores per-frame local point clouds with camera poses instead of fusing observations into one global 3D memory.
- 3.1. Preliminary: Video Diffusion Models: The system uses a standard DiT-based latent diffusion backbone that denoises encoded video latents conditioned on camera and anchor inputs.
- 3.3. Coverage-driven Memory Retrieval: For each target-trajectory chunk, coverage-driven retrieval first filters memories by field-of-view overlap and greedily maximizes newly covered visible area.
- 3.3. Coverage-driven Memory Retrieval: Retrieved memories are rendered into anchor clips, paired with retrieved-to-target relative camera poses, and provided as conditioning alongside the target camera trajectory.
- 3.4. Generation with Multi-Anchor Weaving Controller: The multi-anchor controller jointly attends over anchor latents, estimates pose-aware importance weights, fuses features, and injects the resulting control signal into the backbone.
- 3.5. Iterative Generation: Repeating update, retrieve, and generate steps converts new frames into local memories and supports long-horizon, spatially consistent generation.
4. Experiments
AnchorWeave is evaluated on partial-revisit video generation using RealEstate10K and DL3DV, with quantitative, qualitative, and ablation studies. It improves scene consistency and visual quality across baselines, while local memory, multi-anchor control, and more retrievals provide measurable benefits.
- Evaluation Setup: Evaluation uses 500 videos with 49 target frames and 21 historical-context frames, measuring reconstruction fidelity, consistency, and perceptual quality.PSNR and SSIM assess reconstruction; VBench assesses subject and background consistency, motion smoothness, temporal flickering, aesthetics, and imaging quality.
- Quantitative Comparison: AnchorWeave achieves the best overall performance across reconstruction and perceptual metrics on RealEstate10K and DL3DV.The method improves temporal quality over comparable consistency baselines, including smoother motion and reduced flickering.
- Qualitative Comparison: AnchorWeave preserves scene structure more reliably than baselines, which exhibit blurring, hallucinations, spatial drift, or misaligned details under revisiting.Context-as-Memory maintains consistency but lacks precise camera control, while global point-cloud conditioning can produce severe hallucinations.
- Ablations: Local 3D memory improves both PSNR and SSIM over global memory conditioning.Local memories provide multiple geometric anchors instead of a single anchor rendered from fused global geometry.
- Ablations: Pose-conditioned fusion suppresses misaligned anchors, while joint attention aggregates complementary evidence into sharper, more coherent geometry.Simple averaging and separate attention produce visible artifacts, blurred structures, or inconsistent geometry.
- Ablations and Long-Horizon Examples: Increasing retrievals per chunk consistently improves PSNR and SSIM by providing complementary spatial evidence.The same trend appears qualitatively as reduced misalignment and improved consistency, and long-horizon examples preserve geometry across multiple generated segments.
5. Conclusion
AnchorWeave replaces brittle global 3D reconstruction with clean, view-aligned local geometric memories for long-horizon video generation. Coverage-driven retrieval and adaptive multi-anchor fusion improve visual quality, camera controllability, and long-term consistency across scenes and tasks.
- Conclusion: AnchorWeave replaces a fused global 3D memory with multiple clean, view-aligned local geometric memories.The framework learns to reconcile inconsistencies among retrieved local anchors during generation.
- Conclusion: Coverage-driven retrieval and adaptive multi-anchor fusion provide spatial guidance for long-horizon, camera-controlled generation.The conclusion identifies local geometric conditioning and learned reconciliation as an alternative to global geometric fusion.
- Conclusion: The reported results support improved visual quality, camera controllability, and long-term consistency across diverse scenes and tasks.
A.1. Training/Inference Details
Training stores per-frame local point clouds as spatial memories and optimizes weaving modules on frozen DiT-based video backbones. The configuration uses memory augmentation, frame masking, camera and anchor controls, and fixed training resources.
- Data and Memory: Per-frame local geometry is estimated with TTT3R and stored as local point-cloud memories from 10K training videos.Training uses RealEstate10K and DL3DV.
- Memory Configuration: Memory retrieval uses 8-frame chunks with at most four local point clouds, plus random memory sampling and frame masking for robustness.
- Backbones: AnchorWeave is applied to CogVideoX-I2V-5B and Wan2.2-TI2V-5B while freezing backbone weights and optimizing newly introduced weaving modules.Wan2.2 uses 81-frame videos and CogVideoX uses 49-frame videos.
- Controller: The controller is injected into the first third of backbone layers and combines fused anchor latents with target camera pose embeddings.Both control signals use weight 1.0, and the controller operates during the first 80% of denoising steps.
- Optimization: Training uses Adam at 2×10^-4 for 10K steps with batch size 8 on 8×H100 GPUs, taking approximately one day.
A.2. Baseline Re-implementation
The baselines Context-as-Memory and SPMem are reimplemented on CogVideoX using the same training data as AnchorWeave. Their conditioning mechanisms are reproduced with context-frame, global-point-cloud, and keyframe pathways.
- Context-as-Memory: Context-as-Memory is reimplemented on CogVideoX with the same RealEstate10K and DL3DV training data as AnchorWeave.It selects 12 context frames with high trajectory overlap and fine-tunes the backbone using LoRA.
- SPMem: SPMem is reimplemented on CogVideoX using the same dataset and conditions generation on rendered global point clouds plus keyframe latents.The implementation uses a ControlNet branch and additional cross-attention layers.
B. Coverage-Driven Memory Retrieval Algorithm
The retrieval procedure constructs anchor videos by selecting local memories that provide useful geometric coverage for each target-trajectory chunk. The first rendered frame is always retrieved first as a stable geometric reference.
- The first rendered video frame is enforced as the first retrieval to provide a stable geometric reference.
C. Camera Controllability Comparison with Baselines
AnchorWeave is evaluated for camera controllability under single-image and free-form camera-control settings. It achieves lower camera errors and remains more stable under large viewpoint changes than the compared baselines.
- The evaluation samples 300 videos from DL3DV and RealEstate10K using only the first frame as input.Camera control is measured with rotation and translation errors under the Basic Image-to-Video Camera Control setting.
- AnchorWeave achieves the lowest rotation and translation errors among all baselines.The comparison follows the EPiC evaluation protocol, and lower values are better.
- AnchorWeave outperforms prior approaches even when only a single anchor is provided at test time.The method combines anchor-video visual guidance with explicit camera pose control.
- Under free-form camera trajectories, competing methods exhibit hallucinations, rendering artifacts, blur, degraded details, or motion misalignment.The reported weaknesses include hallucinated structures in ViewCrafter and severe hallucinations in SPMem from imperfect global point-cloud renderings.
- AnchorWeave remains stable under free camera control, preserving visible-region geometry while synthesizing plausible content in newly exposed areas.
F. Long-Horizon Examples
AnchorWeave produces long-horizon explorations across static, panoramic, dynamic, and third-person scenes. The examples report persistent spatial consistency through viewpoint changes and zero-shot generalization to character-centric scenarios.
- The examples generate 241-frame sequences by iteratively producing 81-frame image-to-video segments conditioned on the previous segment’s last frame.Dynamic third-person scenes mask the character during point-cloud memory construction.
- Objects such as tables, chairs, bridges, temples, and cabins remain spatially consistent across long sequences and cross-segment viewpoint changes.
- A 360-degree rotation returns to the original viewpoint while preserving scene content throughout the sequence.
- Point-cloud control supports stable third-person character motion aligned with the camera trajectory while preserving revisited environmental features.
- The third-person scenario is generated zero-shot despite training only on static-scene datasets.The reported training datasets are DL3DV and RealEstate10K, without third-person character or dynamic gameplay supervision.