Source-linked AI summary
CityRAG: Stepping Into a City via Spatially-Grounded Video Generation
Gene Chou, Charles Herrmann, Kyle Genova, Boyang Deng, Songyou Peng, Bharath Hariharan, Jason Y. Zhang, Noah Snavely, Philipp Henzler
TL;DR
CityRAG addresses the challenge of generating navigable, 3D-consistent simulations grounded in real locations while preserving weather and dynamic objects. It retrieves geo-registered video context and trains on temporally unaligned captures to separate static geography from transient attributes. The authors report coherent, physically grounded, minutes-long sequences with stable scene structure and navigation across complex trajectories.
Problem
Generating a 3D-consistent, navigable simulation of a real location requires reconstructing geography while respecting transient conditions such as weather and pedestrians.
Method
CityRAG retrieves geo-registered videos as scene context and uses temporally unaligned training data to disentangle static geometry from transient environmental attributes.
Results
CityRAG demonstrates strong 3D scene understanding, static–dynamic disentanglement without heuristics, and realistic sequences across diverse settings.
Takeaways & Limitations
The approach supports physically grounded walkthroughs that preserve real-location geography and realistic motion while enabling user navigation.
Takeaways & Limitations
CityRAG is not real-time and focuses on spatial grounding rather than the full set of requirements for interactive video world models.
Abstract
from arXiv · showhide
We address the problem of generating a 3D-consistent, navigable environment that is spatially grounded: a simulation of a real location. Existing video generative models can produce a plausible sequence that is consistent with a text (T2V) or image (I2V) prompt. However, the capability to reconstruct the real world under arbitrary weather conditions and dynamic object configurations is essential for downstream applications including autonomous driving and robotics simulation. To this end, we present CityRAG, a video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene, while maintaining learned priors for complex motion and appearance changes. CityRAG relies on temporally unaligned training data, which teaches the model to semantically disentangle the underlying scene from its transient attributes. Our experiments demonstrate that CityRAG can generate coherent minutes-long, physically grounded video sequences, maintain weather and lighting conditions over thousands of frames, achieve loop closure, and navigate complex trajectories to reconstruct real-world geography.
1 Introduction
CityRAG targets 3D-consistent, navigable simulations grounded in real-world geography while preserving transient conditions from an input image. It retrieves geo-registered context and uses temporally unaligned data to separate static scene structure from weather and dynamic objects.
- Motivation: CityRAG generates spatially grounded environments that preserve real buildings, roads, weather, lighting, and dynamic objects from an input image.The goal is a navigable simulation of a specific physical location rather than an unconstrained visual hallucination.
- Motivation: Existing image- and text-conditioned generators lack reliable external-world grounding during inference, while NeRF-like representations require dense same-moment captures and struggle with realistic motion or appearance changes.
- Approach: CityRAG retrieves geo-registered data as scene context while retaining generative priors for complex motion and appearance changes.The retrieved context provides a prior for scene identity, including static structures such as buildings and roads.
- Approach: Temporally unaligned Street View pairs teach the model to disentangle permanent geometry from transient environmental conditions.The dataset contains same-location sequences captured at different times, including differing illumination and traffic conditions.
- Results: Experiments report strong 3D scene understanding, dynamic–static disentanglement without heuristics, and realistic sequences across diverse settings.
2 Related Works
Related work spans scalable video generation, long-term consistency, pose conditioning, external-context conditioning, and retrieval-augmented generation. CityRAG combines these directions with large-scale grounding in real-world geography.
- Video generation: T2V and I2V models are scalable, but this application additionally requires long-term consistency, pose control, and external-context integration.
- Long-term consistency: Long-context, autoregressive, and explicit-memory methods address consistency, yet prior work rarely demonstrates minutes-long videos without significant degradation.
- Pose conditioning: Pose-conditioned models provide camera control and plausible temporal sequences, whereas CityRAG additionally grounds generation in large-scale real-world data.
- Additional context: Reference-to-video and video-to-video approaches use additional context but generally lack demonstrated 3D awareness or require strict adherence to reference videos.
- Retrieval augmentation: Retrieval-augmented visual generation retrieves images or motion examples, while CityRAG retrieves geo-registered data to ground video generation in the real world.
3 Method
CityRAG trains on paired, geo-registered Street View sequences and conditions generation on an input image, user trajectory, and retrieved geographic context. At inference, retrieval and autoregressive frame updates support long, consistent navigation, including stitched arbitrary paths.
- Data: The training corpus contains 5.5M panorama–pose pairs collected across 10 cities, with captures of the same streets at different times when available.
- Data: Training pairs use continuous paths captured at different times with average spatial distance below 5 meters, encouraging static–transient disentanglement.The construction uses N=73 frames and ϵ=5 meters, yielding 1.3M panoramas for training.
- Architecture: The generator conditions on the first image, a trajectory, and geo-registered videos retrieved along that trajectory.The first image supplies scene initialization, including lighting and dynamic-object behavior; the trajectory supplies camera motion.
- Architecture: Geo-registered video context is injected through cross-attention so target frames can attend to the entire conditioning sequence rather than relying only on pixel alignment.Varying the conditioning length forces extraction of global context despite spatial and temporal discrepancies.
- Inference: Inference retrieves Street View data for a user-defined path, generates a segment, updates the first frame and location, and repeats autoregressively.Distinct retrieved videos can be stitched to approximate arbitrary turns and trajectories.
- Inference: 1,000 frames remain consistent in a San Juan example conditioned on four stitched geospatial videos, despite discontinuities in the retrieved context.The result supports robustness to appearance changes and pixel mismatches in geospatial conditions.
4 Experiments
CityRAG is evaluated against video-to-video, image-to-video, and style-transfer baselines on unseen trajectories and neighborhoods. It produces spatially grounded, trajectory-flexible videos with stronger scene fidelity and perceptual similarity than the tested baselines, while generalizing to untrained cities.
- Evaluation setup: The evaluation compares CityRAG with I2V pose control, V2V pose control, and V2V style-transfer baselines on unseen streets and trajectories.Testing includes held-out neighborhoods in trained cities, two cities reserved only for testing, and trajectories containing at least a 45° rotation.
- Qualitative comparisons: CityRAG follows initial weather and dynamic-object conditions while rendering static structures and maintaining realistic motion through challenging turns.In Scene A, the black car continues moving and reappears after leaving the camera’s view; in Scene B, buildings and fences are rendered despite temporal mismatch in the geospatial condition.
- Qualitative comparisons: All baselines fail on the full task: AnyV2V does not move the camera, Gen3C breaks down during turns, and Gen3C V2V supports only limited camera movement.Gen3C I2V remains stable during initial forward motion but degrades when the car turns, while Gen3C V2V struggles more with complex poses.
- Qualitative comparisons: CityRAG follows user-defined trajectories despite mismatched geospatial context and performs a 360° rotation within one sequence.The 360° rotation is double the maximum rotation present in the training set; the caption attributes low visual quality to the temporal VAE.
- Quantitative comparisons: CityRAG outperforms all tested baselines on ground-truth scene fidelity and significantly leads on perceptual metrics including LPIPS and FID.The authors note that generative models can incur pixel-wise error from stochasticity, camera shifts, or plausible geometry hallucinations, even when perceptual similarity is strong.
- Generalization: 15.11 PSNR, 0.461 SSIM, 0.517 LPIPS, and 16.90 FID are reported for untrained cities, with no meaningful performance gap from the full test set.The results suggest generalization across diverse scenes and conditions, according to the authors.
- User study: In the user study, CityRAG is the only method reported to generate videos that are both smooth continuations from first images and faithful renders of real locations.Users rated visual quality, smooth continuation, and physical-location fidelity on a 1–3 scale; Figure 8 plots continuation and location fidelity, with radius indicating visual quality.
- Discussion: CityRAG creates a simulated environment that preserves scene initialization, understands global structure despite imperfect geospatial conditions, and supports arbitrary navigation trajectories.This discussion connects spatial grounding, transient-object animation, and flexible navigation within the reported scope of the experiments.
5 Conclusion
CityRAG emphasizes adherence to real-world locations and uses a data-driven strategy to disentangle static and dynamic scene attributes. The paper presents this grounding as a basis for applications requiring specific environment layouts.
- CityRAG emphasizes adherence to the real world by grounding video generation in physical locations.
- Temporally unaligned training data teaches CityRAG to disentangle static and dynamic attributes through a fully data-driven strategy.
- The method is intended to support applications that rely on specific environment layouts.
A Limitations and Future Work
CityRAG has limitations in autoregressive generation, controllability, real-time interaction, text conditioning, and data coverage. The authors identify these boundaries as directions for future work.
- Autoregression uses only the generated last frame as the next sample’s first frame, although static structures remain stable for dozens of iterations.The authors note that typical I2V models drift after 1 or 2 iterations, whereas CityRAG remains stable, especially in static structures.
- Static-versus-transient disentanglement is completely data-driven, without heuristics for individual objects.The authors suggest fine-grained controls and annotations to improve controllability and customization.
- CityRAG is not real-time, and interactive video worlds require progress beyond spatial grounding, including faster inference and controllability.The project focuses specifically on spatial grounding, realism, and long-horizon temporal consistency.
- Because the training data lack captions, finetuning uses a fixed prompt and the model no longer responds to new text captions.Captioning Street View and text conditioning are left for future work.
- The data exclude snowy, rainy, and nighttime conditions because of hardware and sensor limitations.The authors identify data augmentation and additional modalities such as text as future work.
B Latency and Inference Costs
Inference costs consist of geospatial retrieval and video generation. Retrieval is indexed for fast lookup, while generation remains substantially slower than real-time despite optimization techniques.
- 73 frames at 480p produce a latent size of 18 × 30 × 52 after VAE downsampling and DiT patching.The dimensions are ordered as temporal, height, and width.
- Nearest-neighbor geospatial lookup takes O(log N) after a one-time cKDTree city-map construction, while subsequent relative-pose frames retrieve in O(1).The one-time map-building cost is reported as less than a minute in wall time.
- Video generation costs are comparable to the Wan 2.1 base model and use TeaCache, DeepSpeed-Ulysses, and 40 UniPC inference steps.
- 90 seconds for 73 frames on 8 A100 GPUs corresponds to roughly 0.8 FPS, while 8 B200s reduce this to 30.5 seconds or 2.4 FPS.
- A model distilled to 4 steps could roughly reduce inference time 10 fold, according to the authors’ estimate.Faster inference was not the project’s focus.
C Training Optimization
Training uses v-prediction with a shifted noise schedule, the Muon optimizer, and a fixed learning rate with warmup. The model is trained on 32 A100 GPUs for 20k iterations over one week.
- Training adopts v-prediction with a 3.0 shift toward higher timesteps and uses Muon with a fixed 1e-5 learning rate and warmup.
- The model is trained on 32 A100 GPUs for one week over 20k iterations.
D Architecture Ablations
Architecture ablations compare conditioning strategies, while the user study defines criteria for visual quality, temporal continuation, and static-scene fidelity. The experiments favor residual addition and the proposed cross-attention approach over alternatives.
- Trajectory conditioning: Residual addition achieved the best trajectory-conditioning metrics in the Honolulu ablation.The ablation trained and tested on Honolulu, including streets from unseen neighborhoods.
- Condition representation: Processing condition frames independently through the VAE required three times as many iterations to converge as video-latent processing.The result supports using the Wan base model’s temporal VAE representation for conditioning.
- Geospatial conditioning: The proposed cross-attention conditioning yielded the best adherence to the supplied conditions compared with alternatives such as ControlNet.The comparison used the same first-image and trajectory conditions as the main evaluation setup.
- User study: The user study evaluates methods using head-to-head visual quality judgments and 1-to-3 ratings for temporal continuation and static geographic similarity.Q2 focuses on whether a video appears to continue from the starting frame, while Q3 ignores cars and pedestrians to assess buildings, roads, and layout.
- Related work: Related city-scale reconstruction methods primarily establish large-scale geometry or interactive flythroughs, whereas CityRAG targets grounded video generation with dynamic appearance.The cited reconstruction approaches scale SfM or NeRF-style representations to city-sized regions.
G.1 Privacy and Anonymization
The authors anonymize Street View data and reduce identifiable people in released results, while acknowledging that sensitive information cannot be removed completely.
- Data cleaning: License plates and faces were blurred before collection, and buildings or streets were blurred on request.The authors state that they did not access the raw imagery.
- Presentation safeguards: People were replaced or masked in presented condition images and public geospatial videos where applicable.The authors used synthetic replacements in some images and segmentation-based masking for public release.
- Residual risk: The authors acknowledge that sensitive information cannot be removed 100% and plan to monitor removal requests after release.This bounds the effectiveness of the privacy mitigations.
G.2 Bias in Data Distribution
Although the dataset spans 10 cities across four continents, most data come from Western countries, creating a potential representation bias.
- Distributional limitation: Most collected data are located in Western countries despite coverage of 10 cities across four continents.The authors identify more diverse data collection or algorithmic corrections as future mitigation strategies.