PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding World Simulation Models in a Real-World Metropolis

Junyoung Seo, Hyunwook Choi, Minkyung Kwon, Jinhyeok Choi, Siyoon Jin, Gayoung Lee, Junho Kim, JoungBin Lee, Geonmo Gu, Dongyoon Han, Sangdoo Yun, Seungryong Kim, Jin-Hwa Kim

arXiv:2603.15583v1cs.CV

TL;DR

Existing world simulation models do not generate environments grounded in specific real-world locations. SWM grounds city-scale video generation in street-view imagery and outperforms recent models across cities, maintaining spatial and temporal fidelity over trajectories spanning hundreds of meters.

  • Problem

    World simulation models have not been grounded in specific real-world locations, despite the potential to simulate familiar city environments.

  • Method

    SWM uses retrieval-augmented conditioning on street-view images, cross-temporal pairing, synthetic urban data, view interpolation, and a Virtual Lookahead Sink for city-scale generation.

  • Results

    SWM outperforms recent video world models across Seoul, Busan, and Ann Arbor in visual, temporal, camera-following, and spatial fidelity, with stable generation over hundreds of meters.

  • Takeaways & Limitations

    SWM demonstrates city-scale world simulation grounded in physically existing environments while supporting diverse camera trajectories and text-prompted scene conditions.

  • Takeaways & Limitations

    Long-horizon generation can drift as the camera moves hundreds of meters, making a fixed initial-frame anchor increasingly irrelevant.

Abstract

from arXiv · show

What if a world simulation model could render not an imagined environment but a city that actually exists? Prior generative world models synthesize visually plausible yet artificial environments by imagining all content. We present Seoul World Model (SWM), a city-scale world model grounded in the real city of Seoul. SWM anchors autoregressive video generation through retrieval-augmented conditioning on nearby street-view images. However, this design introduces several challenges, including temporal misalignment between retrieved references and the dynamic target scene, limited trajectory diversity and data sparsity from vehicle-mounted captures at sparse intervals. We address these challenges through cross-temporal pairing, a large-scale synthetic dataset enabling diverse camera trajectories, and a view interpolation pipeline that synthesizes coherent training videos from sparse street-view images. We further introduce a Virtual Lookahead Sink to stabilize long-horizon generation by continuously re-grounding each chunk to a retrieved image at a future location. We evaluate SWM against recent video world models across three cities: Seoul, Busan, and Ann Arbor. SWM outperforms existing methods in generating spatially faithful, temporally consistent, long-horizon videos grounded in actual urban environments over trajectories reaching hundreds of meters, while supporting diverse camera movements and text-prompted scenario variations.

1 Introduction

Seoul World Model (SWM) grounds video world simulation in Seoul’s physically existing urban environment by anchoring generation to location-specific street-view references. It demonstrates city-scale simulation across Seoul, Busan, and Ann Arbor, with improved visual, temporal, camera, and structural fidelity.

  • Motivation: SWM formalizes real-world grounded video world simulation by modeling Seoul, a large and densely structured metropolis.The approach targets navigation through familiar streets and hypothetical scenarios in physically existing environments.
  • Method: SWM fine-tunes a pretrained video world simulation model on 440k Seoul street-view images, real-world driving videos, and additional training data.Street-view photographs provide scalable, location-specific visual references that anchor generated chunks to real-world geometry and appearance.
  • Challenges and design: Retrieval-augmented grounding introduces temporal misalignment because street-view references capture specific moments while simulated scenes remain dynamic.SWM addresses this challenge with cross-temporal pairing.
  • Evaluation: SWM is evaluated in Seoul, Busan, and Ann Arbor, with the latter two cities entirely absent from training and testing cross-city generalization without fine-tuning.The model outperforms recent video world models in visual quality, camera adherence, temporal coherence, and structural fidelity to real locations.

2 Related Work

Related work has progressed from diffusion-based video synthesis toward scalable, long-horizon generation and video world simulation. A parallel direction incorporates 3D geometric reasoning and spatial representations to improve consistency during generation.

  • Video Generation: Diffusion video generation evolved from UNet backbones with temporal modules toward Diffusion Transformers for improved scalability.These models enable high-fidelity video synthesis.
  • Long-Horizon Generation: Long-horizon video generation uses autoregressive and streaming rollouts that condition each new chunk on generated context.These formulations increasingly suffer from exposure bias and error accumulation, motivating persistent global anchors for long-range information.
  • Video World Simulation: Video world simulation models use generative models as dynamic models that predict future observations from past observations and actions.Recent systems support interactive visual observations conditioned on user actions across game environments, autonomous driving, and open-domain settings.
  • Geometric World Models: Geometric approaches improve spatial consistency through depth-based point-cloud rendering, joint video-3D prediction, maintained 3D scene representations, and spatial retrieval.These methods span novel view synthesis and autoregressive world-model generation.

3 Data Construction

SWM training uses aligned street-view references and target videos grounded by camera pose and depth, combining real Seoul imagery, synthetic urban simulations, and public driving videos. Cross-temporal pairing, structured panorama rendering, and freeze-frame interpolation address transient-content ambiguity and sparse spatial sampling.

  • Data sources: SWM builds aligned reference–target pairs using camera poses and depth maps, drawing from Seoul street-view imagery, Unreal Engine simulations, and a public driving-video dataset.These sources provide geometric grounding and increase scenario diversity.
  • Real street-view data: 1.2M panoramic Seoul images are collected with GPS and timestamps, de-identified, and reduced to 440K training images.License plates and pedestrians are blurred before training use.
  • Real street-view data: Training sequences use N consecutive route panoramas, while K nearby panoramas condition generation through forward-facing views with random yaw and direction-matched references.This rendering scheme converts panoramas into pinhole views for supervision and conditioning.
  • Real street-view data: Cross-temporal pairing separates reference and target timestamps, forcing reliance on persistent spatial structure rather than identical transient vehicles or pedestrians.The design mirrors inference, where nearby retrieved images often differ in transient content.
  • View interpolation: An intermittent freeze-frame strategy repeats each keyframe for 4 frames, matching the 3D VAE stride and improving keyframe adherence during sparse-view video interpolation.Sparse street-view spacing of typically 5–20 m otherwise disrupts the temporal continuity learned by video diffusion models.
  • Synthetic urban data: The CARLA synthetic dataset renders 12.7K videos across 6 urban maps and includes pedestrian, vehicle, and free-camera trajectories.Synthetic references are rendered every 10 m with eight directions, using different simulated timestamps from target sequences.

4 Model

SWM autoregressively generates city-grounded videos by conditioning a pretrained DiT on camera trajectories, text prompts, history latents, and retrieved street-view images. Its retrieval framework stabilizes long-horizon generation with a Virtual Lookahead Sink and grounds geometry and appearance through complementary reference pathways.

  • Architecture: SWM autoregressively generates video from a user-specified start location, camera trajectory, and text prompt using a pretrained DiT operating on 3D-VAE-compressed latents.Generation proceeds in chunks, with subsequent chunks additionally conditioned on history latents.
  • Retrieval-Augmented Conditioning: Retrieved street-view images are geo-indexed references that provide both long-horizon stabilization through a Virtual Lookahead Sink and grounding in real-world geometry and appearance.The same retrieval framework supports geometric and semantic referencing.
  • Virtual Lookahead Sink: The Virtual Lookahead Sink dynamically retrieves a street-view image near each chunk’s endpoint, placing a clean future anchor ahead of generation to reduce accumulated drift while remaining spatially relevant.At training, future ground-truth frames are sampled with random temporal offsets; at inference, they are replaced by retrieved street-view images.
  • Geometric and Semantic Referencing: Geometric referencing depth-warps the nearest reference into each target viewpoint for dense spatial layout, while semantic referencing injects all references to preserve complementary appearance details.Each target frame uses one nearest reference for geometric warping, whereas semantic referencing lets each target latent attend to all K references.
  • Cross-Temporal Pairing: Cross-temporal pairing encourages attention to persistent scene structure and away from transient objects when references and targets were captured at different times.This addresses mismatches such as dynamic objects appearing differently in retrieved references and generated frames.

5 Experiments

SWM is evaluated on held-out real-world city benchmarks using visual, temporal, camera-following, and 3D-adherence metrics. It outperforms existing world models and ablations show that cross-temporal pairing, synthetic trajectories, complementary referencing, and the Virtual Lookahead Sink are central to stable, spatially faithful generation.

  • Evaluation setup: Evaluation uses 30-sequence Busan-City-Bench and Ann-Arbor-City-Bench datasets, each containing 365-frame trajectories of approximately 100 meters with test-sequence views excluded from retrieval.The benchmarks assess generalization beyond the Seoul training data while preventing access to ground-truth viewpoints during generation.
  • Evaluation metrics: Visual and temporal quality, camera-following accuracy, and static-region 3D adherence are measured with FID, FVD, VBench Image Quality, RotErr, TransErr, masked PSNR, and LPIPS.RotErr and TransErr quantify target-trajectory following, while masked PSNR and LPIPS are computed only on static regions.
  • Qualitative results: SWM generates text-prompted scenarios while preserving city layout, follows diverse trajectories including pedestrian-style motions, and maintains spatial consistency during long-horizon generation.The qualitative results emphasize controllability, trajectory diversity enabled by synthetic urban data, and reduced error accumulation.
  • Quantitative comparison: SWM achieves the best performance on both Busan-City-Bench and Ann-Arbor-City-Bench across visual and temporal fidelity, camera-following accuracy, and 3D adherence to real locations.Existing world models often drift over long trajectories, causing camera-motion and scene-structure misalignment, blur, reduced motion, or collapse.
  • Ablation studies: Removing cross-temporal pairing causes the largest metric degradation, while removing synthetic data slightly improves FID but substantially harms camera-following accuracy and 3D adherence.The findings indicate that cross-temporal pairing handles mismatched dynamic objects and synthetic data provides diverse training trajectories.
  • Ablation studies: Geometric and semantic referencing are complementary, and the Virtual Lookahead Sink achieves the lowest sliding-window FID and slowest degradation as the camera moves away from the anchor.Without a sink, camera and scene drift; First-Frame and First-Position sinks reduce but do not eliminate drift over distant trajectories.

6 Conclusion … B Evaluation Details

The paper presents Seoul World Model, which grounds video generation in a real city through retrieval-augmented street-view conditioning. Its conclusion highlights cross-temporal pairing, synthetic urban data, and a Virtual Lookahead Sink as solutions to city-scale grounding challenges, while the appendix covers implementation and street-view data processing.

  • 6 Conclusion: Seoul World Model grounds video generation in a real city using retrieval-augmented conditioning on street-view images.The model is presented as a video world model operating beyond imagined environments.
  • 6 Conclusion: Cross-temporal pairing addresses temporal challenges in city-scale grounding.The conclusion identifies this component as part of the combined solution to grounding difficulties.
  • 6 Conclusion: Synthetic urban data addresses spatial challenges in city-scale grounding.The conclusion lists synthetic urban data among the methods used to address grounding challenges.
  • 6 Conclusion: A Virtual Lookahead Sink addresses long-horizon challenges in city-scale grounding.The conclusion identifies the Virtual Lookahead Sink as part of the combined solution for long-horizon generation.
  • 6 Conclusion: The paper encourages further exploration of world simulation operating in the physical world beyond imagined environments.This motivation is stated directly in the conclusion.
  • A Implementation Details: The appendix includes a section on model and training details.This passage identifies subsection A.1 within the implementation details.
  • B Evaluation Details: The merged evaluation-details block is represented here without substantive evaluation content in the supplied passages.No evaluation findings or metrics are provided in the passages for this merged section.
  • A Implementation Details: The appendix includes a section on street-view data processing.This passage identifies subsection A.2 within the implementation details.

C Additional Results and Analyses … Coverage area.

The supplementary material reports additional analyses, implementation details, data-processing procedures, stylization behavior, and the spatial coverage of Seoul street-view data. It also describes limitations and practical mechanisms for training, model conditioning, arbitrary-coordinate initialization, and globally consistent camera geometry.

  • C Additional Results and Analyses: Additional sections cover qualitative results, reference sparsity, an attention-sink ablation, static-scene comparisons, and extended long-horizon evaluation.These analyses are enumerated as supplementary subsections without reported findings in the supplied passages.
  • D Discussions: The discussion includes limitations and failure cases as a dedicated subsection.The supplied passage identifies the subsection but does not provide its specific failure cases.
  • Additional training details.: Teacher-Forcing training injects Gaussian noise with µ=0 and σ=0.1 into conditioning history frames with 50% probability, while mixing Waymo, Seoul street-view, and synthetic data at 20%, 40%, and 40%.The noise reduces the gap between clean training inputs and self-generated inference history; sampling is ratio-based and interleaved.
  • Additional training details.: Under Self-Forcing, causal attention uses self-produced history latents, while Virtual Lookahead and semantic reference tokens remain outside the current generation window but are visible to generated tokens.The supplied passage describes the positional treatment of both conditioning mechanisms.
  • Additional model details.: SWM fine-tunes Cosmos-Predict2.5-2B-I2W, a 2B-parameter DiT with 28 blocks, 16 attention heads, and hidden dimension 2048, using a 16-channel latent space.Camera poses are converted into 6-channel Plücker ray maps and encoded as residual camera embeddings for video and reference tokens.
  • Starting from arbitrary coordinates.: For arbitrary starting coordinates, SWM initializes from the nearest street-view image and generates a buffer chunk that navigates toward the requested point.When the coordinate matches an existing street-view location, its corresponding pinhole image is used directly as the first frame.
  • A.2 Street-View Data Processing: Street-view processing jointly estimates depth and relative camera poses across target and eight-direction reference views, then aligns long-sequence chunks to GPS-based real-world metric coordinates.The resulting database shares a globally consistent coordinate system, making semantic references and the Virtual Lookahead Sink pose-compatible.

A.3 Synthetic Dataset · Target video. · B Evaluation Details

The synthetic dataset uses CARLA to generate realistic, cross-temporally paired urban videos and street-view references, while target videos cover natural pedestrian, vehicle, and diverse free-camera trajectories. Street-view references are rendered densely across roads with varied viewpoints and conditions to support city-scale grounding.

  • A.3 Synthetic Dataset: Cross-temporal pairing renders target videos and street-view references under distinct traffic, lighting, and weather conditions.The data construction divides both targets and references into multiple subsets before rendering.
  • Target video.: Pedestrian and vehicle target videos follow natural motion paths captured with RGBD sensors attached to randomly spawned agents.Sensors are mounted on self-driving pedestrians and vehicles.
  • Target video.: Free-camera trajectories vary initial position, viewing direction, movement speed, and paths while using collision detection.Viewing direction and movement speed are continuously randomized.
  • Target video.: Street-view references are rendered every 10 m along all roads with eight horizontal views uniformly covering 360◦ at each location.Sampling intervals receive slight positional jitter, and lane positions vary on multi-lane roads to reduce the sim-to-real gap.
  • Target video.: The reference set contains 4K street-view positions and 32K reference frames across all maps.The frames result from eight directional views per rendered location.

B.1 Evaluation Metrics · B.2 Baseline Adaptation · C Additional Results and Analyses

The paper evaluates SWM using fidelity, camera-following, and static-scene adherence metrics, while adapting each baseline’s supported camera-conditioning interface to the benchmark inputs. Ground-truth handling and metric computation account for sparse street-view data and dynamic objects.

  • B.1 Evaluation Metrics: SWM evaluation covers visual and temporal fidelity, camera-following accuracy, and 3D adherence using FID/FVD, RotErr/TransErr, and masked PSNR/LPIPS, respectively.FID and FVD assess fidelity; RotErr and TransErr assess camera following; masked PSNR and LPIPS assess 3D adherence.
  • B.1 Evaluation Metrics: For Busan, interpolated videos from sparse ground-truth street-view keyframes serve as FVD ground truth, whereas Ann Arbor uses MARS ground-truth video sequences.Busan-City-Bench lacks ground-truth video; Ann-Arbor-City-Bench directly provides sequences from MARS.
  • B.1 Evaluation Metrics: PSNR and LPIPS compare generated videos with ground-truth videos or image sequences while focusing on static scene structure rather than exact dynamic-object reconstruction.Dynamic objects may differ between generated and ground-truth frames, so reconstruction metrics target static structure preserved from references.
  • B.1 Evaluation Metrics: Dynamic-object masks are extracted per frame with SAM3 text prompts for categories such as pedestrians and vehicles before computing PSNR and LPIPS.The masks are obtained for both generated and ground-truth frames.
  • B.1 Evaluation Metrics: Camera-following metrics use DA3-extracted extrinsics, non-overlapping chunks, first-frame-relative poses, and independently maximum-norm-normalized translation trajectories.RotErr is based on mean geodesic distance between relative rotations; the supplied passage truncates the TransErr definition.
  • B.2 Baseline Adaptation: Because existing world models lack the benchmark’s full input set, each baseline receives the supported subset, with all baselines using the target sequence’s first frame as the starting point.The benchmark inputs include a start frame, camera trajectory, text prompt, and retrieved street-view references.
  • B.2 Baseline Adaptation: Aether, FantasyWorld, and LingBot receive camera trajectories as Plücker ray representations, using model-specific injections and first-frame- or chunk-relative pose conversions.Aether uses 41-frame chunks with 1-frame overlap; FantasyWorld uses Adaptive Layer Normalization; LingBot uses scale-and-shift modulation in each transformer block.
  • B.2 Baseline Adaptation: DeepVerse, Yume1.5, and HY-World1.5 encode camera motion through discrete or textual controls, with HY-World1.5 additionally using continuous PRoPE conditioning.DeepVerse uses 27 action classes; Yume1.5 uses 8 translation and 4 rotation directions; HY-World1.5 uses 9 translation and 9 rotation classes alongside PRoPE.

C.1 View Interpolation … C.5 Comparison with Video Generative Models for Static Scenes

The supplementary sections analyze SWM’s view interpolation design, robustness to sparse references and attention-sink choices, and comparisons with static-scene video models. Together, they show how SWM preserves reference grounding and dynamic plausibility while supporting coherent generation from real-world street-view observations.

  • C.1 View Interpolation: The channel concatenation baseline widens the input projection from 18 to 34 channels but suffers weak keyframe adherence because isolated keyframes do not form valid 4-frame VAE groups.The comparison evaluates channel concatenation against Intermittent Freeze-Frame using PSNR, SSIM, and LPIPS on the Waymo test set.
  • C.1 View Interpolation: Intermittent Freeze-Frame repeats each keyframe four times, matching the 3D VAE’s temporal stride without changing the network architecture.This makes each keyframe a complete 4-frame group and produces smooth motion segments alternating with brief freeze segments during training.
  • C.2 Additional Qualitative Results: Qualitative ablations visualize how individual components affect generation quality, with additional video results provided on the project page.The reported qualitative results are presented in Fig. 16.
  • C.3 Effect of Reference Sparsity: Reducing retrieved references from the default K=5 to K=1 tests SWM under sparse street-view retrieval on Busan-City-Bench.The experiment reports the resulting behavior in Fig. 13.
  • C.3 Effect of Reference Sparsity: As K decreases, mPSNR drops, whereas FID and FVD show no clear degradation because the diffusion model can retain visual plausibility despite weaker location-specific grounding.K=1 achieves the best FID and second-best FVD, while fewer references provide less target-scene coverage.
  • C.4 Additional Ablation on Attention Sink with SF Variant: An additional Self-Forcing ablation compares generation stability across attention-sink types over extended trajectories.The comparison is reported for the SF variant in Fig. 14.
  • C.5 Comparison with Video Generative Models for Static Scenes: GEN3C and VMem provide multi-view conditioning for geometrically consistent static-scene video, but both struggle with temporally inconsistent street-view references and cannot synthesize plausible dynamic motion.GEN3C assumes synchronized static multi-view images, while VMem retrieves pose-similar frames from a geometric-aware memory populated here with retrieved street-view images.
  • C.5 Comparison with Video Generative Models for Static Scenes: The static-scene models obtain worse FID and FVD than SWM, while GEN3C remains competitive with SWM on static-region metrics mPSNR and mLPIPS.These comparisons are reported on both benchmarks, with Table 4 specifically covering Busan-City-Bench.

C.6 Extended Long-Horizon Evaluation · D Discussions

The extended Busan-City-Bench evaluation uses 1,460-frame sequences covering approximately 500 m, revealing larger performance gaps among attention-sink variants. Removing the sink degrades visual quality, while VL Sink provides the strongest real-world appearance grounding and first-position sink improves camera-following accuracy.

  • C.6 Extended Long-Horizon Evaluation: 1,460-frame sequences extend Busan-City-Bench to approximately 500 m per sequence, four times the standard 365-frame benchmark.Table 5 compares attention-sink variants under this extended long-horizon setting.
  • C.6 Extended Long-Horizon Evaluation: Longer horizons make the performance gaps between attention-sink variants more pronounced.The extended benchmark exposes differences that become clearer over increased generation distances.
  • C.6 Extended Long-Horizon Evaluation: 37.37 vs. 25.13 FID shows notable degradation when the attention sink is removed entirely.This result confirms that an attention sink helps maintain visual quality over long distances.
  • C.6 Extended Long-Horizon Evaluation: RotErr 0.019 and TransErr 0.021 give the first-position sink the best camera-following accuracy.These metrics identify the first-position sink as strongest for camera-following accuracy among the compared variants.
  • C.6 Extended Long-Horizon Evaluation: The full model with the VL Sink achieves the best mPSNR and mLPIPS.Its performance indicates the strongest grounding to real-world appearances.
  • C.6 Extended Long-Horizon Evaluation: The ablation compares attention-sink variants specifically on the 1,460-frame Busan-City-Bench sequences.The benchmark sequences are described as four times longer than the standard evaluation.

D.1 Limitations and Failure Cases … D.3 Societal Impact

SWM’s limitations primarily arise from synthetic training videos and temporally inconsistent street-view data, which can cause abrupt object changes. Unlike fixed-route street-view interpolation, SWM supports free-form trajectories and text prompts, while enabling applications grounded in real urban layouts with privacy safeguards.

  • D.1 Limitations and Failure Cases: Synthetic interpolated videos provide effective supervision but remain lower quality than real captured video, which could further improve SWM as it becomes available.SWM’s generation quality is closely tied to training-data quality.
  • D.1 Limitations and Failure Cases: Noisy capture-time metadata can allow temporally inconsistent street-view sequences into training, causing vehicles to appear or disappear abruptly.Street-view images are captured at equal-distance rather than equal-time intervals, so vehicle stops can create large temporal gaps.
  • Relationship to street-view interpolation.: Street-view interpolation synthesizes smooth video along a fixed original route, whereas SWM generates from a user-specified free-form camera trajectory and text prompt using nearby references.The two tasks also differ because SWM’s independently captured references may contain mutually inconsistent dynamic objects.
  • Why is cross-temporal pairing important?: Cross-temporal pairing prevents spurious temporal correlations between retrieved references and target videos, reducing abrupt transitions during inference.Without it, training references captured near the target in space and time could encourage copying dynamic objects into generated frames, conflicting with arbitrary-time references at inference.
  • D.3 Societal Impact: Real-world grounding supports streetscape preview, autonomous-driving evaluation across diverse scenarios, and exploration of familiar places under novel conditions.These applications use real city layouts as the grounding context.
  • D.3 Societal Impact: Street-view data was collected under local regulations and released with faces, license plates, and other sensitive regions blurred, preventing learning of identifiable faces or plates.The training data contains no unblurred personal identifiers.
Loading 2603.15583v1…