PaperlayerView exact revision on arXiv

Source-linked AI summary

MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond

Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhenzhi Wang, Dahua Lin, Bo Dai

arXiv:2309.16553v1cs.CV

TL;DR

City-scale neural rendering lacks a comprehensive, high-quality benchmark, while collecting equivalent real-world data is costly and technically difficult. MatrixCity uses Unreal Engine 5 to construct a large synthetic dataset with aerial and street views, controllable conditions, and extra modalities, then benchmarks representative methods to reveal persistent city-scale challenges.

  • Problem

    Comprehensive, high-quality city-scale neural-rendering data are lacking, while collecting real city-scale datasets is costly, sensitive, and technically difficult.

  • Method

    MatrixCity uses Unreal Engine 5 and an automated plugin to collect controllable aerial and street views with camera poses and additional ground-truth modalities.

  • Results

    Benchmark studies reveal persistent city-scale challenges, including high-rise aerial regions, street-level detail, and naive fusion of aerial and street data.

  • Takeaways & Limitations

    MatrixCity provides a benchmark for studying city-scale neural rendering and supports extended tasks such as depth estimation and inverse rendering.

Abstract

from arXiv · show

Neural radiance fields (NeRF) and its subsequent variants have led to remarkable progress in neural rendering. While most of recent neural rendering works focus on objects and small-scale scenes, developing neural rendering methods for city-scale scenes is of great potential in many real-world applications. However, this line of research is impeded by the absence of a comprehensive and high-quality dataset, yet collecting such a dataset over real city-scale scenes is costly, sensitive, and technically difficult. To this end, we build a large-scale, comprehensive, and high-quality synthetic dataset for city-scale neural rendering researches. Leveraging the Unreal Engine 5 City Sample project, we develop a pipeline to easily collect aerial and street city views, accompanied by ground-truth camera poses and a range of additional data modalities. Flexible controls over environmental factors like light, weather, human and car crowd are also available in our pipeline, supporting the need of various tasks covering city-scale neural rendering and beyond. The resulting pilot dataset, MatrixCity, contains 67k aerial images and 452k street images from two city maps of total size $28km^2$. On top of MatrixCity, a thorough benchmark is also conducted, which not only reveals unique challenges of the task of city-scale neural rendering, but also highlights potential improvements for future works. The dataset and code will be publicly available at our project page: https://city-super.github.io/matrixcity/.

1. Introduction

MatrixCity addresses the lack of comprehensive city-scale neural-rendering benchmarks with a large, controllable synthetic dataset and benchmark study. Its aerial and street views, additional ground-truth modalities, and environmental controls expose challenges that remain unresolved for city-scale rendering.

  • City-scale neural rendering remains underexplored because comprehensive benchmark data are scarce, despite its relevance to aerial surveying, virtual reality, film production, and gaming.
  • MatrixCity is a comprehensive, high-quality synthetic dataset designed for city-scale neural rendering and extended tasks.
  • Benchmark studies identify high-rise aerial regions, street-level detail, and naive fusion of aerial and street data as persistent challenges.
  • The dataset combines complete aerial and street-level city views with depth, normal, and decomposed BRDF materials for multiple tasks.
  • Its Unreal Engine 5 plugin automates high-quality data collection while controlling lighting, weather, and transient human and car crowds.
  • Extensive studies on MatrixCity are intended to reveal key city-scale neural-rendering challenges and facilitate future research.

2. Related work

Related work spans NeRF-based urban reconstruction methods and benchmarks, but existing datasets remain limited in scale, openness, modality, or environmental control. These gaps motivate a more comprehensive city-scale benchmark.

  • Urban-scale NeRF methods include appearance modeling, static-transient separation, spatial block decomposition, and large-scene reconstruction.
  • Existing NeRF benchmarks primarily target single objects, indoor scenes, or outdoor unbounded scenes rather than complete cities.
  • Real-world urban datasets often cover only a few independent scenes, provide restricted access, or release limited scenarios.
  • Datasets commonly provide only aerial or street imagery, which can make building modeling incomplete.
  • Real outdoor data collection makes pedestrian movement, weather, and lighting difficult to control, limiting comprehensive benchmarking.

3. MatrixCity Dataset

MatrixCity is a synthetic city-scale dataset built with Unreal Engine 5, combining aerial and street views with controllable environments and multiple ground-truth properties. Its automated collection and movie-level rendering pipeline supports broad city-scale neural rendering and related tasks.

  • Applications: The dataset's comprehensive city maps and diverse rendered environments are intended to support neural rendering research and applications including virtual reality and autonomous driving.The authors also describe potential use in gaming, virtual studios, and digital-human interaction.
  • Dataset Construction: An automated plugin generates aerial and street camera trajectories, reducing manual setup for dense multi-view city data collection.Aerial trajectories use block geometry and camera height, while street trajectories use manually annotated road endpoints.
  • Quality Control: Movie-level rendering, high scalability settings, anti-aliasing, and disabled motion blur are used to improve image quality and camera-pose accuracy.The pipeline uses Unreal Engine 5's movie render queue rather than a real-time rendering pipeline.
  • Controllable Environments: The dataset supports controlled illumination, fog, weather, pedestrian, vehicle, motion-blur, and defocus-blur conditions.Environmental factors can be adjusted quantitatively or simulated through Unreal Engine functions.
  • Multiple Properties: MatrixCity provides RGB images with depth, normals, and decomposed reflectance components such as diffuse, specular, metallic, and roughness.These properties support inverse rendering, depth estimation, semantic analysis, and other extended tasks.
  • Dataset Statistics: MatrixCity covers two Unreal Engine 5 city scenes totaling 28km^2, with 67k aerial and 452k street-level images.The Small City covers 2.7km^2 and the Big City spans 25.3km^2.

4. Experiments

Experiments benchmark representative neural rendering methods on aerial, street, and fused MatrixCity data, exposing distinct challenges across scales and viewpoints.

  • Datasets and Metrics: The benchmark evaluates rendering with PSNR, SSIM, and LPIPS, and evaluates depth and normals with MSE and MAE.
  • Datasets and Metrics: Experiments use NeRF, DVGO, Instant-NGP, TensoRF, and MipNeRF-360 as representative baselines.
  • Aerial Data: Current methods struggle more with high-rise aerial regions because building occlusions complicate modeling small objects and reflective structures.
  • Street Data: Street-view rendering performs substantially worse than aerial rendering because street data contains more fine-grained detail.
  • Street Data: On Block Small, MLP-based NeRF methods outperform grid-based methods, while MipNeRF-360 partly alleviates unbounded-scene artifacts.
  • Joint Aerial and Street Data: Naively fusing aerial and street data degrades TensoRF and MipNeRF-360, producing dirty aerial ground regions and worse street-view foregrounds.The authors attribute this difficulty to mismatched detail levels and viewing distances, motivating strategies such as finetuning or progressive training.

5. Conclusion

The paper introduces MatrixCity as a realistic, controllable city-scale benchmark with multiple views and ground-truth properties, then evaluates neural rendering methods across view types.

  • MatrixCity is a high-quality, city-scale benchmark with diverse, controllable, and realistic Unreal Engine 5 data.
  • The dataset includes aerial and street views plus depth, normals, and decomposed BRDF materials for extended tasks.
  • The study empirically investigates representative methods on aerial data, street-level data, and their fusion.

Appendix A: Method Details

The appendix details the baseline representations, implementation settings, rendering setup, and aerial block organization used in the benchmark.

  • Baseline Methods: NeRF represents scenes implicitly with a fully connected MLP and is configured with enlarged depth, width, positional encoding, and skip connections.
  • Baseline Methods: DVGO uses optimized density and feature grids with a shallow MLP for view-dependent appearance.
  • Baseline Methods: Instant-NGP uses 16-level multi-resolution hash encoding with 2^22 entries per hash table and resolutions from 16 to 65536.
  • Baseline Methods: TensoRF decomposes a 4D tensor into compact vector and matrix factors, using 500^3 grid resolution and scheduled upsampling.
  • Baseline Methods: MipNeRF-360 combines nonlinear parameterization, coarse-to-fine online distillation, and distortion loss for unbounded artifacts.
  • Implementation: Images are resized to 540 × 960, with most models trained on one A100 GPU and MipNeRF-360 trained on four.
  • Data Collection: Aerial collection divides the city into blocks by building height to better capture details and address height variation in ray sampling.
  • Evaluation Settings: Table 7 compares multi-model block training with single-model whole-city training for aerial data.

Appendix B: More Results on Aerial Data

Larger aerial coverage exposes different scaling behavior: continuous MLP representations lose performance and detail, whereas grid-based methods remain comparatively stable when resolution scales with area.

  • MLP-based NeRF and MipNeRF-360 decline significantly on larger areas despite increased model capacity.
  • Grid-based TensoRF and Instant-NGP show almost no performance drop when grid resolution increases with the modeled area.
  • Expanding the area diminishes detail learning for MLP methods but does not significantly affect detail learning for resolution-scaled grid methods.

Appendix C: More results on Street Data

Appendix C reports an ablation on block size and model capacity using the Small City street data.

  • Table 8 summarizes the ablation settings for the street-data experiments.

Appendix D: Extension Study

Appendix D extends the evaluation with experiments that change illumination and scene dynamics to probe novel view synthesis in real-world urban scenes.

  • Two experiment sets vary lighting conditions and dynamic scenes in urban novel view synthesis.

D.1. Illumination

The illumination study separately examines changes in light direction and intensity, revealing different interpolation behaviors and remaining visual discrepancies.

  • Light-direction interpolation from 0 degree to 90 degree produces lighter shadows but misses the interaction between lighting and building structure.
  • The results imply that decoupling light intensity from scene color is not well executed.

D.2. Dynamic

The dynamic-scene study evaluates novel view synthesis with moving cars and people, while the accompanying figure descriptions distinguish model settings and lighting interpolation examples.

  • D.2. Dynamic: Street views with moving people and traffic are used to model stationary buildings while filtering dynamic traffic with NeRF-W.
  • D.2. Dynamic: Figure 9 compares aerial novel view synthesis methods under multi-model and single-model settings.The multi-model setting divides the Small City into five blocks and trains one model per block, whereas the single-model setting trains one model for the whole Small City.
  • D.2. Dynamic: Figure 10 presents light-angle interpolation in its first row and light-intensity interpolation in its second row.The displayed endpoints are 0 degree and 90 degree for angle, and 1000 and 3000 for intensity.
  • D.2. Dynamic: NeRF-W filtering accidentally removes stationary objects such as parked cars and streetlights, making the decomposition unacceptable.
  • D.2. Dynamic: Figure 11 visualizes novel view synthesis for a stationary scene containing moving cars and people.
Loading 2309.16553v1…