Source-linked AI summary
Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset
Liqiang Lin, Yilin Liu, Yue Hu, Xingguang Yan, Ke Xie, Hui Huang
TL;DR
Outdoor urban-scene research lacks comprehensive data for large-scale perception, reconstruction, and aerial path planning. UrbanScene3D combines synthetic and reconstructed real scenes with aerial observations, ground-truth assets, benchmarks, and an Unreal Engine/AirSim simulator. The platform supports evaluation of path planning and MVS methods and simulation across varied urban environments, while higher-level geometric descriptions remain future work.
Problem
Outdoor urban scenes lack a decent dataset for learning 3D building instance segmentation and face capture challenges from scale, restricted sensing, and incomplete coverage.
Method
UrbanScene3D constructs a large-scale platform combining synthetic and real reconstructed scenes, aerial observations, ground-truth scans or models, path-planning benchmarks, and an Unreal Engine/AirSim simulator.
Results
UrbanScene3D supports evaluation of aerial path-planning and MVS algorithms and enables simulation of autonomous cars and drones in proposed urban environments.
Takeaways & Limitations
The dataset provides varied urban-scene observations and simulation capabilities for research in perception, reconstruction, robotics, autonomous driving, and embodied AI.
Takeaways & Limitations
Future work will add higher-level geometric descriptions such as structural points, cross-sectional profiles, wire-frames, and plane segments.
Abstract
from arXiv · showhide
We present UrbanScene3D, a large-scale data platform for research of urban scene perception and reconstruction. UrbanScene3D contains over 128k high-resolution images covering 16 scenes including large-scale real urban regions and synthetic cities with 136 km^2 area in total. The dataset also contains high-precision LiDAR scans and hundreds of image sets with different observation patterns, which provide a comprehensive benchmark to design and evaluate aerial path planning and 3D reconstruction algorithms. In addition, the dataset, which is built on Unreal Engine and Airsim simulator together with the manually annotated unique instance label for each building in the dataset, enables the generation of all kinds of data, e.g., 2D depth maps, 2D/3D bounding boxes, and 3D point cloud/mesh segmentations, etc. The simulator with physical engine and lighting system not only produce variety of data but also enable users to simulate cars or drones in the proposed urban environment for future research.
1 Introduction
UrbanScene3D addresses the difficulty of capturing and reconstructing large-scale outdoor urban environments by combining synthetic and realistic reconstructed scenes with aerial observations and simulation. It supports evaluation of path planning, multi-view stereo, and autonomous tasks across varied urban settings.
- Outdoor data capture is difficult because environments are extensive, while onboard sensors have limited fields of view and routing choices.
- UrbanScene3D combines professionally designed synthetic scenes with dense, detailed real-world reconstructions produced from aerial images using multi-view stereo.
- Aerial paths, captured image sets, high-precision laser scans, and synthetic ground truth support evaluation of path-planning strategies and MVS algorithms.
- AirSim enables simulation of cars and drones for autonomous tasks in the proposed urban environments.
- The dataset contributes a large-scale benchmark, an aerial path-planning benchmark, and a simulation platform for autonomous driving, robotics, and embodied AI.
2 Related Work
Prior outdoor datasets provide useful sensor data or complete synthetic environments, but they remain limited in coverage, realism, or capture flexibility. Related path-planning methods optimize different objectives under proxy or image-based constraints.
- Ground-based outdoor datasets provide stereo sequences, LiDAR, calibration, and object tracklets, but restricted views and routes make large-scale urban coverage difficult.
- Synthetic CAD datasets offer complete structured environments at lower cost but generally lack geometric and textural detail.
- Existing aerial path-planning methods optimize drone routes using goals and constraints derived from coarse proxy models or top-view images.
- Prior planners target reconstruction quality, shorter paths, fewer sharp turns, or offsite planning from satellite imagery.
3 The UrbanScene3D Dataset
UrbanScene3D combines synthetic and real urban scenes with aerial captures, reconstruction assets, path-planning data, and simulation capabilities. Its design emphasizes broad spatial coverage, multiple observation conditions, and benchmarkable acquisition settings.
- UrbanScene3D contains 10 synthetic and 6 real-world scenes with CAD or reconstructed mesh models and corresponding aerial images.
- The benchmark provides statistics for representative scenes, including triangles, proxy levels, overlap rates, planners, flight paths, and captured images.
- Real scenes are captured by drones along planned paths and reconstructed from aerial images with a commercial MVS solution, while synthetic scenes use artist-built CAD structures.
- Multiple optimized aerial paths support high-quality urban capture and research on both local perception and global understanding of real 3D urban scenes.
- The dataset includes large-scale city scenes, complete real-world scenes, diverse aerial images with shot poses, and flight paths up to 17 km.
- The benchmark varies proxy accuracy and overlap rate to study their influence on reconstruction quality and acquisition efficiency, with hundreds of flight paths.
- Multiple flights across times and lighting conditions increase data variety, including more than 10k real-scene images for future learning-based research.
- The simulator imports real-world scenes, supports drone simulation, and displays scene coverage in real time for UAV exploration research.
4 Scene Acquisition with Aerial Path Planning
UrbanScene3D provides paths, proxies, images, and reconstruction data for evaluating aerial path planning across planners, proxy detail levels, and observation settings. Its proxy hierarchy supports analysis of how scene representation affects planned trajectories and reconstruction quality.
- UrbanScene3D benchmarks path planning using proxies, ground-truth models or point clouds, paths, images, and reconstructed results.
- Four planners generate trajectories across different proxies and overlap rates, producing 100 paths for synthetic scenes and 26 for real scenes.
- Smith et al. optimize viewpoint reconstructability, whereas Zhou et al. use reconstructability to remove redundant viewpoints.
- Zhang et al. incorporate path smoothness to reduce sharp turns and improve capturing efficiency relative to viewpoint-only heuristics.
- UrbanScene3D supplies box, coarse, intermediate, and fine proxies, allowing relationships among proxy detail, paths, and reconstructed-model quality to be studied.
5 Scene Reconstruction Benchmarks
UrbanScene3D evaluates aerial planners by energy use, aerotriangulation, and reconstruction accuracy and completeness. The benchmark shows a trade-off between reconstruction quality and flight efficiency, while finer proxies generally benefit some planners.
- The benchmark evaluates four planners using energy cost, aerotriangulation accuracy, and point-level reconstruction accuracy and completeness.
- Aerotriangulation determines camera poses and produces a sparse environmental point cloud before reconstruction.
- Smith et al. and Zhou et al. are sensitive to proxy detail, while Zhang et al. shows inconsistent aerotriangulation behavior across proxies.
- Smith et al. and Zhou et al. improve in both reconstruction accuracy and completeness as proxies become finer, whereas Zhang et al. is inconsistent.
- Complex geometry and high occlusion reduce reconstruction accuracy and completeness, while oblique photography has low energy cost but produces rougher reconstructions.
- Smith et al. and Zhou et al. generally achieve higher reconstruction accuracy and completeness than oblique photography and Zhang et al., while Zhang et al. uses less energy.
6 Simulator and Applications
UrbanScene3D addresses the lack of large-scale outdoor building-instance data by combining manually labeled urban models with a simulator for varied robotic and vision tasks. Users can vary physical conditions, lighting, and weather to generate diverse data.
- Existing 3D instance-segmentation datasets focus on indoor scenes, leaving limited resources for spacious outdoor urban building-instance segmentation.
- UrbanScene3D manually assigns each building a unique label, creating instance-segmentation maps on 3D textured urban models.
- The AirSim-based simulator supports car and drone simulation with gravity, inertia, and collision handling for autonomous-task data generation.
- Users can manipulate lighting and weather to generate data for depth estimation, navigation, and novel view synthesis across varied conditions.
7 Conclusion and Future Work
UrbanScene3D combines annotated urban data, varied observations, ground truth, and simulation for evaluating path planning and multi-view reconstruction. Future work will add higher-level geometric descriptions and expand the dataset.
- UrbanScene3D provides annotated data, varied observations, reconstruction results, and ground-truth models or scans for evaluating path planning and MVS algorithms.
- Its simulator supports urban-scene capture under different data patterns, lighting conditions, and weather conditions.
- Future extensions include 3D structural points, cross-sectional profiles, wireframes, and plane segments for computer vision and graphics research.