Source-linked AI summary
TorontoCity: Seeing the World with a Million Eyes
Shenlong Wang, Min Bai, Gellert Mattyus, Hang Chu, Wenjie Luo, Bin Yang, Justin Liang, Joel Cheverie, Sanja Fidler, Raquel Urtasun
TL;DR
TorontoCity addresses the lack of large-scale benchmarks combining multiple viewpoints with rich geometry, grouping, and semantics. It constructs map-derived ground truth and aligns heterogeneous imagery, then evaluates diverse tasks whose pilot results show several remain difficult for current methods.
Problem
Existing large-scale datasets use limited viewpoints and lack rich semantics and 3D information at scale.
Method
The paper builds a multi-view GTA benchmark using high-precision maps for ground truth and algorithms for aligning maps with heterogeneous imagery.
Results
Pilot experiments find that semantic segmentation and scene classification work well, while instance segmentation, contour extraction, and height estimation remain challenging.
Takeaways & Limitations
TorontoCity provides a large real-world platform for developing and evaluating methods that leverage different viewpoints, geometry, grouping, and semantics.
Takeaways & Limitations
No network could estimate building height from aerial images alone, indicating that the task may require more sophisticated methods or ground imagery.
Abstract
from arXiv · showhide
In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712.5 $km^2$ of land, 8439 $km$ of road and around 400,000 buildings. Our benchmark provides different perspectives of the world captured from airplanes, drones and cars driving around the city. Manually labeling such a large scale dataset is infeasible. Instead, we propose to utilize different sources of high-precision maps to create our ground truth. Towards this goal, we develop algorithms that allow us to align all data sources with the maps while requiring minimal human supervision. We have designed a wide variety of tasks including building height estimation (reconstruction), road centerline and curb extraction, building instance segmentation, building contour extraction (reorganization), semantic labeling and scene type classification (recognition). Our pilot study shows that most of these tasks are still difficult for modern convolutional neural networks.
1. Introduction
TorontoCity addresses the limited perspectives and sparse large-scale semantics and 3D information in existing datasets by covering the GTA with diverse views, map-derived ground truth, and varied benchmark tasks. Its pilot study finds that current methods perform well on some tasks but that instance segmentation, contour extraction, and height estimation remain difficult.
- Existing large-scale datasets typically use few sensors with similar viewpoints and lack rich semantics and 3D information at scale.
- TorontoCity covers 712.5km^2 of land, 8439km of road, and around 400,000 buildings across the greater Toronto area.
- The benchmark combines aerial, airborne-LIDAR, panorama, stereo, vehicle-LIDAR, GoPro, drone, and 3D-camera views of the city.
- High-precision maps provide ground truth for building height, road centerline and curb extraction, instance segmentation, contour extraction, semantic labeling, and scene classification.The approach avoids infeasible manual labeling and uses map alignment tools to reduce labeling to verification.
- Pilot experiments show that semantic segmentation and scene classification work well, while instance segmentation, contour extraction, and height estimation remain open challenges.
2. Related Work
Prior urban benchmarks address subsets of mapping and scene-understanding tasks, but TorontoCity combines much larger real-world coverage, multiple viewpoints, and diverse semantic and geometric ground truth. The paper also positions high-fidelity maps as a basis for constructing multi-view benchmarks.
- KITTI, Cityscapes, and Aerial-KITTI cover vehicle-based sensing, semantic or instance annotation, and selected aerial-ground reasoning tasks.
- Several multi-perspective benchmarks provide UAV, satellite, handheld, lidar, or stereo data, but some lack semantic ground truth or remain limited in coverage and viewpoints.
- TorontoCity is more than two orders of magnitude bigger than the closely related ISPRS benchmark and adds ground-view imagery, multiple sensors, and diverse semantic and geometric benchmarks.
- Unlike synthetic benchmarks focused on a single view, TorontoCity uses real-world imagery and high-fidelity 3D models of the real world.
- The paper uses high-fidelity maps to create multiple reconstruction, recognition, and reorganization benchmarks from different world views.Earlier work rarely exploited maps as ground truth because accurate maps and well-aligned georeferenced imagery were lacking.
3. TorontoCity at a Glimpse
TorontoCity is a large GTA dataset combining broad geographic coverage, multiple sensing platforms, detailed maps, and urban tasks. Its data support semantic, geometric, grouping, and zoning-oriented evaluation, while map alignment and verification make large-scale labeling feasible.
- The benchmark supports semantic segmentation, road curb and centerline extraction, building instance segmentation, and other tasks described across its data sources and evaluation plan.
- The dataset covers 712.5km^2 of land, 8439km of road, and around 400,000 buildings in the GTA.
- TorontoCity combines aerial imagery from four years, airborne LIDAR, street-view panoramas, vehicle stereo and LIDAR, GoPro data, drone imagery, and 3D-camera data.
- Its maps include 3D buildings, road polylines and curbs, zoning categories, and detailed metadata for poles, lights, and trees.
- High-precision maps replace infeasible manual labeling and provide detailed ground truth across the full GTA.
4. Maps for Creating Large Scale Benchmarks
TorontoCity aligns high-precision maps with aerial and panoramic imagery to create large-scale ground truth despite sensor misalignment. The pipeline combines coarse appearance-based alignment, fine road-curve alignment, and structured matching of road centerlines and curbs.
- Map alignment: Panoramas require alignment because geolocation errors reach 5m, averaging 1.5m, while aerial imagery is already aligned with the maps.Projecting maps directly onto panoramas therefore produces inaccurate ground truth.
- Map alignment: The coarse panorama alignment searches camera offsets using normalized cross correlation combined with a Gaussian prior.The search spans [-10m, 10m] × [-10m, 10m] × [2.2m, 2.6m] at 0.1m steps after ground-plane rectification.
- Map alignment: Fine alignment matches mapped road curves to panorama boundary edges, followed by human verification of successful alignments.The fine search uses a 5cm step over a 2m × 2m area; the reported success rate is 34.35%, with verification taking less than 2s per image.
- Road-map alignment: An MRF aligns curb segments to nearby road centerlines using distance, angular compatibility, and smoothness terms.Dynamic programming performs exact inference over adjacent curb chains, while allowing multiple curbs to match one road.
5. Benchmark Tasks and Metrics
TorontoCity defines diverse benchmarks for geometry, semantics, and grouping across aerial, panoramic, and other imagery. Its tasks evaluate segmentation, topology, contour structure, zoning, and building height with task-specific metrics.
- Benchmark scope: The benchmark supports submissions using any subset of aerial images, panoramas, Go-Pro, and stereo imagery.It is designed to test joint reasoning about geometry, semantics, and grouping.
- Segmentation: Building footprint and road segmentation use mean Intersection-Over-Union from a top-down view.Building instance segmentation is defined as a separate benchmark task.
- Contour structure: Building structured contours combine segmentation IoU with geometric similarity between estimated and ground-truth polygon turning functions.The metric targets polygon structure rather than only blob-like segmentation overlap.
- Road topology: Road topology evaluates curb and centerline polylines using precision and recall at 0.5m and 2.0m distance thresholds.Estimated and ground-truth polylines are discretized into 10cm intervals.
- Additional tasks: Additional planned tasks include tree detection and species recognition, normal estimation, facade parsing, and traffic-object detection and segmentation.These extensions use the dataset’s accurate 3D building models and broad urban coverage.
6. Experimental Evaluation
The pilot evaluates TorontoCity across segmentation, zoning, contour, centerline, curb, and height tasks using aerial and ground-view imagery. Performance varies substantially: some tasks are nearly solved, while instance-level, geometric, zoning, and height estimation remain difficult.
- Evaluation Setup: The pilot uses non-overlapping 50 km^2 training, 25 km^2 validation, and 50 km^2 testing regions, with 56K streetview images.All reported numbers are evaluated on the testing set.
- Semantic Segmentation: Semantic segmentation performs well with both FCN-8 and ResNet56 networks.ResNet56 outputs tend to be sharper, whereas FCN outputs tend to be smoother.
- Instance Segmentation: Instance segmentation performance is low despite connected-component labeling and morphological opening, leaving substantial room for new methods.The dataset’s more than 400,000 buildings provide a large platform for developing such methods.
- Road Centerlines and Curbs: ResNet achieves the highest scores for road centerlines and curbs, while morphological filtering improves both networks, but substantial improvement remains possible.Centerlines are derived from the medial axis of road masks, and curbs from road contours.
- Contours and Zoning: Building contours are reasonable but unsatisfactory after polygon simplification, and urban zoning segmentation remains extremely hard from aerial views alone.Building polygons average 13 vertices after applying the Ramer-Douglas-Peucker algorithm with a 0.5m threshold.
- Ground-View Road Segmentation: 97.21% average IoU, 98.64% average pixel accuracy, and 87.53% average top-down IoU show that ground-view road segmentation is nearly solved.The result uses 1,000 training, 200 validation, and 800 testing panoramas.
- Building Height: No network estimates building height from aerial images alone, indicating that the task remains unresolved or requires more sophisticated methods.Ground imagery is suggested as a logical first step.
7. Conclusions
TorontoCity is a large, multi-view benchmark designed for joint reasoning about geometry, grouping, and semantics. Its map-derived ground truth supports diverse urban tasks, while the pilot study finds that most remain difficult for modern convolutional networks.
- Benchmark Scope: TorontoCity covers the full GTA with 712.5km^2 of land, 8439km of road, and around 400,000 buildings.It includes views captured from airplanes, drones, and cars.
- Ground Truth and Tasks: The benchmark uses high-precision maps to create ground truth because human annotation is infeasible at this scale.Its tasks include height estimation, road centerline and curb extraction, instance segmentation, contour extraction, semantic labeling, and scene classification.
- Conclusions: The pilot study shows that most TorontoCity tasks remain difficult for modern convolutional networks.The benchmark is intended to support joint reasoning about geometry, grouping, and semantics.