Source-linked AI summary
Playing for Benchmarks
Stephan R. Richter, Zeeshan Hayder, Vladlen Koltun
TL;DR
Existing benchmarks provide only partial combinations of task coverage, annotation density, temporal consistency, and environmental diversity for visual perception. This paper introduces a virtual-world benchmark and a source-independent collection method, finding that it is challenging while closely matching reference scene statistics.
Problem
Visual perception benchmarks lack a single dataset combining broad low- and high-level task coverage, dense per-frame ground truth, temporal structure, and diverse environmental conditions.
Method
The paper instruments a photorealistic virtual world without source-code access to collect video-rate ground truth for multiple tasks, including dense correspondences and object-level 3D layouts.
Results
The benchmark is challenging for state-of-the-art methods, while its scene-category distribution nearly matches Cityscapes with Jensen-Shannon divergence 0.003.
Takeaways & Limitations
Shared, temporally consistent ground truth across tasks and conditions is intended to support robust broad-competence visual perception systems.
Takeaways & Limitations
The collection methodology must handle meshes that appear or disappear, duplicate segment IDs, motion, and level-of-detail changes across consecutive frames.
Abstract
from arXiv · showhide
We present a benchmark suite for visual perception. The benchmark is based on more than 250K high-resolution video frames, all annotated with ground-truth data for both low-level and high-level vision tasks, including optical flow, semantic instance segmentation, object detection and tracking, object-level 3D scene layout, and visual odometry. Ground-truth data for all tasks is available for every frame. The data was collected while driving, riding, and walking a total of 184 kilometers in diverse ambient conditions in a realistic virtual world. To create the benchmark, we have developed a new approach to collecting ground-truth data from simulated worlds without access to their source code or content. We conduct statistical analyses that show that the composition of the scenes in the benchmark closely matches the composition of corresponding physical environments. The realism of the collected data is further validated via perceptual experiments. We analyze the performance of state-of-the-art methods for multiple tasks, providing reference baselines and highlighting challenges for future research. The supplementary video can be viewed at https://youtu.be/T9OybWv923Y
1. Introduction
The paper introduces a high-resolution video benchmark that provides dense ground truth across complementary low- and high-level visual perception tasks. It uses realistic simulated-world data collected across diverse conditions and evaluates state-of-the-art methods to expose challenges for future research.
- The benchmark is designed to support broad-competence perception systems that integrate temporal information and complementary tasks into comprehensive environment models.
- The benchmark evaluates optical flow, visual odometry, semantic instance segmentation, object detection and tracking, and 3D scene layout on shared video data.Ground truth is available for every frame, including pixel-level segmentations and subpixel-accurate correspondences.
- 254,064 fully annotated frames span 184 kilometers of driving, riding, and walking in varied environmental conditions.
- A new methodology collects video-rate ground truth from simulated worlds without access to their source code or content.The approach supports dense correspondences and instance-level 3D layouts.
- Experiments on semantic segmentation, semantic instance segmentation, visual odometry, and optical flow indicate that the benchmark is challenging and offers directions for future work.
2. Background
Prior benchmarks separately provide broad task coverage, accurate annotations, or environmental diversity, but each has important limitations. The paper combines these characteristics in one large-scale virtual-world benchmark and extends them with temporally consistent 3D and dense correspondence ground truth.
- KITTI evaluates multiple low- and high-level tasks on video but has sparse optical-flow ground truth, limited detection and segmentation annotations, and fair-weather data from one town.
- Cityscapes provides highly accurate instance-level annotations, but only 5,000 frames are annotated at high accuracy and annotation takes more than 90 minutes per frame.
- HCI and Oxford RobotCar address changing environmental conditions through recordings across seasons, nights, rain, and snow, while retaining ground-truth acquisition challenges.
- The benchmark integrates broad task coverage, accurate instance annotations, and diverse conditions in one large-scale dataset, then adds temporally consistent object-level ground truth and 3D scene layouts at video rate.
- Scaling comprehensive virtual benchmarks remains difficult because realistic large-scale content creation requires detailed geometry, materials, lighting, and object animation.
- Earlier game-based data collection could not operate at video rate or provide instance-level segmentations and dense correspondences, motivating the new approach.
3. Data Collection Methodology
The data-collection system instruments a photorealistic game without source access, extracting rendering resources and reconstructing object identities, geometry, motion, and dense correspondences across frames. It combines runtime shader and bytecode techniques with matching and inverse-rendering procedures to produce rich ground truth at video rate.
- Data collection setup: Grand Theft Auto V supplies a functioning city and surroundings in a photorealistic three-dimensional world for benchmark generation.
- Runtime instrumentation: Runtime shader augmentation tags pixels with rendering-resource IDs, depth, and transparency values despite unavailable shader source code.The system operates directly on bytecode and uses dynamic software updating.
- Object identification: Tracking resources and transformation matrices yields temporally consistent semantic patches and clusters object parts into individual instances.Majority voting over constituent patches resolves conflicting semantic labels.
- 3D scene layout: Recorded meshes and transformations recover camera position and produce object 3D bounding boxes in the camera reference frame.
- Challenges: The methodology must handle disappearing meshes, duplicate segment IDs, camera and object motion, and level-of-detail replacements across frames.
- Temporal association: Weighted matching associates meshes across consecutive frames by minimizing motion while pruning mismatched classes and dynamic-object segment IDs.Class-dependent speed limits constrain plausible associations.
- Dense correspondences: Inverse transformation of recorded rendering pipelines recovers dense surface correspondences for rigid objects from pixel depth and stored matrices.
- Dense correspondences: Offline shader slicing and rasterizer inversion provide exact subpixel-accurate correspondences for nonrigid objects.The method maps camera-space points back to object-local coordinates through selectively executed shader slices.
4. Dataset
VIPER is a large, fully annotated video benchmark spanning diverse conditions and multiple visual-perception tasks. Its scene statistics and perceptual ratings are compared with established real and synthetic datasets to assess realism.
- Dataset construction: Every video frame has pixelwise semantic categories, dense instance segmentations, instance-level boundaries, tracking IDs, and 3D scene layout annotations.The annotations support consistent evaluation across multiple visual-perception tasks on the same data.
- Statistical analysis: VIPER’s category-count distribution closely matches Cityscapes, with JSD (Cityscapes ∥VIPER) = 0.003 versus 0.67 for COCO and 0.69 for KITTI.Its instance-count distribution also matches Cityscapes more closely: JSD values are 0.02 for VIPER, 0.19 for COCO, and 0.30 for KITTI.
- Statistical analysis: VIPER provides instance-level labels for 11 semantic categories, compared with 10 in Cityscapes and 7 in KITTI.The number of instances with pixel-level segmentation masks for each class is more than an order of magnitude higher.
- Statistical analysis: 0.02 is the JSD between VIPER and Cityscapes vehicle-distance distributions, compared with 0.12 between KITTI and Cityscapes.This analysis evaluates the realism of 3D scene layouts using vehicle distance from the camera.
- Perceptual experiment: At 8 seconds, VIPER was judged more realistic than other synthetic datasets in 75% to 94% of comparisons.At 125 milliseconds, VIPER exceeded other synthetic datasets in 60% to 73% of comparisons; real Cityscapes versus VIPER reached 89% at 8 seconds.
5. Baselines and Analysis
The benchmark provides baselines across semantic, motion, and geometric perception tasks, while analyses reveal realistic but demanding conditions, distance effects, and environmental challenges.
- Evaluation scope: The evaluation covers semantic segmentation, semantic instance segmentation, visual odometry, and optical flow using representative state-of-the-art methods.The goals include validating realism, assessing difficulty, providing reference baselines, and characterizing method performance.
- Semantic segmentation: 69% mean IoU was achieved by PSPNet on VIPER, compared with above 80% on Cityscapes, indicating greater semantic-segmentation difficulty.PSPNet remained ahead of the ResNet-50 baseline by approximately 14 points on both datasets.
- Semantic instance segmentation: 12.6 AP for BAIS on VIPER versus 17.4 on Cityscapes shows that VIPER is also more challenging for semantic instance segmentation.Models trained only on day and sunset images lose performance in other conditions, particularly at night.
- Optical flow: Optical-flow performance is challenged most by adverse weather, ground and vehicle motion, very large objects, and very large displacements.The benchmark uses a weighted area under the curve over 0–5 px thresholds, emphasizing lower-threshold accuracy.
- Visual odometry: Visual-odometry accuracy is highest in easy daytime settings but is much lower than corresponding KITTI results across conditions.Snow, night, sunset lens flares, and rain reflections introduce keypoint or pose-estimation difficulties.
- Distance analysis: All instance-segmentation methods perform best within 10 meters of the camera, with accuracy deteriorating as distance increases.The analysis measures performance as a function of object distance under different conditions.
- Future directions: Integrating visual odometry with semantic analysis is proposed as a future direction because semantic information can help handle dynamic objects and scale drift.The benchmark’s integrated ground truth supports developing such techniques.
6. Conclusion
The paper presents a large visual-perception benchmark with dense, multimodal ground truth across environmental conditions. It is intended to support methods that exploit temporal structure and complementary perception tasks, with withheld test labels enabling public evaluation.
- Contribution: The benchmark provides ground-truth data for low-level and high-level vision tasks across more than 250 thousand video frames in different environmental conditions.The conclusion emphasizes the breadth of annotated video data.
- Scientific impact: The benchmark is designed to support techniques that leverage temporal structure and complementary visual-perception tasks.The authors aim to encourage robust broad-competence systems that model their environments.
- Release and evaluation: Withheld test ground truth will support a public evaluation server and leaderboard, alongside future baselines and integrated-task challenges.Planned challenges include temporally consistent instance segmentation, tracking, and 3D layout estimation in video.