Source-linked AI summary
Playing for Data: Ground Truth from Computer Games
Stephan R. Richter, Vibhav Vineet, Stefan Roth, Vladlen Koltun
TL;DR
Pixel-accurate semantic labels are expensive because they require extensive boundary tracing. The paper reconstructs associations from commercial-game rendering to propagate labels rapidly, producing 25 thousand labeled images and improving real-world segmentation while reducing hand-labeling needs.
Problem
Creating large datasets with pixel-accurate semantic labels is costly because accurate object-boundary tracing requires substantial human effort.
Method
The approach detours game rendering, hashes geometry, textures, and shaders into persistent object signatures, and propagates labels across images, scenes, sessions, and time.
Results
25 thousand images were labeled in 49 hours, and game data improved semantic-segmentation accuracy while models using 1/3 of CamVid outperformed models using the complete training set.
Takeaways & Limitations
Game-generated data can supplement real-world training images and reduce the need for expensive conventional semantic-segmentation labeling.
Abstract
from arXiv · showhide
Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.
1 Introduction
The paper addresses the cost of pixel-accurate semantic labeling by extracting reusable object information from commercial game rendering, enabling rapid annotation and useful synthetic training data.
- Motivation: Pixelwise semantic labeling is difficult because tracing accurate object boundaries requires substantial human effort.CamVid labeling reportedly required 60 minutes per image, while Cityscapes required 90 minutes per image.
- Motivation: Commercial open-world games offer realistic appearance, environments, object behavior, and interactions, but their internal operation and content are largely inaccessible.The approach targets games such as Grand Theft Auto, Watch Dogs, and Hitman.
- Approach: Detouring records and modifies rendering commands, hashes rendering resources, and propagates pixel-accurate object labels across scenes, sessions, time, and shared instances.The method uses a wrapper between the game and operating system to access communication with graphics hardware without game source code.
- Results: 25 thousand images from Grand Theft Auto V were densely labeled in 49 hours, roughly three orders of magnitude faster than similarly dense semantic-segmentation annotation.Propagation reduced average annotation time per image as new object labels spread across images.
- Results: Models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.The result indicates that acquired synthetic data can reduce the amount of hand-labeled real-world data required.
2 Related Work
Prior work used synthetic data for benchmarking, object- or task-specific training, and depth-based semantic understanding; this paper instead trains semantic segmentation from realistic game imagery and labels.
- Synthetic data: Synthetic data has long supported computer-vision benchmarking, especially optical-flow evaluation where accurate real-world ground truth is difficult to obtain.MPI-Sintel additionally provides ground truth for depth estimation and related tasks.
- Synthetic data: Rendered parametric models and 3D objects have been used to train systems for pose estimation, detection, shape-from-shading, optical flow, stereo disparity, and scene flow.These approaches span object-level and whole-environment renderings.
- Computer games: Computer-game tools have supported surveillance evaluation, pedestrian-detector training, and autonomous-driving research, but this work targets semantic segmentation with realistic game worlds.The paper distinguishes its use of commercial-game data for training from prior simulator and game-tool applications.
- Depth-based methods: Indoor scene models synthesize semantic labels with depth maps but provide no appearance cues, limiting trained models to depth inputs.The paper instead uses modern computer games to improve segmentation on real-world benchmarks from regular color images.
3 Breaking the Curse of Dataset Annotation
The paper breaks the annotation bottleneck by intercepting rendering communication from a commercial game and converting resource metadata into persistent, pixel-linked patches. These patches support label propagation across frames and instances, enabling dense, diverse datasets with far less manual effort.
- 3.1 Data acquisition: The approach extracts pixel-level annotation cues from off-the-shelf games without access to their source code or content.The system intercepts and records communication between the game and graphics hardware, using a wrapper around the graphics API.
- 3.1 Data acquisition: Resource identities are made persistent across frames and gameplay sessions by hashing the memory content associated with meshes, textures, and shaders.Per-frame lookup tables map volatile application resource IDs to 128-bit hash keys.
- 3.1 Data acquisition: A second rendering pass encodes mesh, texture, and shader IDs into pixels, producing resource ID maps that link image surfaces to rendering resources.The first pass preserves the conventional color and buffer outputs; the second pass stores resource IDs at each pixel.
- 3.2 Semantic labeling: Images are decomposed into patches sharing a common mesh–texture–shader combination, whose boundaries support pixel-accurate labeling without manual object tracing.These patches are linked across images and generally remain within a single object, while their boundaries coincide with semantic-class boundaries.
- 3.3 Dataset and analysis: 98.7% of frames were more than 90% pre-annotated when reached, while the collected images remained highly variable in MTS usage and content.26.5% of MTS combinations occurred in only one image, the median occurrence was four frames, and sampled images varied in content and layout.
4 Semantic Segmentation
The experiments evaluate joint training with game data on CamVid and KITTI semantic segmentation. Game data improves accuracy and reduces the amount of labeled real-world data needed.
- Training procedure: The training procedure first jointly trains on real and synthetic images, then fine-tunes on real images only.The first stage uses minibatches containing four real and four synthetic images; the second stage runs for 4K iterations.
- CamVid dataset: Training with synthetic images increases CamVid mean IoU by 3.9 percentage points.The comparison uses the full CamVid training set augmented with synthetic images.
- CamVid dataset: Game data with one-third of the CamVid training set surpasses full-CamVid training without game data.This result indicates that synthetic images can reduce the amount of hand-labeled real-world training data required.
- CamVid dataset: Synthetic-data training outperforms Kundu et al. by 2.8 percentage points on CamVid without temporal cues.Kundu et al. used a larger ConvNet and whole video sequences, whereas this comparison considers individual frames only.
- KITTI dataset: The KITTI model trained with game data outperforms the model trained without game data by 2.6 percentage points.The evaluation uses the Ros et al. split of 100 training images and 46 test images.
5 Discussion
The approach produces dense semantic annotations from a realistic open-world game and improves semantic segmentation on real-world images. The authors also identify extensions to video and other dense prediction tasks.
- Discussion: The approach produces dense pixel-level annotations for 25 thousand images from a realistic open-world game.The presented data also increases semantic segmentation performance on real-world images and reduces conventional labeling needs.
- Discussion: Game-derived data improves KITTI performance by 2.6 percentage points over training without game data.This result is reported in the KITTI dataset table.
- Discussion: The approach could be extended to densely annotated video, multiple dense prediction problems, and instance-level segmentation.Listed applications include optical flow, scene flow, depth estimation, boundary detection, stereo reconstruction, intrinsic image decomposition, and visual odometry.