Source-linked AI summary
Photorealistic Image Synthesis for Object Instance Detection
Tomas Hodan, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta N. Sinha, Brian Guenter
TL;DR
Object-instance detection needs large amounts of expensive real annotated imagery, and synthetic-only training can perform poorly on real images. The paper synthesizes photorealistic scene images using physics-based configurations and physically based rendering, improving Faster R-CNN by 24% on Rutgers APC and 11% on LineMod-Occluded.
Problem
Object-instance detection requires large amounts of expensive real annotated images, while models trained only on synthetic images can show a performance drop on real images.
Method
The approach renders 3D objects in complete scenes, generates plausible object and camera configurations with physics simulation, and uses physically based rendering.
Results
24% absolute mAP@.75IoU improvement was achieved on Rutgers APC and 11% on LineMod-Occluded over rendering objects on random photographs.
Takeaways & Limitations
The approach demonstrates the benefit of highly photorealistic synthetic images for training Faster R-CNN object detectors on real images.
Abstract
from arXiv · showhide
We present an approach to synthesize highly photorealistic images of 3D object models, which we use to train a convolutional neural network for detecting the objects in real images. The proposed approach has three key ingredients: (1) 3D object models are rendered in 3D models of complete scenes with realistic materials and lighting, (2) plausible geometric configuration of objects and cameras in a scene is generated using physics simulations, and (3) high photorealism of the synthesized images achieved by physically based rendering. When trained on images synthesized by the proposed approach, the Faster R-CNN object detector achieves a 24% absolute improvement of mAP@.75IoU on Rutgers APC and 11% on LineMod-Occluded datasets, compared to a baseline where the training images are synthesized by rendering object models on top of random photographs. This work is a step towards being able to effectively train object detectors without capturing or annotating any real images. A dataset of 600K synthetic images with ground truth annotations for various computer vision tasks will be released on the project website: thodan.github.io/objectsynth.
1. INTRODUCTION
The paper addresses the expense of acquiring real annotated images for object-instance detection by synthesizing photorealistic training images. Its approach combines complete scene models, physics-based configuration, and physically based rendering, improving Faster R-CNN performance on real images.
- Motivation: Object-instance detection requires recognizing specific objects and estimating their 2D bounding boxes, while CNN training requires large amounts of expensive real annotated imagery.
- Motivation: Synthetic training data scales with minimal human effort but commonly produces a performance drop when models trained only on synthetic images are tested on real images.
- Approach: Physically based rendering simulates light transport and complex illumination effects, producing images that can be difficult to distinguish from photographs.
- Approach: The proposed synthesis renders 3D objects inside complete scenes with realistic materials and lighting, simulates plausible object and camera configurations, and uses physically based rendering.
- Results: 24% and 11% absolute mAP@.75IoU gains were achieved on Rutgers APC and LineMod-Occluded, respectively, over rendering objects on random photographs.The comparison uses real test images and Faster R-CNN trained on the two synthetic-image alternatives.
2. RELATED WORK
Related work synthesizes training images through object rendering, complete-scene generation, domain adaptation, domain randomization, and physically based rendering. The paper positions its contribution at the intersection of photorealistic rendering and object understanding.
- Rendering Objects: Prior object-rendering methods commonly paste OpenGL-rendered 3D objects onto randomly selected real photographs for detection, segmentation, viewpoint, or pose tasks.
- Rendering Scenes: Complete-scene approaches generate rendered imagery and ground-truth maps, but commercial game engines cannot be customized to insert new 3D object models.
- Domain Adaptation: Domain adaptation bridges synthetic and real domains through real-domain retraining, domain-invariant features, or learned mappings, while domain randomization varies rendering parameters.
- PBR-based Approaches: Physically based rendering has supported intrinsic decomposition, semantic segmentation, normal estimation, boundary detection, and gaze estimation, but prior work focused on scene understanding or eyes rather than object understanding.
3. PROPOSED APPROACH
The proposed pipeline synthesizes objects in detailed indoor scenes by modeling materials and lighting, arranging objects through physics simulation, generating camera views, and rendering with Arnold physically based rendering.
- Object Modeling: The system uses 15 LineMod and 14 Rutgers APC 3D object models with manually assigned material properties.
- Scene Modeling: Objects are arranged in six furnished scenes whose geometry, materials, lighting, and clutter are designed to represent realistic work, household, and shelf environments.
- Scene and Object Composition: Physically simulated objects fall under gravity and undergo mutual collisions after initialization above selected stages, producing plausible arrangements.
- Scene and Object Composition: Cameras target randomly selected objects with randomized azimuth, elevation, and distance, and full RGB rendering is retained only when the focused object is at least 30% visible.
- Scene Modeling: Objects are rendered in six scenes, including models from both LineMod and Rutgers APC.
- Rendering: Arnold physically based rendering produces low-, medium-, and high-quality images, with average rendering times of 15, 120, and 720 seconds, respectively.
4. EXPERIMENTS
Experiments compare photorealistic rendering with baseline composites, evaluate rendering quality, and test scene context using Faster R-CNN on LineMod-Occluded and Rutgers APC. Photorealistic images and accurate context improve detection, while quality benefits vary by scene.
- Experimental Design: The experiments evaluate PBR images against random-photograph composites, compare high and low PBR quality, and assess scene context.
- Experimental Setup: Evaluation uses mAP@.75IoU on real RGB test images from LM-O and RU-APC, with Faster R-CNN trained on synthetic images.
- 4.2. Importance of PBR Quality: On LM-O, high-quality PBR images improved mAP@.75IoU by almost 6% over low-quality PBR images.
- 4.3. Importance of Scene Context: In-context RU-APC training images improved performance by 13% with Inception-ResNet-v2 and 16% with ResNet-101 over out-of-context images.
5. CONCLUSION
The paper concludes that highly photorealistic synthetic images benefit Faster R-CNN object detection and outlines future expansion and dataset release.
- The proposed photorealistic synthesis approach benefits training the Faster R-CNN object detector.
- Future work will explore photorealistic rendering for training models on other vision tasks.
- A dataset of 600K photorealistic images will be released on the project website.
- The authors acknowledge contributions from K. Bekris and C. Mitash to the RU-APC dataset.
SUPPLEMENTARY MATERIAL
The supplementary material documents example images, pose generation, detector results, per-class scores, and physically based rendering quality settings.
- The supplement provides examples of baseline and high-quality physically based rendering images, pose generation, and detector outputs.
- It includes per-class detection scores in Tables 2 and 3 and describes the physically based rendering quality settings.
- PBR Quality Settings: Increasing Arnold ray counts reduces aliasing and illumination noise, while increasing reflection depths improves rendering accuracy.