Source-linked AI summary
Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation
Jonathan Tremblay, Thang To, Stan Birchfield
TL;DR
Accurate robotic detection and pose estimation need ground-truth data, but real-world collection is time-consuming, error-prone, expensive, and difficult to scale. FAT addresses this gap with a photorealistic synthetic dataset containing varied household scenes, rich annotations, and multiple sensor modalities, supporting research in robotic scene understanding.
Problem
Robotic perception requires accurate detection and pose estimation of multiple household objects, but real-world ground-truth collection is costly, error-prone, time-consuming, and does not scale to datasets needed for deep neural networks.
Method
FAT synthetically combines YCB household-object models with virtual environments, physics-based falling and collisions, photorealistic rendering, stereo RGBD cameras, and comprehensive scene annotations.
Results
FAT provides 61,500 snapshots with varied backgrounds, poses, occlusions, and lighting conditions, alongside statistics quantitatively confirming this variety.
Takeaways & Limitations
The dataset supports research on object detection, pose estimation, segmentation, depth estimation, and monocular, stereo, and RGBD sensing for robotic manipulation.
Abstract
from arXiv · showhide
We present a new dataset, called Falling Things (FAT), for advancing the state-of-the-art in object detection and 3D pose estimation in the context of robotics. By synthetically combining object models and backgrounds of complex composition and high graphical quality, we are able to generate photorealistic images with accurate 3D pose annotations for all objects in all images. Our dataset contains 60k annotated photos of 21 household objects taken from the YCB dataset. For each image, we provide the 3D poses, per-pixel class segmentation, and 2D/3D bounding box coordinates for all objects. To facilitate testing different input modalities, we provide mono and stereo RGB images, along with registered dense depth images. We describe in detail the generation process and statistical analysis of the data.
1. Introduction
Robotic manipulation needs accurate multi-object detection and pose estimation, but real-world ground-truth collection is costly, error-prone, and difficult to scale. FAT addresses this gap with photorealistic synthetic scenes and comprehensive annotations.
- Motivation: Real-world ground-truth acquisition is time-consuming, error-prone, expensive, and difficult to scale for deep-learning datasets.These constraints limit evaluation across new object categories and environmental conditions.
- Motivation: Synthetic data is proposed as an efficient alternative when computer-vision ground truth is laborious to collect.The paper identifies 3D detection and pose estimation as natural applications because accurate ground truth is difficult to acquire.
- Contribution: FAT extends existing pose datasets in quantity and variety, including extreme lighting variations and multiple modalities.The comparison concerns datasets with accurate poses for multiple occluded objects, including T-LESS and YCB-Video.
2. Falling Things Dataset
FAT is generated in Unreal Engine 4 by dropping YCB objects into varied virtual environments and recording multimodal sensor data. Its single-object and mixed-object splits provide diverse poses, occlusions, and scene conditions.
- Scene generation: FAT uses three high-fidelity virtual environments and 15 locations spanning indoor and outdoor backgrounds, terrain, lighting, and shadows.The environments are a kitchen, sun temple, and forest.
- Object generation: Twenty-one YCB household objects are randomly initialized, then allowed to fall, collide, and settle under gravity in each scene.Object coordinate frames are centered and aligned before simulation.
- Sensor modalities: The stereo RGBD camera design supports monocular RGB, stereo RGB, and RGBD modalities for cross-modality research.The system uses paired cameras, enabling researchers to choose among these sensor inputs.
- Dataset composition: 61,500 unique frames are divided into 31,500 single-object frames and 30,000 mixed-object frames.Mixed scenes contain uniformly sampled sets of 2–10 objects, with replacement allowing repeated categories.
- Dataset statistics: Visibility statistics count occurrences above 25% and 75% visibility, while smaller objects are occluded more often than larger objects.Occurrences below 25% visibility are omitted from the displayed statistics.
- Dataset statistics: Mustard-bottle statistics show nonuniform yaw and pitch, uniform roll, camera distances slightly beyond 0.5 m–1.5 m, significant occlusion, and broadly centered centroids.Yaw modes arise because objects often rest on their front or back after falling; rolling or sliding explains distance-range extensions.
3. Conclusion
The paper presents FAT as a high-fidelity dataset for robotics perception, covering household objects across varied backgrounds, poses, occlusions, and lighting conditions. It is intended to support research across detection, pose, depth, segmentation, and sensor modalities.
- Conclusion: FAT targets object detection, pose estimation, segmentation, depth estimation, and sensor-modality research for robotic manipulation.The dataset focuses on household items from the YCB dataset.
- Conclusion: The dataset combines high-fidelity rendering with broad variation in backgrounds, poses, occlusions, and lighting conditions.The paper states that dataset statistics quantitatively confirm this variety.
A. Sample Images
Sample FAT images demonstrate variation in object poses, backgrounds, scene composition, and lighting conditions.
- Sample Images: Sample images illustrate varied object poses, backgrounds, scene composition, and lighting conditions.The displayed examples use random center crops.
B. YCB Objects
The dataset uses 21 household objects from the YCB dataset, including items such as a pitcher base, bleach cleanser, cracker box, and power drill.
- 21 YCB household objects are used to create the dataset, spanning items such as cans, boxes, tools, kitchenware, and produce.