Source-linked AI summary
LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes
Pat Marion, Peter R. Florence, Lucas Manuelli, Russ Tedrake
TL;DR
Task-specific RGBD segmentation networks need large labeled datasets, but existing datasets do not generally meet that need and per-image labeling is costly. LabelFusion creates pixelwise labels and object poses from real RGBD scenes using reconstruction, human-assisted mesh fitting, and label rendering. It scales to over a million labeled object instances in a few days, and the dataset supports findings about training-data variety and generalization.
Problem
Task-specific RGBD networks may need tens to hundreds of thousands of labeled examples, while existing datasets generally do not satisfy the need for large, representative data.
Source: §I · p. 1Method
LabelFusion reconstructs multi-view RGBD scenes, uses human-assisted ICP fitting of object meshes, and renders projected meshes into pixelwise labels and object poses.
Source: §I · p. 1Results
Networks trained on multi-object scenes performed significantly better than those trained on single-object scenes, and about 50 backgrounds yielded above-50% IoU on entirely novel scenes.
Source: §V · p. 6Takeaways & Limitations
The dataset enables evaluation of how training-data type and quantity affect generalization for practical robotic-manipulation segmentation networks.
Source: B. Empirical Evaluations: How Much… · p. 5, §V · p. 6
Abstract
from arXiv · showhide
Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.
I. INTRODUCTION
Practical RGBD segmentation networks need large, task-specific labeled datasets, but labeling images individually is prohibitively time-consuming. The paper introduces a scalable real-data pipeline and evaluates how training-data characteristics affect generalization.
I. INTRODUCTION
- Task-specific networks may require tens to hundreds of thousands of labeled examples, making once-per-image human labeling prohibitive.Prior work has explored simulated data, while this method targets scaling to these quantities with real data.Source: §I · p. 1
- The pipeline fuses multi-view RGBD into a dense reconstruction, fits object meshes with ICP assistance, and renders projected meshes as labels, enabling labeling once per scene.The paper reports over 1,000,000 labeled object instances collected in a few days.Source: §I · p. 1
- The paper evaluates how multi-object scenes, background variety, and views per scene affect generalization, alongside reviewing related work on RGBD labeling, pose estimation, and data requirements.The related-work review organizes prior research into those three areas.Source: §I · p. 1, §II · p. 1
A. Methods for Generating Labeled RGBD Datasets
Prior RGBD dataset-generation methods differ in annotation scope and scalability: some provide pose labels without pixelwise labels, while others provide pixelwise masks but have constraints on scene composition or calibration.
A. Methods for Generating Labeled RGBD Datasets
- T-LESS provides about 49K RGBD images with 6DOF pose labels, but requires calibrated equipment and manual mesh alignment; motion-capture approaches offer high-fidelity poses with scaling constraints.The cited constraints include restricted environments, object markers, and time-intensive setup.Source: A. Methods for Generating Labeled RGBD… · p. 2
- Background subtraction can produce pixelwise labels, but its cited implementation uses one object per scene and requires accurately calibrated robot-arm motion.It generated 130,000 labeled images for 39 objects and can label deformable objects.Source: A. Methods for Generating Labeled RGBD… · p. 2
- SceneNN and ScanNet annotate dense 3D reconstructions with generic classes, not specific-object geometric models or 6DOF poses; LabelFusion instead targets application-specific objects and labels.Their stated focus is scene-understanding benchmarks.Source: A. Methods for Generating Labeled RGBD… · p. 2
B. Object-Specific Pose Estimation in Clutter for Robotic
The paper situates its pipeline among methods for estimating object poses in clutter and motivates real-data generalization experiments by noting that robot vision differs from ImageNet-style imagery.
B. Object-Specific Pose Estimation in Clutter for Robotic
- In cluttered RGBD scenes, cited pose pipelines use pixelwise segmentation to crop point clouds before ICP registration to known meshes; other approaches learn pose directly or bypass pose estimation.The paper describes its labels as useful for training segmentation networks for object-specific manipulation in clutter.Source: B. Object-Specific Pose Estimation in… · p. 2
C. Empirical Evaluations of Data Requirements for Image Segmentation Generalization
- Robotic visual data and desired object specificity differ from ImageNet-style images, and the authors report no prior generalization studies at the scale of real data used here.Related work had examined data requirements using simulation and augmenting small real datasets with simulated data.Source: C. Empirical Evaluations of Data… · p. 2
III. DATA GENERATION PIPELINE
The data-generation pipeline collects RGBD views, reconstructs a scene, fits object meshes, and renders labels. Data collection can use either an automated arm or a hand-carried sensor.
III. DATA GENERATION PIPELINE
- The pipeline comprises RGBD collection, dense 3D reconstruction, mesh generation, human-assisted annotation, and rendering labeled images.The overview shows user-initialized mesh alignment followed by ICP refinement and rendering masks into RGB images.Source: §III · p. 2, Figure 1 · p. 3
A. RGBD Data Collection
- RGBD capture can use an automated arm to reduce human workload or a hand-carried sensor that avoids sophisticated equipment and supports collection across varied environments.The paper reports using the handheld approach for 112 scenes and the arm-mounted approach for 26 scenes.Source: A. RGBD Data Collection · pp. 2–3
- In the arm-mounted setup, the sensor can scan without requiring knowledge of the transform between robot and camera because processing is done in the camera frame.Typical logs averaged 120 seconds, with data captured at 30 Hz.Source: A. RGBD Data Collection · p. 3
B. Dense 3D Reconstruction
The pipeline uses ElasticFusion for dense reconstruction and camera tracking, while its modular design permits other reconstruction methods with camera pose tracking. Object meshes support faster annotation, but must be rigid and sufficiently high quality for ICP alignment.
B. Dense 3D Reconstruction
- ElasticFusion reconstructs scenes in real time on a desktop with an NVIDIA GTX 1080 GPU, but large flat surfaces lacking RGB or depth texture can cause failures.The pipeline is modular, so any reconstruction method providing camera pose tracking can replace ElasticFusion.Source: B. Dense 3D Reconstruction · p. 3
C. Object Mesh Generation
- Meshes speed annotation by aligning object models instead of manually segmenting the 3D reconstruction, and the pipeline requires rigid objects.Meshes must be sufficiently high quality for ICP alignment; RGB textures are unnecessary.Source: C. Object Mesh Generation · pp. 3–4
D. Human Assisted Annotation
Human-assisted initialization lets ICP align object meshes in cluttered reconstructions, where tested global registration methods did not provide satisfactory results. Annotation takes about 30 seconds per object and yields over 80% IoU against polygon-drawn labels with approximately four orders of magnitude less human effort per image.
D. Human Assisted Annotation
- Tested global registration methods,, did not align known objects satisfactorily, in part because many scene points belonged to none of the objects.The pipeline therefore uses human input to assist registration.Source: D. Human Assisted Annotation · p. 4
- Three corresponding scene and mesh points initialize object pose, after which ICP refines alignment on a pointcloud cropped to within 1 cm of the model.A single click can segment the table when objects are on a flat surface.Source: D. Human Assisted Annotation · p. 4
- 30 seconds per object; labels achieve over 80% IoU against polygon-drawn image labels with approximately four orders of magnitude less human effort per image.The annotation is also faster and more accurate than manually aligning full object meshes, which can take several minutes per object.Source: D. Human Assisted Annotation · p. 4
E. Rendering of Labeled Images and Object Poses
After 3D annotation, object poses are projected into each logged camera frame to render pixelwise labels. The discussion emphasizes that one 3D labeling effort covers thousands of images, with about 30 seconds of annotation per object per scene.
E. Rendering of Labeled Images and Object Poses
- Object meshes projected into rendered RGB images produce pixelwise labels, while reconstruction camera poses and aligned object models provide object poses for each image frame.The section includes Table I, titled “Dataset Description.”Source: E. Rendering of Labeled Images and… · p. 4, Table I · p. 4
F. Discussion
- The method requires no sophisticated calibration, handles arbitrary rigid objects in general environments, and labels thousands of same-scene RGBD images from one 3D annotation effort.Human annotation takes 30 seconds per object per scene.Source: F. Discussion · p. 4
IV. RESULTS
The results section first evaluates the LabelFusion data-generation pipeline, then examines how different aspects of training data relate to generalization.
IV. RESULTS
- The results section evaluates the effectiveness of the LabelFusion data-generation pipeline.This is the first of the two announced analyses.Source: §IV · p. 4
- The paper uses data generated by the pipeline for empirical experiments on the generalization value of different aspects of training data.These experiments follow the pipeline evaluation.Source: §IV · p. 4
- The announced experiments aim to quantify how different aspects of training data relate to generalization.The passage identifies the experimental scope but reports no findings.Source: §IV · p. 4
A. Evaluation of Data Generation Pipeline
LabelFusion generated a large, varied set of labeled RGBD images, while requiring about 30 seconds of human time per object per scene. Most pipeline components ran at approximately real-time, and parallelizable postprocessing could keep pace with a single sensor.
A. Evaluation of Data Generation Pipeline
- Over 352,000 labeled RGBD images and 1,000,000+ object instances were generated, including scenes spanning varied environments, lighting, and clutter.Over 200,000 images were generated in approximately one day by two people.Source: A. Evaluation of Data Generation… · p. 4
- Example labeled scenes cover heavy multi-object clutter, low light, motion blur, varied object distances, and 25 environments collected with a hand-carried sensor.The pipeline’s example performance figure shows network (e) on a multi-object test scene.Source: Figure 2 · p. 5, Figure 4 · p. 5
- Human labeling took approximately 30 seconds per object per scene; a small cluster of 2–4 desktop machines could post-process data at real-time rates.Postprocessing took several times real-time but was parallelizable; the paper estimates that one to two people and a handful of computers could support 30 Hz data generation.Source: Figure 3 · p. 4, A. Evaluation of Data Generation… · p. 5
B. Empirical Evaluations: How Much Data Is Needed For Practical Object-Specific Segmentation?
The experiments vary training-scene composition, background diversity, and views per scene to examine segmentation generalization. They find advantages for multi-object training in multi-object and novel-background settings, while the effect of sampling rate depends on camera motion.
B. Empirical Evaluations: How Much Data Is Needed For Practical Object-Specific Segmentation?
- The evaluation varies multi-object versus single-object training, number of background environments, and views per scene using ResNet segmentation networks and common test sets.Source: B. Empirical Evaluations: How Much… · p. 5
- Average per-object IoU increased 190% with multi-object training despite fewer labeled pixels, and IoU performance increased 369% with equal scene counts versus single-object training.After 18 multi-object scenes, adding 18 single-object scenes had no noticeable effect on multi-object generalization.Source: B. Empirical Evaluations: How Much… · pp. 5–6
- Training on 1, 2, 5, 10, 25, or 50 backgrounds produced approximately logarithmic performance growth on 11 unseen environments.Multi-object training with occlusions also generalized better to novel backgrounds than single-object training.Source: B. Empirical Evaluations: How Much… · p. 6
- On single-object tests, the multi-object-training advantage was absent; single-object training scenes were sufficient for IoU above 60%.Source: B. Empirical Evaluations: How Much… · p. 6
- Higher effective sensor rates monotonically improved segmentation, with heavily diminished returns after 0.3 Hz for slower robot-arm-mounted data and more gains for faster hand-carried data.The reported camera motion velocities were approximately 0.03 m/s for robot-arm-mounted data and 0.05–0.17 m/s for hand-carried data.Source: B. Empirical Evaluations: How Much… · p. 6
V. CONCLUSION
LabelFusion generates RGBD images with per-pixel labels and ground-truth object poses using only minutes of human labeling time per scene. Experiments show that multi-object training scenes and varied backgrounds improve segmentation generalization.
V. CONCLUSION
- LabelFusion produces RGBD data with per-pixel labels and ground-truth object poses, requiring only a few minutes of human time for a scene containing thousands of images.Source: §V · p. 6
- The authors provide LabelFusion as open-source software and supply an example dataset generated with the pipeline.Source: §V · p. 6
- Networks trained on multi-object scenes performed significantly better than networks trained on single-object scenes across novel multi-object and new-background single-object scenes.The recommendation is to prioritize multi-object data collection across varied backgrounds for the greatest generalization gains.Source: §V · p. 6
- Approximately 50 training backgrounds enabled single-object-scene networks to exceed 50% IoU on entirely novel scenes.Source: §V · p. 6
- The authors hope the pipeline and segmentation analysis will guide data collection for generalization in robotic manipulation settings.Source: §V · p. 6