Source-linked AI summary
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, Silvio Savarese
TL;DR
6D pose estimation from RGB-D data must reconcile complementary color and depth information while remaining accurate and fast in cluttered scenes. DenseFusion processes the modalities separately, fuses dense per-pixel features, and performs learned iterative refinement. It outperforms prior approaches on YCB-Video and LineMOD and supports robot grasping and manipulation.
Problem
Existing methods struggle to use complementary RGB-D information while satisfying accuracy and real-time inference requirements, particularly under heavy occlusion.
Method
DenseFusion processes color and depth separately, fuses them through dense per-pixel feature embeddings, and integrates neural iterative pose refinement.
Results
DenseFusion outperforms state-of-the-art methods on YCB-Video and LineMOD, including 3.5% higher pose accuracy and 200x faster inference than PoseCNN after ICP refinement.
Takeaways & Limitations
The method is sufficiently robust for highly cluttered scenes and was used by a real robot to grasp and manipulate objects from estimated poses.
Takeaways & Limitations
For symmetric objects, multiple canonical frames make the original pose loss ambiguous, so DenseFusion uses a different objective.
Abstract
from arXiv · showhide
A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information from the RGB image and depth separately or use costly post-processing steps, limiting their performances in highly cluttered scenes and real-time applications. In this work, we present DenseFusion, a generic framework for estimating 6D pose of a set of known objects from RGB-D images. DenseFusion is a heterogeneous architecture that processes the two data sources individually and uses a novel dense fusion network to extract pixel-wise dense feature embedding, from which the pose is estimated. Furthermore, we integrate an end-to-end iterative pose refinement procedure that further improves the pose estimation while achieving near real-time inference. Our experiments show that our method outperforms state-of-the-art approaches in two datasets, YCB-Video and LineMOD. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.
1. Introduction
DenseFusion addresses the difficulty of achieving accurate, fast 6D pose estimation from RGB-D data, especially in cluttered and occluded scenes. It combines per-pixel RGB-D fusion with end-to-end iterative refinement and reports stronger accuracy and speed than prior methods.
- RGB-D pose estimation must handle occlusion, sensor noise, lighting variation, and real-time speed requirements simultaneously.
- Classical RGB-D methods rely on handcrafted features and fixed correspondence procedures, limiting performance under heavy occlusion and lighting variation.
- Deep RGB-D methods such as PoseCNN and MCN use elaborate post-hoc refinement steps that cannot be jointly optimized and are too slow for real-time applications.
- DenseFusion embeds and fuses RGB values with point clouds at the per-pixel level, explicitly modeling local appearance and geometry for heavy occlusion.
- The integrated iterative refinement procedure improves pose estimation while maintaining real-time inference speed.
- 3.5% higher pose accuracy and 200x faster inference than PoseCNN after ICP refinement were reported across YCB-Video and LineMOD.
2. Related Work
Prior 6D pose methods use RGB, depth, or RGB-D data with differing representations and fusion strategies. DenseFusion builds on these approaches by combining appearance and geometry while replacing naive fusion and costly refinement with learned alternatives.
- RGB-only methods become unreliable with low-texture or low-resolution inputs because they lack depth information.
- Voxel-based depth methods encode geometry effectively but can be prohibitively expensive, with one method taking nearly 20 seconds per frame.
- Point-cloud methods achieve strong performance in urban driving, but generic object pose estimation requires both geometric and appearance information.
- RGB-D methods have used hardcoded features or surrogate objectives rather than directly optimizing the true 6D pose estimation objective.
- DenseFusion differs from PointFusion by using local feature fusion instead of naive concatenation and adds iterative refinement to improve pose estimation.
3. Model
DenseFusion estimates known-object 6D poses by separately encoding RGB and depth while fusing their embeddings densely at corresponding pixels. Per-pixel pose predictions are confidence-weighted and refined iteratively to address clutter, occlusion, and the inefficiency of post-hoc refinement.
- Architecture: The model segments known objects from RGB images, then supplies masked depth pixels as 3D points and image patches to pose estimation.Segmentation produces object masks and bounding boxes before pose estimation operates on each segmented object.
- Dense Feature Extraction: RGB and depth are processed separately into color and geometric embeddings because their information resides in different spaces.The depth channel is converted into a 3D point cloud, while image features preserve pixel structure.
- Pixel-wise Dense Fusion: Dense fusion associates each 3D point with its corresponding image pixel, combining their features for pixel-wise pose prediction.The mapping uses projection onto the image plane with known camera intrinsics, rather than relying only on a global feature.
- 6D Object Pose Estimation: Each fused feature produces a pose and confidence score, allowing the network to select the highest-confidence prediction among per-pixel hypotheses.This design can favor visible object regions and reduce effects from occlusion and segmentation noise.
- 6D Object Pose Estimation: The pose loss compares transformed model points with ground-truth points, using a closest-point objective for symmetric objects.The symmetric-object objective avoids ambiguous canonical-frame supervision.
- Iterative Refinement: A neural iterative refinement module updates the initial pose through repeated residual estimation and avoids dependence on costly ICP post-processing.The refinement network reuses image embeddings and fuses them with geometric features from the newly transformed point cloud.
4. Experiments
DenseFusion is evaluated on YCB-Video and LineMOD using benchmark comparisons, model variants, occlusion analysis, runtime measurements, and robotic grasping. The experiments show gains from dense fusion and iterative refinement, robustness under occlusion, near-real-time efficiency, and practical grasping performance.
- Experimental setup: The experiments use YCB-Video and LineMOD, comparing DenseFusion with state-of-the-art methods and four model variants.YCB-Video contains objects with varying shapes, textures, and occlusion conditions; LineMOD supports broader comparisons with existing methods.
- Metrics: ADD-S evaluates YCB-Video pose accuracy, including the percentage of predictions below the 2cm tolerance used for most robot grippers.LineMOD uses ADD for non-symmetric objects and ADD-S for its two symmetric objects.
- Ablations: Dense fusion outperforms global fusion-by-concatenation, while iterative refinement further improves pose estimation over the per-pixel model.Among the variants, Ours (Iterative) achieves the best performance; refinement particularly helps texture-less symmetric objects such as bowl, banana, and extra large clamp.
- Robustness to occlusion: 2% overall accuracy loss for Ours (per-pixel) and Ours (iterative) under increasing occlusion contrasts with significant degradation for PointFusion and PoseCNN+ICP.The occlusion level is estimated from each object instance’s invisible surface percentage.
- Time efficiency: 200x faster inference than PoseCNN+ICP enables DenseFusion to reach 16 FPS, with segmentation, pose estimation, and refinement contributing similarly to runtime.PoseCNN+ICP spends most of its time on post-processing ICP.
- Robotic grasping: The robot succeeds on 73% of grasps using DenseFusion pose estimates, although banana-model mismatch hinders orientation estimation and causes some failures.The deployment uses five YCB objects and evaluates grasping on a table.
5. Conclusion
DenseFusion estimates 6D poses from RGB-D images by fusing color and depth densely, improving robustness to occlusion and supporting robotic grasping and manipulation.
- DenseFusion fuses color and depth into dense features weighted by prediction confidence for 6D pose estimation.
- The method outperforms previous approaches across several datasets and is significantly more robust against occlusions.
- A robot successfully uses the estimated poses to grasp and manipulate objects.
6. Supplementary Materials
The supplementary materials define an occlusion measurement, describe its computation, document the robot platform, and show refinement under blur and low light.
- Invisible surface percentage quantifies how occluded an object is from the camera viewpoint.It is used in Section 4.5.
- The occlusion measure compares sensor depth with model-projected depth after transforming and projecting sampled object-surface points.Pixels whose depth discrepancy exceeds a margin are treated as occluded.
- The robot experiment uses a Toyota HSR equipped with an Asus Xtion RGB-D sensor, holonomic base, and two-finger gripper.The model was trained on YCB-Video without finetuning and tolerated the camera difference.
- Iterative refinement improves pose accuracy in LineMOD scenes containing blur and low-light conditions.