Source-linked AI summary
FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation
Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, Jian Sun
TL;DR
Single-image RGB-D pose estimation still lacks a fully effective way to combine appearance and geometry. FFB6D performs bidirectional fusion throughout both networks and selects texture- and geometry-informed keypoints; it reports state-of-the-art results across several benchmarks.
Problem
It remains unknown how to fully leverage complementary RGB appearance and depth geometry information for 6D pose estimation.
Method
FFB6D fuses RGB and geometry features bidirectionally at every encoding and decoding layer and uses SIFT-FPS to select 3D keypoints from texture and geometry.
Results
The approach achieves state-of-the-art 6D pose estimation performance on the YCB-Video, LineMOD, and Occlusion LineMOD datasets.
Takeaways & Limitations
The proposed fusion network and keypoint-selection algorithm provide a pose-estimation approach that leverages rich appearance and geometry representations from a single RGBD image.
Takeaways & Limitations
Separately extracted RGB and point-cloud features can fail on similar-appearance or reflective-surface objects, motivating communication between the branches.
Abstract
from arXiv · showhide
In this work, we present FFB6D, a Full Flow Bidirectional fusion network designed for 6D pose estimation from a single RGBD image. Our key insight is that appearance information in the RGB image and geometry information from the depth image are two complementary data sources, and it still remains unknown how to fully leverage them. Towards this end, we propose FFB6D, which learns to combine appearance and geometry information for representation learning as well as output representation selection. Specifically, at the representation learning stage, we build bidirectional fusion modules in the full flow of the two networks, where fusion is applied to each encoding and decoding layer. In this way, the two networks can leverage local and global complementary information from the other one to obtain better representations. Moreover, at the output representation stage, we designed a simple but effective 3D keypoints selection algorithm considering the texture and geometry information of objects, which simplifies keypoint localization for precise pose estimation. Experimental results show that our method outperforms the state-of-the-art by large margins on several benchmarks. Code and video are available at \url{https://github.com/ethnhe/FFB6D.git}.
1. Introduction
6D pose estimation benefits from combining RGB appearance with depth geometry, but existing approaches do not fully exploit their complementarity. FFB6D addresses this with full-flow bidirectional fusion and texture-geometry-aware keypoint selection.
- 6D pose estimation supports augmented reality, autonomous driving, and robotic grasping but remains difficult under sensor noise, lighting variation, and occlusion.
- RGB-only methods lose geometry through perspective projection, limiting performance on poorly lit, low-contrast, and textureless objects.
- Existing RGB-depth methods either use time-consuming, non-end-to-end refinement or separately extract RGB and point-cloud features, causing degeneration in similar-appearance and reflective-surface cases.
- FFB6D fuses RGB and geometry features bidirectionally at every encoding and decoding layer, allowing the two branches to exchange complementary information throughout representation learning.
- SIFT-FPS selects 3D keypoints using object texture and geometry, avoiding non-salient smooth-surface points that are difficult to localize.
- The method reports state-of-the-art performance on the YCB-Video, LineMOD, and Occlusion LineMOD datasets without requiring time-consuming post-refinement.
2. Related Work
Prior RGB-D pose-estimation methods commonly separate appearance and geometry processing or rely on point-cloud-only reasoning. FFB6D instead uses bidirectional fusion modules as communication bridges between the two feature-extraction networks.
- Holistic RGB methods directly predict pose parameters, while dense-correspondence methods recover poses through pixel-to-mesh matching and PnP.
- Point-cloud-only approaches use 3D ConvNets or point-cloud networks, but sparsity, non-texture, and reflective surfaces limit their performance.
- Traditional and recent data-driven RGB-D methods may use templates, surrogate objectives, or RGB initialization followed by ICP or MCN refinement, which is time-consuming and not end-to-end optimizable.
- Separately extracting RGB and point-cloud features prevents the two networks from sharing information and limits learned-representation capacity.
- FFB6D adds bidirectional fusion modules throughout the network flow so each branch receives supplementary information from the other branch.
3. Proposed Method
FFB6D combines RGB appearance and point-cloud geometry through bidirectional fusion across the feature-extraction flow, then uses dense RGBD features and selected 3D keypoints for pose recovery.
- Full Flow Bidirectional Fusion Network: Given an RGBD image, the framework extracts appearance features with a CNN and geometric features with a point cloud network, then fuses them bidirectionally.Point-to-pixel and pixel-to-point modules communicate between the two branches during feature extraction.
- Pixel-to-point fusion: Pixel-to-point fusion gathers neighboring RGB features for each point and combines the integrated appearance feature with the point feature through concatenation and a shared MLP.Global RGB feature squeezing is avoided because background pixels and multiple objects can discard detailed information.
- Coordinate alignment: Nearest interpolation resizes the XYZ map to match deeper RGB feature maps, avoiding boundary noise introduced by mean-based coordinate pooling.The XYZ map is maintained because convolution reduces feature-map height and width during encoding.
- Point-to-pixel fusion: Point-to-pixel fusion gathers nearest point features for each image feature using its XYZ coordinate, then concatenates the integrated geometry feature with the color feature through a shared MLP.This transfers geometric information from the point cloud network to the CNN rather than using only a global point feature.
- Pose estimation: Dense appearance and geometry embeddings are matched by projecting points onto the image plane, concatenated into RGBD features, and supplied to segmentation and 3D keypoint detection.The detected per-object keypoints are used with least-squares fitting to recover pose parameters R and T.
- Keypoint selection: The keypoint-selection procedure improves on surface-based Farthest Point Sampling by selecting 3D keypoints using the object's texture and geometry information.The selected keypoints are detected per object before least-squares pose recovery.
4. Experiments
Experiments evaluate FFB6D on three pose-estimation benchmarks using ADD-family metrics, ablations, and efficiency analyses. The method improves benchmark performance, robustness to occlusion, and runtime relative to cited baselines.
- Implementation: The evaluation uses ResNet34-PSPNet for RGB features, RandLA-Net for 12,288 sampled depth points, and bidirectional fusion across encoding and decoding layers.Training jointly optimizes semantic segmentation, center-point voting, and 3D-keypoint voting with a weighted multi-task loss.
- Benchmark setup and metrics: FFB6D is evaluated on YCB-Video, LineMOD, and Occlusion LineMOD using dataset-specific ADD-family metrics.YCB-Video reports ADD-S and ADD(S) AUC; LineMOD and Occlusion LineMOD report ADD-0.1d accuracy.
- Benchmark results: 1.1% on ADD-S and 1.0% on ADD(S) AUC: FFB6D advances single-view state-of-the-art results on YCB-Video without iterative refinement.With iterative refinement, it also achieves the best performance and exceeds methods requiring time-consuming post-refinement even without refinement.
- Robustness: FFB6D does not suffer a performance drop as occlusion increases, unlike previous methods evaluated by ADD-S under 2 cm accuracy.The authors attribute this robustness to using texture and geometry information for locating 3D keypoints in highly occluded scenes.
- Benchmark results: 4.7%: FFB6D advances state-of-the-art on Occlusion LineMOD without iterative refinement, while also outperforming the state-of-the-art on LineMOD.Occlusion LineMOD evaluates heavily occluded multi-object scenes.
- Ablation and efficiency: Full-flow fusion yields the highest ablation performance, bidirectional fusion outperforms one-way fusion, and SIFT-FPS improves keypoint localization over distance-only FPS.Compared with PVN3D, FFB6D uses fewer parameters and runs 2.5 times faster; adding DenseFusion after full-flow fusion provides no gain.
5. Conclusion
FFB6D combines bidirectional RGB–geometry fusion throughout representation learning with SIFT-FPS keypoint selection for 6D pose estimation. It outperforms previous approaches on several benchmark datasets and may generalize to other RGBD tasks.
- FFB6D performs full-flow bidirectional fusion to extract rich appearance and geometry information from a single RGBD image.
- SIFT-FPS selects 3D keypoints using object texture and geometry information to simplify keypoint localization.
- FFB6D outperforms previous approaches by remarkable margins on several benchmark datasets.
- The proposed fusion network may generalize to RGBD applications including 3D object detection, 3D instance semantic segmentation, and salient object detection.
A.1. Details of the Network Architecture
The network uses separate RGB and point-cloud branches, exchanging information bidirectionally at every encoding and decoding stage before pose prediction and fitting.
- FFB6D uses ResNet34 and PSPNet for RGB appearance processing and RandLA-Net for point-cloud geometry representation learning.
- Point-to-pixel and pixel-to-point fusion modules communicate information between the two branches at each encoding and decoding layer.
- The network predicts semantic labels, center-point offsets, and 3D keypoint offsets from concatenated appearance and geometry features.
- MeanShift clustering separates object instances and supports keypoint voting before least-squares fitting recovers pose parameters.
A.2. Implementation: Different Representation Learning Frameworks
The implementation compares multiple RGBD representation-learning frameworks and evaluates them on LineMOD and Occlusion-LineMOD using ADD-0.1d, while describing CNN–point-cloud feature construction.
- CNN-R⊕D lifts depth pixels into XYZ and normal maps, concatenates them with RGB, and processes each point through ResNet34-PSPNet.
- The architecture uses ResNet34 and PSPNet for RGB encoding and decoding, with RandLA-Net for point-cloud feature extraction.
- LineMOD and Occlusion-LineMOD evaluations report ADD-0.1d, with symmetric objects highlighted in the quantitative tables.
- During decoding, voxel size increases while feature dimensions decrease, and geometry features are obtained by trilinear interpolation before concatenation with CNN appearance features.
A.3.1 Quantitative result on the LineMOD dataset.
Additional quantitative 6D pose-estimation results on the LineMOD dataset are presented in Table 10 using the ADD-0.1d metric.
- Additional 6D pose-estimation results on LineMOD are reported in Table 10.
- Table 10 contains quantitative evaluation results for the LineMOD dataset.
- The LineMOD results are presented as further results of the paper’s 6D pose-estimation evaluation.
A.3.2 Quantitative result on the Occlusion-LineMOD dataset.
The paper reports additional Occlusion-LineMOD results and qualitative YCB-Video pose results, including comparisons with prior work under occlusion and similar appearance conditions.
- The model is trained on LineMOD and evaluated on Occlusion-LineMOD only for testing, following the state of the art.
- Qualitative 6D pose results are provided on the YCB-Video dataset.
- Compared with PVN3D using the DenseFusion architecture, FFB6D is more robust to occlusion and objects with similar appearance.