Source-linked AI summary
Foundational feature fusion for conditional flow matching in 6D pose estimation
Amir Hamza, Davide Boscaini, Fabio Poiesi
TL;DR
Existing 6D pose methods depend on task-specific overlap supervision and simple multimodal fusion, limiting conditioning without retraining. FunFlow6D uses frozen geometric and appearance foundation features with gated cross-attention, and outperforms prior methods across four BOP datasets while simplifying supervision and training. Its pipeline still relies on external pose estimation, multiple denoising steps, and segmentation masks.
Problem
Existing methods require task-specific overlap-aware encoders, whose conditioning can be unreliable and must be retrained for each new object.
Method
FunFlow6D conditions conditional flow matching on frozen dGeDi and DINOv2 features, reduces semantic features with UMAP, and fuses modalities using gated cross-attention.
Results
FunFlow6D outperforms state-of-the-art competitors on four BOP datasets while halving trained parameters and removing per-instance pre-training.
Takeaways & Limitations
The method provides stronger robustness to symmetries and occlusions with a simplified training recipe and weaker supervision signals.
Takeaways & Limitations
The pipeline relies on external RANSAC-based pose estimation, multiple denoising steps, and segmentation masks that are not jointly optimized.
Abstract
from arXiv · showhide
Conditional flow matching has enabled a step forward in object 6D pose estimation, achieving state-of-the-art performance by progressively denoising and registering object representations to observed scenes. Existing methods require training task-specific encoders supervised on object-scene overlap and rely on trivial feature fusion strategies to resolve pose ambiguities. We present FunFlow6D, a novel flow matching-based formulation that leverages features from geometric and appearance foundation models for pose estimation, eliminating the need for task-specific encoder training. We also introduce a cross attention-based fusion mechanism that dynamically combines geometric and appearance features to provide richer conditioning for the flow matching module. Experiments on four datasets from the BOP benchmark show that FunFlow6D outperforms the previous state of the art while reducing supervision requirements and memory overhead. Extensive ablations validate the contribution of each proposed component. Project website: https://tev-fbk.github.io/FunFlow6D/.
1 Introduction
FunFlow6D addresses unreliable, per-instance conditioning in 6D pose estimation by combining frozen geometric and appearance features with adaptive fusion. It reports stronger accuracy with less supervision and no per-instance encoder training.
- Motivation: 6D pose estimation supports spatial applications but must remain accurate under clutter and occlusion.The setting uses RGBD observations together with a prior CAD model or reconstructed mesh.
- Limitations of prior methods: Existing overlap-aware encoders provide unreliable conditioning and require retraining for each new object.On LM-O, overlap-derived correspondences average only 3.5 inliers.
- Proposed approach: FunFlow6D conditions flow matching on frozen zero-shot geometric features that capture local 3D structure without per-instance fine-tuning.The method is designed to meet geometric distinctiveness, local structure preservation, and adaptive fusion requirements.
- Results: FunFlow6D outperforms state-of-the-art methods on four BOP datasets while halving trained parameters and removing per-instance pre-training.The reported evaluation covers diverse object categories, occlusion levels, textures, and clutter conditions.
- Proposed approach: UMAP reduces appearance features while preserving local neighborhoods and retaining task-relevant information that PCA may discard.This dimensionality-reduction choice is presented as a core contribution for semantic conditioning.
- Proposed approach: A gated-attention fusion module adapts flow conditioning to local object structure by enriching interaction between semantic and geometric cues.The module replaces fixed fusion with learned per-point modality weighting.
2 Related work
Prior 6D pose methods use correspondence, generative, and multimodal-fusion strategies, but key approaches rely on object-specific overlap encoders or fixed feature fusion. FunFlow6D instead uses foundational geometric and appearance features within conditional flow matching.
- Pose-estimation approaches: Instance-level 6D pose estimation methods use explicit supervision for known objects, with indirect methods matching correspondences and direct methods regressing transformations.Flose is identified as a nearby generative approach based on conditional flow matching.
- Generative pose estimation: Flose transports Gaussian noise onto an object CAD surface through conditional flow matching, improving robustness to clutter and occlusion without explicit correspondence estimation.Its conditioning still depends on an overlap-aware encoder retrained for every new object.
- FunFlow6D: FunFlow6D uses frozen dGeDi and DINOv2 features, reduces semantic features with UMAP, and fuses modalities with gated cross-attention.The overview places these operations within feature extraction before conditional generative denoising and pose estimation.
- Per-point descriptors: Foundation-model descriptors provide per-point signals intended to discriminate local surface neighborhoods and generalize to unseen objects without retraining.The related work distinguishes appearance and geometric encoders as complementary sources of these descriptors.
- Feature fusion: Existing feature fusion methods combine semantic and geometric modalities through closed-form operations or training-based schemes.Examples include channel concatenation in FreeZe and point-wise summation in Flose.
3 Method
FunFlow6D estimates 6D pose by extracting frozen geometric and appearance descriptors, adaptively fusing them, and conditioning a flow that canonicalizes observed points before rigid alignment.
- Problem formulation: The method estimates a rigid transformation from an RGBD scene observation and a known object point cloud, using segmentation and depth back-projection to form the target cloud.The target is partial, occluded, and noisy; supervision also uses its ground-truth canonicalized form.
- Feature extraction: A frozen multi-scale geometric encoder supplies local descriptors at 30% and 40% of the object diameter, combining fine surface cues with broader context.The encoder is applied to both query and target point clouds and is intended to improve robustness to partiality and local noise.
- Feature extraction: Frozen vision features are lifted onto query and target points, with query features precomputed from multi-view renderings and target features obtained from the RGBD crop.Query renderings are processed offline, while target pixel features are transferred through depth-induced pixel-to-point correspondences.
- Feature extraction: UMAP reduces appearance features to the geometric descriptor dimension while preserving local manifold structure and fitting the embedding only on query features.The same embedding is then applied to target instances so both clouds occupy a shared low-dimensional space.
- Feature fusion: A bidirectional gated cross-attention block exchanges geometric and semantic information, learning per-point modality weights before producing fused descriptors for both clouds.The two branches swap query and key-value modalities; gated outputs are projected, normalized, concatenated, and mapped back to D dimensions.
- Conditional flow matching: Conditional flow matching transports Gaussian noise toward the concatenated query and canonicalized target, while the learned velocity is conditioned on fused descriptors and positional encodings.At inference, only the target portion evolves; its canonicalized estimate induces point correspondences used for Procrustes alignment and subsequent refinement.
- Pose recovery: The induced correspondences yield a camera-to-canonical rigid transform through orthogonal Procrustes, with sampled hypotheses rejecting outliers before ICP refinement.The robust estimation procedure targets errors from imperfect flow predictions, including those associated with heavy occlusion or symmetry ambiguity.
4 Results
FunFlow6D improves correspondence discriminativity and 6D pose accuracy by combining frozen geometric and appearance features with UMAP reduction and learned fusion. Across BOP datasets, it outperforms competing methods while using fewer trainable models and parameters.
- Feature discriminativity analysis: 6.96 inliers: dGeDi approximately doubles correspondence quality over overlap-aware features without task-specific fine-tuning.Other foundation models do not match this result; Utonia reaches only 2.5 inliers.
- Feature discriminativity analysis: 3.21 average inliers: DINOv2 surpasses TIPSv2 at 2.40, DINOv3 at 1.20, and EUPE at 0.70 among appearance encoders.The passage attributes the gap to DINOv2’s dense, spatially localized self-distilled features.
- Feature discriminativity analysis: 25.61 inliers: UMAP far outperforms PCA and other dimensionality-reduction techniques at the same target dimensionality.PCA reaches 3.21, while No-PCA reaches 0.99 without normalization and 1.44 with normalization.
- 6D pose estimation accuracy: FunFlow6D uses approximately 9× fewer parameters than HccePose(BF) and less than half the parameters of Flose.HccePose(BF) trains one network per object, totaling 34 models, while other methods train one network per dataset.
- 6D pose estimation accuracy: FunFlow6D consistently outperforms competing methods on LM-O, TUD-L, IC-BIN, and YCB-V in Average Recall.The largest improvement occurs on LM-O, which contains severe occlusions; gains use frozen encoders without per-instance training or fine-tuning.
- Inference time breakdown: 381.11 ms, or 37.52% of per-image time, is spent on 50-step Euler integration in FunFlow6D.The full reported per-image runtime is 1015.90 ms on the stated hardware, excluding object instance segmentation.
5 Conclusion
FunFlow6D fuses appearance and geometric foundational features in conditional flow matching for known-object 6D pose estimation from RGBD scenes. It outperforms state-of-the-art competitors on four BOP datasets, but retains external pose estimation, multi-step inference, and segmentation dependencies.
- FunFlow6D conditions flow matching on fused appearance and geometric foundational features for 6D pose estimation from RGBD observations.
- RANSAC remains an external module because flow matching does not guarantee a valid rigid transformation.Enforcing rigidity during denoising is identified as a future direction for a fully end-to-end and more efficient pipeline.
- Multiple denoising steps increase inference latency, while segmentation quality constrains performance through a separately optimized pipeline stage.The authors propose single-step alternatives and scene-level prediction as future directions.