Source-linked AI summary
OCA: ODE-Driven Cross-Attention for Image-to-Point-Cloud Registration
Pei An, Jiaqi Yang, Yulong Wang, Siwen Quan, Liangliang Nan
TL;DR
Attention ambiguity from the cross-modal gap complicates reliable 2D–3D correspondence learning in I2P registration. OCA models ideal feature interaction with ODEs and refines features and attention through propagation. The authors report registration-recall gains of up to 5%, 9%, and 15% in standard, fine-tuning, and zero-shot settings, respectively.
Problem
The modality gap can produce spurious 2D–3D correspondences with high similarity scores, impeding reliable cross-modal correspondence learning.
Source: §1 · p. 2Method
OCA uses assignment ODEs to approximate ideal feature interaction and refine I2P feature representations by propagating features and the attention matrix.
Source: §1 · p. 3Results
Up to 5%, 9%, and 15% registration-recall improvements are reported under standard, fine-tuning, and zero-shot settings, respectively.
Source: §1 · p. 3Takeaways & Limitations
Experiments on four public datasets and five representative baselines demonstrate OCA's effectiveness and generalizability within I2P registration frameworks.
Source: §1 · p. 3, §6 · p. 15In textureless scenes, larger errors in the initial sparse attention make ODE propagation unstable and inaccurate.
Source: §5.2 · p. 13
Abstract
from arXiv · showhide
Cross-attention is a crucial component in learning-based image-to-point-cloud (I2P) registration. Although existing cross-attention mechanisms have achieved promising progress, attention ambiguity remains a fundamental challenge that hinders the learning of discriminative 2D-3D correspondences. To address this problem, we revisit cross-attention and establish ordinary differential equations (ODEs) to model the ideal I2P feature interaction. Based on this formulation, we develop an ODE-driven cross-attention (OCA) module that refines feature representations and attention matrices through ODEs. In practice, OCA can be seamlessly integrated into existing I2P registration frameworks. To validate its effectiveness, we incorporate OCA into five state-of-the-art baselines and evaluate on four public benchmark datasets. Experimental results demonstrate that OCA improves registration recall by up to 5\%, 9\%, and 15\% under the standard, fine-tuning, and zero-shot settings, respectively.
1 Introduction
The introduction frames attention ambiguity from the cross-modal gap as a barrier to reliable 2D–3D correspondence learning and presents OCA as an ODE-based refinement of features and attention. It situates OCA among prior cross-attention approaches and reports gains across evaluation settings.
1 Introduction
- Learning-based I2P registration estimates camera pose from image–point-cloud correspondences, with feature interaction bridging modalities to produce discriminative, modality-invariant features.The task supports applications including visual localization, state estimation, colorization, and SLAM.Source: §1 · p. 1
- The modality gap can produce spurious 2D–3D matches with high similarity, impeding reliable correspondence learning despite prior efforts to mitigate ambiguity.Prior approaches include feature-manifold alignment, uncertainty correction, covariance alignment, and keypoint-based learning.Source: §1 · p. 2
- Up to 5%, 9%, and 15% registration-recall improvements are reported under standard, fine-tuning, and zero-shot settings, respectively.The evaluation spans four public datasets and five state-of-the-art baselines.Source: §1 · p. 3
- OCA models ideal feature interaction with assignment ODEs, then refines I2P features and attention by propagating them through ODEs.The authors describe it as a lightweight, non-parametric module that complements existing cross-attention in I2P frameworks.Source: §1 · p. 3
2 Related works
- I2P cross-attention evolved from keypoint matching to patch-based interaction, a practical compromise with lower computational burden than attending over all pixels and points.Patch-based interaction is widely used in state-of-the-art frameworks.Source: §2 · pp. 3–4
- Recent methods address ambiguity with prior-knowledge-based approaches or prior-knowledge-free networks; OCA strengthens the latter and outperforms representative prior-free methods.The prior-knowledge-based category is sensitive to the quality and generalization of pretrained visual foundation models, whereas prior-free methods avoid external pretrained models.Source: §2 · p. 4
3 Rethinking cross-attention from ODEs
The section models ideal image-to-point feature interaction with assignment ODEs, relates their discrete updates to transformer cross-attention, and derives convergence conditions that motivate strategies for reducing attention ambiguity.
3.1 Assignment ODEs
- Ground-truth assignments make corresponding pixel and point feature derivatives equal, yielding modality-invariant features, but are unavailable before registration.The assignment ODEs replace this ideal interaction with time-varying features and an estimated assignment matrix.Source: §3.1 · pp. 4–5
- Assignment ODEs use row-normalized feature correlations as an approximation of the unknown ground-truth assignment matrix.The model treats features and assignments as time-varying variables to mimic the ideal interaction.Source: §3.1 · p. 5
3.2 Relation of cross-attention and assignment ODEs
- Euler discretization makes the assignment ODE updates approximately equivalent to standard transformer cross-attention when softmax scores are scaled by 1/√c.This connection allows ODE analysis to address ambiguity inherent in cross-attention.Source: §3.2 · p. 5
3.3 Convergence analysis of assignment ODEs
- Convergence can occur when ρ′(A(t))=0, including when A(t) is a permutation matrix, or when the two feature Gram matrices share an eigenvalue.The second condition does not contribute to improving I2P registration performance in practice.Source: §3.3 · p. 6
- The analysis motivates sparse attention and an initial attention matrix close to Agt, strategies standard cross-attention struggles to ensure.Softmax does not guarantee sparsity, while L2-normalized features make a near-ground-truth initial attention difficult to ensure.Source: §3.3 · p. 6
4 Refining cross-attention with ODEs
The section turns the convergence analysis into OCA, combining attention initialization and ODE propagation to refine features and attention, then embedding the module in I2P registration.
4.1 Feasible solutions from ODE perspective
- Top-K pruning sparsifies each attention row but can discard correct matches and cause early gradient vanishing; temperature-scaled softmax offers a safer approximation.The temperature γ≥1 amplifies row-wise differences to approximate sparse attention.Source: §4.1 · pp. 6–7
- OCA initializes attention from L2-normalized features, matching the feature-distance basis of correspondences, then integrates assignment dynamics to suppress incorrect-match similarity.The paper states that normalization makes this attention computation more accurate than the original assignment-ODE formulation.Source: §4.1 · p. 7
4.2 ODE-driven based cross-attention
- OCA unifies the feasible solutions in two stages: initialize attention from normalized features and prune it, then iteratively propagate attention and features with discretized ODE updates.The updates use normalized feature matrices and sparse attention.Source: §4.2 · p. 8
- OCA outputs weighted averages of the initial and propagated features and is lightweight and non-parametric.The weighting factor is ω.Source: §4.2 · p. 8
4.3 OCA-embedded I2P registration
- The embedded framework places OCA after feature interaction and uses two-stage training: train from scratch with τ=0, then fine-tune with the original τ.The default I2P registration loss is used in both stages.Source: §4.3 · p. 8
5 Experiments and Discussions
Experiments evaluate OCA with five baselines across four public datasets under standard, fine-tuning, and zero-shot settings. Results support its effectiveness, generalizability, and low propagation cost, with a failure case in extremely texture-less scenes.
5.2 Comparisons
- OCA significantly improves inlier ratio and registration recall on 7-Scenes, with recall gains above 3% for Matr, Flow-I2P, and CA-I2P.The authors attribute the gains to OCA rather than extra training epochs.Source: §5.2 · p. 10, Table 1 · p. 10
- Fine-tuning results show significant gains on TUM and state-of-the-art performance on ScanNet, while gains are smaller on less-textured, sparser RGBD-v2.All compared methods exceed 65% recall on ScanNet.Source: §5.2 · p. 10, Table 2 · p. 10, Table 3 · p. 11
- In zero-shot ScanNet tests, Zero-OCA gains 15% recall with Flow-I2P and OCA gains 12% with Bridge, although both reduce inlier ratios.The authors relate the lower inlier ratios to propagation retaining high-confidence correspondences, which lead to precise pose estimation and improved recall.Source: §5.2 · pp. 11–12, Table 5 · p. 11
- Compared with lightweight DDPM and flow matching models and Diff-Reg, OCA is reported as more effective on TUM.The comparison is reported in Table 6.Source: Table 6 · p. 11, §5.2 · p. 13
- OCA’s failure case is extremely texture-less scenes, where errors in the initial sparse attention matrix make ODE propagation unstable and inaccurate.The authors propose future feedback to assess initial attention accuracy and a more stable propagation strategy.Source: §5.2 · pp. 13–14, Figure 6 · p. 13
5.3 Ablation studies
- On 7-Scenes, increasing propagation iterations from 1 to 4 lowers inlier ratio from 57.1% to 43.3% while raising recall from 48.9% to 57.5%.At T ≥5 both metrics drop; the authors set T to 3, choose τ=0.10 because large time steps destabilize assignment ODEs, and report 6 ms propagation cost.Source: §5.3 · p. 14, Table 7 · p. 14, Table 8 · p. 14
6 Conclusions
The conclusion presents assignment ODEs as a model of ideal cross-modal feature interaction and derives OCA by analyzing their stability. It reports effectiveness across four public datasets and five baselines, while noting failure under heavy noise when initialization is not sufficiently informative.
6 Conclusions
- OCA enhances correspondence learning through iterative refinement in existing I2P registration frameworks, with effectiveness and generalizability demonstrated across four public datasets and five representative baselines.The conclusion frames the approach as revisiting cross-attention from an ODE perspective and connecting assignment ODEs to conventional cross-attention.Source: §6 · pp. 14–15
- The assignment ODEs approximate ideal cross-modal feature interactions, and their stability analysis motivates the OCA module.Source: §6 · p. 15
- OCA fails in heavily noised cases because ODE propagation relies on a reasonably informative initialization.The authors propose extending the theoretical framework and developing a more robust ODE-based feature interaction mechanism.Source: §6 · p. 15
A. Derivation of Eq. 5 and discussion of convergence condition (c2)
The appendix derives the attention-matrix dynamics by differentiating its factorization and discusses why convergence condition (C2) contributes little in practical learning. The discussion attributes this limited contribution to a demanding eigenvalue condition and weak correlation with the ground-truth assignment.
A. Derivation of Eq. 5 and discussion of convergence condition (c2)
- For A(t) = x(t)y(t)^T, differentiating the attention matrix applies the product rule to the time-varying feature factors.The derivative is the sum of the derivative of x(t) multiplied by y(t)^T and x(t) multiplied by the derivative of y(t)^T.Source: A. Derivation of Eq. 5 and discussion… · p. 15
- Substituting the feature derivatives yields dA(t)/dt = ρ(A(t))Y(t) + X(t)ρ(A(t)).The derivation identifies this expression as Eq. 14, from which the paper's Eq. 5 is directly derived.Source: A. Derivation of Eq. 5 and discussion… · p. 15
- Condition (C2) contributes little because the Sylvester equation requires X(t) and Y(t) to share an eigenvalue, and its solution is weakly correlated with A_gt.The passage notes that deep-network learning makes the shared-eigenvalue condition difficult to ensure.Source: A. Derivation of Eq. 5 and discussion… · p. 15
B. Explanation of existing solutions from an ODE perspective
The appendix interprets existing cross-attention solutions through ODE strategies: priors can encourage sparse attention or align initial attention with the ground truth. Because predicted priors can be erroneous, the paper motivates a prior-knowledge-free module and reports qualitative improvements with OCA.
B. Explanation of existing solutions from an ODE perspective
- Geometric priors such as depth and surface normals can reduce the cross-modal gap and support strategy (S2), which seeks an initial attention matrix close to the ground truth.Depth features rely on approximate 3D similarity between reconstructed scenes and point clouds; surface normals approximately follow a 3D rotation transformation.Source: B. Explanation of existing solutions… · p. 16
- Semantic priors restrict valid correspondences to regions with identical labels, while topological priors restrict them to overlapping regions; both encourage sparse attention for strategy (S1).The passage says topological priors have stronger generalization capability than semantic priors.Source: B. Explanation of existing solutions… · p. 16
- OCA significantly increases inlier counts in qualitative comparisons across 7-Scenes, ScanNet, and RGBD-v2 with five representative baselines.The baselines are Matr, Flow-I2P, Bridge, LDF-I2P, and CA-I2P.Source: Figure 9 · p. 17
- Because geometric, semantic, and topological priors can contain prediction errors, the paper proposes a prior-knowledge-free module to enhance I2P cross-attention.The listed errors include inaccurate depth, erroneous normals, incorrect labels, and improperly segmented regions.Source: B. Explanation of existing solutions… · p. 17
- With OCA, virtual objects in the AR visualization are rendered with high spatiotemporal consistency with the real world across continuous timestamps.Source: Figure 10 · p. 18
C. Visualization of input data
The passage identifies small image–point-cloud overlap, noisy point clouds, and low-texture indoor scenes as real-world I2P registration challenges.
C. Visualization of input data
- Real-world I2P registration faces small image–point-cloud overlap, range-sensor noise in 3D point clouds, and low texture in some indoor scenes.The experiments validate OCA under these challenging conditions.Source: C. Visualization of input data · p. 18
D. More qualitative comparisons
Qualitative evaluations across challenging scenes show OCA suppressing outliers and increasing inliers, with robust performance across scenarios and an augmented-reality application.
D. More qualitative comparisons
- OCA suppresses outliers and increases inliers through attention-matrix propagation across challenging scenes.The qualitative comparison uses five representative baselines on 7-Scenes, ScanNet, and RGBD-v2.Source: D. More qualitative comparisons · p. 18
- The reported qualitative behavior is robust across diverse complex scenarios and five representative baseline methods.The baselines are Matr, Flow-I2P, Bridge, LDF-I2P, and CA-I2P.Source: D. More qualitative comparisons · p. 18
- In an augmented-reality application, OCA improves registration recall and camera-pose accuracy, enabling high-quality virtual-object rendering.Source: D. More qualitative comparisons · p. 18
E. In-depth discussion of other propagating methods
Alternative attention-propagation approaches DDPM and flow matching degrade I2P registration performance, while a modified Diff-Reg also performs below Matr+OCA.
E. In-depth discussion of other propagating methods
- DDPM and flow matching both degrade I2P registration performance when used to propagate the attention matrix.The paper attributes this to their lack of an effective mechanism for enforcing attention-matrix sparsity and accuracy.Source: E. In-depth discussion of other… · pp. 18–19
- Without metric depth, modified Diff-Reg denoises the attention matrix with a U-Net but lacks a mechanism to enforce its sparsity and accuracy.The original Diff-Reg uses accurate metric depth as a geometric prior to lift correspondences and identify outliers with ICP.Source: E. In-depth discussion of other… · p. 19
- The modified Diff-Reg performs worse than Matr+OCA under the comparison described.Source: E. In-depth discussion of other… · p. 19
F. Extended comparisons
Extended comparisons report that OCA brings a classical baseline close to a prior-knowledge-based method on TUM and improves both metrics for that method, while also improving 3DMatch results.
F. Extended comparisons
- On TUM, Matr+OCA achieves Registration Recall close to prior-knowledge-based Top-I2P, and OCA further improves Top-I2P's Inlier Ratio and Registration Recall.Top-I2P uses 2D and 3D Segment Anything Models to partition the inputs and predict topological relationships for feature matching.Source: F. Extended comparisons · p. 19
- On 3DMatch, integrating OCA improves both GeoTransformer's Inlier Ratio and Registration Recall.Source: F. Extended comparisons · p. 20
- The 3DMatch result suggests OCA can enhance intra-modal geometric correspondence estimation, beyond cross-modal matching.Source: F. Extended comparisons · p. 20
G. More analysis of parameters γ and ω
The parameter discussion explains how γ controls attention sparsity and how ω governs the fusion of initial and propagated features, with excessive propagation-related weighting degrading performance.
G. More analysis of parameters γ and ω
- Setting γ=2 balances Inlier Ratio and Registration Recall; increasing γ through 4 improves recall but lowers inlier ratio, while γ>4 lowers both metrics.γ controls the sparsity of ρsparse(A[k]); excessive sparsity makes it sensitive to the accuracy of the initial matching matrix.Source: G. More analysis of parameters γ and ω · p. 20
- Performance degrades for ω≥0.20, so the fusion of initial and propagated 2D and 3D features requires care.The fusion is (1−ω)X[0]+ωX[T] and (1−ω)Y[0]+ωY[T].Source: G. More analysis of parameters γ and ω · p. 20
- The paper links ODE propagation to impaired loss backpropagation and notes that coupled feature propagation can induce gradient instability.This analysis motivates choosing ω suitably in practical applications.Source: G. More analysis of parameters γ and ω · p. 20