Source-linked AI summary
Learning Two-View Correspondences and Geometry Using Order-Aware Network
Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, Hongen Liao
TL;DR
Two-view correspondence estimation must reject outliers while modeling both local and global spatial context. The Order-Aware Network hierarchically clusters unordered correspondences, correlates the clusters, and upsamples them to infer inlier probabilities and regress the essential matrix. Experiments on indoor and outdoor datasets report significantly improved relative pose estimation and correspondence accuracy over state-of-the-art methods.
Problem
Existing PointCN-like approaches inadequately capture local neighborhoods and complex global relations among unordered sparse correspondences, although both contexts matter for outlier rejection.
Method
Order-Aware Network uses learnable DiffPool clustering, Order-Aware Filtering for cluster correlation, and DiffUnpool recovery to predict inliers and regress the essential matrix.
Results
The method significantly improves relative pose estimation accuracy on both outdoor and indoor datasets compared with state-of-the-art methods.
Takeaways & Limitations
Hierarchical order-aware processing provides a learned way to exploit local and global context in sparse two-view correspondences.
Takeaways & Limitations
DiffPool's permutation invariance means pooled features can correspond to multiple input arrangements, constraining direct recovery without spatial information.
Abstract
from arXiv · showhide
Establishing correspondences between two images requires both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential matrix. Specifically, this proposed network is built hierarchically and comprises three novel operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These clusters are in a canonical order and invariant to input permutations. Next, the clusters are spatially correlated to form the global context of correspondences. After that, the context-encoded clusters are recovered back to the original size through a proposed upsampling operator. We intensively experiment on both outdoor and indoor datasets. The accuracy of the two-view geometry and correspondences are significantly improved over the state-of-the-arts. Code will be available at https://github.com/zjhthu/OANet.git.
1. Introduction
Two-view geometry estimation depends on rejecting outliers from putative correspondences, but prior PointCN-style methods inadequately model local neighborhoods and complex global relations. Order-Aware Network addresses these issues with permutation-invariant clustering, order-aware processing, and hierarchical feature recovery, improving relative pose estimation on indoor and outdoor datasets.
- Two-view geometry supports Structure from Motion and visual Simultaneous Localization and Mapping, requiring accurate outlier rejection before relative pose recovery.
- PointCN-like methods process each correspondence independently for local features and use mean-variance normalization for global context, overlooking richer point relations.
- Sparse matches lack well-defined neighbors and stable relations, making local-context encoding and correspondence-relation modeling challenging.
- Order-Aware Filtering correlates pooled clusters through spatially shared perceptrons to model global correspondence context.
- DiffPool learns a soft assignment from unordered correspondences to canonically ordered clusters, while DiffUnpool recovers correspondence-level representations using input spatial information.
- The proposed network significantly improves relative pose estimation accuracy over state-of-the-art methods on both outdoor and indoor datasets.
2. Related Work
Related work applies deep learning to feature matching and outlier rejection, while this paper focuses on learning-based rejection for relative pose estimation. Existing approaches include CNN feature extraction, differentiable RANSAC-like selection, PointNet-style correspondence classification, and graph-based or bilateral constraints.
- Deep learning has been applied to dense and sparse geometric matching, with sparse methods often replacing handcrafted features such as SIFT using CNNs.
- Putative correspondences contain outliers, especially in wide-baseline cases, motivating RANSAC, USAC, bilateral filtering, statistical smoothness, and graphical-model approaches.
- DSAC differentiates probabilistic RANSAC selection, while PointCN formulates rejection as inlier/outlier classification with essential-matrix regression and Context Normalization.
- Sparse correspondences have no well-defined neighbors, so geometric deep-learning approaches must avoid relying solely on heuristic neighborhood definitions.
3. Order-Aware Network
Order-Aware Network addresses unordered sparse correspondences by hierarchically modeling local and global context, then restoring correspondence-level representations for inlier classification and essential-matrix regression.
- 3.1. Problem Formulation: The network predicts correspondence inlier probabilities and regresses the essential matrix through a weighted eight-point algorithm.Its objective combines classification and essential-matrix losses.
- 3.2. Differentiable Pooling Layer: DiffPool learns soft assignments that map unordered correspondences into fewer clusters to capture local context.The assignment matrix is generated by a permutation-equivariant network and normalized row-wise.
- 3.2. Differentiable Pooling Layer: Permutation-invariant DiffPool maps inputs into a particular learned canonical cluster order, independent of input permutation.This canonical order is determined by the parameters of hpool(·).
- 3.3. Differentiable Unpooling Layer: Order-Aware DiffUnpool learns assignments from level-l features so upsampled clusters retain the previous level’s order information.The design addresses the plain unpooling implementation’s inability to align recovered features with the preceding stage.
- 3.3. Differentiable Unpooling Layer: The unpooling design supports variable input sizes by pooling to fixed clusters and upsampling back to the original correspondence count.The paper gives N = 2000 and M = 500 as an example while stating that fewer or more keypoints can also be handled.
- 3.4. Order-Aware Filtering Block: The Spatial Correlation layer models relations between canonically ordered clusters, complementing PointCN to capture global context.It applies weight-sharing perceptrons along the spatial dimension and is assembled with PointCN in the Order-Aware Filtering block.
4. Experiments
Experiments evaluate Order-Aware Network on outdoor YFCC100M and indoor SUN3D scenes, including ablations and comparisons with prior models. The method improves relative-pose estimation and correspondence quality, with gains from order-aware unpooling, filtering, and iterative refinement.
- Datasets and setup: Experiments use outdoor YFCC100M and indoor SUN3D datasets, testing both known and unknown scenes under shared evaluation settings.YFCC100M uses 68 training sequences and four unknown test sequences; SUN3D uses 239 training sequences and 15 unknown test scenes.
- Datasets and setup: mAP under 5° is the default metric, with angular differences measuring rotation and translation errors and results reported with or without RANSAC.Changing RANSAC’s inlier threshold from 0.01 to 0.001 substantially improves the baseline, motivating separate post-processing reports.
- Ablation studies: 5.23% improvement over the baseline is achieved by Order-Aware DiffUnpool on unknown scenes without RANSAC, while plain DiffUnpool yields negligible improvement.The ablation compares PointCN + UnB with the baseline and PointCN + UnA under the same setting.
- Ablation studies: 3.77% improvement on unknown scenes without RANSAC comes from replacing second-level PointCN blocks with Order-Aware Filtering blocks.The comparison is between PointCN + UnB + OF and PointCN + UnB.
- Ablation studies: The larger three-level model drops on unknown scenes, whereas the iterative network raises mAP from 33.68% to 39.33% without RANSAC.The authors therefore retain the two-level architecture and use detached gradients between iterative stages.
- Comparison to other baselines: 15.78% and 7.03% improvements over PointCN are reported on outdoor and indoor unknown scenes without RANSAC, respectively.The method achieves the best results under all tested settings and remains effective with RANSAC post-processing; visualizations cover difficult scenes including wide baselines and repetitive structures.
5. Conclusion
The Order-Aware Network learns two-view correspondences and geometry through hierarchical operations that capture local and global context. These operations significantly improve relative pose estimation accuracy on outdoor and indoor datasets.
- Order-Aware Network learns two-view correspondences and geometry using hierarchical context modeling.Its operations include DiffPool, Order-Aware DiffUnpool, and Order-Aware Filtering blocks.
- DiffPool and Order-Aware DiffUnpool cluster meaningful nodes to capture local context.
- Order-Aware Filtering blocks capture the global context of correspondences.
- The proposed operations significantly improve relative pose estimation accuracy on outdoor and indoor datasets.
A.1 Weighted Eight-Point Algorithm
The weighted eight-point algorithm constructs a correspondence matrix and estimates the essential matrix by weighting each correspondence according to its predicted inlier probability.
- Each correspondence ci = (xi 1, yi 1, xi 2, yi 2) contributes one row to matrix X ∈R^N×9.
- The traditional eight-point algorithm minimizes ∥X^T XVec(E)∥ to recover the essential matrix E.
- The weighted formulation minimizes ∥X^T diag(w)XVec(E)∥, where w contains neural-network-predicted inlier probabilities.
- The resulting linear least-square problem has a closed-form solution obtained through eigenvalue decomposition.
A.2 Outlier rejection results of different methods
The paper reports correspondence outlier-rejection results using precision, recall, and F-score, with inliers defined by a symmetric epipolar-distance threshold.
- Precision, recall, and F-score are provided for each method in Table 4.
- Table 4 uses an inlier threshold of 10−4 symmetric epipolar distance.