PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Resolution Correspondence Networks

Xinghui Li, Kai Han, Shuda Li, Victor Adrian Prisacariu

arXiv:2006.08844v2cs.CV

TL;DR

Dense pixel-wise correspondence is central to several vision tasks, yet high-resolution 4D correlation processing is computationally expensive. DualRC-Net uses coarse and fine feature maps in a coarse-to-fine pipeline, achieving state-of-the-art results on HPatches, InLoc, and Aachen Day-Night while avoiding fine-resolution 4D convolutions.

  • Problem

    High-resolution dense correspondence requires reliable matching, but existing 4D-convolution methods incur prohibitively high memory and computation costs.

  • Method

    DualRC-Net forms and refines a full 4D correlation tensor from coarse features, then uses it to guide dense matching in fine-resolution feature maps.

  • Results

    DualRC-Net achieves state-of-the-art results across HPatches, InLoc, and Aachen Day-Night.

  • Takeaways & Limitations

    The dual-resolution design combines robust neighbourhood consensus with fine-resolution matching while avoiding expensive 4D convolutions on fine-resolution features.

  • Takeaways & Limitations

    The technology can be misused for unauthorized localization or to guide UAV attacks, making application-specific legislation potentially necessary.

Abstract

from arXiv · show

We tackle the problem of establishing dense pixel-wise correspondences between a pair of images. In this work, we introduce Dual-Resolution Correspondence Networks (DualRC-Net), to obtain pixel-wise correspondences in a coarse-to-fine manner. DualRC-Net extracts both coarse- and fine- resolution feature maps. The coarse maps are used to produce a full but coarse 4D correlation tensor, which is then refined by a learnable neighbourhood consensus module. The fine-resolution feature maps are used to obtain the final dense correspondences guided by the refined coarse 4D correlation tensor. The selected coarse-resolution matching scores allow the fine-resolution features to focus only on a limited number of possible matches with high confidence. In this way, DualRC-Net dramatically increases matching reliability and localisation accuracy, while avoiding to apply the expensive 4D convolution kernels on fine-resolution feature maps. We comprehensively evaluate our method on large-scale public benchmarks including HPatches, InLoc, and Aachen Day-Night. It achieves the state-of-the-art results on all of them.

1 Introduction

Dense pixel-wise correspondence is important for computer-vision tasks, but existing 4D-convolution approaches are expensive at high resolution. DualRC-Net addresses this with dual-resolution, coarse-to-fine matching.

  • Dense image correspondences support 3D reconstruction, camera motion estimation, and relocalization.
  • Existing neighbourhood-consensus methods construct full 4D correlation tensors and refine them with 4D convolutions, but high-resolution processing is prohibitively expensive.
  • DualRC-Net extracts coarse- and fine-resolution feature maps using an FPN-like backbone for coarse-to-fine correspondence estimation.
  • The coarse maps form a full 4D correlation tensor, while refined coarse matches guide correspondence extraction from fine-resolution features.
  • DualRC-Net reports state-of-the-art results on HPatches, InLoc, and Aachen Day-Night.

2 Related work

Correspondence research spans sparse keypoint pipelines and end-to-end dense matching. DualRC-Net retains neighbourhood-consensus filtering while using dual resolutions to improve dense matching efficiency and accuracy.

  • Correspondence estimation underpins tasks including 3D reconstruction, image retrieval, and robot relocalization.
  • Traditional methods detect salient keypoints, describe them, and match candidate pairs using confidence or validity checks.
  • NCNet combines detection, description, and matching end to end by constructing a dense 4D matching tensor and applying 4D convolutions.
  • DualRC-Net uses coarse-resolution features for refined 4D matching and fine-resolution features for denser correspondence, unlike Sparse-NCNet’s 3D sub-manifold projection.
  • Siamese CNN pose-regression methods are typically applied to small viewpoint changes and have limited generalization to new scenes.

3 Method

DualRC-Net estimates dense correspondences progressively: coarse features form and refine a complete 4D matching tensor, while fine features produce localised dense matches guided by the coarse scores.

  • Coarse matching and consensus: Coarse features form a complete 4D correlation tensor, whose neighbourhood consistency is refined using learnable 4D convolutions.Each tensor element represents cosine correlation between a feature pair, while local 4D neighbourhoods encode matching consistency between image regions.
  • Coarse matching and consensus: The correlation tensor is symmetrised and filtered with soft mutual nearest-neighbour constraints before guiding subsequent matching.The filtering retains high scores when the corresponding match is strong in both matching directions.
  • Fine-resolution dense matching: The refined coarse scores are interpolated and up-sampled to mask fine-resolution score maps, focusing fine matching on likely local regions.Nearest-neighbour up-sampling aligns the coarse score map with the fine map, followed by element-wise multiplication.
  • Dual-resolution feature extraction: The method extracts interlocked coarse- and fine-resolution feature maps with an FPN-like backbone.The fine map preserves higher spatial resolution while incorporating contextual information from deeper feature layers.
  • Fine-resolution dense matching: For each fine source feature, the method selects the highest-scoring target location and retains matches satisfying mutual nearest-neighbour consistency.The resulting intersection of matches from both directions forms the final dense correspondence set.
  • Fine-resolution dense matching: Selecting the top 50% of coarse matches restricts fine queries to corresponding local regions, reducing queries by half and increasing speed.Each coarse location corresponds to an r × r fine-region neighborhood, which is searched only for selected coarse matches.
  • Training objective: Training uses sparse keypoint annotations with probability-map supervision, Gaussian-smoothed ground truth, and an orthogonal regularizer for one-to-one matching.The overall loss combines the correspondence loss with the orthogonal loss using λ = 0.05.

4 Experimental results

DualRC-Net is evaluated on HPatches, InLoc, and Aachen Day-Night under substantial viewpoint and illumination variation, achieving strong correspondence and relocalization performance. Across comparisons, it substantially outperforms competing methods while using a higher-resolution matching representation more efficiently.

  • DualRC-Net is evaluated on HPatches, InLoc, and Aachen Day-Night, which contain substantial viewpoint and illumination variations.
  • HPatches: On HPatches, DualRC-Net significantly outperforms NCNet and Sparse-NCNet under illumination change and is competitive under viewpoint change.Sparse-NCNet performs slightly better below 6 pixels, whereas DualRC-Net performs better above 6 pixels.
  • HPatches: A 400 × 300 feature map requires 1600 × 1200 input for DualRC-Net, compared with 3200 × 2400 for NCNet.Sparse-NCNet obtains the same feature-map resolution with a 3200 × 2400 input by reducing the final convolution stride.
  • HPatches: DualRC-Net substantially outperforms D2-Net, DEFL, R2D2, SuperPoint, and SuperPoint+SuperGlue on HPatches.
  • DualRC-Net achieves the state-of-the-art result on InLoc for distance threshold > 0.3m and on Aachen Day-Night for thresholds (0.5m, 2°) and (5m, 10°).InLoc evaluates long-term indoor relocalization, while Aachen Day-Night evaluates outdoor relocalization under illumination change.

5 Conclusion

The conclusion presents DualRC-Net as an end-to-end dual-resolution architecture for dense pixel-wise correspondences. Its coarse-to-fine design improves matching reliability and localization accuracy while avoiding expensive fine-resolution 4D convolutions, and it achieves state-of-the-art results across three benchmarks.

  • DualRC-Net establishes dense pixel-wise correspondences through an end-to-end coarse-to-fine dual-resolution architecture.
  • The model refines a full coarse 4D correlation tensor with learnable neighbourhood consensus before guiding fine-resolution matching.
  • The design improves matching reliability and localization accuracy without applying expensive 4D convolution kernels to fine-resolution feature maps.
  • DualRC-Net achieves state-of-the-art results on HPatches, InLoc, and Aachen Day-Night.

Broader Impact

The proposed model is positioned as useful for applications including image alignment, retrieval, reconstruction, and pose estimation, while also carrying misuse risks that may require supervision.

  • Vision-only correspondence and localisation could be misused to locate people or property without permission or guide terrorist UAV attacks.
  • The paper states that these negative impacts relate more to application domains than to the technology itself and may require legislation and strict supervision.

Appendices

The appendices provide additional experiments and analyses covering FPN alternatives, neighbourhood-consensus comparisons, and qualitative benchmark results.

  • Appendix A evaluates five alternatives to the FPN-like structure for fusing dual-resolution feature maps.
  • Appendix B compares DualRC-Net with other neighbourhood-consensus-based methods in greater detail.
  • Appendix C qualitatively compares DualRC-Net with state-of-the-art methods on three benchmarks.

A Investigation on more variants of FPN structure

The appendix evaluates five FPN-like dual-resolution feature-extractor variants on HPatches and InLoc, finding broadly similar performance and selecting the simplest suitable default.

  • FPN variants: Five FPN variants fuse backbone outputs differently, while 1 × 1 convolutions align all feature-map channels to 1024.
  • Evaluation: The variants were evaluated on HPatches and InLoc using the alternative FPN-based dual-resolution feature extractors.
  • HPatches: All five variants have similar overall HPatches performance; type (d) very slightly leads but requires more 3 × 3 convolution parameters than type (a).
  • InLoc: On InLoc, type (b) is marginally behind, while types (a), (c), (d), and (e) have entangled curves without a consistent winner.
  • Selection: Type (a) was selected because, except for type (b), it is the simplest architecture among the five options.
  • Channel comparison: For type (e), increasing channels does not affect performance, whereas type (a) with 1024 channels performs better than its 256-channel version under severe illumination changes at smaller thresholds.

B Comparison with other neighbourhood consensus based methods

DualRC-Net is compared with Sparse-NCNet and NCNet under matched image or feature-map settings, showing competitive performance with differing trade-offs across illumination and viewpoint changes.

  • Matched input images: With 1600 × 1200 input images, DualRC-Net achieves the best HPatches results except under huge illumination changes below a 3-pixel threshold.
  • Matched feature maps: With 200 × 100 relocalisation feature maps, DualRC-Net performs best under huge illumination variations but worse than both alternatives under huge viewpoint variations.
  • Overall comparison: Overall, DualRC-Net performs on par with Sparse-NCNet and NCNet in the reported comparisons.
  • Efficiency: DualRC-Net processes image pairs in 2.05s with 1232MB GPU memory, compared with 0.82s/680MB for Sparse-NCNet and 4.15s/7868MB for NCNet.
  • Resolution trade-off: Larger relocalisation resolution improves illumination-change performance but worsens viewpoint-change performance at small thresholds.

C Qualitative comparison

DualRC-Net is qualitatively compared with Sparse-NCNet and NCNet on HPatches, InLoc, and Aachen Day-Night, using selected top matches and benchmark-specific assessment procedures. It performs best under challenging HPatches illumination changes, substantially outperforms NCNet under viewpoint changes, and produces reliable correspondences across difficult indoor and outdoor scenes.

  • Comparison setup: Comparisons use pretrained Sparse-NCNet and NCNet models, with DualRC-Net evaluated at 1600×1200 input resolution versus 3200 × 2400 for both baselines.The qualitative comparisons cover HPatches, InLoc, and Aachen Day-Night.
  • HPatches benchmark: Top 6000 matches are judged correct when their re-projection error is less than 3 pixels, with correct matches marked green and incorrect matches red.This criterion is used for the HPatches qualitative comparison.
  • HPatches benchmark: DualRC-Net performs the best under huge illumination changes on HPatches.Figure 13 reports correct/total matches for each image pair.
  • HPatches benchmark: Under huge viewpoint changes on HPatches, DualRC-Net substantially outperforms NCNet and performs on par with Sparse-NCNet.Figure 14 also reports correct/total matches for each image pair.
  • InLoc benchmark: On InLoc, visual inspection indicates reliable correspondences for indoor scenes with very large viewpoint and illumination changes and repetitive patterns.No ground truth is provided, and the comparison selects the top 500 matches for each method.
  • Aachen Day-Night benchmark: On Aachen Day-Night, visual inspection indicates reliable correspondences for outdoor scenes with very large viewpoint changes, day-night illumination changes, and repetitive patterns.No ground truth is available, and the comparison selects the top 500 matches for each method.
Loading 2006.08844v2…