PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions

Ignacio Rocco, Relja Arandjelović, Josef Sivic

arXiv:2004.10566v1cs.CV

TL;DR

The paper addresses inaccurate correspondence localisation and the high memory and inference costs of Neighbourhood Consensus Networks. Sparse-NCNet sparsifies the correlation tensor for 4D submanifold sparse-convolution processing, adds higher-resolution inputs and two-stage relocalisation, and reports strong benchmark performance with over 10× efficiency gains.

  • Problem

    Neighbourhood Consensus Networks provide strong correspondence estimation but are limited by large memory consumption, long inference time, and poorly localised matches.

  • Method

    Sparse-NCNet stores only promising correspondences in a sparse correlation tensor, processes it with a 4D submanifold sparse CNN, and uses higher resolution plus two-stage relocalisation.

  • Results

    Sparse-NCNet achieves equivalent matching results to NCNet while using up to 20× less memory and running up to 10× faster, and outperforms state-of-the-art results on two challenging benchmarks.

  • Takeaways & Limitations

    The end-to-end pipeline makes neighbourhood-consensus correspondence estimation more computationally efficient and supports downstream applications such as camera pose estimation and 3D reconstruction.

  • Takeaways & Limitations

    NCNet’s dense correlation tensor can require 3.4GB for 200×150 feature maps and over 50GB for subsequent 4D CNN processing, limiting feature-resolution increases.

Abstract

from arXiv · show

In this work we target the problem of estimating accurately localised correspondences between a pair of images. We adopt the recent Neighbourhood Consensus Networks that have demonstrated promising performance for difficult correspondence problems and propose modifications to overcome their main limitations: large memory consumption, large inference time and poorly localised correspondences. Our proposed modifications can reduce the memory footprint and execution time more than $10\times$, with equivalent results. This is achieved by sparsifying the correlation tensor containing tentative matches, and its subsequent processing with a 4D CNN using submanifold sparse convolutions. Localisation accuracy is significantly improved by processing the input images in higher resolution, which is possible due to the reduced memory footprint, and by a novel two-stage correspondence relocalisation module. The proposed Sparse-NCNet method obtains state-of-the-art results on the HPatches Sequences and InLoc visual localisation benchmarks, and competitive results in the Aachen Day-Night benchmark.

1 Introduction

Sparse-NCNet adapts dense Neighbourhood Consensus Networks to address their memory, speed, and correspondence-localisation limitations. It sparsifies tentative matches and uses submanifold sparse convolutions, while higher-resolution processing and two-stage relocalisation improve localisation.

  • Motivation: Extreme illumination and viewpoint changes remain challenging because local-feature methods can miss detections.
  • Motivation: Dense feature extraction avoids the detection stage but is memory intensive and limited in localisation accuracy by grid sampling.
  • Proposed approach: Sparse-NCNet extends NCNet’s jointly trainable feature extraction, matching, and filtering while targeting high memory use, slow execution, and poor localisation.
  • Proposed approach: The model uses a 4D CNN on a sparse correlation tensor containing only the most promising correspondences.
  • Proposed approach: Up to 10× faster execution and 20× lower memory use are reported with equivalent results and no performance decrease versus NCNet.
  • Results: A two-stage relocalisation module improves output correspondence localisation, while the method reports state-of-the-art results on HPatches Sequences and InLoc and competitive Aachen Day-Night results.

2 Related work

Related work spans trainable local features, dense feature matching, incorrect-match filtering, and methods for improving feature localisation. Sparse-NCNet builds on NCNet while applying sparse 4D convolutional processing to dense-feature correspondences and introducing two-stage relocalisation.

  • Trainable local features and filtering: Trainable local-feature methods increasingly learn detection and description jointly, but their matches are often filtered separately using nearest-neighbour or learned procedures.
  • Dense feature matching: Dense features can improve robustness to illumination changes, although their coarse localisation is problematic when correspondences are used directly for 3D vision tasks.
  • Learning to filter incorrect matches: Existing filtering methods include heuristic ratio tests, mutual nearest neighbours, clustering, and graph-based approaches, often designed for keypoint features rather than dense grids.
  • Neighbourhood consensus: Sparse-NCNet builds on NCNet’s dense-feature match filtering while jointly learning feature extraction, matching, and filtering in one pipeline.
  • Improved feature localisation: NCNet relocalisation can increase output resolution only twofold, whereas Sparse-NCNet combines hard relocalisation with soft interpolation for sub-feature-grid accuracy.
  • Sparse convolutional networks: Submanifold sparse convolutions keep active sites constant across layers, preserving sparsity while filtering correspondences represented as points in 4D image-coordinate space.

3 Sparse Neighbourhood Consensus Networks

Sparse-NCNet preserves NCNet’s dense feature processing while replacing exhaustive correspondence storage with a sparse, symmetric correlation tensor processed by submanifold sparse convolutions. A guided two-stage relocalisation module addresses coarse feature-grid localisation, while the sparse design reduces resource demands and enables higher-resolution matching.

  • 3.1 Review: Neighbourhood Consensus Networks: NCNet extracts dense feature maps, stores all pairwise matches in a 4D correlation tensor, and filters them with a 4D CNN.The dense tensor supports neighbourhood consensus filtering but grows quadratically with the number of dense features.
  • 3.1 Review: Neighbourhood Consensus Networks: The original NCNet is memory intensive, slow, and poorly localised, limiting practical use and feature-map resolution.A 200×150 tensor requires 3.4GB alone, while subsequent 4D CNN processing requires more than 50GB; a 100×75 example takes approximately 10 seconds on a Tesla T4.
  • 3.2 Sparse-NCNet: Efficient Neighbourhood Consensus Networks: Sparse-NCNet retains only the top K cosine-similarity matches per feature in both directions, then combines the one-sided tensors into a symmetric sparse correlation tensor.This retains multiple matching hypotheses rather than only the best match while avoiding storage of all possible correspondences.
  • 3.2 Sparse-NCNet: Efficient Neighbourhood Consensus Networks: For 100×75 feature maps with K = 10, sparse storage is 3.43MB versus 215MB dense, yielding a 12× processing-time reduction.The sparse representation stores at most h × w × K × 2 elements, rather than h × w × h × w.
  • 3.2 Sparse-NCNet: Efficient Neighbourhood Consensus Networks: Submanifold sparse convolutions process the sparse correlation tensor while preserving active sites, and a permutation-invariant CNN applies the 4D neighbourhood-consensus filter in both input-order directions.The same output matches are obtained regardless of the order of the input images.
  • 3.3 Match relocalisation by guided search: The two-stage relocalisation module uses high-resolution features for guided search: hard relocalisation selects the best local match, then soft relocalisation refines it to sub-feature-grid accuracy.High-resolution features are used only for relocalisation; Sparse-NCNet processes downsampled features at h × w resolution, and the relocalised matches lie on a 2h × 2w grid.

4 Experimental evaluation

The evaluation uses HPatches, InLoc, and Aachen Day-Night to assess matching and camera localisation under viewpoint and illumination changes. Sparse-NCNet retains nearly the same matching quality with lower resource use, while its higher-resolution variant achieves the best results.

  • Evaluation benchmarks: Sparse-NCNet is evaluated on HPatches Sequences, InLoc, and Aachen Day-Night benchmarks.These cover direct matching, indoor 6-dof localisation, and outdoor 6-dof localisation under challenging day-night illumination changes.
  • Evaluation benchmarks: HPatches uses 108 planar-scene sequences to measure mean matching accuracy under viewpoint and illumination variations.The benchmark includes 56 viewpoint-change sequences and 52 illumination-change sequences.
  • Results: Sparse-NCNet keeps almost the same matching performance as NCNet while significantly reducing processing time and memory consumption.The comparison includes variants without relocalisation and with hard relocalisation only.
  • Results: The two-stage relocalisation module improves performance with only a minor increase in processing time.This comparison is reported between methods C1 and B1.
  • Results: The higher-resolution Sparse-NCNet variant produces the best results while remaining 30% faster and 3× more memory efficient than the best NCNet variant.It processes 200×150 feature maps, enabled by the reduced memory consumption.

Peak VRAM (MB)

Sparse-NCNet matches NCNet while reducing execution time and memory consumption, and its higher-resolution, two-stage configuration achieves the best reported matching results.

  • Sparse-NCNet obtains equivalent results to NCNet without relocalisation and with hard relocalisation while reducing execution time and memory consumption.The comparison covers both A1 versus A2 and B1 versus B2.
  • The two-stage H+S relocalisation improves matching accuracy with only a minor increase in execution time.
  • Higher-resolution processing produces the best results while remaining faster and more memory efficient than NCNet.Reduced memory consumption makes the higher-resolution configuration possible.
  • Sparse-NCNet obtains the best overall MMA on HPatches, with a large margin over R2D2.
  • On InLoc, hard-relocalised Sparse-NCNet at 100×75 resolution matches NCNet while running almost 7× faster and using 6.5× less memory.
  • Two-stage relocalisation at 200 × 150 resolution obtains the best InLoc results and sets a new state-of-the-art.Original NCNet cannot be used at this higher resolution because of its excessive memory requirements.

5 Conclusion

The paper develops Sparse Neighbourhood Consensus Networks to make correspondence estimation efficient and practical while retaining robust matching performance and end-to-end trainability.

  • Sparse-NCNet overcomes the original NCNet’s main limitations while jointly performing feature extraction, matching, and robust match filtering efficiently.
  • The model outperforms state-of-the-art results on two challenging matching benchmarks.
  • The end-to-end trainable pipeline can incorporate additional modules for downstream tasks such as camera pose estimation or 3D reconstruction.

Appendices

The appendices provide operational insights into Sparse-NCNet and additional qualitative benchmark results.

  • Appendix A presents insights into how Sparse-NCNet operates.
  • Appendix B contains additional qualitative results on the HPatches Sequences benchmark.
  • Appendices C and D provide additional qualitative results on InLoc and Aachen Day-Night, respectively.

A Insights about Sparse-NCNet

Sparse-NCNet’s additional matches tend to appear near strong matches, illustrating propagation of matching information through neighbouring features.

  • The appendix compares the top 100, 400, and 1600 matches produced by Sparse-NCNet.
  • Many additional matches appear close to the initial high-ranking matches as N increases.
  • Match strength is encoded by color, with more yellow indicating stronger matches and new matches demonstrating information propagation.

B HPatches Sequences benchmark

On HPatches, Sparse-NCNet is evaluated against trainable local-feature methods using Mean Matching Accuracy and qualitative correspondence visualisations. It achieves the largest fraction of correct matches for both 2K and 6K outputs while producing densely packed correspondences.

  • Evaluation metric: Mean Matching Accuracy measures the fraction of correct matches under different pixel-tolerance thresholds on HPatches Sequences.The metric uses a ground-truth homography and an indicator for positive correctness.
  • Correspondence structure: Unlike methods selecting local extrema, Sparse-NCNet produces densely packed correspondence sets through information propagation in local neighbourhoods.The comparison includes SuperPoint, D2-Net, and R2D2.
  • Qualitative results: Sparse-NCNet achieves the largest fraction of correct matches for both the top 2K and top 6K matches at t = 3 pixels.Correct and incorrect correspondences are visualised in green and red, respectively, and fractions are reported in percentage and absolute values.

C InLoc benchmark

InLoc qualitative results show Sparse-NCNet producing coherent matches between cellphone query images and earlier 3D-scanned database images despite differing illumination and scene changes. RANSAC within the PnP solver can remove likely outliers and yield well-aligned estimated poses.

  • Qualitative matching: Most of the top 500 unverified correspondences are coherent, while likely outliers can be removed by RANSAC within the PnP pose solver.The query and database images differ in illumination and may contain displaced objects because they were captured months apart.
  • Pose estimation: Rendered images from estimated query 6-dof poses are well aligned with the query images, indicating low translation and rotation errors.The poses are estimated by running RANSAC+PnP on Sparse-NCNet matches.

D Aachen day-night benchmark

Aachen day-night qualitative results examine matches between night query images and day database images. Sparse-NCNet produces mostly coherent and apparently correct correspondences despite strong illumination changes, without geometric verification.

  • Qualitative matching: Most of the top 500 correspondences are coherent and seem correct despite strong illumination changes between night queries and day database images.The displayed correspondences were not geometrically verified.
  • Evaluation setup: The image pairs use night queries and their top matching daytime database images selected by Sparse-NCNet’s average matching score.The correspondences are overlaid for visual inspection.
Loading 2004.10566v1…