Source-linked AI summary
Neighbourhood Consensus Networks
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, Josef Sivic
TL;DR
Reliable dense image correspondence is difficult under strong appearance changes and repetitive patterns. The paper introduces an end-to-end trainable neighbourhood-consensus network that evaluates dense match patterns before assignment without a global geometric model, and reports strong results across category- and instance-level tasks.
Problem
Reliable dense correspondences are difficult because corresponding scene elements can differ strongly in appearance and repetitive patterns create ambiguous matches.
Method
An end-to-end trainable CNN analyzes local consensus patterns in dense 4D correspondences, uses weak image-pair supervision, and integrates differentiable matching components.
Results
The model achieves state-of-the-art results across category- and instance-level matching, including PF Pascal and InLoc, with about 3% improvement on the reported category-level dataset.
Takeaways & Limitations
Neighbourhood consensus can be learned end to end for dense matching across substantially different matching tasks without requiring a global geometric model.
Takeaways & Limitations
The method has quadratic O(N^2) complexity, limiting currently handled image resolution to 1600 × 1200px or 3200 × 2400px with 4-D max+argmax pooling, and can fail on scale-changing repetition or consistent incorrect groups.
Abstract
from arXiv · showhide
We address the problem of finding reliable dense correspondences between a pair of images. This is a challenging task due to strong appearance differences between the corresponding scene elements and ambiguities generated by repetitive patterns. The contributions of this work are threefold. First, inspired by the classic idea of disambiguating feature matches using semi-local constraints, we develop an end-to-end trainable convolutional neural network architecture that identifies sets of spatially consistent matches by analyzing neighbourhood consensus patterns in the 4D space of all possible correspondences between a pair of images without the need for a global geometric model. Second, we demonstrate that the model can be trained effectively from weak supervision in the form of matching and non-matching image pairs without the need for costly manual annotation of point to point correspondences. Third, we show the proposed neighbourhood consensus network can be applied to a range of matching tasks including both category- and instance-level matching, obtaining the state-of-the-art results on the PF Pascal dataset and the InLoc indoor visual localization benchmark.
1 Introduction
Image correspondence remains difficult under appearance changes and repetitive or textureless regions, where individual feature matching can fail. The paper proposes learning neighbourhood consensus over dense matches before assignment to resolve these ambiguities without a global geometric model.
- Visual correspondences support 3D reconstruction, visual localization, and object recognition, but trainable features have yielded only modest gains over hand-crafted features.
- Nearest-neighbour matching of individual features followed by separate geometric disambiguation struggles with repetitive or textureless image regions.
- NC-Net analyzes local neighbourhood patterns across dense correspondences so unique matches can support nearby ambiguous matches.
- The method learns neighbourhood consensus constraints directly from data before hard correspondence assignment, reducing errors from early ambiguous matches.
- The proposed CNN learns local geometric constraints without a global model, trains with image-pair weak supervision, and targets category- and instance-level matching.
2 Related work
Prior correspondence systems use hand-crafted or trainable descriptors with nearest-neighbour matching and often separate geometric post-processing. This work connects differentiable dense matching with neighbourhood-consensus filtering for broader matching conditions than optical flow or fixed geometric alignment.
- Matching with hand-crafted image descriptors: Traditional systems use hand-crafted invariant descriptors, nearest-neighbour candidates, and ambiguity filters such as ratio tests or mutual-nearest-neighbour checks.
- Matching with trainable descriptors: Trainable descriptor methods commonly produce sparse CNN features, compare them with learned or Euclidean similarity, and apply geometric consistency separately.
- Trainable image alignment: End-to-end alignment methods estimate affine, perspective, or thin-plate-spline transformations from all pairwise feature matches.
- Match filtering by neighbourhood consensus: Neighbourhood-consensus methods filter matches using local distances, angles, or counts of consistent neighbours, sometimes with scale-invariant or grid-based neighbourhoods.
- Flow and disparity estimation: Unlike optical flow and disparity estimation, this work addresses matching with large viewpoint or appearance changes, including indoor localization and category-level matching.
3 Proposed approach
The proposed pipeline extracts dense descriptors, computes all pairwise matches, and uses differentiable soft mutual-nearest-neighbour filtering with a 4D neighbourhood consensus CNN to retain spatially supported correspondences.
- Dense feature extraction and matching: Dense descriptors are extracted from both images, and all pairwise feature similarities are stored in a 4D correlation tensor.The tensor represents matches using feature positions from both images.
- Neighbourhood consensus network: The network evaluates matches before hard assignment, avoiding early decisions that can be incorrect for repetitive or textureless features.Correct matches are expected to have coherent supporting matches in the 4D correspondence space.
- Neighbourhood consensus network: 4D convolutions analyze local neighbourhoods in both images to learn patterns distinguishing reliable from unreliable matches.The first-layer filters span the Cartesian product of local neighbourhoods in the two images.
- Soft mutual nearest neighbour filtering: Soft mutual-nearest-neighbour filtering provides a differentiable gating mechanism that downweights non-reciprocal matches while preserving global reciprocity constraints.Unlike hard filtering, the soft formulation remains suitable for insertion during training and evaluation.
4 Experimental results
The experiments evaluate NC-Net for category-level semantic matching and instance-level indoor localization, using dense correspondences for semantic transfer and camera pose estimation. NC-Net improves matching quality in both settings, including challenging repetitive and textureless scenes, but has quadratic complexity and identified failure modes.
- Category-level matching: NC-Net obtains approximately 3% improvement over state-of-the-art methods on PF-Pascal semantic keypoint transfer.Performance is measured by the percentage of correctly matched annotated keypoints (PCK).
- Category-level matching: Dense correspondences from NC-Net transfer annotated keypoints between semantically related images despite large appearance changes and non-rigid geometric deformations.The method is evaluated on category-level matching, where images contain different instances from the same object category.
- Instance-level matching: NC-Net is inserted into the InLoc pipeline to produce correspondences used for 6DoF camera pose estimation in indoor visual localization.The evaluation uses database images and smartphone query images captured at a different time.
- Instance-level matching: InLoc+NC-Net provides a much larger and locally consistent set of inlier matches than InLoc+MNN, including in low-textured regions.Both methods use the same CNN features.
- Instance-level matching: DensePE+NC-Net improves over SparsePE and DensePE, while InLoc+NC-Net achieves state-of-the-art results on the indoor localization benchmark.The comparisons cover sparse DoG+SIFT matching and CNN feature matching used in the prior InLoc approach.
- Limitations: NC-Net remains challenged by large-scale changes in repetitive patterns and locally consistent groups of incorrect matches.Its quadratic O(N^2) complexity limits current processing to 1600 × 1200px, or 3200 × 2400px with 4-D max+argmax pooling.
5 Conclusion
The paper concludes that NC-Net learns local correspondence patterns without a global geometric model, can use weak supervision, and performs strongly across two distinct matching tasks. It identifies broader opportunities for end-to-end learning in challenging correspondence problems.
- Conclusion: NC-Net learns local correspondence patterns without requiring a global geometric model.The same model supports reliable matches for instance-level camera pose estimation and category-level semantic object alignment.
- Conclusion: The model trains effectively with weak supervision and achieves state-of-the-art performance on two different matching tasks.The evaluated tasks are indoor localization and semantic object alignment.
- Conclusion: The results open possibilities for end-to-end learning in 3D category-level matching and visual localization across day/night illumination.
Appendices
The appendices provide technical details, additional experiments, and dataset information supporting the main paper. They cover implementation, qualitative matching results, and the Indoor Venues Dataset.
- Appendices: Appendices A and B describe implementation details for 4D convolutions and feature relocalization.
- Appendices: Appendices C and D provide additional experimental results for category-level and instance-level matching.
- Appendices: Appendix E provides additional details about the Indoor Venues Dataset used for the experiments.
A Implementation of 4D convolutions
Because the framework did not support 4D convolutions, the authors implemented them by aggregating multiple 3D convolutions. This extends an earlier strategy that constructed 3D convolutions from multiple 2D convolutions.
- Implementation of 4D convolutions: 4D convolutions are implemented by aggregating the results of multiple 3D convolutions.The implementation was needed because the authors' PyTorch framework did not natively support 4D convolutions.
- Implementation of 4D convolutions: The implementation follows a precedent in which multiple 2D convolutions were aggregated to obtain 3D convolutions.
B Feature relocalization
The relocalization method recovers sub-grid feature positions from a downsampled 4-D correlation map, improving localization precision without retaining the higher-resolution map throughout processing.
- The approach targets pose-estimation settings where precisely localized features are needed, while avoiding the full computational cost of doubling both feature-map dimensions.Doubling h and w increases memory consumption and computation time sixteenfold.
- Higher-resolution features create a 2h × 2w × 2h × 2w correlation map, which is downsampled to h × w × h × w before neighbourhood-consensus processing.The downsampling uses 4-D max-pooling with kernel size 2.
- The max-pooling operation records the arg-max offsets δa, δb, δc, δd for each pooled correlation entry.
- Final matches are computed from the downsampled map and then relocalized using the extracted offsets to reduce localization error.The resulting feature positions are computed from these shifts after matching.
C Additional results for category-level matching
Additional PF-Pascal examples show NC-Net transferring semantic keypoints despite appearance changes, non-rigid deformation, and occasional ambiguity in the ground-truth annotations.
- Randomly sampled PF-Pascal test examples compare automatically transferred keypoints with ground-truth correspondence annotations.
- Some ground-truth keypoints are ambiguous, such as a cat keypoint that could lie on either the face’s right side or its back.
D Additional results for instance-level matching
Qualitative indoor-localization examples show NC-Net succeeding where several baselines fail, with broader and more geometrically consistent inlier matches supporting more accurate pose estimation.
- NC-Net successfully localizes query images within 2 meters and 10◦ in examples where the baseline method fails.
- NC-Net is compared qualitatively against DensePE, InLoc, and InLoc+MNN using localization-error annotations and green inlier matches.
- NC-Net’s inlier matches tend to cover more of the scene and be more geometrically consistent than baseline matches, enabling more accurate pose estimation.
- Figure 10 reports consistent improvements over DensePE, InLoc, and the InLoc+MNN baseline, which uses the same ResNet-101 dense features as NC-Net.
E The Indoor Venues Dataset (IVD)
The Indoor Venues Dataset provides weakly supervised training data for instance-level matching, drawn from diverse user-uploaded indoor imagery across European venues and cities.
- IVD contains 3861 positive image pairs from 89 venues across six European cities, with 10% reserved for validation and the remainder for training.
- The images are user-uploaded Google Maps content selected to represent realistic indoor scenes from venues such as restaurants, cafes, and museums.
- More than 50% of downloaded images were non-relevant, so a fixed ImageNet-pretrained ResNet-50 classifier filtered food, indoor, outdoor, and other categories.
- Curated indoor images were sampled into query and same-venue candidate pairs, then manually inspected to retain pairs depicting the same scene region.
- The dataset is publicly available from the authors’ research webpage.
- The dataset examples illustrate variability in illumination, viewpoint, scene layout, and occlusion across venues and cities.