PaperlayerView exact revision on arXiv

Source-linked AI summary

Patch2Pix: Epipolar-Guided Pixel-Level Correspondences

Qunjie Zhou, Torsten Sattler, Laura Leal-Taixe

arXiv:2012.01909v3cs.CV

TL;DR

Existing correspondence networks are limited by memory-driven low-resolution matching, while fully supervised methods require precise ground-truth correspondences. Patch2Pix uses weakly supervised detect-to-refine matching to regress pixel-level correspondences from patch proposals and reject outliers. The refinement improves correspondence networks across image matching, homography estimation, and localization, and generalizes to fully supervised proposals without retraining.

  • Problem

    Correspondence networks unify matching stages but suffer from memory-limited resolution and imprecise pixel-level matching, while ground-truth correspondence supervision can introduce training bias.

  • Method

    Patch2Pix first generates patch-level match proposals, then progressively regresses pixel-level matches within local patches and predicts confidence scores for outlier rejection using epipolar supervision.

  • Results

    Patch2Pix consistently improves correspondence-network performance on image matching, homography estimation, and localization, with refinement gains also transferring to SuperPoint + SuperGlue proposals without retraining.

  • Takeaways & Limitations

    The refinement network learns to predict matches from local patch pairs across indoor and outdoor scenes, independent of how the proposals were obtained.

  • Takeaways & Limitations

    Patch2Pix cannot generate new proposals, so performance suffers when NCNet fails to produce enough valid proposals.

Abstract

from arXiv · show

The classical matching pipeline used for visual localization typically involves three steps: (i) local feature detection and description, (ii) feature matching, and (iii) outlier rejection. Recently emerged correspondence networks propose to perform those steps inside a single network but suffer from low matching resolution due to the memory bottleneck. In this work, we propose a new perspective to estimate correspondences in a detect-to-refine manner, where we first predict patch-level match proposals and then refine them. We present Patch2Pix, a novel refinement network that refines match proposals by regressing pixel-level matches from the local regions defined by those proposals and jointly rejecting outlier matches with confidence scores. Patch2Pix is weakly supervised to learn correspondences that are consistent with the epipolar geometry of an input image pair. We show that our refinement network significantly improves the performance of correspondence networks on image matching, homography estimation, and localization tasks. In addition, we show that our learned refinement generalizes to fully-supervised methods without re-training, which leads us to state-of-the-art localization performance. The code is available at https://github.com/GrumpyZhou/patch2pix.

1. Introduction

Image correspondence is central to geometric vision tasks, but existing pipelines and correspondence networks face robustness, resolution, and supervision challenges. Patch2Pix addresses these issues by refining patch-level proposals into pixel-level matches while rejecting outliers under epipolar supervision.

  • Image correspondences support tasks including Structure-from-Motion and Simultaneous Localization and Mapping.
  • The classical pipeline detects and describes local features, matches nearest neighbors, and rejects outlier correspondences.
  • Hand-crafted features are vulnerable to extreme illumination changes, motion blur, and repetitive or weakly textured scenes.
  • Correspondence networks perform matching within a single network but are constrained by memory bottlenecks that limit matching resolution and pixel-level accuracy.Low-resolution matching makes each correspondence uncertain within its local patches and can reduce relative pose accuracy.
  • Patch2Pix first predicts patch-level match proposals, then refines them into pixel-level matches using epipolar geometry and jointly rejects outliers.The method is trained without pixel-wise ground-truth correspondences.

2. Related Work

Prior work separately learned local features, matching, and outlier rejection, while correspondence networks unified these stages but remained limited in resolution and matching precision. Patch2Pix instead refines match locations directly at image resolution.

  • Learned Local Features: Learned local-feature methods improve robustness over hand-crafted features, but higher-resolution or per-pixel representations increase computation and memory costs.
  • Matching and Outlier Rejection: Nearest-neighbor correspondences are commonly filtered using mutual consistency, matching scores, learned outlier labels, or probabilities for RANSAC weighting.
  • End-to-End Matching: End-to-end correspondence networks combine feature detection, matching, and outlier rejection in one forward pass, but correlation-map memory limits matching resolution.
  • End-to-End Matching: DualRC-Net combines coarse- and fine-resolution matching scores, whereas Patch2Pix refines match locations with regression layers at image resolution.
  • Full versus Weak Supervision: Most local-feature methods use exact correspondences for supervision, while CAPS uses epipolar geometry as weak supervision.

3. Patch2Pix: Match Refinement Network

Patch2Pix uses a two-stage detect-to-refine pipeline: a correspondence network proposes patch-level matches, then regressors refine them to pixel-level matches and reject unreliable proposals.

  • Existing correspondence networks lose pixel-level accuracy because memory limits force downscaled feature maps and weak supervision does not distinguish good from bad matches.
  • Patch2Pix first predicts patch-level match proposals, then uses classification and regression to refine each proposal within local patches.The classification head estimates proposal confidence, while regression locates a pixel-level match.
  • The refinement stage searches S × S local patch pairs to compensate for localization errors caused by downscaled feature maps.The local patches are chosen with S > 2^L−1 so they cover a larger region than the original feature-map neighborhood.
  • Two same-architecture regressors progressively update match locations and confidence scores, producing final pixel-accurate matches.The fine-level regressor operates on patches recentered at the mid-level matches.
  • The pixel-level loss combines confidence classification and geometric accuracy terms as Lpixel = αLcls + Lgeo, with α empirically set to 10.The geometric supervision uses epipolar consistency measured by Sampson distance from the relative camera pose.

4. Implementation Details

Patch2Pix is trained using match proposals from an adapted NCNet on MegaDepth with 60,661 matching pairs and separate mid- and fine-level thresholds.

  • Patch2Pix is trained on MegaDepth using 60,661 constructed matching pairs and proposals detected by an adapted NCNet.The mid-level thresholds are 50, while the fine-level thresholds are 5 for both classification and geometric losses.

5. Evaluation on Geometrical Tasks

Patch2Pix is evaluated on image matching, homography estimation, and localization benchmarks, where it refines patch-level proposals into pixel-level correspondences and rejects outliers using confidence scores. Across these tasks, refinement improves matching and geometric estimation, generalizes to SuperPoint + SuperGlue proposals without retraining, and achieves strong localization results.

  • 5.1. Image Matching: Patch2Pix refines NCNet patch proposals into pixel-level correspondences, with confidence thresholds controlling the quantity–quality trade-off of retained matches.The HPatches image-matching evaluation uses c = 0.5/0.9 and compares against NCNet, SparseNCNet, local-feature methods, and SuperPoint + SuperGlue.
  • 5.1. Image Matching: Patch2Pix achieves the best weakly supervised image-matching performance under illumination changes and, at c = 0.9, under viewpoint changes.It outperforms all fully supervised methods under illumination variations, while remaining below SuperPoint + SuperGlue for viewpoint changes.
  • 5.2. Homography Estimation: 15 percent improvement in overall homography accuracy within 1 pixel demonstrates that learned pixel-level refinement improves geometric estimation beyond patch-level matching.Under viewpoint variations, Patch2Pix is best among weakly supervised methods at 1-pixel error and approaches SuperPoint + SuperGlue overall accuracy.
  • 5.3. Outdoor Localization on Aachen Day-Night: On Aachen Day-Night, Patch2Pix outperforms the other two weakly supervised methods on night-time local-feature evaluation and is competitive with strong fully supervised methods.With SuperGlue proposals, it is competitive with SuperGlue on day-time images and slightly better on night queries.
  • 5.4. Indoor Localization on InLoc: 14.5 % better than SparseNCNet on DUC2 at the finest error shows Patch2Pix’s advantage on challenging indoor localization.Patch2Pix is best among weakly supervised methods and becomes the overall best method when using SuperPoint + SuperGlue proposals.
  • Generalization: Patch2Pix improves SuperPoint + SuperGlue and SuperPoint + CAPS without retraining, indicating refinement generalizes across proposal sources and scene types.The refinement network is trained on NCNet-type proposals but predicts matches from local patch pairs independently of how those pairs were obtained.

6. Conclusion

Patch2Pix introduces a two-stage detect-to-refine paradigm: semantic patch-level proposals are refined into detailed pixel-level correspondences. Its epipolar-geometry-supervised refinement improves geometry-task performance, generalizes across proposal types, and achieves state-of-the-art long-term localization without retraining.

  • 6. Conclusion: Patch2Pix predicts patch-level match proposals before refining them into pixel-level matches, separating semantic capture from local structural refinement.The refinement network regresses match locations from CNN features and jointly predicts confidence scores for outlier rejection.
  • 6. Conclusion: Epipolar geometry weakly supervises the network to identify geometrically consistent correspondences.The supervision is applied to the refinement process rather than requiring ground-truth correspondences.
  • 6. Conclusion: The refinement network consistently improves the correspondence-network baseline across multiple geometry tasks.The reported tasks include image matching, homography estimation, and localization-related evaluation.
  • 6. Conclusion: Patch2Pix generalizes from correspondence-network proposals to other proposal types during testing.This enables refinement of proposals produced by methods other than the training proposal source.
  • 6. Conclusion: Applying the refinement to the best fully supervised method without retraining achieves state-of-the-art results on challenging long-term localization tasks.The conclusion specifically reports this result for localization.

Supplementray Material

The supplementary material documents Patch2Pix's architecture, training data, implementation details, ablations, and experimental setups. It extends the main paper with reproducibility and additional analysis.

  • Supplementray Material: The supplement provides architecture details for the backbone, regressors, and adapted NCNet baseline.It also describes training data and other implementation details.
  • Supplementray Material: The supplement reports ablation studies on architecture and training hyper-parameters.These studies are presented in Section B.
  • Supplementray Material: Additional experimental setups cover homography estimation and outdoor localization.The supplied passage truncates the final description of the outdoor-localization setup.

A. Implementation Details.

Patch2Pix uses a frozen truncated ResNet34 backbone, adapted NCNet proposals, and two regressors that refine local patch matches while predicting confidence for rejection. Training uses MegaDepth image pairs and expanded proposal samples.

  • Backbone: A truncated ResNet34 extracts five feature maps with channel dimensions [3, 64, 64, 128, 256].The final map retains 1/8 input resolution by preventing further stride downscaling, and the ImageNet-pretrained backbone is frozen.
  • Regressor: The mid-level and fine-level regressors aggregate collected patch-pair features and output confidence c plus local-match coordinates δ_i.Both regressors share the architecture shown in Figure 5.
  • Our Adapted NCNet: Adapted NCNet exhaustively matches the two final 1/8-resolution feature maps, reduces the correlation map with MaxPool4D, and applies 4D convolution for neighborhood consensus.Raw matches are obtained from row-wise and column-wise maxima of the matching score map.
  • Our Adapted NCNet: Inference retains mutually matched patches and rejects those below match-score threshold c = 0.9.The authors report that this threshold produced the best performance across tasks.
  • Training Data Processing: Training uses MegaDepth images from 196 scenes, retaining aspect ratios between [1.3, 1.7] and pairs with more than 35% visual overlap.At most 500 pairs are randomly selected per scene.
  • Training Details: For each training pair, 400 NCNet proposals undergo expansion to produce 3200 matches for the two regressors.The regressors are optimized with Adam, starting at learning rate 5e−4 for five epochs and then 1e−4 until convergence.

B. Training Ablation Study.

The ablation study evaluates feature collection, loss thresholds, and local patch expansion on HPatches homography estimation. Using all pre-final-layer features and selected thresholds improves results, while patch expansion trades slower convergence for better final accuracy.

  • B. Training Ablation Study: HPatches ablations compare correctly estimated homographies at average corner-error thresholds of 1, 3, and 5 pixels.The variants are evaluated at confidence threshold 0.5 and grouped for comparison in Table 4.
  • Training Hyper-parameters: Using features from all layers before the final feature map f4 gives the best results in feature-collection experiments.Other settings are held identical within group I.
  • Training Hyper-parameters: Thresholds of 400 and 50 perform best for viewpoint changes, while threshold 50 is better under illumination changes and overall more promising.These comparisons concern the mid-level regressor's threshold settings.
  • Training Hyper-parameters: b_θcls = b_θgeo = 50 and e_θcls = e_θgeo = 5 overall outperform the other threshold settings.The authors state that this setting is especially effective under viewpoint changes.
  • Effect of Local Patch Expansion: Patch expansion makes the best model learn faster at the same training duration, but without expansion convergence is faster at similar accuracy.With expansion, convergence is slower but reaches better accuracy.

C. Experiment Details.

The experiments specify evaluation procedures for homography estimation, localization, and image resizing. They also document a quantization step needed to integrate pairwise-regressed matches into triangulation-based localization pipelines.

  • Homography Estimation Details: Homography correctness is measured by the percentage of estimates whose average four-corner error is below 1, 3, or 5 pixels.Corners are transformed with the estimated homography and compared against ground-truth corners; homographies are estimated using pydegensac's findHomography.
  • Quantization Details: Localization pipelines require co-occurring keypoints across retrieval pairs for reliable 3D reconstruction and triangulation.Pairwise regression can produce slightly different positions for the same image across pairs such as (A, B) and (A, C).
  • Quantization Details: Matches within 4 pixels are represented by their mean location, sacrificing some pixel-level accuracy for compatibility with triangulation.Duplicate matches are then removed by retaining the match with the highest confidence score.
  • Image Size and Filtering: Patch2Pix benefits from 1024-pixel images, whereas SparseNCNet performs best at 1600 pixels.The complete results also show method-specific effects of outlier-filtering thresholds: some methods improve at 0.75, while D2Net and R2D2 perform better without thresholding.

D. Qualitative Results.

The qualitative evaluation visualizes Patch2Pix matches across internet, HPatches, PhotoTourism, Aachen Day-Night, and InLoc image pairs. It uses green and red overlays to distinguish inliers from other matches in localization examples.

  • Figure 6: Figure 6 visualizes Patch2Pix matches on internet, HPatches, and PhotoTourism image pairs using NCNet proposals and a 0.25 confidence threshold.Inliers are identified with pydegensac's homography or fundamental-matrix estimation routines, using a threshold of 2 for HPatches homographies.
  • Figures 7–9: Figures 7–9 visualize Patch2Pix matches on Aachen Day-Night and InLoc using both NCNet and SuperPoint + SuperGlue proposals.
  • InLoc Benchmark: Table 5 reports the percentage of correctly localized queries under specific error thresholds for methods evaluated within the HLOC pipeline.The table marks the best results and each method's best setting across the benchmark.
  • Localization Visualizations: For a randomly selected query, database images are chosen by the localization solver's inlier counts.Inlier matches are plotted in green, other matches in red, and the inlier numbers are reported.
Loading 2012.01909v3…