PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Hough Matching Networks

Juhong Min, Minsu Cho

arXiv:2103.16831v1cs.CV

TL;DR

Visual correspondence under large image variations still requires geometric matching. This paper introduces convolutional Hough matching, which distributes candidate-match similarities over a geometric transformation space and evaluates them convolutionally. CHMNet outperforms state-of-the-art methods on standard semantic correspondence benchmarks.

  • Problem

    Geometric matching remains essential for reliable visual correspondence under large variations, including viewpoint and illumination changes, blur, occlusion, and lack of texture.

    Source: §1 · p. 1

  • Method

    CHM distributes candidate-match similarities over a geometric transformation space using convolutional evaluation and a trainable semi-isotropic high-dimensional kernel.

    Source: §1 · p. 1

  • Results

    CHMNet outperforms state-of-the-art methods on standard semantic correspondence benchmarks, including a 9%p PCK (αbbox = 0.1) gain over [47] on SPair-71k (F).

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    CHM provides a convolutional Hough-transform perspective for geometric matching with interpretable high-dimensional kernels.

    Source: §6 · p. 8

    Training assumes keypoint-match annotations for each image pair.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

Despite advances in feature representation, leveraging geometric relations is crucial for establishing reliable visual correspondences under large variations of images. In this work we introduce a Hough transform perspective on convolutional matching and propose an effective geometric matching algorithm, dubbed Convolutional Hough Matching (CHM). The method distributes similarities of candidate matches over a geometric transformation space and evaluate them in a convolutional manner. We cast it into a trainable neural layer with a semi-isotropic high-dimensional kernel, which learns non-rigid matching with a small number of interpretable parameters. To validate the effect, we develop the neural network with CHM layers that perform convolutional matching in the space of translation and scaling. Our method sets a new state of the art on standard benchmarks for semantic visual correspondence, proving its strong robustness to challenging intra-class variations.

1. Introduction

Geometric matching remains essential for visual correspondence under large image variations, but earlier Hough-based neural methods are limited by non-learnability and position-invariant voting. CHM uses convolutional voting in a geometric transformation space, and CHMNet outperforms state-of-the-art methods on standard semantic correspondence benchmarks.

1. Introduction

  • Earlier neural Hough-matching methods are neither fully differentiable nor learnable and are vulnerable to background clutter because their Hough space is position-invariant.Geometric matching remains essential for image pairs with variations such as viewpoint, illumination, blur, occlusion, and lack of texture.Source: §1 · p. 1
  • CHM distributes candidate-match similarities over a geometric transformation space and evaluates them convolutionally with a learnable semi-isotropic high-dimensional kernel.Its position-aware surrounding context makes the output translation-equivariant in transformation space and brings robustness to background clutter.Source: §1 · p. 1, Figure 1 · p. 1
  • CHMNet performs geometric matching in translation and scaling space and outperforms state-of-the-art methods on standard semantic correspondence benchmarks.The approach is reported as robust to challenging intra-class variations.Source: §1 · p. 2

2. Related Work

Prior work applies Hough voting to geometric matching and 4D convolution to visual correspondence, but these approaches have limitations in local consensus, parameter cost, and transformation scope. CHM combines high-dimensional Hough voting with a lightweight trainable layer and a translation-and-scaling CHMNet.

Hough transformation

  • Earlier geometric Hough matching methods seek global offset consensus rather than local, match-specific consensus, making them less accurate and more vulnerable to clutter.The literature also describes Hough matching as effective for correspondence with intra-class variations.Source: §2 · p. 2

Semantic visual correspondence

  • Semantic correspondence methods progressed from hand-crafted descriptors to CNN-based correlation and geometric models, including approaches using 2D convolutions on correlation tensors.The surveyed approaches include learned global transformation estimation, offset-aware kernels, and coarse-to-fine matching.Source: §2 · p. 2

4D convolution for visual correspondence

  • CHM extends Hough matching to high-dimensional convolution with one interpretable semi-isotropic kernel per layer, using few parameters while learning a reliable voting strategy.Prior 4D-convolution approaches use many parameters and consider only translation.Source: §2 · p. 2

Related work contributions

  • CHM introduces a Hough-transform perspective on convolutional matching and performs high-dimensional voting convolutionally.The method is developed as a trainable layer with a semi-isotropic high-dimensional kernel for non-rigid matching.Source: §2 · p. 2
  • CHMNet performs geometric matching in translation and scaling space using 6D convolution.The paper reports state-of-the-art results on standard semantic correspondence benchmarks and robustness to intra-class variations.Source: §2 · p. 2

3. Convolutional Hough Matching

Convolutional Hough Matching turns geometric voting over candidate correspondences into a trainable convolutional layer. Its local, position-sensitive voting and compact kernel extend Hough matching across transformation dimensions while retaining interpretable weights.

3.1. Hough matching & its convolutional extension

  • A global Hough voting map cannot assess each candidate match's reliability and can raise scores for distant outliers sharing an inlier's offset.The Hough space represents possible translation and scale offsets between regions, where candidate matches cast votes.Source: §3.1 · p. 3
  • CHM creates a local voting space for each candidate match, using neighboring regions to produce a match-specific score from votes at zero offset.This replaces the global voting map shared across candidate matches.Source: §3.1 · p. 3
  • The generalized convolutional form reduces to 4D convolution when restricted to center translation and can extend beyond 6D with rotation or shear.Figure 2 depicts geometric voting in 6D space, including translation and scale.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Convolutional Hough matching layer

  • The CHM layer learns a convolution kernel, relaxing full isotropy so weights can depend on distances from the kernel position as well as match offset.Its position-sensitive isotropic kernel shares parameters for matching triplets of offset and distances from the kernel position.Source: §3.2 · p. 4
  • CHM layers can be stacked and used with neural-network layers that compute image correlations, while parameter sharing keeps their 6D kernels compact and interpretable.Each kernel element represents a voting weight for a neighboring match in local offset space.Source: §3.2 · p. 4
  • The learned kernel visualization assigns higher weights to matches with smaller offsets and closer distances.The visualization decomposes the 6D kernel into maps of parameter values for shared offsets.Source: §3.2 · p. 4

4. Convolutional Hough Matching Networks

CHMNet combines high-dimensional correlation computation, convolutional Hough matching, and flow formation with keypoint transfer. Its pipeline builds multi-scale correlations, votes over translation and scale, and converts matching scores into dense correspondences for training against annotated keypoints.

4.1. High-dimensional correlation computation

  • Multi-scale feature pairs produce 4D correlation tensors, which are interpolated to a common size and assembled into a 6D tensor for CHM.The feature maps are resized at multiple scales, with reduced channel dimensions for the subsequent correlation computation.Source: §4.1 · p. 4

4.2. Convolutional Hough Matching

  • CHM convolves the 6D correlation tensor to vote over translation and scale, then uses max-pooling to select dominant scale votes before another CHM stage.The first CHM uses a semi-isotropic kernel; the 6D input is reduced to 4D by scale-space max-pooling.Source: §4.2 · pp. 4–5

4.3. Flow formation & keypoint transfer

  • Softmax-normalized correlation scores and kernel soft-argmax transfer regular-grid coordinates into a dense flow field at sub-pixel resolution.A Gaussian kernel is centered at the highest-scoring match, and the probability map weights the transferred coordinates.Source: §4.3 · p. 5
  • The soft sampler assigns a keypoint match using distance-weighted transferred coordinates, addressing the fine-localization limits of discrete sampling.Weights are inversely proportional to distance from the queried keypoint, within a threshold.Source: §4.3 · p. 5

4.4. Training objective

  • Training assumes annotated keypoint pairs and minimizes the average Euclidean distance between predicted and ground-truth keypoints.The predicted pairs are obtained using the keypoint-transfer scheme.Source: §4.4 · p. 5

5. Experimental Evaluation

CHMNet outperforms prior methods on the reported semantic-correspondence benchmarks, while its ablations support scale-space voting and a lightweight kernel design. Results also show a trade-off between memory and computation and characterize the model’s behavior on background clutter.

5. Experimental Evaluation

  • Evaluation uses SPair-71k’s 70,958 pairs with large viewpoint and scale variation, plus PF-PASCAL and PF-WILLOW, whose pairs have smaller such variations.On SPair-71k, models are evaluated either finetuned on SPair-71k or transferred from PF-PASCAL; PF-PASCAL-trained models are also tested on PF-WILLOW.Source: §5 · p. 6, §5.1 · p. 6

5.1. Results and analysis

  • 9%p higher PCK (αbbox = 0.1) than [47] on SPair-71k (F); on PF-PASCAL, CHMNet improves by 4.4%p and 0.9%p at αimg ∈{0.05, 0.1}.The reported comparison is from Table 1; the paper also reports robust transfer performance on SPair-71k (T) and PF-WILLOW.Source: §5.1 · p. 6, Table 1 · p. 6
  • 19.6 vs. 44.9 GFLOPs for CHM’s {1,1} channels versus {16,16,1} in 4D-convolution models, though CHM uses more memory than some compared models.An optimized nD-convolution implementation enables 54ms inference without increasing FLOPs or memory.Source: §5.1 · p. 6
  • CHMNet outperforms comparison methods on precision-recall curves, and its matching scores discriminate semantic parts from background clutter.The clutter evaluation assigns matches using each method’s own keypoint-transfer scheme and uses segmentation masks or bounding boxes to distinguish true from false positives because dense-flow annotations are absent.Source: §5.1 · p. 7

5.2. Ablation study and analysis

  • PCK rises from 43.9 to 44.5 on SPair-71k when the proposed semi-isotropic kernel replaces the full isotropic kernel, and reaches 46.4 with 6D voting.The comparison uses models with 110 versus 1,250 parameters for the first kernel comparison; the 6D result extends voting into scale-space.Source: §5.2 · p. 7
  • Maximum votes concentrate around the center scale on PF-PASCAL and PF-WILLOW, but span different scales on SPair-71k.The paper relates this pattern to the datasets’ smaller versus larger scale variations, respectively.Source: §5 · p. 6, §5.2 · p. 7
  • Removing CHM or replacing it with global offset voting produces unreliable matches with background outliers, while omitting the last CHM layer harms refinement of upsampled correlation scores.The reported module ablation also attributes reliable keypoint transfer to kernel G and soft sampler A(k), which suppress noisy scores and aggregate neighborhood transfers.Source: Table 3 · p. 8
  • Increasing channel size rarely improves performance and typically harms prediction quality for the tested k4D-4D kernels.The paper argues that high-dimensional convolution learns a reliable voting strategy rather than diverse correlation-tensor patterns; CHM layers have 247–654 times fewer parameters than prior 4D-convolution layers.Source: Table 3 · p. 8, §5.2 · p. 8

6. Conclusion

The conclusion presents CHMNet as a high-dimensional geometric matching model and reports benefits across standard semantic-correspondence benchmarks. It frames CHM as both a generalization of existing 4D convolutions and a Hough-transform perspective on convolutional matching.

6. Conclusion

  • CHMNet uses CHM in a high-dimensional geometric transformation space to establish visual correspondence, and experiments on standard benchmarks demonstrate benefits.The conclusion describes CHM as generalizing existing 4D convolutions and offering a Hough-transform perspective with interpretable high-dimensional kernels.Source: §6 · p. 8

Appendix A. Additional results and analyses

Additional analyses relate scale-space voting patterns to dataset scale variation and inspect learned kernel weights. The proposed kernel’s parameter sharing is reported to benefit efficiency and accuracy.

Appendix A. Additional results and analyses

  • Scale-space vote winners come from diverse positions for SPair-71k, but mostly from original scales for PF-PASCAL and PF-WILLOW, consistent with their relative scale variations.Maxpooled positions can also depend on the scales of object parts.Source: Appendix A · p. 12
  • Learned k6D-4D kernel weights are higher for matches with large offsets and closer distance, and several of its 4D maps show similar patterns.The fully isotropic kernel cannot express equally diverse weight patterns because of its parameter-sharing constraint.Source: Appendix A · p. 12
  • The proposed kernel’s parameter-sharing strategy benefits efficiency and accuracy in the reported Table 2 results.Source: Appendix A · p. 12

Appendix B. More implementation details

The appendix describes coordinate normalization and training settings, then details an efficient implementation of 4D and higher-dimensional convolution. The revised 4D implementation reduces convolution operations and runtime relative to the original.

Coordinate normalization

  • Coordinate normalization bounds intermediate output coordinates to [−1, 1] to ensure numerical stability of loss gradients.The coordinates P form a dense regular grid used for flow formation.Source: Appendix B · p. 12

Hyperparameters

  • Training uses learning rates of 1e-3 for CHM layers and 1e-5 for the backbone, with batch size 16, threshold τ = 0.1, and Gaussian-kernel standard deviation 17.The Gaussian kernel is specified as G ∈ R30×30.Source: Appendix B · p. 12

Implementation of high-dimensional convolution

  • The faster 4D implementation reshapes the minibatch correlation tensor, applies 3D convolutions to k copies, then reshapes and combines their outputs.The original implementation instead performs H rounds of 3D convolutions.Source: Appendix B · p. 12
  • The revised 4D convolution uses H times fewer 3D convolution operations than the original implementation, with about 0.7 ms versus 150 ms on the stated machine.For 6D convolution with input size R16×15×15×3×15×15×3, the implementation takes about 180 ms on the same machine.Source: Appendix B · p. 12

Implementation details

  • The appendix also describes manually implementing parameter-sharing kernels and constructing the isotropic kernel by assigning parameters into a zero-filled high-dimensional kernel.The text notes that channel sizes are omitted for brevity.Source: Appendix B · p. 12

Appendix C. Qualitative results

Qualitative examples show CHMNet finding reliable matches across multiple instances and background clutter, and aligning objects across challenging image differences. The appendix also visualizes failure cases, scale-space pooling positions, and learned high-dimensional kernels.

Appendix C. Qualitative results

  • CHMNet's top 300 confident matches lie mostly on common instances, while replacing CHM layers with regularized Hough matching or removing them yields noisy or failed localization.The examples include multiple instances and PF-PASCAL pairs, where CHMNet also distinguishes semantic parts from background clutter.Source: Appendix C · p. 13, Figure A1 · p. 13, Figure A8 · p. 16
  • Warped source images align to target objects with large viewpoint, illumination, and scale differences, using correspondences and thin plate spline transformations.The qualitative comparisons show PF-PASCAL and SPair-71k examples against DHPF, ANC-Net, HPF, DCCNet, and NCNet.Source: Appendix C · p. 13, Figure A9 · p. 16, Figure A10 · p. 17
  • The appendix identifies representative CHMNet failure cases under extreme viewpoint changes, large intra-class variation, and deformation.Ground-truth keypoints are shown as circles and predicted keypoints as crosses, with lines depicting matching error.Source: Figure A2 · p. 13, Appendix C · p. 13
  • Scale-space visualizations encode max-pooled positions through circle size, with large circles indicating position (2) and smaller circles indicating other positions.The examples show given source keypoints, predicted target keypoints, and ground-truth target keypoints.Source: Figure A3 · p. 14, Appendix C · p. 14
  • Learned-kernel visualizations encode larger weights with darker offsets and decompose high-dimensional kernels into 4D kernels for display.The illustrated k6D-4D ψ kernel has four 4D kernels with 55 parameters each; the full and isotropic variants have nine with 625 and three with 15, respectively.Source: Figure A4 · p. 14, Figure A5 · p. 15, Figure A6 · p. 15
Loading 2103.16831v1…