PaperlayerView exact revision on arXiv

Source-linked AI summary

GIFT: Learning Transformation-Invariant Dense Visual Descriptors via Group CNNs

Yuan Liu, Zehong Shen, Zhixuan Lin, Sida Peng, Hujun Bao, Xiaowei Zhou

arXiv:1911.05932v1cs.CV

TL;DR

Local correspondence estimation needs distinctive descriptors that tolerate geometric changes, but conventional CNN descriptors are not naturally invariant and feature pooling may reduce distinctiveness. GIFT uses group convolutions on features from transformed images, then bilinear pooling, and outperforms traditional and learned descriptors on correspondence benchmarks.

  • Problem

    Local correspondence estimation needs descriptors that encode distinctive patterns while tolerating geometric and photometric changes, whereas CNN descriptors are not naturally invariant to geometric transformations.

    Source: §1 · p. 1

  • Method

    GIFT processes features extracted from transformed images with group convolutions that encode group-feature structures, then bilinearly pools intermediate representations into invariant descriptors.

    Source: §1 · p. 2

  • Results

    GIFT outperforms traditional and recent learned descriptors on HPSequence and SUN3D correspondence estimation and shows robustness to extreme scale and orientation changes on new datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GIFT descriptors can be used with an interest-point detector for sparse matching or a uniformly sampled grid for dense matching, without requiring covariant patch detectors.

    Source: §1 · p. 2

    GIFT considers only the rotation-and-scaling subgroup of affine transformations, and its discrete-sampling boundary effects may affect matching beyond reasonable scale and rotation ranges.

    Source: §3 · p. 3, §3.4 · p. 5

Abstract

from arXiv · show

Finding local correspondences between images with different viewpoints requires local descriptors that are robust against geometric transformations. An approach for transformation invariance is to integrate out the transformations by pooling the features extracted from transformed versions of an image. However, the feature pooling may sacrifice the distinctiveness of the resulting descriptors. In this paper, we introduce a novel visual descriptor named Group Invariant Feature Transform (GIFT), which is both discriminative and robust to geometric transformations. The key idea is that the features extracted from the transformed versions of an image can be viewed as a function defined on the group of the transformations. Instead of feature pooling, we use group convolutions to exploit underlying structures of the extracted features on the group, resulting in descriptors that are both discriminative and provably invariant to the group of transformations. Extensive experiments show that GIFT outperforms state-of-the-art methods on several benchmark datasets and practically improves the performance of relative pose estimation.

1 Introduction

Local correspondence estimation needs descriptors that remain distinctive while tolerating geometric and photometric changes. GIFT uses group convolutions rather than pooling alone to build discriminative descriptors invariant to a transformation group, and is reported to outperform traditional and learned descriptors on correspondence benchmarks.

1 Introduction

  • Reliable local correspondences require descriptors that encode distinctive patterns while remaining invariant to geometric and photometric changes.Source: §1 · p. 1
  • Traditional pipelines rely on covariant patch detectors, while dense CNN descriptors remain sensitive to viewpoint changes and augmentation does not guarantee invariance on unseen data.Reliable covariant patches can be scarce, and hand-crafted detectors are sensitive to artifacts and lighting.Source: §1 · p. 1
  • GIFT treats features from transformed images as group-defined patterns and uses group convolutions to preserve their local structure before bilinear pooling yields invariant descriptors.Pooling alone may reduce feature distinctiveness; GIFT descriptors can be used with interest-point detectors or a uniform grid.Source: §1 · p. 2
  • GIFT outperforms traditional and recent learned descriptors on HPSequence and SUN3D correspondence estimation, and is robust to extreme scale and orientation changes on new datasets.The unoptimized implementation runs at approximately 15 fps on a GTX 1080 Ti GPU.Source: §1 · p. 2

2 Related work

Prior correspondence methods use covariant patch detectors, pool features from transformed inputs, or extract descriptors directly from CNN feature maps. GIFT instead processes transformed-image features with separate group CNNs, exploiting their group structure rather than pooling it away.

2 Related work

  • Traditional matching pipelines detect covariant patches before extracting invariant descriptors, but reliably detected patches are scarce and consistent scale or orientation remains challenging.Source: §2 · p. 2
  • Methods such as A-SIFT search transformed patches, while other approaches pool transformed-patch features to obtain invariance; GIFT uses group convolutions to retain feature structure.The passage identifies pooling as a potential loss of descriptor distinctiveness.Source: §2 · p. 2
  • CNN feature-map descriptors are not naturally invariant to geometric transformations, and augmentation cannot guarantee invariance on unseen data.UCN uses a spatial transformer to normalize patches, but learning an invariant transformer is as difficult as learning a viewpoint-covariant detector.Source: §2 · p. 2
  • Unlike Group Equivariant CNNs that apply group convolutions over the product of translation and geometric transformation groups, GIFT uses a vanilla CNN followed by separate group CNNs.The paper describes this separation as more efficient than the original Group Equivariant CNN.Source: §2 · p. 3, Figure 1 · p. 3

3 Method

GIFT represents transformed-image features as a function on a transformation group, then uses group CNNs and bilinear pooling to produce descriptors that are discriminative and invariant to transformations in the considered group.

3.1 Group feature extraction

  • GIFT extracts a group feature by transforming the image, processing it with a vanilla CNN, and sampling the feature map at the transformed interest-point location.The resulting feature is equivariant: transforming the image permutes the group feature, and matched points’ group features are related by such a permutation.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 Group convolution layer

  • Discrete group convolutions encode local structures in group features while preserving their equivariance, allowing multiple group convolution layers to be stacked.The convolution uses learnable weights over a set of transformations around the identity; for the 2D translation group, it reduces to conventional 2D convolution.Source: §3.2 · p. 4

3.3 Group bilinear pooling

  • Two group CNNs process each group feature, and bilinear pooling combines their outputs into the GIFT descriptor, which is invariant to transformations in the group.The authors use bilinear pooling because it collects second-order feature statistics and generalizes statistics used by several prior invariant-descriptor methods.Source: §3.3 · pp. 4–5

3.4 Implementation details

  • With five sampled scales and five orientations, GIFT’s group feature is a tensor whose entries represent an interest point across transformations; discrete sampling weakens the formal equivariance guarantees near range boundaries.The authors report that this boundary effect does not obviously affect matching when scale and rotation changes remain in a reasonable range.Source: §3.4 · p. 5
  • The implementation uses six group convolution layers for state-of-the-art comparisons and produces a 128-dimensional, L2-normalized descriptor; training minimizes a triplet loss with hard-negative mining.The group is sampled at five scales and five orientations, and the triplet-loss margin is set to 0.5.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate GIFT on correspondence estimation, including extreme scale and rotation changes, dense matching, and relative pose estimation. GIFT outperforms state-of-the-art correspondence methods, while GIFT-F outperforms other methods on most MVS sequences.

4.1 Datasets and Metrics

  • Evaluation spans HPSequences, SUN3D, synthetic extreme-scale and rotation sets, and MVS; PCK uses a five-pixel threshold and pose uses rotation error.GIFT trains on warped MS-COCO images and is finetuned on GL3D for relative pose estimation.Source: §4.1 · pp. 5–6

4.2 Ablation study

  • GIFT-1 leads all datasets except Illum-HP, bilinear pooling beats the tested alternatives, and performance rises with more group-convolution layers.The layer comparison tests 1, 3, and 6 layers; subsequent experiments use GIFT-6.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.3 Comparison with state-of-the-art methods

  • GIFT outperforms Superpoint, GeoDesc, and LF-Net on the evaluated datasets, including HPSequences, SUN3D, and extreme-rotation and scaling sets.The comparison uses the same keypoints as the compared method; Table 4 summarizes PCK results.Source: §4.3 · pp. 7–8, Table 4 · p. 8
  • GIFT’s PCK drops slowly as synthetic scaling and rotation increase on HPatches.GIFT-SP uses Superpoint keypoints and GIFT-DoG uses DoG keypoints.Source: §4.3 · p. 8, Figure 5 · p. 9

4.4 Performance for dense correspondence estimation

  • For dense correspondence estimation, GIFT outperforms Vanilla CNN and Daisy on HPSequence, ES-HP, and ER-HP.Figure 4 displays only correctly estimated correspondences, with matched points sharing a color across the two images.Source: §4.4 · p. 8, Table 5 · p. 8, Figure 4 · p. 8

4.5 Performance for relative pose estimation.

  • On most MVS sequences, GIFT-F outperforms the other methods for relative pose estimation using rotation error.GIFT-F is finetuned on GL3D; translations are not evaluated because they are up-to-scale.Source: §4.5 · pp. 8–9, Table 6 · p. 9

4.6 Running time

  • For 480×360 images with 1024 interest points, GIFT-6 takes about 65.2 ms on an Intel i7 3.7GHz CPU and GTX 1080 Ti GPU.Image warping takes 32.5 ms, vanilla-CNN processing 27.5 ms, and group-feature embedding 5.2 ms.Source: §4.6 · p. 9

5 Conclusion

GIFT is a dense descriptor with provable invariance to a transformation group. Group CNNs encode stable, discriminative group-feature structures, and the paper reports state-of-the-art correspondence-estimation performance across several datasets.

  • GIFT group features are stable under transformations and discriminative among different interest points.Source: §5 · p. 9
  • Group CNNs encode those structures, and bilinear pooling constructs the transformation-invariant descriptors.Source: §5 · p. 9
  • GIFT achieves state-of-the-art correspondence-estimation performance on HPSequence, SUN3D, and datasets with extreme scale and orientation changes.Source: §5 · p. 9

Related papers

Loading 1911.05932v1…