PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Correspondence Network

Christopher B. Choy, JunYoung Gwak, Silvio Savarese, Manmohan Chandraker

arXiv:1606.03558v3cs.CV

TL;DR

Visual correspondence methods often optimize surrogate patch similarity, which lacks a direct metric interpretation and scales inefficiently. UCN learns a task-specific metric space with a fully convolutional architecture and correspondence contrastive loss, achieving strong geometric and semantic correspondence results across multiple datasets and settings.

  • Problem

    Prior CNN correspondence features optimize surrogate patch similarity rather than directly learning a metric space, and patch comparisons require O(n^2) feed-forward passes.

  • Method

    UCN uses deep metric learning, a fully convolutional network, correspondence contrastive loss, active hard-negative mining, and a convolutional spatial transformer.

  • Results

    UCN achieves state-of-the-art sparse SFM and dense geometric or semantic correspondence performance across KITTI, PASCAL, and CUB, including rigid and non-rigid bodies.

  • Takeaways & Limitations

    The learned features outperform prior approaches on geometric and semantic correspondence tasks, even without spatial priors or global optimization.

Abstract

from arXiv · show

We present a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, spanning across rigid motions to intra-class shape or appearance variations. In contrast to previous CNN-based approaches that optimize a surrogate patch similarity objective, we use deep metric learning to directly learn a feature space that preserves either geometric or semantic similarity. Our fully convolutional architecture, along with a novel correspondence contrastive loss allows faster training by effective reuse of computations, accurate gradient computation through the use of thousands of examples per image pair and faster testing with $O(n)$ feed forward passes for $n$ keypoints, instead of $O(n^2)$ for typical patch similarity methods. We propose a convolutional spatial transformer to mimic patch normalization in traditional features like SIFT, which is shown to dramatically boost accuracy for semantic correspondences across intra-class shape variations. Extensive experiments on KITTI, PASCAL, and CUB-2011 datasets demonstrate the significant advantages of our features over prior works that use either hand-constructed or learned features.

1 Introduction

Visual correspondence spans low-level geometric matching to high-level semantic alignment, but prior approaches use specialized features or surrogate patch similarity. UCN instead learns a shared correspondence metric with efficient fully convolutional computation.

  • Motivation: Visual correspondence supports tasks from 3D reconstruction and image retrieval to semantic alignment across object instances.The correspondence setting ranges from sparse matching of projections of the same 3D point to semantic analogs such as bird beaks or car headlights.
  • Limitations of prior methods: Traditional methods rely on hand-designed features or domain-specific priors, while CNN activations are often optimized for surrogate patch similarity.Such activations do not necessarily form a metric space with an explicit distance interpretation.
  • Limitations of prior methods: O(n^2) feed-forward passes are required to compare n patches with n patches in another image using typical patch-similarity methods.Overlapping patch regions also require repeated feature extraction.
  • UCN framework: UCN directly learns a feature space preserving geometric or semantic similarity and supports invariance to projective, shape, and appearance variations.At test time, correspondence becomes nearest-neighbor search in feature space rather than pairwise patch-similarity evaluation.
  • UCN framework: The fully convolutional architecture, active hard-negative mining, correspondence contrastive loss, and convolutional spatial transformer target efficient training and patch normalization.The spatial transformer is designed to compensate for affine transformations and mimic normalization used by descriptors such as SIFT.
  • Results: UCN reports state-of-the-art sparse, dense geometric, and semantic correspondence performance across KITTI, PASCAL, and CUB datasets.The evaluations cover rigid and non-rigid bodies as well as intra-class shape or appearance variations.

2 Related Works

Related work spans hand-designed descriptors, CNN similarity measures, metric learning, spatial transformation, and fully convolutional architectures. UCN combines these lines by learning dense correspondence features and using correspondence-specific training objectives.

  • Correspondence features: Hand-designed features such as SIFT, HOG, SURF, and DAISY have served as basic representations across computer vision applications.
  • UCN positioning: UCN's system overview combines dense feature extraction, a convolutional spatial transformer, L2 normalization, and correspondence contrastive loss.The loss uses positive and negative correspondence pairs to organize features in the learned space.
  • CNN similarity measures: Prior CNN approaches commonly use Siamese networks or intermediate activations to measure patch similarity or provide generic correspondence features.These approaches include applications to driving data, stereo matching, and semantic correspondence.
  • Metric learning: Neural metric-learning work uses learned mappings, contrastive losses, triplet losses, and mini-batch training to preserve semantic distances.
  • Spatial transformations: Spatial transformer networks learn transformations such as zooming, rotation, or arbitrary object transformations, complementing CNNs' partial built-in invariances.
  • Fully convolutional networks: Fully convolutional architectures reuse activations for overlapping regions, improving efficiency for dense feature extraction.UCN applies this efficiency principle to correspondence learning.

3 Universal Correspondence Network

The Universal Correspondence Network combines metric learning, fully convolutional feature extraction, correspondence contrastive loss, hard-negative mining, and a convolutional spatial transformer for efficient visual correspondence.

  • Fully Convolutional Feature Learning: Fully convolutional learning reuses activations, trains on thousands of correspondences per image pair, and extracts dense features from arbitrary-sized images.These properties improve gradient estimation and resource efficiency compared with patch-based methods.
  • Testing Efficiency: O(n) feed-forward passes replace O(n^2) testing for n keypoints, making the network faster than patch-similarity methods.The efficiency comes from extracting feature maps rather than comparing every patch pair independently.
  • Correspondence Contrastive Loss: Metric learning directly maps corresponding points to neighboring feature representations, while separating non-corresponding pairs by a margin.The correspondence contrastive loss encourages positive pairs to be close and negative pairs to be at least margin m apart.
  • Hard Negative Mining: Active hard-negative mining selects violating nearest-neighbor pairs because random negatives are generally too far apart to contribute to training.The mined negative must be far from the ground-truth correspondence location.
  • Convolutional Spatial Transformer: The convolutional spatial transformer applies an independent transformation to each keypoint’s features, mimicking patch normalization for scale, rotation, and related transformations.Unlike a global spatial transformer, it estimates transformations convolutionally for individual feature outputs and is especially important across large intra-class shape variations.

4 Experiments

Experiments evaluate UCN on geometric, semantic, and camera-motion correspondence tasks using KITTI, PASCAL, CUB, and related benchmarks. UCN generally improves matching accuracy over prior features, while the convolutional spatial transformer is especially beneficial under larger shape variation.

  • Experimental setup: UCN is evaluated on geometric correspondence, semantic correspondence, and camera localization using KITTI, MPI-Sintel, PASCAL-Berkeley, and CUB datasets.Performance is measured primarily with PCK, while camera localization uses essential-matrix decomposition errors.
  • Geometric correspondence: At PCK@10px, raw UCN nearest-neighbor correspondences outperform state-of-the-art methods on KITTI Flow 2015 and MPI-Sintel without global optimization.Several comparison methods use additional global optimization, whereas UCN uses raw nearest-neighbor matches.
  • Geometric correspondence: On KITTI raw sequences, UCN features outperform SIFT, DAISY, and KAZE with dense extraction, and remain significantly better when prior features use sparse SIFT keypoints.The convolutional spatial transformer further improves the KITTI raw PCK curves.
  • Semantic correspondence: On PASCAL-Berkeley, nearest-neighbor UCN matches outperform prior methods by a large margin for most classes and overall, even without spatial priors or global optimization.The evaluation uses PCK with a variable image-size-normalized threshold α·L.
  • Semantic correspondence: On CUB, UCN outperforms WarpNet by a large margin, while the convolutional spatial transformer significantly improves performance on both PASCAL and CUB.The transformer’s benefit is attributed to estimating optimal patch normalization under large shape deformations.
  • Camera motion estimation: For camera motion estimation, dense UCN features remain competitive with sparse SIFT despite SIFT being designed for this setting, and UCN outperforms patch-similarity features.Camera rotation and translation errors are reported as angular deviations from ground truth.

5 Conclusion

The paper presents deep metric learning for visual correspondence estimation and combines it with several architectural and training innovations. Experiments show improved speed or accuracy over prior approaches across geometric and semantic correspondence tasks, without spatial priors or global optimization.

  • Conclusion: UCN learns correspondence-specific feature spaces through deep metric learning rather than optimizing a surrogate patch-similarity objective.The framework targets both geometric and semantic correspondences.
  • Conclusion: The correspondence contrastive loss, fully convolutional architecture, active hard-negative mining, and convolutional spatial transformer improve training efficiency, gradient computation, testing speed, or patch normalization.These components provide the capabilities highlighted by the authors’ conclusion.
  • Conclusion: UCN features perform better than prior state-of-the-art methods on geometric and semantic correspondence tasks, even without spatial priors or global optimization.The conclusion also identifies rigid and non-rigid motion or shape estimation as future application areas.

A.1 Network Architecture

The network is implemented as a fully convolutional feature extractor with correspondence-specific layers operating on features and correspondence coordinates. Dense feature extraction and normalization support accurate localization and matching.

  • Backbone and normalization: The architecture uses an ImageNet-pretrained GoogLeNet trunk from conv1 through inception_4a with stride 2 in the bottom two layers and stride 1 thereafter.Channel-wise L2 normalization gives every feature unit norm and stabilizes gradients during training.
  • Correspondence layers: After inception_4a, the correspondence contrastive loss consumes features from both images together with their respective correspondence coordinates.Correspondences are densely sampled from optical flow or matched keypoints, with random negative coordinates added for sparse semantic annotations.
  • Network design: The model includes hard-negative mining, a convolutional spatial transformer, and correspondence contrastive loss, with implementation components including K-NN and channel-wise L2 normalization layers.The network remains fully convolutional rather than using flattening or fully connected layers.

A.2 Convolutional Spatial Transformer

The convolutional spatial transformer applies affine transformations across local regions and uses bilinear interpolation to produce output features from sampled input values.

  • The convolutional spatial transformer comprises multiple affine spatial transformers, with the number determined by image size.
  • Affine parameters θij map sampled input coordinates to target grid coordinates.
  • Bilinear interpolation computes each output value from four neighboring input values.
  • The formulation includes gradients with respect to transformation parameters and input features for learning.

A.3 Additional tests for semantic correspondence

Additional PASCAL VOC tests compare UCN with FlowWeb and provide further qualitative semantic matching results.

  • UCN outperforms FlowWeb on the PASCAL VOC comparison.
  • The FlowWeb comparison uses class-based data splitting because FlowWeb is optimized for an unsupervised setting.
  • Additional qualitative semantic matching results are provided in Figures A2 and A3.

A.4 Additional KITTI Raw Results

Additional KITTI raw-sequence results use a selected subset of driving data and examine PCK across camera baselines and semantic classes.

  • The KITTI subset contains 9268 frames, representing 15 minutes of driving, with Velodyne, stereo RGB, and GPS-IMU inputs.
  • Proprietary NEC segmentation data supports evaluation across different semantic classes.
  • Sequences 17, 18, and 60 were excluded because their scenes were mostly static, while sequence 93 was excluded for noisy GPS-IMU inputs.
  • PCK at 30 pixels is plotted across camera baselines, and semantic-class PCK is evaluated on KITTI raw sequences.

A.5 KITTI Dense Correspondences

The appendix presents additional qualitative nearest-neighbor correspondence results for UCN on KITTI images.

  • The results use nearest-neighbor matches to illustrate correspondence behavior.
  • The qualitative examples concern universal correspondence network outputs on KITTI images.
  • Additional examples are shown in Figure A6.

A.6 Sintel Dense Correspondences

This section presents additional qualitative nearest-neighbor correspondence results on Sintel images using the universal correspondence network. The visualizations compare query points in one frame with predicted matches in the next frame.

  • A.6 Sintel Dense Correspondences: Additional qualitative results visualize nearest-neighbor matches from the universal correspondence network on Sintel images.The results are presented in Fig. A7.
  • A.6 Sintel Dense Correspondences: Each visualization row places query points from framet beside predicted keypoint matches from framet+1.This layout is explicitly labeled in the accompanying text.
  • A.6 Sintel Dense Correspondences: The dense feature visualizations use Sintel image pairs separated by 1 frame difference.The figure descriptions show query points on the left and nearest-neighbor matches on the right.
Loading 1606.03558v3…