Source-linked AI summary
Learning Dense Correspondence via 3D-guided Cycle Consistency
Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qixing Huang, Alexei A. Efros
TL;DR
Dense correspondence across different object instances is hard to supervise because ground-truth labels are unavailable. The paper uses CAD-linked cycle consistency to train a ConvNet, and reports strong correspondence performance without CAD models at test time.
Problem
Dense visual correspondence across different object instances lacks readily obtainable ground-truth labels, although consistency across instances provides a possible supervisory signal.
Method
The method links real image pairs through two rendered views of an appropriate 3D CAD model and trains flows in a ground-truth-anchored correspondence 4-cycle.
Results
The consistency-supervised network significantly outperforms other methods except on bicycle and motorbike, where SIFT flow has a slight advantage.
Takeaways & Limitations
Cycle consistency can provide meta-supervision for learning tasks without direct labels, and CAD models can support correspondence learning between 2D images of different object instances.
Takeaways & Limitations
The matchability formulation assumes every pixel in each synthetic view is matchable in its corresponding real image.
Abstract
from arXiv · showhide
Discriminative deep learning approaches have shown impressive results for problems where human-labeled ground truth is plentiful, but what about tasks where labels are difficult or impossible to obtain? This paper tackles one such problem: establishing dense visual correspondence across different object instances. For this task, although we do not know what the ground-truth is, we know it should be consistent across instances of that category. We exploit this consistency as a supervisory signal to train a convolutional neural network to predict cross-instance correspondences between pairs of images depicting objects of the same category. For each pair of training images we find an appropriate 3D CAD model and render two synthetic views to link in with the pair, establishing a correspondence flow 4-cycle. We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth. At test time, no CAD models are required. We demonstrate that our end-to-end trained ConvNet supervised by cycle-consistency outperforms state-of-the-art pairwise matching methods in correspondence-related tasks.
1. Introduction
Dense correspondence across different object instances lacks readily obtainable ground-truth labels, despite its importance for several vision tasks. The paper uses 3D CAD models and cycle consistency to train a ConvNet end-to-end, then predicts real-image correspondences without CAD models at test time.
- Motivation: Dense visual correspondence estimates pixel-wise matches between images of visually similar objects or scenes, but suitable training data is difficult to obtain.The task supports optical flow, stereo matching, recognition, segmentation, and depth estimation through correspondence and label transfer.
- Motivation: Existing learning methods exploit shared structure for same-instance correspondence, while SIFT flow remained difficult to challenge for cross-instance correspondence.The cited same-instance approaches use commonality of scene structure as supervision, but do not solve correspondence across different object or scene instances.
- Approach: The paper links two real images through two rendered views of an appropriate 3D CAD model, whose synthetic-to-synthetic correspondence provides ground-truth supervision.The rendering engine precisely determines where shape points move across viewpoints, while the network must learn to apply this information to real images.
- Approach: Cycle consistency connects synthetic-to-real, real-to-real, and real-to-synthetic flows into a 4-cycle whose composition is trained against the known synthetic correspondence.Direct synthetic supervision alone does not work well because synthetic and real images differ visually and cross-instance ground truth is unavailable.
- Contributions: The paper introduces cycle consistency as meta-supervision, presents an end-to-end deep network for dense cross-instance correspondence, and uses widely available CAD models to learn from 2D images.At test time, the method does not require CAD models.
2. Related work
Prior correspondence methods span pairwise optimization, image-set alignment, deep learning for same-instance matching, and image-shape alignment through 3D models. The paper builds on these lines while addressing cross-instance correspondence with an end-to-end learned approach.
- Cross-instance pairwise correspondence: SIFT Flow and related methods compute cross-instance dense correspondence by matching local features with smoothness, displacement, proposal, or classifier-based priors.DSP Matching accelerates SIFT Flow while modestly improving accuracy; other methods extend PatchMatch or aggregate pixel-level classifiers.
- Collection correspondence: Collection correspondence methods align entire image sets through sequential entropy minimization, low-rank modeling, or collection-level flow estimation.These approaches pose correspondence jointly across image sets rather than only between image pairs.
- Deep learning for correspondence: Deep correspondence methods learn same-instance matching from synthetic optical-flow data, reconstructed 3D scenes, or stereo pairs, but remain limited to the same physical object or scene.Deep features from object classification have also been used for intra-class alignment, with performance similar to SIFT flow.
- Image-shape correspondence: Image-shape alignment methods establish image correspondence through intermediate 3D shapes, including 2D-to-CAD matching and CAD-assisted object detection.Recent depth-based methods also apply deep learning to 3D-related correspondence settings.
3. Approach
The approach jointly learns dense correspondence flow and matchability using cycle-consistency supervision from synthetic views linked to real image pairs. An encoder-decoder CNN predicts these quantities, while differentiable composition and interpolation enable end-to-end optimization.
- Learning dense correspondence and matchability: The CNN predicts both a dense flow field between image pairs and a matchability map indicating whether source pixels have correspondences in the target.The flow is a 2D offset field, while matchability handles cases where pairwise correspondence is undefined.
- Cycle-consistency supervision: For each real training pair, two CAD-rendered views provide synthetic ground truth that supervises a cycle through synthetic-to-real, real-to-real, and real-to-synthetic predictions.The predicted path is required to be consistent with the rendering engine’s synthetic-to-synthetic flow and matchability.
- Consistency objectives: Flow composition adds the first flow to a second flow sampled at the displaced location, whereas matchability composition depends on both matchability and flow.This distinction makes matchability supervision sensitive to the predicted transitive path rather than directly applying synthetic labels to real pairs.
- Consistency objectives: The training objective combines truncated Euclidean flow loss with per-pixel cross-entropy matchability loss under cycle consistency.The truncated flow loss uses T = 15 pixels and is reported as more robust to spurious outliers during noisy early training.
- Consistency objectives: Because multiplicative matchability composition can exploit clean synthetic backgrounds, the objective requires care to avoid unconstrained real-image background predictions.Zero synthetic-background matchability can make the composed value zero regardless of later predictions.
- Continuous approximation: Bilinear interpolation converts discrete CNN predictions into continuous flow and matchability functions, making transitive composition differentiable for stochastic-gradient training.The interpolation uses the four neighboring pixels of each continuous point.
- Network architecture: The network uses shared-weight feature encoding followed by separate flow and matchability decoders, and the same network is applied to all three cycle edges during training.The architecture has an 8-layer feature encoder and 9-layer decoder branches for each output.
4. Experimental Evaluation
The evaluation tests cycle-consistency training through feature visualization, keypoint transfer, matchability prediction, and shape-to-image segmentation transfer. Across these tasks, the consistency-supervised network generally improves over initialization and baselines, while retaining category- and viewpoint-related limitations.
- Training visualization: Cycle-consistency training makes transitive correspondence trajectories increasingly agree with synthetic ground truth during training.Individual correspondences along the cycle also become more semantically plausible.
- Feature visualization: The learned car feature embedding appears viewpoint-sensitive despite no explicit viewpoint-estimation objective.This suggests viewpoint becomes an implicit cue for correspondence and matchability.
- Keypoint transfer: Consistency fine-tuning significantly improves the initialized network and is more effective than synthetic direct supervision for adapting to real images.The comparison is based on the performance gap between the final consistency-fine-tuned network and synthetic fine-tuning.
- Keypoint transfer: The final consistency-supervised network outperforms the baselines on keypoint transfer overall, except for bicycle and motorbike, where SIFT flow has a slight advantage.Example visualizations show more accurate transfers overall, though SIFT flow wins in one displayed case.
- Matchability prediction: Matchability prediction reaches 67.8% mean accuracy versus 57.1% for SIFT flow, outperforming it on all categories except bicycle and motorbike.Chance performance is 50%, and matchability is evaluated from predicted correspondence existence.
- Shape-to-image segmentation transfer: The learned flows produce more accurate shape-to-image segmentation transfer than SIFT flow in the toy experiment.The experiment transfers per-pixel labels from rendered shapes to real images using predicted flow fields.
5. Discussion
The paper finds cycle consistency effective for learning dense cross-instance correspondences and presents it as a potentially general form of indirect supervision for tasks with few or no direct labels.
- Cycle consistency served as a supervisory signal for learning dense cross-instance correspondences.
- The authors suggest applying the same idea when ground truth is known for one or more edges along a cycle.
- The paper aims to encourage further work on tasks with little or no direct labels using cycle consistency or other indirect supervision.