Source-linked AI summary
WarpNet: Weakly Supervised Matching for Single-view Reconstruction
Angjoo Kanazawa, David W. Jacobs, Manmohan Chandraker
TL;DR
Matching objects across category and pose variations is difficult without labor-intensive part annotations. WarpNet learns a spatial prior from artificially warped training pairs, improving matching and supporting single-view reconstructions comparable to supervised methods.
Problem
Matching across fine-grained categories and pose variations is difficult with appearance features alone, while part annotations and CAD models are costly, sparse, and not scalable.
Method
WarpNet predicts correspondences between two images and trains without supervised keypoint annotations using artificially warped image pairs with known thin-plate spline transformations.
Results
WarpNet consistently outperforms other methods in PCK evaluation and produces reconstructions nearly as good as those obtained using supervised annotations.
Takeaways & Limitations
The learned matching prior enables part-annotation-free single-view reconstruction in fine-grained datasets.
Takeaways & Limitations
The TPS prior may hallucinate birds of similar pose across wide baselines, and reconstruction quality depends on choosing better image subsets.
Abstract
from arXiv · showhide
We present an approach to matching images of objects in fine-grained datasets without using part annotations, with an application to the challenging problem of weakly supervised single-view reconstruction. This is in contrast to prior works that require part annotations, since matching objects across class and pose variations is challenging with appearance features alone. We overcome this challenge through a novel deep learning architecture, WarpNet, that aligns an object in one image with a different object in another. We exploit the structure of the fine-grained dataset to create artificial data for training this network in an unsupervised-discriminative learning approach. The output of the network acts as a spatial prior that allows generalization at test time to match real images across variations in appearance, viewpoint and articulation. On the CUB-200-2011 dataset of bird categories, we improve the AP over an appearance-only network by 13.6%. We further demonstrate that our WarpNet matches, together with the structure of fine-grained datasets, allow single-view reconstructions with quality comparable to using annotated point correspondences.
1. Introduction
The paper addresses matching and single-view reconstruction without part annotations despite appearance and shape variation. WarpNet learns unsupervised warps from fine-grained dataset structure and uses them as spatial priors for matching and reconstruction.
- Problem: Single-view reconstruction requires matching keypoints across fine-grained objects despite large intra-class shape and interclass appearance variations.Existing approaches rely on labor-intensive part annotations or 3D CAD models, which are sparse and not scalable.
- Matching: The network output provides a spatial prior for matching across appearance, pose, articulation, and shape variations.Fine-grained datasets support matching through appearance similarity within categories, global shape similarity across categories, and propagation across both variations.
- Approach: WarpNet predicts correspondences between object images across category and pose variations without supervised keypoint annotations.The architecture learns to warp points on one object into corresponding points on another.
- Training: Artificial correspondences are generated from fine-grained dataset structure and known thin-plate spline transformations to train WarpNet.WarpNet is a Siamese network that receives two images and learns from warped image pairs.
- Reconstruction: WarpNet matches support single-view reconstruction without part annotations, producing reconstructions nearly as good as those using supervised annotations.The reconstruction procedure also uses match propagation and heuristics based on fine-grained dataset structure.
2. Related Work
Prior work uses supervised appearance or geometric methods for matching, often requiring part, transformation, or CAD supervision. This paper instead targets deformable objects across instances and categories using learned semantic warps and dataset-derived artificial correspondences.
- Supervised matching: Supervised matching methods learn similarity metrics for same-instance viewpoints or dense flow, whereas this task matches deformable objects across instances and appearances.The task requires semantic understanding of object shape beyond local appearance.
- Spatial priors: Spatial-prior methods improve matching with labeled part distributions, but annotations are laborious and cover only a few nameable parts.Sparse annotations can be insufficient for reconstruction.
- Unsupervised matching: Unsupervised dense-matching methods are primarily geometric or emphasize global consistency, while WarpNet learns category-specific semantic properties for pairwise matching.The approaches are described as complementary, with different evaluation settings.
- Single-view reconstruction: Single-view reconstruction has progressed from ground-truth part correspondences to training-time annotations, whereas this work requires no part annotations during training or testing.The target setting is reconstruction from a single image using collections of similar objects.
- Transformation learning: Unlike transformation-learning work that uses ground-truth parameters or discretized rigid classes, WarpNet predicts continuous thin-plate spline parameters from artificial correspondences.The predicted transformation is directly used for matching rather than serving only as a feature-learning pretext.
- Transformation learning: WarpNet extends spatial transformer networks by predicting an unsupervised warping function that aligns two object instances.Its design uses a Siamese architecture and a matching objective.
3. Learning without Part Annotations
WarpNet learns to align objects across category, pose, shape, articulation, and appearance variations without supervised keypoint annotations. It is trained using artificial TPS-based correspondences and produces a spatially informed warp for matching real images.
- Architecture and objective: WarpNet predicts correspondences between two images, including objects from different categories or poses, without requiring supervised annotations.The network parameterizes the correspondence function as a thin-plate spline transformation.
- Training data: Artificial correspondences are created by applying spatial and chromatic transformations that reflect non-rigid shape changes and articulations in birds.Exemplar TPS transformations are computed between silhouettes using neighborhood relationships in the fine-grained dataset.
- Architecture and objective: During training, WarpNet receives an image pair and keypoint locations, then learns a warp that minimizes distances between corresponding points after transformation.The model is trained through point distances rather than direct supervision of TPS coefficients.
- Architecture and objective: The Siamese architecture extracts tied-weight convolutional features, combines them by subtraction, and predicts a deformed grid defining the TPS warp.Feature subtraction is used instead of concatenation to avoid substantially increasing the number of parameters and destabilizing training.
- Generalization: On real images unseen during training, WarpNet accounts for differences in bird shape, articulation, and appearance when computing object-to-object warps.The output is used to align variations such as head or tail orientation and differences between fat and skinny or small and large birds.
- Generalization: Qualitative comparisons evaluate WarpNet against an ILSVRC-trained CNN, emphasizing its relative robustness to appearance, pose, and articulation changes.The comparison is presented through sample matches rather than a quantitative result in the supplied passages.
4. Matching and Reconstruction
WarpNet supplies matches for reconstruction by combining appearance with a learned spatial prior and by propagating correspondences through structured subsets of fine-grained bird images. The reconstruction procedure addresses sparse viewpoints, deformability, dataset scale, and outlier selection.
- Matching with WarpNet: WarpNet matches combine appearance similarity with a symmetric spatial prior derived from transformations computed in both directions.Matches are ranked with a ratio test that discards points similar to many others, using densely extracted foreground keypoints.
- Single-view reconstruction: Good matches provide the key input for part-annotation-free single-view reconstruction, replacing the need for expensive supervised inputs such as part annotations or CAD models.The paper presents this as a new reconstruction problem enabled by its matching approach.
- Single-view reconstruction: CUB-200-2011 provides only 60 images per category, while preferred viewpoints and large within-category deformations make straightforward matching and viewpoint coverage difficult.The reconstruction pipeline therefore needs matches across a well-distributed set of viewpoints.
- Single-view reconstruction: Selecting reconstruction images is challenging because CUB-200-2011 contains nearly 12,000 images, creating computational demands and increased exposure to outliers.The paper contrasts this scale with earlier reconstruction settings containing fewer than 1,000 bird images.
- Single-view reconstruction: Three dataset-structure heuristics select related bird categories using shared keywords, ornithological taxonomy, or nearest neighbors in a species-similarity tree.All three heuristics target matched keypoints from multiple subcategories to improve viewpoint coverage.
- Single-view reconstruction: The reconstruction pipeline retains high-precision matches between pose-graph neighbors, propagates them through a virtual view network, and ignores pairs with too few surviving matches.It uses an 85% precision threshold and requires at least 50 surviving matches for retained image pairs.
5. Experiments
Experiments on CUB-200-2011 evaluate WarpNet for correspondence matching and weakly supervised reconstruction. WarpNet improves matching over appearance-based baselines and produces reconstructions close to supervised results, while exemplar-TPS training is important for high-precision matching.
- Experimental setup: Experiments use CUB-200-2011 bird images, pose-graph neighbors, annotated or expanded correspondences, and reconstruction comparisons against supervised, VGG-M, and DSP methods.The test set contains 5000 image pairs with more than 50k ground-truth matches; evaluation uses PCK and precision-recall metrics.
- Match evaluation: WarpNet-coseg outperforms baseline VGG-M by 10.8%, showing robustness when ground-truth segmentations are replaced by co-segmentation outputs.The experiment tests applicability without ground-truth segmentation masks.
- Match evaluation: WarpNet achieves 53.4% AP, a 13.6% increase over VGG-M conv4 appearance matching.Its higher recall is attributed to the learned spatial prior, which does not use part annotations.
- Match evaluation: WarpNet consistently outperforms other methods on PCK, with a growing advantage over VGG-M conv4 as the matching tolerance α increases.At 85% precision, the comparison emphasizes recall useful for reconstruction because outliers must be avoided.
- Choice of transformations: Exemplar-TPS transformations are critical: WarpNet outperforms AffineNet, whose predicted warps are helpful generally but not precise enough to improve recall at high precision.AffineNet uses affine transformations for artificial training data while retaining TPS output capacity.
- Single-view object reconstruction: WarpNet reconstructions are most consistent with the supervised approach, whereas VGG-M and DSP produce noisy results and struggle with articulated tails and wings.The supervised method uses annotated correspondences and serves as an upper bound; WarpNet uses no part correspondences.
- Single-view object reconstruction: A limitation is that the TPS prior can hallucinate similarly posed birds across wide baselines, although better image selection may reduce this problem.The reconstruction pipeline also faces challenges from matching errors and extreme outlier points in competing methods.
6. Conclusions and Future Work
The paper concludes that WarpNet enables annotation-free matching and reconstruction by predicting warps between objects and exploiting fine-grained dataset structure. It reports improved matching and reconstruction quality comparable to supervised methods, while identifying image-subset selection and incremental reconstruction as future challenges.
- Conclusions: WarpNet predicts a function that warps one object into another and can be trained without supervised part annotations.The method exploits fine-grained dataset structure to train the network and uses its output as a spatial prior.
- Conclusions: The approach achieves significant improvements over prior state-of-the-art without part annotations and produces reconstructions similar in quality to supervised methods.The conclusion presents matching and reconstruction as the two demonstrated outcomes.
- Future work: Future work includes selecting optimal image subsets and determining an image-addition order for incremental reconstruction with bundle adjustment.These are identified as key challenges for extending the reconstruction pipeline.