Source-linked AI summary
Repeatability Is Not Enough: Learning Affine Regions via Discriminability
Dmytro Mishkin, Filip Radenovic, Jiri Matas
TL;DR
The paper addresses the gap between geometric repeatability and reliable matching by learning affine regions with descriptor-based objectives. It introduces the hard negative-constant loss and a weakly supervised AffNet estimator, which improves affine-covariant matching, image retrieval, and wide-baseline performance.
Problem
Geometric repeatability does not guarantee that local regions are reliably matched, so affine-shape learning must account for descriptor matchability.
Method
The paper learns affine shape and related parameters weakly supervised with the hard negative-constant descriptor loss, without requiring precisely aligned patches.
Results
AffNet improves affine-detector repeatability, image retrieval, and wide-baseline matching, outperforming prior affine-shape methods and improving the state of the art in retrieval.
Takeaways & Limitations
Learning local regions should optimize descriptor matchability alongside geometric properties rather than relying on repeatability alone.
Abstract
from arXiv · showhide
A method for learning local affine-covariant regions is presented. We show that maximizing geometric repeatability does not lead to local regions, a.k.a features,that are reliably matched and this necessitates descriptor-based learning. We explore factors that influence such learning and registration: the loss function, descriptor type, geometric parametrization and the trade-off between matchability and geometric accuracy and propose a novel hard negative-constant loss function for learning of affine regions. The affine shape estimator -- AffNet -- trained with the hard negative-constant loss outperforms the state-of-the-art in bag-of-words image retrieval and wide baseline stereo. The proposed training process does not require precisely geometrically aligned patches.The source codes and trained weights are available at https://github.com/ducha-aiki/affnet
1 Introduction
The paper argues that affine-feature learning should optimize descriptor matchability, not geometric repeatability alone, and proposes weakly supervised affine-shape learning with a descriptor-based loss. AffNet is presented as improving affine estimation, image retrieval, and wide-baseline matching while avoiding the feature-count losses of classical affine adaptation.
- Motivation: Affine covariance supports robust matching across wide baselines and provides stronger geometric constraints than scale-covariant features.The Hessian-Affine detector with RootSIFT is described as the image-retrieval gold standard.
- Motivation: Classical affine adaptation fails in 20%-40% of cases and is not robust to significant illumination change.These failures reduce detected-feature counts and repeatability, motivating alternatives for correspondence-heavy pipelines.
- Contribution: Geometric repeatability alone is insufficient for successful matching because learned affine shapes must steer estimators toward discriminative regions.The paper therefore argues for optimizing a descriptor-related loss when learning affine shape.
- Contribution: The hard negative-constant loss combines advantages of triplet and contrastive positive losses for descriptor-based registration and learning.The paper also proposes learning affine shape, orientation, and potentially other geometric or appearance parameters without precise ground truth.
- Contribution: AffNet significantly outperforms prior affine-shape methods and improves image retrieval by a large margin without significantly reducing detected-feature counts.This makes it suitable for pipelines that need affine invariance only occasionally.
- Related work: Recent related methods learn descriptors, orientations, or translation-covariant detectors, but affine-shape learning remains comparatively unaddressed.The paper distinguishes its approach from methods requiring exact patch geometry, predefined anchors, many sampled shapes, or dense same-image negatives.
2 Learning affine shape and orientation
The paper learns affine regions by optimizing descriptor-based matchability rather than geometric repeatability alone, using affine parameterizations, descriptor losses, and the HardNegC objective. AffNet predicts affine transformations from imperfectly aligned transformed patches and is trained without precise geometric ground truth.
- Affine shape parametrization: An affine local frame has six parameters; translation (x, y) comes from the detector, while the method focuses on the affine transformation matrix A.The affine matrix is parameterized using scale, orientation, and a unit-determinant affine shape with a residual shape component.
- Affine shape parametrization: Different affine transformation parameterizations significantly influence the performance of CNN-based local-geometry estimators.The paper reports this comparison in Table 2.
- Descriptor losses: HardNegC treats the hardest negative distance as constant while optimizing a triplet-style descriptor loss.Its loss uses matching-descriptor distance and hardest-negative distance, with the derivative through the negative distance set to zero.
- Descriptor losses: 50% correct matches remain possible even with perfect geometric registration, showing that repeatability alone does not ensure descriptor matchability.Descriptor-driven optimization can raise matching to up to 90% under illumination changes, although independent region deformation increases geometric error.
- Descriptor losses: PosDist tends to squash regions and produce larger geometric error, whereas HardNeg and HardNegC preserve more non-collapsed regions and achieve better registration behavior.HardNeg gives the best number of surviving features and geometric error in the cited experiment; HardNegC performs slightly worse geometrically but slightly better on matching score.
- AffNet training procedure: AffNet training applies independent random affine transformations to matching patches, predicts transformations toward an unknown canonical shape, and feeds spatially normalized patches to a descriptor network.Descriptors form an n × n distance matrix used to construct triplets for HardNegC; training uses 10M pairs from UBC Phototour despite imperfect alignment and affine noise.
3 Empirical evaluation
Experiments show that AffNet’s descriptor-based learning and parameterization choices improve matching, repeatability, feature survival, wide-baseline stereo, and image retrieval.
- Learning losses and descriptors: HardNegC was the only tested loss without non-converged results; converged losses and descriptors achieved comparable performance in this experiment.The authors attribute this to the CNN producing the same affine transformation for a patch.
- Parameterization: Residual-shape parameterization performed best overall, while jointly learning affine shape and orientation failed or underperformed the two-stage procedure.The two-stage procedure learns affine shape first, then estimates orientation on the normalized patch.
- Repeatability: AffNet slightly outperformed Baumberg on viewpoint changes and performed nearly like plain Hessian under illumination changes, while increasing correspondences more substantially.The repeatability evaluation used HSequences, comprising illumination and viewpoint subsets, with correctness defined by overlap error below 40%.
- Repeatability: 0.52, 0.47, and 0.56 were AffNet repeatabilities for DoG, Harris, and Hessian detectors, respectively, outperforming Baumberg for all three detectors.The corresponding Baumberg values were 0.51, 0.44, and 0.52.
- Feature survival: 25% more features survived with AffNet because its estimated shapes were less elongated than Baumberg’s, whose average axis ratio was 1.99 versus 1.63 for AffNet.Features were rejected for excessive elongation, boundary contact, or failure to converge within sixteen iterations.
- Stereo and retrieval: AffNet outperformed Baumberg in registered image pairs and/or correct inliers across all tested wide-baseline stereo datasets, and achieved the best local-feature retrieval results on Oxford5k and Paris6k.HesAffNet increased used features from 12.5M to 17.5M on Oxford5k and from 15.6M to 21.2M on Paris6k.
4 Conclusions
The paper concludes that descriptor matchability should guide local-feature learning alongside repeatability, with weakly supervised AffNet improving several affine-feature benchmarks.
- The method learns affine feature shape in a weakly supervised manner.
- Descriptor matchability, not only repeatability, should be considered when learning a feature detector.