Source-linked AI summary
LATCH: Learned Arrangements of Three Patch Codes
Gil Levi, Tal Hassner
TL;DR
Local binary descriptors are efficient but can lose performance because pixel-pair comparisons are sensitive to noise and local appearance changes. LATCH replaces those comparisons with learned, supervised arrangements of pixel-patch triplets and is reported to outperform other binary descriptors while narrowing the gap with larger histogram-based methods. It retains binary descriptors’ fast matching and small storage requirements, with a minor extraction-time increase.
Problem
Local descriptors must represent keypoints discriminatively and invariantly while keeping computation, storage, and matching costs low; existing binary descriptors are vulnerable to noise and local appearance changes.
Method
LATCH compares triplets of pixel patches and uses labeled training data to select informative, low-correlation triplet arrangements for binary descriptor bits.
Results
LATCH outperforms other binary descriptors by wide margins and narrows their performance gap with larger histogram-based representations.
Takeaways & Limitations
LATCH retains binary descriptors’ fast matching and small storage requirements while providing improved reported performance, at a minor increase in extraction-time computation.
Takeaways & Limitations
The reported reconstruction evaluation focuses on descriptor quality rather than final reconstruction quality, which could be improved by subsequent dense multi-view stereo processing.
Abstract
from arXiv · showhide
We present a novel means of describing local image appearances using binary strings. Binary descriptors have drawn increasing interest in recent years due to their speed and low memory footprint. A known shortcoming of these representations is their inferior performance compared to larger, histogram based descriptors such as the SIFT. Our goal is to close this performance gap while maintaining the benefits attributed to binary representations. To this end we propose the Learned Arrangements of Three Patch Codes descriptors, or LATCH. Our key observation is that existing binary descriptors are at an increased risk from noise and local appearance variations. This, as they compare the values of pixel pairs; changes to either of the pixels can easily lead to changes in descriptor values, hence damaging its performance. In order to provide more robustness, we instead propose a novel means of comparing pixel patches. This ostensibly small change, requires a substantial redesign of the descriptors themselves and how they are produced. Our resulting LATCH representation is rigorously compared to state-of-the-art binary descriptors and shown to provide far better performance for similar computation and space requirements.
1. Introduction
LATCH addresses the efficiency–discriminativeness trade-off in local image descriptors by replacing noise-sensitive pixel-pair comparisons with learned comparisons of pixel-patch triplets. It is reported to outperform related binary descriptors and narrow the gap with histogram-based methods while retaining compact binary representations.
- Local descriptors must be discriminative and transformation-invariant while remaining efficient to compute, store, and match.
- Binary descriptors reduce computational and storage costs through compact strings and fast comparisons, but can be less descriptive than histogram-based representations.
- Pixel-pair comparisons are sensitive to noise and local appearance changes, while smoothing can discard information in high-frequency keypoint regions.
- LATCH compares triplets of pixel patches, giving each binary bit more spatial support and reducing sensitivity to noise.
- Supervised learning selects informative, low-correlation triplet arrangements from labeled training data for descriptor computation.
- LATCH is evaluated extensively and reported to outperform comparable binary descriptors, narrowing their performance gap with histogram-based methods.
2. Related Work
Related work spans compact binary keypoint descriptors, histogram-based representations, and patch-comparison codes. LATCH differs by directly comparing learned triplets of patches for sparse keypoints rather than using pixel pairs, hand-crafted patterns, or costly floating-point pipelines.
- Binary keypoint descriptors were developed for compact storage and efficient matching, commonly using pixel-intensity comparisons and binary strings.
- ORB adds rotation invariance and unsupervised selection of pixel pairs, while BRISK and FREAK use structured sampling patterns and binary comparisons.
- LDA-Hash, DBRIEF, and BinBoost obtain binary outputs through floating-point descriptors, projections, filters, or learned hash functions, increasing computational cost.
- LATCH uses supervised learning to select efficient patch-triplet arrangements directly, rather than learning projections or relying on hand-crafted sampling.
- TPLBP compares similarities among three patches, but uses a hand-crafted, parameter-controlled ring scheme for per-pixel codes.
- Unlike TPLBP, LATCH is designed as a sparse keypoint descriptor rather than a per-pixel code pooled over image regions.
3. Method
LATCH replaces vulnerable pixel-pair comparisons with learned comparisons among three pixel patches, using labeled data to select informative and low-correlation arrangements. The resulting descriptor is evaluated against binary and larger histogram-based methods on Oxford benchmark conditions.
- From pixel pairs to patch triplets: Pixel-pair bits can be noise-sensitive because each bit depends on two specific pixels, while pre-smoothing may lose high-frequency information.LATCH addresses this by comparing pixel patches instead of individual pixels.
- From pixel pairs to patch triplets: Each LATCH bit uses an anchor patch and two companion patches, comparing the anchor’s similarity to both companions.Patch similarity is computed with the Frobenius norm before producing the binary value.
- Learning patch triplet arrangements: 56,000 candidate triplet arrangements were randomly generated and evaluated over labeled window pairs to assess their binary responses.The benchmark contains over 400k windows in each of the Liberty, Notre Dame, and Yosemite collections.
- Evaluation: On the Oxford Bikes and Leuven curves, LATCH outperforms methods that are not an order of magnitude slower, especially at high precision.The Oxford benchmark includes blur in Bikes and lighting changes in Leuven.
- Learning patch triplet arrangements: Arrangement quality rewards matching binary values for same pairs and differing values for not-same pairs.Arrangements with highly correlated responses are skipped using threshold τ = 0.2.
- Learning patch triplet arrangements: The learning procedure selects arrangements directly rather than learning projections or filters, while retaining comparative or better performance according to the paper.This distinguishes LATCH from more computationally expensive learned binary representations.
4. Experimental results
Experiments across Oxford, Learning Local Descriptors, and reconstruction tests evaluate LATCH against binary and histogram-based descriptors. LATCH generally improves descriptor performance while retaining substantially lower extraction or matching costs than floating-point alternatives.
- Run times: LATCH requires an order of magnitude less extraction time than some floating-point descriptor alternatives.Extraction times were averaged over 250K patches using the implementations and settings described for the experiments.
- Oxford benchmark: LATCH outperforms almost all alternatives, including SIFT and SURF, on the Oxford benchmark’s recall versus 1-precision measure.The benchmark covers eight subsets with zoom, viewpoint, lighting, compression, and blur variations.
- Learning Local Descriptors benchmark: LATCH outperforms other binary descriptors by noticeable margins on both Notre-Dame and Liberty same/not-same tests.Evaluation reports accuracy, area under the ROC curve, and 95% error-rate using a learned distance threshold.
- Design analysis: Larger LATCH descriptors and patches generally improve performance, while gains from patches larger than 9 × 9 or 11 × 11 decay.The analysis varies descriptor sizes from 4 to 64 bytes and compares multiple patch sizes against the default 7 × 7 configuration.
- Design analysis: The proposed learning method outperforms the compared alternatives, and combining correlated-triplet elimination provides a further performance boost.Random triplet selection performs substantially worse than the learned methods.
- Multi-view 3D reconstruction: LATCH and SIFT produce qualitatively comparable 3D reconstructions, while LATCH matching is consistently an order of magnitude faster.The reconstruction experiment focuses on descriptor quality rather than densifying surfaces with a subsequent multi-view stereo step.
5. Conclusions
LATCH is a binary local descriptor designed to retain binary descriptors’ speed and compact storage while narrowing their performance gap with larger histogram-based representations. It compares learned arrangements of pixel patches rather than individual pixel pairs and outperforms other binary descriptors in the authors’ tests.
- 5. Conclusions: LATCH preserves the fast matching and small storage requirements associated with binary descriptors.The conclusion places these properties alongside improved descriptor performance.
- 5. Conclusions: LATCH outperforms other binary descriptors by wide margins and closes the gap with much larger, more expensive histogram-based representations.The stated comparison concerns the authors’ tests and does not claim identical performance in every setting.
- 5. Conclusions: LATCH compares pixel patches in triplets and selects discriminative triplet arrangements to reduce sensitivity to noise and local appearance changes.This replaces individual pixel-pair sampling with patch-based comparisons and learned triplet selection.