PaperlayerView exact revision on arXiv

Source-linked AI summary

SOLAR: Second-Order Loss and Attention for Image Retrieval

Tony Ng, Vassileios Balntas, Yurun Tian, Krystian Mikolajczyk

arXiv:2001.08972v5cs.CVcs.LG

TL;DR

Global descriptors had not explored correlations between features within a feature map, while second-order similarity had been used for local patch descriptors. SOLAR combines second-order spatial attention with descriptor similarity for retrieval and matching, and the authors report significant improvements and state-of-the-art results on public benchmarks.

  • Problem

    Global descriptors had not explored correlations between features within a feature map, while second-order similarity had only recently been applied to local patch descriptors.

    Source: §1 · p. 1

  • Method

    SOLAR re-weights feature maps using second-order spatial attention and combines it with second-order descriptor similarity for retrieval and matching.

    Source: §1 · p. 2

  • Results

    The authors report significant improvements over state of the art on public retrieval and matching benchmarks, including state-of-the-art retrieval performance on RParis+1M.

    Source: §1 · p. 2, §7 · p. 14

  • Takeaways & Limitations

    The authors report that combining second-order spatial information and descriptor similarity generalizes to both global and local descriptor learning.

    Source: §1 · p. 2, §7 · p. 14

    The formulation assumes each feature location corresponds to pixels within a rectangular receptive field in the input image.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Recent works in deep-learning have shown that second-order information is beneficial in many computer-vision tasks. Second-order information can be enforced both in the spatial context and the abstract feature dimensions. In this work, we explore two second-order components. One is focused on second-order spatial information to increase the performance of image descriptors, both local and global. It is used to re-weight feature maps, and thus emphasise salient image locations that are subsequently used for description. The second component is concerned with a second-order similarity (SOS) loss, that we extend to global descriptors for image retrieval, and is used to enhance the triplet loss with hard-negative mining. We validate our approach on two different tasks and datasets for image retrieval and image matching. The results show that our two second-order components complement each other, bringing significant performance improvements in both tasks and lead to state-of-the-art results across the public benchmarks. Code available at: http://github.com/tonyngjichun/SOLAR

1 Introduction

Global descriptors compactly aggregate feature maps but had not explored within-map feature correlations, while second-order similarity had improved local matching descriptors. SOLAR combines second-order spatial attention and descriptor similarity to improve descriptors for retrieval and matching.

1 Introduction

  • Global descriptors had not explored correlations between features within a feature map, despite providing compact aggregation.Bilinear features compute second-order correlations but expand feature dimensions, requiring dimensionality reduction.Source: §1 · p. 1
  • SOLAR combines second-order spatial attention and descriptor similarity to improve image features for retrieval and matching.Its spatial attention re-weights feature maps, while descriptor-space similarity makes distances between clusters consistent.Source: §1 · p. 2
  • The work combines second-order attention across consecutive feature maps at different resolutions and studies its effects through ablations.Source: §1 · p. 2
  • The authors report significant improvements over state of the art on public image-retrieval and image-matching benchmarks.Source: §1 · p. 2

2 Related Work

Prior retrieval methods either aggregate local descriptors or generate global descriptors in one CNN pass. SOLAR relates these lines of work to second-order attention and similarity, applying their combination to descriptor learning.

2 Related Work

  • Local aggregation detects and describes local features, then combines them into a compact vector using methods such as BoW, VLAD, or Fisher Kernels.CNN-based local descriptors improved large-scale retrieval benchmarks.Source: §2 · p. 3
  • Global single-pass methods generate a descriptor in one CNN forward pass, with pooling choices including Max-pooling, SPoC, CroW, R-MAC, and GeM.GeM was reported to perform well in work optimizing a differentiable approximation of average precision.Source: §2 · p. 3
  • Second-order attention had been used in vision tasks but not descriptor learning, while second-order similarity had been introduced for local patches.SOLAR combines second-order spatial attention with descriptor loss for learning global image representations for retrieval.Source: §2 · p. 3

3 Method

The method extends GeM pooling with second-order spatial attention and trains global descriptors using combined first- and second-order losses, followed by whitening and normalization.

3 Method

  • The method extends state-of-the-art GeM pooling with second-order spatial pooling and then applies second-order similarity loss, whitening, and normalization.Source: §3 · p. 4

3.1 Preliminaries

  • GeM pooling aggregates a CNN feature map into a single descriptor using a learnable parameter p.The input image is processed by an FCN to produce feature map f.Source: §3.1 · p. 4

3.2 Second-Order Spatial Pooling

  • SOA uses a non-local block to correlate each feature location with the whole map, addressing GeM’s lack of relative spatial contributions.The attention map has shape hw × hw, and the resulting features reflect correlations across spatial locations.Source: §3.2 · pp. 4–5

3.3 Second-Order Similarity Loss

  • The global-descriptor objective combines first- and second-order losses, extending second-order similarity to global descriptors and hard-mining negative pairs.The first-order term is triplet loss; the combined objective is balanced by λ.Source: §3.3 · p. 6

3.4 Descriptor Whitening

  • A bias-enabled fully connected layer after GeM learns descriptor whitening end-to-end, followed by ℓ2 normalization.The authors report this whitening approach as superior to a linear projection in their experiments.Source: §3.4 · p. 6

3.5 Network Architecture and Training

  • SOA blocks can be inserted at different CNN feature maps, and the training pipeline combines triplet loss with whitening and ℓ2 normalization.Hard negatives are mined each epoch from a random pool, excluding negatives from the same scene or landmark class.Source: Figure 2 · p. 5, §3.5 · p. 6

4 Results on Large-Scale Image Retrieval

The section evaluates SOLAR on large-scale image retrieval, comparing it with local-aggregation and global single-pass methods. It reports strong benchmark results and qualitative examples of attention and retrieval behavior.

4.1 Datasets

  • The evaluation uses GL18 for training and ROxf/RPar for testing, with each retrieval dataset assessed at Easy, Medium, and Hard difficulty levels.ROxf contains 4,993 images and RPar 6,322; each has 70 queries, and R1M adds one million distractor images.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Comparison to the State-of-the-Art on Image Retrieval

  • SOLAR beats the baseline by 3.6% mAP on ROxf-Hard and 3.0% on RPar-Hard; it also exceeds DELF-D2R-R-ASMK* on three reported mAP comparisons.With R1M distractors, SOLAR is state of the art among global single-pass methods, with mAP gains over the SOTA on four Medium/Hard datasets.Source: Table 1 · p. 8, §4.2 · p. 8
  • With R1M distractors, SOLAR exceeds the global single-pass SOTA in mAP by 4.0% on ROxf-Medium, 4.2% on ROxf-Hard, 1.9% on RPar-Medium, and 3.6% on RPar-Hard.The reported comparisons use ROxf-RPar and their R1M-distractor sets under Medium and Hard protocols.Source: §4.2 · p. 8, Table 1 · p. 8
  • SOA blocks add 7.4% inference time over GeM; SOLAR uses 7.7GB on R1M versus 27.6GB for DELF-D2R-R-ASMK* and processes images faster.Reported processing times are 0.15s per image for SOLAR and over 1.5s for DELF-D2R-R-ASMK* on a Titan Xp GPU.Source: §4.2 · p. 8
  • Freezing backbone weights while training SOAs and whitening reportedly makes training converge faster and yields higher benchmark performance.The authors also report reduced training time under this setting.Source: §4.2 · p. 9

4.3 Qualitative Retrieval Results

  • For background locations, attention is sparsely distributed within the main landmark; landmark locations highlight distinctive regions, including features outside their receptive field.The attention patterns are reported as consistent in difficult examples involving occlusion, distant landmarks, extensive background, and nighttime images.Source: §4.3 · p. 9, Figure 3 · p. 9
  • On challenging ROxf-Hard examples, SOLAR places more correct retrievals near the top than GeM, which tends to rank images with merely similar features higher.Figure 4 compares the query and top-seven results for the baseline and SOLAR; green and red borders mark correct and incorrect retrievals.Source: §4.3 · p. 10, Figure 4 · p. 10

5 Ablation Study

The ablation study tests how second-order attention and similarity affect global and local descriptors. It finds that the components complement each other for retrieval, and that SOAs also improve local patch descriptors across HPatches tasks.

5.1 Optimal Feature Contribution

  • SOLAR outperforms GeM across most tested p values, especially on Hard examples, and its learned p corresponds to its mAP-curve peak while GeM’s is sub-optimal.As p increases beyond roughly 20, mAP rapidly decreases; Fig. 5 compares mAP against p on ROxf-RPar.Source: §5.1 · p. 11, Figure 5 · p. 11

5.2 Impact of Second-Order Components on Image Retrieval

  • SOLAR, combining SOS and SOA, gives the best retrieval mAPs, with SOS adding more than 1% on SOA versus less than 1% on the baseline.SOA4, SOA5, and SOA4,5 improve mAP by 0.93%, 1.15%, and 1.78%, respectively; early SOA insertion is too computationally expensive for high-resolution retrieval images.Source: §5.2 · pp. 11–12, Table 2 · p. 12

5.3 Generalisation to Image Matching with Local Descriptors

  • UBC Patches results are omitted because performance has saturated and FPR@95 has limitations; UBC is used for training, while HPatches evaluates local descriptors.Models are trained on UBC’s liberty subset and selected by average FPR@95 on notredame and yosemite.Source: §5.3 · pp. 12–13, Figure 6 · p. 13
  • Up to 1.75% over SOSNet, SOAs generally improve HPatches Patch Retrieval mAP; SOA6 is the exception, while other SOA configurations show consistent ordering across verification and matching tasks.Consecutive-level and cross-scale insertions are beneficial; SOA6’s 8 × 8 feature map provides less informative spatial correlation and makes optimisation more difficult.Source: §5.3 · pp. 13–14, Figure 6 · p. 13

6 Implementation Details

The implementation fine-tunes GeM+SOLAR on GL18 and trains SOSNet+SOAs for local descriptors.

6 Implementation Details

  • GeM+SOLAR fine-tunes the SOAs and whitening layer for up to 50 epochs on GL18, using triplets with one positive and five hard negatives per anchor.Training uses Adam, batch size 8, margin 1.25 for triplet loss, and λ = 10 for SOS loss; testing averages descriptors from three scales.Source: §6 · p. 14
  • SOSNet+SOAs uses the re-implemented SOSNet settings, with SOAs inserted and trained under identical settings.GeM+SOLAR fine-tuning takes roughly 12 hours across four 1080Ti GPUs; each SOSNet+SOAs training takes roughly five hours on one 1080Ti GPU.Source: §6 · p. 14

7 Conclusion

SOLAR combines second-order spatial attention with descriptor similarity for global retrieval and extends the approach to local patch descriptors. The paper reports state-of-the-art results on its retrieval and HPatches benchmarks.

7 Conclusion

  • SOLAR improves RParis+1M by 3.6% over similar global single-pass methods and by 3.5% over local aggregation methods, while using a fraction of their time and memory costs.The paper also reports a 1.75% improvement for local descriptors on HPatches.Source: §7 · p. 14

1 Results Reported in FPR@95 on UBC Patches

The supplementary UBC Patches results compare SOSNet with configurations that insert SOA blocks after different L2-Net layers. SOAs generally improve results slightly, while SOA6 performs worse than the baseline.

1 Results Reported in FPR@95 on UBC Patches

  • SOA blocks generally improve UBC Patches results slightly with few extra parameters, but SOA6 configurations perform noticeably worse than the baseline.The authors suspect low-resolution, higher-level feature maps create optimisation constraints; insertions at consecutive feature levels perform noticeably better than the compared non-consecutive configurations.Source: §1 · pp. 18–19, Table 1 · p. 18

2 L2-Net Architecture

The section presents the L2-Net architecture used by SOSNet and the ablation study, and describes the implementation’s normalization, activation, descriptor, and dropout choices.

2 L2-Net Architecture

  • Table 2 presents the L2-Net convolutional kernel parameters and intermediate feature-map dimensions to clarify SOA block insertions.The table does not provide complete architecture details; it refers readers to Tian et al. [46] for normalization, activation, and model variations.Source: Table 2 · p. 19
  • The architecture table organizes layers by kernel, stride, output shape, and cumulative parameter count.Output shape is listed as [h, w, c].Source: §2 · p. 19
  • SOSNet and SOSNet+ patches pass through InstanceNorm, convolution layers followed by BatchNorm and ReLU except Layer-7, then a 128-dimensional descriptor receives ℓ2-norm.Dropout at rate 0.1 is inserted between Layer-6 and Layer-7 during training to prevent over-fitting.Source: §2 · p. 19

3 Second-order attention maps on patches

On HPatches patch correspondences, second-order attention maps emphasize connected structures and remain consistent across viewpoint and illumination changes.

3 Second-order attention maps on patches

  • In patches, second-order attention at a spatial location focuses on similar or connected structures, unlike attention on large images.The passage attributes this contrast to patches having less semantic and color information and fewer distinctive textures.Source: §3 · pp. 19–20
  • The figure visualizes attention maps at four selected pixel locations for SOSNet+, SOA3,4,5 on reference and hard-correspondence patches.The examples are from HPatches sequences with viewpoint changes and illumination changes.Source: §3 · p. 19, Figure 1 · p. 20
  • Attention maps remain consistent between reference patches and hard correspondences across viewpoint and illumination changes at SOA3,4,5.The examples compare viewpoint changes in the top row and illumination changes in the bottom row.Source: §3 · p. 20, Figure 1 · p. 20

Related papers

Loading 2001.08972v5…