Source-linked AI summary
DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features
Min Yang, Dongliang He, Miao Fan, Baorong Shi, Xuetong Xue, Fu Li, Errui Ding, Jizhou Huang
TL;DR
Image retrieval commonly uses global features to find candidates and local features to re-rank them, a two-stage process involving repeated ranking and costly spatial verification. DOLG fuses orthogonal local components with a global descriptor in an end-to-end single-stage model. The paper reports that it significantly outperforms previous two-stage state-of-the-art methods and improves state-of-the-art performance on Revisited Oxford and Paris.
Problem
Two-stage retrieval ranks candidates with global features and then uses costly local-feature spatial verification, with errors able to accumulate across stages.
Source: §1 · p. 2Method
DOLG extracts local components orthogonal to a global descriptor, concatenates them as complementary information, and aggregates them into a compact descriptor for end-to-end single-stage retrieval.
Source: §1 · p. 2Results
Takeaways & Limitations
The results support orthogonal local–global fusion as an effective single-stage image retrieval approach on Revisited Oxford and Paris.
Source: §1 · p. 2, §6 · p. 8The authors did not extensively tune GeM, ArcFace-loss, or dilation-rate hyperparameters, instead following settings from DELG and ASPP.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Image Retrieval is a fundamental task of obtaining images similar to the query one from a database. A common image retrieval practice is to firstly retrieve candidate images via similarity search using global image features and then re-rank the candidates by leveraging their local features. Previous learning-based studies mainly focus on either global or local image representation learning to tackle the retrieval task. In this paper, we abandon the two-stage paradigm and seek to design an effective single-stage solution by integrating local and global information inside images into compact image representations. Specifically, we propose a Deep Orthogonal Local and Global (DOLG) information fusion framework for end-to-end image retrieval. It attentively extracts representative local information with multi-atrous convolutions and self-attention at first. Components orthogonal to the global image representation are then extracted from the local information. At last, the orthogonal components are concatenated with the global representation as a complementary, and then aggregation is performed to generate the final representation. The whole framework is end-to-end differentiable and can be trained with image-level labels. Extensive experimental results validate the effectiveness of our solution and show that our model achieves state-of-the-art image retrieval performances on Revisited Oxford and Paris datasets.
1. Introduction
Prior image retrieval methods combine global-feature candidate search with local-feature re-ranking, but the two-stage process incurs costly ranking and can accumulate errors. DOLG fuses local and global information into a compact descriptor for single-stage retrieval, and the paper reports that it significantly outperforms prior two-stage state-of-the-art methods.
1. Introduction
- Two-stage retrieval ranks images twice and uses costly spatial verification, while errors in both stages can accumulate.The paper motivates a unified single-stage solution to address these issues.Source: §1 · p. 2
- DOLG decomposes local components orthogonal to the global feature, concatenates them as complementary information, and aggregates them into a compact descriptor.The framework is end-to-end learnable and jointly learns local and global features.Source: §1 · p. 2
- Multi-atrous convolutions followed by self-attention are used to extract representative local features.The paper describes these components as improving local-feature learning.Source: §1 · p. 2
- DOLG significantly outperforms previous two-stage state-of-the-art image retrieval methods.The paper reports results on Revisited Oxford and Paris.Source: §1 · p. 2
2. Related Work
Related work covers local and global image representations and approaches that combine them. DOLG draws on attentive local features and global descriptors while distinguishing its orthogonal fusion and single-stage retrieval from prior joint methods.
2.1. Local feature
- DELF uses attentive local descriptors but a single-scale feature map; DOLG uses multi-atrous convolutions to simulate an image pyramid and handle scale variation.Earlier local retrieval methods also used handcrafted features, aggregation, and spatial verification.Source: §2.1 · p. 2
2.2. Global feature
- Global descriptors evolved from aggregation methods to differentiable sum- and GeM-pooling, with CNN retrieval models trained using ranking- or classification-based losses.DOLG uses ArcFace loss and explores pooling schemes for performance improvement.Source: §2.2 · pp. 2–3
2.3. Joint local and global CNN features
- Prior joint approaches include distilling local and global features into a compact descriptor and DELG's end-to-end joint training; DELG nevertheless retains two-stage retrieval.DOLG instead uses orthogonal global–local fusion for single-stage retrieval.Source: §2.3 · p. 3, Figure 2 · p. 3
3. Methodology
DOLG combines global and local image features in a compact descriptor, using a ResNet-based global branch, an attention-equipped local branch, and orthogonal fusion. The network is trained end to end with image-level labels and an ArcFace margin loss.
3.1. Overview
- The global branch uses GeM pooling and a fully connected layer, while a local branch after Res3 supplies features for a final compact descriptor.The GeM exponent p is set to 3.0; values above 1 focus pooling more on salient feature points.Source: §3.1 · p. 3
3.2. Local Branch
- Three dilated-convolution layers and a global-average-pooling branch form a feature pyramid; self-attention then models the importance of local feature points.The dilated branches provide different spatial receptive fields to handle scale variation among image instances.Source: §3.2 · p. 3
3.3. Orthogonal Fusion Module
- Each local feature point is projected onto the global feature, and subtracting that projection yields a component orthogonal to the global feature.The module then appends the global feature to the orthogonal components and aggregates the result into a 512-dimensional descriptor.Source: §3.3 · p. 4
3.4. Training Objective
- Training uses image-level labels with one L2-normalized N-class prediction head and ArcFace margin loss.ArcFace adjusts the cosine similarity for the ground-truth class using a margin.Source: §3.4 · p. 4
4. Experiments
DOLG is evaluated against retrieval methods on Revisited Oxford and Paris, with ablations examining implementation choices and fusion components. The reported comparisons favor DOLG, including against two-stage methods.
4.1. Implementation Details
- Experiments train on GLDv2-clean and evaluate mAP on Medium and Hard splits of revisited Oxford and Paris; inference averages normalized features from five image-pyramid scales.Training uses 80% of GLDv2-clean for training and 20% for validation, with ResNet50 and ResNet101 experiments.Source: §4.1 · p. 5
4.2.1 Comparison with State-of-the-art Methods
- DOLG’s mP@10 is reported as better than two-stage DELGr on both Roxf and Rpar.The passage links the single-stage result to end-to-end training and avoiding error accumulation.Source: §4.2.1 · p. 5, Table 2 · p. 6
- DOLG and two-stage DELGr outperform official two-stage DELG with “+1M” distractors, while the comparison notes different training durations for the official and re-implemented models.DELGr and DOLG were trained for 100 epochs, versus 25 for official DELG; the passage attributes lower robustness of original DELG features partly to this difference.Source: §4.2.1 · pp. 5–6
- 80.5% vs. 77.51% mAP on Roxf-Medium and 58.82% vs. 54.76% on Roxf-Hard compare R50-DOLG with DELG; DOLG also improves over DELG re-ranking across reported splits.R50-DOLG also exceeds R50-How by up to 1.1% on Roxf-Medium and 8.21% on Rpar-Medium; R50-DOLG and R101-DOLG report further gains over matching-depth DELG variants.Source: §4.2.1 · p. 5, Table 1 · p. 6
4.2.2 Ablation Studies
- Among tested fusion locations, f3-only performs best; adding f3, f4, or both improves over the global-only setting, while f3 and f4 together do not improve over f3 alone.The passage attributes f3’s advantage to sufficient spatial resolution and network depth, and notes that using both branches increases model complexity.Source: §4.2.2 · p. 7, Table 3 · p. 7
- GeM pooling in the global branch combined with average pooling in the orthogonal fusion module is the best tested pooling combination.Table 4 compares different pooling functions; AVG denotes ordinary global average pooling.Source: §4.2.2 · p. 7, Table 4 · p. 7
- Local-feature fusion improves medium-split mAP from 78.2% to 80.5% on Roxf and 89.0% to 89.8% on Rpar; removing self-attention also notably lowers performance.Removing multi-atrous slightly lowers mAP on Medium and Hard splits, especially Hard, but slightly improves easy-case performance.Source: §4.2.2 · p. 7, Table 5 · p. 7
- Orthogonal fusion outperforms direct concatenation and Hadamard-product fusion in the reported comparison.The method decomposes local features into components parallel and orthogonal to the global feature, then fuses the orthogonal component with the global feature.Source: §4.2.2 · pp. 7–8, Table 6 · p. 8
5. Discussions
The discussion reports that several hyperparameters were not extensively tuned and that aggregation was chosen for proof of concept. The current modules add limited computational overhead, and the retrieval process remains single-stage.
5. Discussions
- The authors did not extensively tune GeM, ArcFace margin-loss, or dilation-rate hyperparameters, instead following DELG and ASPP practices to assess their building blocks.They note that tuning might improve performance.Source: §5 · p. 8
- Aggregation A was chosen from GeM and average pooling for proof of concept; exploring more complicated learning-based aggregation is left to future work.With average pooling, the fusion module can pool local features first, then project and subtract, simplifying computation.Source: §5 · p. 8
- Average pooling is linear, making orthogonal fusion equivalent to pooling local features before projection and subtraction, which can further simplify computation.The module is described as sufficiently simple yet effective.Source: §5 · p. 8
- Compared with DELG and DELF, extra computation comes from the Multi-Atrous and orthogonal fusion modules, but the authors describe the overhead as quite limited.The retrieval process can be completed in a single stage.Source: §5 · p. 8
6. Conclusion
The paper presents orthogonal fusion of local and global features for single-stage image retrieval. Its local branch models scale variation and descriptor significance, and experiments report improved state-of-the-art performance on Roxf and Rpar.
6. Conclusion
- The authors report significantly improved state-of-the-art performance on Roxf and Rpar with orthogonal local-global fusion for single-stage retrieval.They describe this as a first attempt to fuse local and global features orthogonally for this task.Source: §6 · p. 8
- The local branch uses a multi-atrous module to simulate a spatial feature pyramid for scale variation and self-attention to model each local descriptor’s significance.Source: §6 · p. 8
- The orthogonal fusion module combines complementary local and global information to mutually reinforce each other and produce a representative final descriptor through objective-oriented training.Source: §6 · p. 8