Source-linked AI summary
Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking
Xi Chen, Xu Chen, Xiamgyang Jia, Wei Wang, Xu Zhang, Zhenyuan Shun
TL;DR
RS image–text retrieval must balance fine-grained alignment with efficiency in complex scenes, while existing approaches can involve costly interaction or substantial data and computation. FTF uses pre-computed multi-granular recall followed by parameter-free BTIB reranking within top-K candidates. Experiments on three public benchmarks report competitive retrieval performance with substantially reduced inference cost.
Problem
Dense objects and complex backgrounds in RS imagery make it difficult to achieve fine-grained cross-modal alignment and efficient retrieval together.
Source: §1 · p. 1Method
FTF recalls candidates using pre-computed multi-granular embeddings, then applies parameter-free BTIB text-guided reranking within the top-K set and uses I2M Loss to optimize alignment and intra-modal consistency.
Source: §1 · pp. 2–3Results
Across three public benchmarks, FTF maintains competitive retrieval performance while substantially reducing inference cost.
Source: §5 · p. 18Takeaways & Limitations
Candidate size offers a flexible balance: smaller sets bring stronger speedups, while larger sets recover more retrieval accuracy.
Source: §4.3 · p. 13
Abstract
from arXiv · showhide
Remote sensing (RS) image-text retrieval plays a critical role in understanding massive RS imagery. However, the dense multi-object distribution and complex backgrounds in RS imagery make it difficult to simultaneously achieve fine-grained cross-modal alignment and efficient retrieval. Existing methods either rely on complex cross-modal interactions that lead to low retrieval efficiency, or depend on large-scale vision-language adaptation, which requires massive data and computational resources. To address these issues, we develop a fast-then-fine (FTF) retrieval framework centered on interaction-free multi-granular recall and parameter-free text-guided cross-modal reranking with the Balanced Text-guided Interaction Block (BTIB). Specifically, interaction-free multi-granular recall pre-computes gallery embeddings and matches them with the corresponding query embedding without full-gallery token-level text-image interaction, while text-guided cross-modal reranking with BTIB refines local semantic correspondence within the top-K candidate set. Furthermore, an inter- and intra-modal loss is designed to jointly optimize cross-modal alignment across multi-granular representations. Extensive experiments on public benchmarks demonstrate that the FTF achieves competitive retrieval accuracy while significantly improving retrieval efficiency compared with existing methods.
1 Introduction
Remote sensing image–text retrieval must balance fine-grained alignment with efficiency amid high-resolution scenes containing many objects and complex backgrounds. FTF combines multi-granular candidate recall with top-K text-guided reranking, alongside a loss and parameter-free module intended to support this balance.
1 Introduction
- Dense objects, fine structural details, and complex backgrounds make it difficult for RS image–text retrieval to achieve both efficient search and fine-grained alignment.Existing interaction-heavy methods can incur computational overhead, while VLM-based methods typically require massive data and computational resources.Source: §1 · pp. 1–2
- FTF recalls candidates using pre-computed multi-granular gallery embeddings, then restricts costly text–image interaction to the top-K candidates for fine-grained refinement.This design avoids full-gallery token-level interaction while keeping query-conditioned reranking within a controllable candidate budget.Source: §1 · p. 2, Figure 1 · p. 2
- BTIB combines text-conditioned visual normalization with visual-token balancing normalization to refine candidate-level semantic correspondence without additional learnable parameters.The contribution description identifies these as complementary normalization operations.Source: §1 · p. 2
- FTF introduces I2M Loss to jointly optimize cross-modal alignment and intra-modal neighborhood consistency across visual granularities, and experiments on three benchmarks report competitive accuracy with substantially lower inference cost.The experiments cover RSITMD, RSICD, and UCM.Source: §1 · p. 3
2 Related Work
Prior RS retrieval work strengthens fine-grained matching through complex interactions or adapted vision–language representations, with efficiency, data, and compute costs remaining concerns. Efficient retrieval and reranking approaches offer alternatives, while FTF uses multi-granular recall and top-K parameter-free refinement.
2.1 Task-Specific RS Retrieval Methods
- Task-specific RS methods improve fine-grained alignment through cross-modal matching and relation modeling, but applying complex interactions across the full candidate space limits retrieval efficiency.The cited examples include attention-based matching, local-to-global matching, and multi-granular feature learning.Source: §2.1 · p. 3
2.2 VLM-Based Methods
- VLM adaptation can improve RS cross-modal representations, while MLLM-assisted captioning or entity extraction adds inference steps and these approaches typically require substantial data and computational resources.The passage names CLIP-Base, PETL, SkyScript, GRAFT, RemoteCLIP, GeoRSCLIP, Caption2CLIP, and GraVLM as examples.Source: §2.2 · p. 3
2.3 Efficient Cross-Modal Retrieval
- Dual-encoder retrieval scales through pre-computed embeddings but may lose fine-grained accuracy; FTF instead uses multi-granular CLIP recall and parameter-free BTIB reranking only on top-K candidates.This differs from CFITR’s hash-based coarse retrieval and Transformer-based fine matching.Source: §2.3 · p. 4
3 Methods
FTF formulates RS image–text retrieval as ranking gallery images for a textual query, then separates efficient multi-granular candidate recall from text-guided reranking. Its I2M Loss jointly optimizes cross-modal alignment and intra-modal structure across the recall and reranking representations.
3.1 Problem Formulation
- The objective is to rank gallery images by relevance to a text query, using v1 for efficient top-K selection and v2 and v3 for coarse- and fine-grained reranking.The three visual embeddings are learned alongside text representations.Source: §3.1 · p. 4
3.2 Overview
- FTF first selects top-K candidates through embedding similarity, then applies BTIB to those query–candidate pairs, restricting fine-grained interaction to the candidate set.The framework avoids token-level text–image interaction across the full gallery.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
3.3 Interaction-Free Multi-Granular Recall
- Recall embeddings combine coarse scene semantics and fine patch details for pre-computable gallery matching; mean and max pooling retain complementary information across visual-token window scales.The stated pooling windows are average-pooling sizes on the visual token grid.Source: §3.3 · pp. 5–6
3.4 Text-Guided Cross-Modal Reranking with BTIB
- BTIB uses bidirectional text–visual token matching to generate coarse and fine candidate embeddings, then combines their query similarities to refine the ranking.Its balanced correspondence emphasizes token pairs distinctive from both textual and visual perspectives.Source: §3.4 · pp. 7–8
3.5 Inter- and Intra-Modal Loss
- I2M Loss aligns recall and reranking representations with text while preserving within-modality neighborhood structure across granularities.Its inter-modal term aligns text with v1–v3; its intra-modal term uses v1 neighborhoods to guide v2 and v3.Source: §3.5 · pp. 8–9
4 Experiments
Across RSITMD, RSICD, and UCM, FTF offers competitive retrieval accuracy with candidate-size-dependent efficiency, while its two-stage design limits BTIB reranking to top-K candidates. Experiments examine candidate coverage, component ablations, accuracy–efficiency trade-offs, and qualitative retrieval behavior.
4.1 Experimental Settings
- Experiments use RSITMD, RSICD, and UCM; retrieval is evaluated with I2T and T2I Recall@K and mean Recall, alongside latency and throughput.Both branches initialize from pretrained CLIP ViT-B/32 and are adapted on each dataset’s training split.Source: §4.1 · p. 9
4.2 Comparison with State-of-the-Art Methods
- FTF (Full) reaches mR 51.73; FTF (K=50) achieves 12.4 ms latency and 80.6 q/s with mR 48.65, while K=200 reaches mR 51.47 at 39.6 ms.The K setting controls the trade-off between efficiency and accuracy; the full-gallery result is competitive with strong baselines but has relatively high latency.Source: §4.2 · pp. 10–11, Table 1 · p. 10, Figure 3 · p. 11
4.3 Generalization and Efficiency on Additional Datasets
- On RSICD, FTF (Full) reaches mR 37.15 versus FSSN’s 37.14; on UCM, it reaches 57.44 versus FSSN’s 58.38, while pruned variants retain competitive accuracy.On RSICD, increasing K from 50 to 200 raises mR from 35.30 to 36.89; candidate-pruned FTF has much lower latency than full-gallery reranking.Source: §4.3 · p. 12, Table 2 · p. 12
- FTF latency grows 1.30×, 1.23×, and 1.15× from UCM to RSICD for K=50, 100, and 200, respectively, versus about 5.0×–5.2× for compared full-gallery baselines.The authors attribute the slower growth to restricting BTIB reranking to a fixed-size top-K set while recall scans the expanding gallery.Source: Figure 5 · p. 13, §4.3 · p. 13
4.4 Stage-1 Candidate Coverage
- At K=200, stage-one average Cov@200 is 97.3% on RSITMD, 95.8% on RSICD, and 98.2% on UCM, retaining most ground-truth matches for reranking.Coverage@K counts queries with at least one ground-truth match retained in the candidate set.Source: §4.4 · p. 14, Table 3 · p. 14
4.5 Ablation Study
- Using both coarse and fine branches performs best; removing BTIB lowers mR from 50.49 to 46.67, and removing I2M’s intra-modal term lowers it to 48.72.BTIB outperforms no reranking, score-level fusion, and one-directional text-to-image softmax; unlike lightweight cross-attention, it adds no parameters and has lower reranking cost.Source: §4.5 · pp. 14–15, Table 4 · p. 14, Table 5 · p. 15
4.6 Efficiency–Accuracy Balance Analysis
- As K increases from 50 to 300, mR rises to 51.62 while latency increases from 12.4 ms to 57.5 ms and throughput falls from 80.6 q/s to 17.4 q/s.Accuracy gains gradually saturate after K=200, which the authors identify as a practical balance between retrieval accuracy and inference cost.Source: §4.6 · pp. 15–16, Figure 6 · p. 15
4.7 Qualitative Analysis
- Qualitative examples show reranking promoting recalled ground-truth images from ranks 39 to 2 and 67 to 5, while combined coarse and fine cues capture scene semantics and local details.For two challenging queries, the proposed method ranks the ground-truth image first, whereas FSSN and EBAKER return related scenes that miss detailed spatial constraints.Source: §4.7 · pp. 16–17, Figure 7 · p. 16, Figure 8 · p. 17
5 Conclusions
FTF combines interaction-free multi-granular recall with candidate-level BTIB reranking, while I2M jointly optimizes inter-modal alignment and intra-modal consistency. Experiments on three public benchmarks show competitive retrieval performance with substantially reduced inference cost.
5 Conclusions
- FTF recalls top-K candidates from pre-computed gallery embeddings, avoiding costly full-gallery token-level text–image interaction.BTIB then refines semantic correspondence within the candidate set.Source: §5 · p. 18
- Across three public benchmarks, FTF maintains competitive retrieval performance while substantially reducing inference cost.Source: §5 · p. 18
- Parameter-free BTIB refines candidate-set semantic correspondence, while I2M jointly optimizes inter-modal alignment and intra-modal structural consistency.Source: §5 · p. 18
- Candidate size flexibly balances retrieval accuracy, latency, and throughput across deployment requirements.Source: §5 · p. 18