PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

Zhiqiang Yuan, Wenkai Zhang, Changyuan Tian, Xuee Rong, Zhengyuan Zhang, Hongqi Wang, Kun Fu, Xian Sun

arXiv:2204.09860v1cs.CVcs.IRcs.MM

TL;DR

RSCTIR methods often emphasize global image features while neglecting local object relationships and saliency. GaLR combines global and optimized local representations through MIDF and DREA, then applies MR during inference; experiments report state-of-the-art performance on public datasets.

  • Problem

    Existing RSCTIR methods mainly focus on global image features, while local object relationships and salient-instance information receive less attention.

    Source: §I · p. 1

  • Method

    GaLR uses DREA-assisted graph convolution for local representations, MIDF for dynamic global-local fusion, and MR for bidirectional similarity-based post-processing.

    Source: §I · p. 2, B. Optimize Local Representation by… · p. 5

  • Results

    Quantitative analyses on multiple public remote-sensing text-image datasets demonstrate the effectiveness and state-of-the-art performance of GaLR.

    Source: §V · p. 14

  • Takeaways & Limitations

    MR further improves retrieval results without additional training, while GaLR combines complementary global and local information for remote-sensing retrieval.

    Source: F. Ablation Studies of Structures · p. 12, §V · p. 14

Abstract

from arXiv · show

Cross-modal remote sensing text-image retrieval (RSCTIR) has recently become an urgent research hotspot due to its ability of enabling fast and flexible information extraction on remote sensing (RS) images. However, current RSCTIR methods mainly focus on global features of RS images, which leads to the neglect of local features that reflect target relationships and saliency. In this article, we first propose a novel RSCTIR framework based on global and local information (GaLR), and design a multi-level information dynamic fusion (MIDF) module to efficaciously integrate features of different levels. MIDF leverages local information to correct global information, utilizes global information to supplement local information, and uses the dynamic addition of the two to generate prominent visual representation. To alleviate the pressure of the redundant targets on the graph convolution network (GCN) and to improve the model s attention on salient instances during modeling local features, the de-noised representation matrix and the enhanced adjacency matrix (DREA) are devised to assist GCN in producing superior local representations. DREA not only filters out redundant features with high similarity, but also obtains more powerful local features by enhancing the features of prominent objects. Finally, to make full use of the information in the similarity matrix during inference, we come up with a plug-and-play multivariate rerank (MR) algorithm. The algorithm utilizes the k nearest neighbors of the retrieval results to perform a reverse search, and improves the performance by combining multiple components of bidirectional retrieval. Extensive experiments on public datasets strongly demonstrate the state-of-the-art performance of GaLR methods on the RSCTIR task. The code of GaLR method, MR algorithm, and corresponding files have been made available at https://github.com/xiaoyuan1996/GaLR .

I. INTRODUCTION

RSCTIR addresses the need to retrieve relevant remote-sensing images from rapidly growing image collections using text. Existing approaches leave gaps in local object relationships, redundant-target modeling, and bidirectional similarity information, motivating GaLR and MR.

Motivation

  • RSCTIR enables text-based retrieval of relevant images from rapidly expanding remote-sensing data.Its utility in human-computer interaction has driven increasing attention to the task.Source: §I · p. 1

Research gaps

  • Existing embedded-based methods mainly use global image features, often neglecting local object relationships and fine-grained target awareness.Global features capture overall image content, whereas local features represent relationships between objects.Source: §I · p. 2
  • Redundant targets in remote-sensing images increase graph-modeling pressure and hinder attention to salient instances.The paper addresses this issue with a denoised representation matrix and enhanced adjacency matrix, or DREA.Source: §I · p. 2
  • Previous frameworks overlook bidirectional ranking information in the cross-modal similarity matrix during inference.MR uses candidates from the other modality for reverse retrieval and combines bidirectional factors to optimize results.Source: §I · p. 2

Proposed approach

  • GaLR combines global and local information through MIDF, while DREA improves local representations and MR performs post-processing without extra training.MIDF uses local information to correct global information and global information to supplement local information.Source: §I · p. 2

Results

  • Experiments on RSICD and RSITMD report state-of-the-art accuracy for GaLR.The comparison is presented across two popular remote-sensing image-text datasets.Source: Figure 1 · p. 2

B. Graph Neural Network (GNN)

The visual representation pipeline augments global image encoding with detector-based local features and graph convolutional modeling. MIDF dynamically combines both feature levels while addressing the differing information content of global and local representations.

Framework

  • The framework includes visual and text representations followed by MR post-processing, which improves retrieval accuracy without additional training.The architecture is organized into visual representation, text representation, and post-processing components.Source: §III · p. 3, Figure 2 · p. 4

Visual representation

  • GaLR adds a local-feature branch to traditional global visual encoding to represent relationships between important image objects.Global-only encoding can miss distinctions between salient objects and redundant information.Source: 1) Visual Representation · pp. 3–4
  • MVSA-based convolutional encoding extracts global features, while a DOTA-pretrained detector extracts objects for local representation.The detected objects are subsequently used to construct target relationships.Source: 1) Visual Representation · p. 4
  • A graph convolutional network models relationships among the objects extracted from each remote-sensing image.The resulting local feature is denoted as v_l in the formulation.Source: 1) Visual Representation · p. 4
  • Global features summarize the image, whereas local features provide finer target-level content and object relationships.Local features may be weaker when images contain few or no detectable objects.Source: 1) Visual Representation · p. 4

Multi-level fusion

  • MIDF dynamically weights and adds global and local features, using local information to correct global information and global information to supplement local information.This design accounts for images where local features contain little information and global features are especially important.Source: 1) Visual Representation · p. 4

2) Text Representation:

The model represents text with a recurrent encoder and represents image regions with a GCN optimized by DREA. DREA reduces redundant target relations and strengthens relationships involving salient objects for local visual encoding.

Text Representation

  • Text words are embedded and processed by a recurrent network to obtain the text representation.The representation is produced through an RNN and multilayer perceptron pipeline.Source: 2) Text Representation · p. 4

Local Representation

  • A GCN constructs local image representations from detected-object features and their adjacency relationships.Object position, category, probability, and area are embedded as node inputs, while the adjacency matrix models object relations.Source: B. Optimize Local Representation by… · p. 5

DREA Optimization

  • DREA addresses redundant RS targets that can overload GCN modeling and divert attention from useful instances.Redundant targets increase training and inference costs and can prevent the network from focusing on informative features.Source: B. Optimize Local Representation by… · p. 5
  • Similar nearby objects are filtered and integrated into a representation that reflects the surrounding area.This reduces local-target redundancy and computational complexity.Source: B. Optimize Local Representation by… · p. 5
  • The enhanced adjacency relationship boosts larger-area objects and weakens relationships involving smaller-area objects.Area ranking determines relationship gains, changing the model’s focus toward prominent targets for more effective local representations.Source: B. Optimize Local Representation by… · p. 5

C. Multi-Level Information Dynamic Fusion (MIDF)

MIDF combines global and local visual information through feature retransformation and dynamic fusion. Local features correct global representations, global features supplement local representations, and learned weights produce the fused representation.

Feature Retransformation

Information Complementarity

  • Local features generate a mask that filters global features while global features directly supplement local features.This bidirectional complementarity addresses redundancy in global information and limited information in local features.Source: C. Multi-Level Information Dynamic… · p. 6

Feature Retransformation

  • Self-attention discovers internal correlations, while guided attention uses one feature to guide representation of the other.These modules optimize the interaction between global and local information.Source: C. Multi-Level Information Dynamic… · p. 6

Dynamic Fusion

  • The mixed visual information is transformed into learnable dynamic weights that generate the fused features.The dynamic fusion stage combines the retransformations using weights derived from their superposition.Source: C. Multi-Level Information Dynamic… · p. 6

D. Multivariate Rerank (MR)

MR post-processes cross-modal retrieval by using top-k candidates for reverse search and combining bidirectional ranking evidence with a confidence component. The resulting similarity supports a finer-grained secondary ranking.

Algorithm Overview

  • MR uses candidates from the similarity matrix for reverse retrieval and combines multiple ranking factors to optimize results.It integrates image-to-text and text-to-image ranking information and adds a significance component.Source: D. Multivariate Rerank (MR) · pp. 6–7

Forward Ranking

  • For image-to-text retrieval, MR selects the top k nearest texts and records each candidate’s ranking position.The normalized i2t component gives higher values to higher-ranked candidate texts.Source: D. Multivariate Rerank (MR) · p. 7

Reverse Ranking

  • For each retrieved candidate, MR performs reverse search in the opposite modality using nearest neighbors.The reverse-search component provides secondary similarity confirmation that can correct the original retrieval component.Source: D. Multivariate Rerank (MR) · p. 7

Confidence Weighting

  • The significance component measures confidence from the candidate’s similarity relative to similarities with all images.This confidence is used as a weighting term for the final similarity.Source: D. Multivariate Rerank (MR) · p. 7

Final Similarity

  • The final MR similarity weights three reranking components and combines forward and reverse search results for secondary ranking.The paper states that MR accounts for more factors than the original similarity and produces a more fine-grained ranking result.Source: D. Multivariate Rerank (MR) · p. 7

E. Objective Function

The objective uses triplet loss to align paired image-text samples more closely than unpaired samples by a prescribed margin.

Triplet Loss

  • Triplet loss minimizes the distance between paired samples relative to unpaired samples by a fixed margin.The loss pushes negative samples away from the anchor while bringing positive pairs closer.Source: E. Objective Function · p. 7

F. Training and Inference Procedure

GaLR extracts global and local visual representations, fuses them dynamically, and optimizes image-text alignment with triplet loss. During inference, MR post-processes the similarity matrix for secondary optimization.

Training

  • GaLR detects image objects, constructs relation matrices with DREA, applies GCN for local features, and combines them with CNN-derived global features through MIDF.The text representation is extracted with a bidirectional GRU.Source: F. Training and Inference Procedure · pp. 7–8

Inference

  • MR post-processes the accumulated similarity matrix and returns the final image-text similarity matrix.The inference procedure initializes S, computes similarities, and applies MR to produce S_mr.Source: F. Training and Inference Procedure · p. 8

Training

  • The bidirectional GRU extracts text features while triplet loss optimizes the visual and textual representations.The loss is calculated from distances between single-modal visual and text features.Source: F. Training and Inference Procedure · p. 8

Inference

  • During inference, GaLR computes batch cosine similarities and appends them to the full test-set similarity matrix.The resulting matrix is then passed to MR to obtain a secondary-optimized similarity matrix.Source: F. Training and Inference Procedure · p. 8

IV. EXPERIMENTS RESULTS AND ANALYSIS

The experiments compare GaLR with existing retrieval models and analyze its components, reranking strategy, visualizations, and retrieval time across remote sensing text-image datasets.

Experiment scope

  • The experiments evaluate GaLR through comparisons, ablations, visualization experiments, and retrieval-time analysis on several remote sensing text-image datasets.The study also examines the reasons for the reported performance of GaLR and MR.Source: §IV · p. 8

A. Dataset and Evaluation Metrics

The evaluation uses RSICD and RSITMD, datasets containing one remote sensing image paired with five sentences per sample. Performance is measured with recall-at-k and mean recall.

Datasets

  • RSICD contains 10,921 samples at 224×224 pixels, while RSITMD contains 4,743 samples at 256×256 pixels.RSITMD is described as having more fine-grained text representation than RSICD.Source: A. Dataset and Evaluation Metrics · p. 8
  • Each dataset sample contains one remote sensing image and five corresponding sentences.Figure 5 illustrates samples from RSICD and RSITMD.Source: A. Dataset and Evaluation Metrics · p. 8
  • Both datasets are partitioned into 80% training, 10% validation, and 10% test sets.The experiments follow Yuan et al.’s data-partitioning approach.Source: A. Dataset and Evaluation Metrics · p. 8

Evaluation metrics

  • R@k measures the proportion of ground-truth items recalled among the top k results, with k set to 1, 5, and 10.mR averages multiple R@k values to reflect overall performance.Source: A. Dataset and Evaluation Metrics · p. 8

Implementation details

  • All experiments run on a single NVIDIA RTX 3090 GPU, with inputs uniformly scaled to 256 × 256 pixels.Rotation and flip augmentations are used to improve robustness.Source: B. Implementation Details · pp. 8–9

C. Comparisons With the SoTA Methods

GaLR combines global and local information, with DREA improving local representations and MR refining similarity rankings. Experiments report strong results on RSICD and RSITMD, while ablations examine local-feature weighting and reranking parameters.

Comparisons With the SoTA Methods

  • GaLR without MR reaches mR 18.62 on RSICD, and adding MR raises it to 18.96.The reported improvement is attributed to MR’s secondary sorting.Source: C. Comparisons With the SoTA Methods · p. 9
  • GaLR without MR reaches 31.00 on RSITMD, while GaLR with MR achieves mR 31.41 and the first position among multiple indicators.The reported RSITMD result is described as SoTA performance by the authors.Source: C. Comparisons With the SoTA Methods · p. 9

Performance Analysis of DREA Mechanism

  • Increasing the local-feature proportion from the baseline to 0.6 improves LF w/o DREA accuracy from 27.46 to 29.87.The comparison uses the global-only AMFMN visual coding branch as the baseline.Source: D. Performance Analysis of DREA… · p. 10
  • With DREA, retrieval improvement is significant when the local-feature proportion is between 0.2 and 0.6, with highest accuracy at w_l 0.4.The results indicate that DREA improves local-feature representation under the tested weights.Source: D. Performance Analysis of DREA… · p. 10

Rerank Versus MR

  • On RSITMD, source similarity gives mR 31.00, reranking adds 0.21 points, and MR reaches 31.41.The authors attribute MR’s larger gain to mining similarity-matrix information more diversely than reverse ranking alone.Source: E. Rerank Versus MR · p. 10
  • MR’s parameter analysis varies k, w_c1, and w_c2 on RSITMD to assess retrieval behavior and method stability.Figure 6 compares original, reranked, and MR results as these parameters change.Source: E. Rerank Versus MR · p. 11, Figure 6 · p. 11

F. Ablation Studies of Structures

The ablations examine how MVSA, DREA, MIDF, and MR affect retrieval, while visual comparisons assess GaLR against AMFMN and LW-MCR. Results support combining global and local features, dynamically fusing them, and reranking with bidirectional similarity information.

Ablation design

  • Eight control experiments isolate the effects of MVSA, DREA, MIDF, and MR on retrieval effectiveness.The configurations compare global-only, local-only, fused, dynamically fused, and reranked variants.Source: F. Ablation Studies of Structures · p. 11

Module effects

Visual comparison

  • GaLR’s visual retrieval results better represent queried instances than AMFMN and LW-MCR, while LW-MCR can provide finer-grained feature registration but produces false retrievals.Source: G. Visual Analysis of Recall Results · p. 13

H. Qualitative Analysis of Semantic Localization

Semantic localization experiments compare global-only inference, added local features, and MIDF optimization using a fine-grained scene description. The qualitative analysis shows that local representations reduce attention to non-ground-truth regions, while GaLR remains close to competing methods in retrieval time.

Semantic localization setup

Compared configurations

Visualization

Qualitative findings

Retrieval time

  • GaLR’s retrieval time is almost equal to other models overall, slightly below AMFMN in ET and slightly below AMFMN but above MTFN in IT.Source: §I · p. 14

Conclusion

  • Quantitative analysis on multiple remote-sensing text-image datasets demonstrates the effectiveness of the proposed method for remote-sensing retrieval.Source: §V · p. 14

Related papers

Loading 2204.09860v1…