PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

Zhiqiang Yuan, Wenkai Zhang, Changyuan Tian, Xuee Rong, Zhengyuan Zhang, Hongqi Wang, Kun Fu, Xian Sun

arXiv:2204.09860v1cs.CVcs.IRcs.MM

TL;DR

RSCTIR methods often emphasize global image features while neglecting local object relationships and saliency. GaLR combines global and optimized local representations through MIDF and DREA, then applies MR during inference; experiments report state-of-the-art performance on public datasets.

  • Problem

    Existing RSCTIR methods mainly focus on global image features, while local object relationships and salient-instance information receive less attention.

  • Method

    GaLR uses DREA-assisted graph convolution for local representations, MIDF for dynamic global-local fusion, and MR for bidirectional similarity-based post-processing.

  • Results

    Quantitative analyses on multiple public remote-sensing text-image datasets demonstrate the effectiveness and state-of-the-art performance of GaLR.

  • Takeaways & Limitations

    MR further improves retrieval results without additional training, while GaLR combines complementary global and local information for remote-sensing retrieval.

Abstract

from arXiv · show

Cross-modal remote sensing text-image retrieval (RSCTIR) has recently become an urgent research hotspot due to its ability of enabling fast and flexible information extraction on remote sensing (RS) images. However, current RSCTIR methods mainly focus on global features of RS images, which leads to the neglect of local features that reflect target relationships and saliency. In this article, we first propose a novel RSCTIR framework based on global and local information (GaLR), and design a multi-level information dynamic fusion (MIDF) module to efficaciously integrate features of different levels. MIDF leverages local information to correct global information, utilizes global information to supplement local information, and uses the dynamic addition of the two to generate prominent visual representation. To alleviate the pressure of the redundant targets on the graph convolution network (GCN) and to improve the model s attention on salient instances during modeling local features, the de-noised representation matrix and the enhanced adjacency matrix (DREA) are devised to assist GCN in producing superior local representations. DREA not only filters out redundant features with high similarity, but also obtains more powerful local features by enhancing the features of prominent objects. Finally, to make full use of the information in the similarity matrix during inference, we come up with a plug-and-play multivariate rerank (MR) algorithm. The algorithm utilizes the k nearest neighbors of the retrieval results to perform a reverse search, and improves the performance by combining multiple components of bidirectional retrieval. Extensive experiments on public datasets strongly demonstrate the state-of-the-art performance of GaLR methods on the RSCTIR task. The code of GaLR method, MR algorithm, and corresponding files have been made available at https://github.com/xiaoyuan1996/GaLR .

I. INTRODUCTION

RSCTIR addresses the need to retrieve relevant remote-sensing images from rapidly growing image collections using text. Existing approaches leave gaps in local object relationships, redundant-target modeling, and bidirectional similarity information, motivating GaLR and MR.

  • Motivation: RSCTIR enables text-based retrieval of relevant images from rapidly expanding remote-sensing data.Its utility in human-computer interaction has driven increasing attention to the task.
  • Research gaps: Existing embedded-based methods mainly use global image features, often neglecting local object relationships and fine-grained target awareness.Global features capture overall image content, whereas local features represent relationships between objects.
  • Research gaps: Redundant targets in remote-sensing images increase graph-modeling pressure and hinder attention to salient instances.The paper addresses this issue with a denoised representation matrix and enhanced adjacency matrix, or DREA.
  • Research gaps: Previous frameworks overlook bidirectional ranking information in the cross-modal similarity matrix during inference.MR uses candidates from the other modality for reverse retrieval and combines bidirectional factors to optimize results.
  • Proposed approach: GaLR combines global and local information through MIDF, while DREA improves local representations and MR performs post-processing without extra training.MIDF uses local information to correct global information and global information to supplement local information.
  • Results: Experiments on RSICD and RSITMD report state-of-the-art accuracy for GaLR.The comparison is presented across two popular remote-sensing image-text datasets.

B. Graph Neural Network (GNN)

The visual representation pipeline augments global image encoding with detector-based local features and graph convolutional modeling. MIDF dynamically combines both feature levels while addressing the differing information content of global and local representations.

  • Framework: The framework includes visual and text representations followed by MR post-processing, which improves retrieval accuracy without additional training.The architecture is organized into visual representation, text representation, and post-processing components.
  • Visual representation: GaLR adds a local-feature branch to traditional global visual encoding to represent relationships between important image objects.Global-only encoding can miss distinctions between salient objects and redundant information.
  • Visual representation: MVSA-based convolutional encoding extracts global features, while a DOTA-pretrained detector extracts objects for local representation.The detected objects are subsequently used to construct target relationships.
  • Visual representation: A graph convolutional network models relationships among the objects extracted from each remote-sensing image.The resulting local feature is denoted as v_l in the formulation.
  • Visual representation: Global features summarize the image, whereas local features provide finer target-level content and object relationships.Local features may be weaker when images contain few or no detectable objects.
  • Multi-level fusion: MIDF dynamically weights and adds global and local features, using local information to correct global information and global information to supplement local information.This design accounts for images where local features contain little information and global features are especially important.

2) Text Representation:

The model represents text with a recurrent encoder and represents image regions with a GCN optimized by DREA. DREA reduces redundant target relations and strengthens relationships involving salient objects for local visual encoding.

  • Text Representation: Text words are embedded and processed by a recurrent network to obtain the text representation.The representation is produced through an RNN and multilayer perceptron pipeline.
  • Local Representation: A GCN constructs local image representations from detected-object features and their adjacency relationships.Object position, category, probability, and area are embedded as node inputs, while the adjacency matrix models object relations.
  • DREA Optimization: DREA addresses redundant RS targets that can overload GCN modeling and divert attention from useful instances.Redundant targets increase training and inference costs and can prevent the network from focusing on informative features.
  • DREA Optimization: Similar nearby objects are filtered and integrated into a representation that reflects the surrounding area.This reduces local-target redundancy and computational complexity.
  • DREA Optimization: The enhanced adjacency relationship boosts larger-area objects and weakens relationships involving smaller-area objects.Area ranking determines relationship gains, changing the model’s focus toward prominent targets for more effective local representations.

C. Multi-Level Information Dynamic Fusion (MIDF)

MIDF combines global and local visual information through feature retransformation and dynamic fusion. Local features correct global representations, global features supplement local representations, and learned weights produce the fused representation.

  • Feature Retransformation: MIDF first retransforms global and local features before dynamically fusing the resulting multi-level information.The retransformation stage uses self-attention and guided attention to improve feature interaction.
  • Information Complementarity: Local features generate a mask that filters global features while global features directly supplement local features.This bidirectional complementarity addresses redundancy in global information and limited information in local features.
  • Feature Retransformation: Self-attention discovers internal correlations, while guided attention uses one feature to guide representation of the other.These modules optimize the interaction between global and local information.
  • Dynamic Fusion: The mixed visual information is transformed into learnable dynamic weights that generate the fused features.The dynamic fusion stage combines the retransformations using weights derived from their superposition.

D. Multivariate Rerank (MR)

MR post-processes cross-modal retrieval by using top-k candidates for reverse search and combining bidirectional ranking evidence with a confidence component. The resulting similarity supports a finer-grained secondary ranking.

  • Algorithm Overview: MR uses candidates from the similarity matrix for reverse retrieval and combines multiple ranking factors to optimize results.It integrates image-to-text and text-to-image ranking information and adds a significance component.
  • Forward Ranking: For image-to-text retrieval, MR selects the top k nearest texts and records each candidate’s ranking position.The normalized i2t component gives higher values to higher-ranked candidate texts.
  • Reverse Ranking: For each retrieved candidate, MR performs reverse search in the opposite modality using nearest neighbors.The reverse-search component provides secondary similarity confirmation that can correct the original retrieval component.
  • Confidence Weighting: The significance component measures confidence from the candidate’s similarity relative to similarities with all images.This confidence is used as a weighting term for the final similarity.
  • Final Similarity: The final MR similarity weights three reranking components and combines forward and reverse search results for secondary ranking.The paper states that MR accounts for more factors than the original similarity and produces a more fine-grained ranking result.

E. Objective Function

The objective uses triplet loss to align paired image-text samples more closely than unpaired samples by a prescribed margin.

  • Triplet Loss: Triplet loss minimizes the distance between paired samples relative to unpaired samples by a fixed margin.The loss pushes negative samples away from the anchor while bringing positive pairs closer.

F. Training and Inference Procedure

GaLR extracts global and local visual representations, fuses them dynamically, and optimizes image-text alignment with triplet loss. During inference, MR post-processes the similarity matrix for secondary optimization.

  • Training: GaLR detects image objects, constructs relation matrices with DREA, applies GCN for local features, and combines them with CNN-derived global features through MIDF.The text representation is extracted with a bidirectional GRU.
  • Inference: MR post-processes the accumulated similarity matrix and returns the final image-text similarity matrix.The inference procedure initializes S, computes similarities, and applies MR to produce S_mr.
  • Training: The bidirectional GRU extracts text features while triplet loss optimizes the visual and textual representations.The loss is calculated from distances between single-modal visual and text features.
  • Inference: During inference, GaLR computes batch cosine similarities and appends them to the full test-set similarity matrix.The resulting matrix is then passed to MR to obtain a secondary-optimized similarity matrix.

IV. EXPERIMENTS RESULTS AND ANALYSIS

The experiments compare GaLR with existing retrieval models and analyze its components, reranking strategy, visualizations, and retrieval time across remote sensing text-image datasets.

  • Experiment scope: The experiments evaluate GaLR through comparisons, ablations, visualization experiments, and retrieval-time analysis on several remote sensing text-image datasets.The study also examines the reasons for the reported performance of GaLR and MR.

A. Dataset and Evaluation Metrics

The evaluation uses RSICD and RSITMD, datasets containing one remote sensing image paired with five sentences per sample. Performance is measured with recall-at-k and mean recall.

  • Datasets: RSICD contains 10,921 samples at 224×224 pixels, while RSITMD contains 4,743 samples at 256×256 pixels.RSITMD is described as having more fine-grained text representation than RSICD.
  • Datasets: Each dataset sample contains one remote sensing image and five corresponding sentences.Figure 5 illustrates samples from RSICD and RSITMD.
  • Datasets: Both datasets are partitioned into 80% training, 10% validation, and 10% test sets.The experiments follow Yuan et al.’s data-partitioning approach.
  • Evaluation metrics: R@k measures the proportion of ground-truth items recalled among the top k results, with k set to 1, 5, and 10.mR averages multiple R@k values to reflect overall performance.
  • Implementation details: All experiments run on a single NVIDIA RTX 3090 GPU, with inputs uniformly scaled to 256 × 256 pixels.Rotation and flip augmentations are used to improve robustness.

C. Comparisons With the SoTA Methods

GaLR combines global and local information, with DREA improving local representations and MR refining similarity rankings. Experiments report strong results on RSICD and RSITMD, while ablations examine local-feature weighting and reranking parameters.

  • Comparisons With the SoTA Methods: GaLR without MR reaches mR 18.62 on RSICD, and adding MR raises it to 18.96.The reported improvement is attributed to MR’s secondary sorting.
  • Comparisons With the SoTA Methods: GaLR without MR reaches 31.00 on RSITMD, while GaLR with MR achieves mR 31.41 and the first position among multiple indicators.The reported RSITMD result is described as SoTA performance by the authors.
  • Performance Analysis of DREA Mechanism: Increasing the local-feature proportion from the baseline to 0.6 improves LF w/o DREA accuracy from 27.46 to 29.87.The comparison uses the global-only AMFMN visual coding branch as the baseline.
  • Performance Analysis of DREA Mechanism: With DREA, retrieval improvement is significant when the local-feature proportion is between 0.2 and 0.6, with highest accuracy at w_l 0.4.The results indicate that DREA improves local-feature representation under the tested weights.
  • Rerank Versus MR: On RSITMD, source similarity gives mR 31.00, reranking adds 0.21 points, and MR reaches 31.41.The authors attribute MR’s larger gain to mining similarity-matrix information more diversely than reverse ranking alone.
  • Rerank Versus MR: MR’s parameter analysis varies k, w_c1, and w_c2 on RSITMD to assess retrieval behavior and method stability.Figure 6 compares original, reranked, and MR results as these parameters change.

F. Ablation Studies of Structures

The ablations examine how MVSA, DREA, MIDF, and MR affect retrieval, while visual comparisons assess GaLR against AMFMN and LW-MCR. Results support combining global and local features, dynamically fusing them, and reranking with bidirectional similarity information.

  • Ablation design: Eight control experiments isolate the effects of MVSA, DREA, MIDF, and MR on retrieval effectiveness.The configurations compare global-only, local-only, fused, dynamically fused, and reranked variants.
  • Module effects: 2.63 points: adding MVSA increases the mR indicator from g1 to g2.
  • Module effects: 0.95 points: DREA improves retrieval when comparing l3 and l4, although it harms performance when only local features are used.
  • Module effects: Global-only features outperform local-only features because global representations retain more of the remote-sensing image information.
  • Module effects: Combining global and local features significantly improves retrieval accuracy, indicating that both information levels contribute to retrieval.
  • Module effects: MIDF raises accuracy by 0.15 points without requiring manual trade-off-parameter tuning, while MR further optimizes f3 retrieval through multivariate similarity reranking.
  • Visual comparison: GaLR’s visual retrieval results better represent queried instances than AMFMN and LW-MCR, while LW-MCR can provide finer-grained feature registration but produces false retrievals.

H. Qualitative Analysis of Semantic Localization

Semantic localization experiments compare global-only inference, added local features, and MIDF optimization using a fine-grained scene description. The qualitative analysis shows that local representations reduce attention to non-ground-truth regions, while GaLR remains close to competing methods in retrieval time.

  • Semantic localization setup: Semantic localization identifies the region in a large scene that best matches a text query using multi-scale sliding-window image slices.
  • Compared configurations: The experiment compares global-only inference, GaLR with local information, and GaLR with MIDF optimization.
  • Visualization: Fig. 8 uses image and heatmap rows, with red indicating higher likelihood, blue indicating lower likelihood, and red boxes marking ground truth.
  • Qualitative findings: Adding local representations enlarges low-attention regions outside ground truth, weakens attention in another non-ground-truth area, and produces fewer false detections.
  • Retrieval time: GaLR’s retrieval time is almost equal to other models overall, slightly below AMFMN in ET and slightly below AMFMN but above MTFN in IT.
  • Conclusion: Quantitative analysis on multiple remote-sensing text-image datasets demonstrates the effectiveness of the proposed method for remote-sensing retrieval.
Loading 2204.09860v1…