PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

Zhiqiang Yuan, Wenkai Zhang, Kun Fu, Xuan Li, Chubo Deng, Hongqi Wang, Xian Sun

arXiv:2204.09868v1cs.CVcs.MM

TL;DR

The paper addresses limitations in remote-sensing text–image retrieval caused by multiscale scarcity, redundant targets, coarse text, and strong intraclass similarity. It proposes AMFMN with adaptive feature matching and dynamic margins, and reports improved retrieval accuracy across several datasets.

  • Problem

    Remote-sensing retrieval methods and traditional datasets struggle with multiscale and redundant image targets, coarse text, and high intraclass similarity.

  • Method

    AMFMN uses multiscale visual self-attention, visual-guided text representation, multisource feature matching, redundant-feature filtering, and similarity-based dynamic triplet margins.

  • Results

    On RSICD, AMFMN variants outperform other models, while on RSITMD and Sydney, the proposed variants achieve leading comprehensive retrieval performance.

  • Takeaways & Limitations

    Direct image–text similarity with multiscale visual features and visual-guided text representation improves retrieval for images containing multiscale and redundant targets.

Abstract

from arXiv · show

Remote sensing (RS) cross-modal text-image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multi-scale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multi-scale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multi-scale feature inputs, favors multi-source retrieval methods, and can dynamically filter redundant features. AMFMN employs the multi-scale visual self-attention (MVSA) module to extract the salient features of RS image and utilizes visual features to guide the text representation. Furthermore, to alleviate the positive samples ambiguity caused by the strong intraclass similarity in RS image, we propose a triplet loss function with dynamic variable margin based on prior similarity of sample pairs. Finally, unlike the traditional RS image-text dataset with coarse text and higher intraclass similarity, we construct a fine-grained and more challenging Remote sensing Image-Text Match dataset (RSITMD), which supports RS image retrieval through keywords and sentence separately and jointly. Experiments on four RS text-image datasets demonstrate that the proposed model can achieve state-of-the-art performance in cross-modal RS text-image retrieval task.

I. INTRODUCTION

Cross-modal RS text–image retrieval is motivated by the difficulty of efficiently matching flexible text queries to images containing multiscale, redundant, and highly similar content. The paper addresses these issues with AMFMN, adaptive-margin triplet loss, and the fine-grained RSITMD dataset.

  • Research motivation: Cross-modal retrieval directly matches text and images, avoiding information loss from caption generation and intermediate text-similarity matching.Traditional caption-based retrieval is time-consuming to annotate and can lose information across two stages.
  • Research motivation: RS images contain many irrelevant background features and multiscale targets, while natural-scene methods inadequately filter redundancy or exploit scale information.These characteristics make salient-feature extraction important for RS retrieval.
  • Research motivation: Strong intraclass similarity creates soft positive samples, making the training objective ambiguous when similar negative images resemble the ground truth.The paper replaces a fixed triplet-loss margin with a margin adapted to prior sample-pair similarity.
  • Proposed contributions: The paper constructs RSITMD as a fine-grained, more challenging dataset with scene changes, detailed captions, and keyword attributes for multiple retrieval tasks.The dataset is intended to reduce intraclass similarity and supports keyword-based retrieval in addition to sentence-based retrieval.
  • Proposed contributions: AMFMN adapts to multiscale features, supports multisource retrieval, dynamically filters redundant features, and uses visual features to guide text representation.Its MVSA module extracts salient image features, while the broader network addresses multimodal matching.

B. Audio–Image Cross-Modal RS Image Retrieval

The paper situates AMFMN within cross-modal RS retrieval and describes a direct image–text similarity framework. Its formulation combines projected visual and textual features, visual guidance, and dynamic fusion of sentence and keyword information.

  • Text–image retrieval: Earlier RS text–image systems often rely on coarse generated captions, whose intermediate information loss can reduce retrieval accuracy.Direct cross-modal similarity methods are more established in natural scenes than in RS scenes.
  • Architecture: The complete AMFMN architecture includes CNN and GRU embeddings, MVSA salient-feature extraction, VGMF text fusion, and dynamic-margin triplet loss.These components form the paper’s stated five-part implementation framework.
  • Formulation: AMFMN maps image and text features into a shared embedding space and computes cross-modal similarity through an elementwise product.The visual and textual inputs are separately projected before similarity calculation.
  • Visual guidance: Visual features guide text representation so the encoder can dynamically account for salient objects in the image.The paper describes this image-guided representation as more reasonable than text encoding independent of visual information.
  • Visual-guided multimodal fusion: The VGMF module dynamically fuses sentence and keyword features to support multiple retrieval inputs and produce robust text representations.Sentence features capture relationships among objects, whereas keywords emphasize vital image targets.

B. Unimodal Embedding

The unimodal embedding stage extracts global and multilevel visual features while representing sentences and keywords as modality-specific embeddings. For images, multiscale extraction addresses redundancy and the loss of small-target information in single-scale global features.

  • The unimodal embedding stage covers visual, sentence, and keyword embeddings.
  • Visual Embedding: A pretrained ImageNet CNN extracts global RS image features v(g) from input image I using parameters θI.
  • Visual Embedding: Global features alone can retain redundant target information and inadequately represent multiscale RS images, especially as small targets diminish through convolutional layers.
  • Visual Embedding: The visual encoder also extracts feature maps from each convolutional layer for multilevel image representation.
  • Visual Embedding: Feature maps from early layers are upsampled and concatenated as low-level features, while later-layer maps are sampled and connected as high-level features.

2) Sentence Embedding:

Sentence and keyword inputs are converted into sequential representations with shared embedding parameters and recurrent modeling. The resulting global features provide separate text representations for sentence- and keyword-based retrieval.

  • Sentence Embedding: A sentence S is split into T words, which are converted into dense vectors with the embedding matrix We.
  • Sentence Embedding: A bidirectional GRU models the embedded sentence while preserving temporal information across word positions.
  • Sentence Embedding: The sentence feature at each step averages the forward and backward hidden states, which are then used to represent global sentence information.
  • Keywords Embedding: Keywords are embedded with the same parameter matrix We used for sentences and modeled sequentially because keyword phrases can contain fine-grained and target words.
  • Multiscale Visual Self-Attention: The MVSA motivation combines multiscale image processing with redundant-target filtering so small targets remain represented and salient image features are extracted.

1) Multiscale Feature Fusion:

The multiscale feature fusion stage aligns low- and high-level image features, combines them with residual preservation of high-level information, and prepares the representation for redundancy filtering.

  • Multiscale Feature Fusion: Low-level features are transformed with 3×3 convolutions and PReLU to match the high-level feature-map size.
  • Multiscale Feature Fusion: The transformed low- and high-level features are concatenated into a joint representation, while the mean high-level feature is added residually to prevent its loss.
  • Redundant Feature Filtering: Because joint features can contain redundant or misleading representations, the method applies a secondary transformation to restrain useless information.
  • Redundant Feature Filtering: L2-regularized joint features pass through independent 1×1 convolutions to produce joint information features and a feature gate vector.
  • Redundant Feature Filtering: Elementwise multiplication suppresses useless joint representations, after which a sigmoid-generated salient mask gates the global image vector to produce the final image feature.

D. Visual-Guided Multimodal Fusion

Visual-guided multimodal fusion uses visual information to dynamically combine sentence and keyword representations. It evaluates soft, fusion, and similarity attention mechanisms to guide text features.

  • Visual-Guided Multimodal Fusion: VGMF dynamically fuses sentence and keyword features because sentences capture object relations while keywords capture vital image targets.
  • Visual-Guided Attention: The VGA module provides visual-guided information for other modalities and includes soft attention, fusion attention, and similarity attention.
  • Soft Attention: Soft attention transforms visual features through a linear layer and activation function to create a gate controlling the text output.
  • Fusion Attention: Fusion attention concatenates visual and text features into a joint representation used to generate denoising self-attention.
  • Similarity Attention: Similarity attention applies cosine similarity between transformed visual and text features, then uses a sigmoid to control and refocus the converted text information.
  • Visual-Guided Attention: After VGA, the model obtains sentence and keyword features guided by visual information.

2) Dynamic Fusion of Multimodal Information:

The method dynamically fuses sentence and keyword information while replacing the fixed triplet-loss margin with a similarity-aware margin. This shifts optimization from binary matching toward learning degrees of image-text similarity, reducing ambiguity from soft positive samples.

  • Dynamic Fusion of Multimodal Information: The VGMF module dynamically fuses visual-guided sentence and keyword features into the final text representation.Keywords supplement isolated target information that may be absent from query sentences.
  • Optimized Triplet Loss: Traditional triplet loss uses a consistent margin for every negative sample, which is unsuitable for remote-sensing images containing soft positive samples.Strong intraclass similarity makes some negative images visually similar to the ground truth.
  • Optimized Triplet Loss: The proposed loss uses prior sample-pair similarity to learn the degree of image-text matching rather than only whether pairs match.This transforms hard-label optimization into soft-label optimization.
  • Optimized Triplet Loss: The dynamic margin αct is derived from prior similarity and makes margins smaller for more similar pairs while preserving larger margins for unrelated pairs.BLEU and METEOR indicators are used to calculate text similarity in training samples.
  • Optimized Triplet Loss: Parameters β and γ control the dynamic-margin curve, with γ setting the maximum margin and β controlling its decay or curvature.For positive β, the curve becomes convex and approaches zero as similarity approaches one.

IV. DATASET FOR RS IMAGE-TEXT MATCH

The paper introduces RSITMD to address limited fine-grainedness and high intraclass similarity in existing remote-sensing image-caption datasets. RSITMD combines diversified captions with keywords, supporting sentence, keyword, and joint retrieval.

  • Existing Dataset Limitations: Existing Sydney, UCM, and RSICD caption datasets retain high sentence or image similarity, limiting their suitability for fine-grained image-text retrieval.Sydney is small and highly similar, UCM has similar sentences, and RSICD contains repeated captions with low fine graininess.
  • RSITMD Construction: RSITMD provides more fine-grained captions and lower sentence similarity than the existing Sydney, UCM, and RSICD datasets.Its similarity visualization curve is closer to the ideal diagonal pattern.
  • RSITMD Construction: RSITMD contains 4,743 images and 23,715 captions, with one to five fine-grained keywords assigned to each image.The dataset combines images selected from RSICD with images from Google Earth.
  • Annotation Strategy: Different annotators describe the same image using attributes such as color, size, and adjacency to increase caption diversity and distinguish intraclass images.The annotation guidance prioritizes fine-grained object attributes and discourages coarse wording.
  • Dataset Comparison: RSITMD’s diversity score is 4.6, compared with 1.83 for Sydney, 0.97 for UCM, and 1.67 for RSICD.The authors use the ratio of completely inconsistent sentences to images as the diversity measure.
  • Retrieval Tasks: RSITMD supports retrieval using sentences or keywords individually and jointly, with keywords supplementing target information absent from sentences.This reflects the different information needs of multi-target descriptions and single-target queries.

V. EXPERIMENTS RESULTS AND ANALYSIS

Experiments evaluate AMFMN on Sydney, UCM, RSICD, and RSITMD using standard recall metrics, with RSITMD additionally supporting joint keyword-and-sentence retrieval. The study also uses fixed train-validation-test splits and reports an ablation of the dynamic variable triplet loss.

  • Experimental Setup: AMFMN is evaluated on the Sydney, UCM, RSICD, and RSITMD datasets to verify its effectiveness for remote-sensing retrieval.RSITMD is used for the joint keyword-and-sentence retrieval experiment.
  • Experimental Setup: Each dataset is divided into 80% training, 10% validation, and 10% test samples.The split is applied before evaluating the retrieval models.
  • Evaluation Metrics: The experiments use R@1, R@5, R@10, and mR, where mR averages six recall rates across image-to-text and text-to-image retrieval.R@K measures the proportion of ground truth items appearing among the top K results.
  • Implementation: The implementation uses ResNet-18 for visual features, with 512-dimensional visual embeddings, 300-dimensional word embeddings, and a 512-dimensional bidirectional-GRU hidden layer.Images are resized and cropped or rotated for training augmentation.

C. Comparisons With the State-of-the-Art Approaches

AMFMN is evaluated against established retrieval methods across four RS image-text datasets and multiple input modes. It achieves strong results on RSICD, RSITMD, and keyword-image retrieval, while joint sentence-keyword input further improves retrieval.

  • Cross-dataset comparison: AMFMN is compared with VSE++, CAMP, MTFN, and other baselines on RSICD, RSITMD, UCM, and Sydney.The experiments use ResNet-18 in place of the object-detection network and adapt inputs when only sentences or keywords are available.
  • RSICD results: 1.95 mR points: AMFMN-fusion improves over MTFN on RSICD, with all three AMFMN variants outperforming the other compared models.AMFMN-fusion is the best-performing AMFMN variant in the reported RSICD comparison.
  • RSITMD results: AMFMN-sim achieves the highest reported RSITMD comprehensive performance, while the other AMFMN variants also remain ahead of the other models.Except for sentence-recall R@1, AMFMN reports the best results for the other stated RSITMD indicators.
  • Sydney results: 51.72 mR: AMFMN-sim ranks first on the Sydney comparison, while AMFMN-soft and AMFMN-fusion tie for second.The reported Sydney result is presented as evidence of the method’s robustness.
  • Keyword retrieval: 20.80 mR: AMFMN is optimal for keyword-image retrieval on RSITMD, exceeding MTFN by 3.03 points.The keyword retrieval experiment uses 21,403 keywords from RSITMD.
  • Multisource input: Joint sentence-keyword retrieval improves the reported result by 0.86 points over sentence-image retrieval, whereas keyword-only retrieval is less effective.The paper attributes the joint-input gain to keywords supplementing information missing from the sentence.

E. Ablation Studies

The ablation studies examine the dynamic variable triplet loss, visual feature extraction, retrieval inputs, visual guidance, and adaptive fusion. Results consistently support the contributions of these components, with the loss outperforming fixed-margin alternatives and the full configuration benefiting from combined modules.

  • Loss-function ablation: The dynamic variable triplet loss Lct outperforms traditional triplet loss L across tested maximum margins, while β has no significant effect on mR.The experiments vary constant margins, γ, and β; recall is lowest at β = 3, but β is generally insensitive.
  • MVSA ablation: 1.18 points: replacing global image features v(g) with MVSA features Fv improves the retrieval indicator in the corresponding ablations.The same conclusion is reported when comparing the sentence-based configurations m2 and m4.
  • Input ablation: Sentence retrieval is more accurate than keyword retrieval in the compared configurations, supporting the greater fine-grained information available from sentences.This comparison is made for both global image features and MVSA-extracted features.
  • Retrieval configuration: Using different retrieval methods improves retrieval accuracy across the m3–m5 configurations.These configurations combine MVSA visual features with sentence and keyword representations in different ways.
  • Cross-modal fusion: Visual-guided text representation improves results for the same image features, and adding the adaptive gate further improves the full model.The comparisons identify gains from the VGA mechanism and from the proposed information-fusion mechanism.

F. Saliency Mask Visualization

The saliency-mask analysis shows MVSA concentrating on image regions and relationships relevant to the query. Retrieval examples further indicate complementary strengths for keyword, sentence, and joint inputs, with joint retrieval combining their benefits.

  • Saliency masks: MVSA saliency masks adaptively highlight image regions relevant to query statements, enabling salient visual features to guide analysis.The visualization examines five typical images and the relationship between salient masks and described content.
  • Saliency masks: The masks attend to both primary objects and fine-grained context, such as tennis courts with roads, or playgrounds with nearby runways and houses.The examples indicate attention to adjacent relationships as well as individual targets.
  • Retrieval examples: Keyword retrieval can produce category-consistent results even when early ranks are not the exact ground truth, despite limited fine-grained expressions.The example describes the first two incorrect results as still belonging to the playground category.
  • Retrieval examples: Sentence retrieval generally provides higher accuracy than keyword retrieval but can still confuse visually similar or semantically distant categories.The reported example includes a non-ground-truth rank 1 that is visually similar and a rank 5 result classified far from the ground truth.
  • Retrieval examples: Joint sentence-keyword retrieval combines the advantages of both inputs, improving accuracy while keeping erroneous results within the correct image category.The examples are presented as evidence that finer-grained combined information benefits retrieval.

H. Explore Fast Locate Using Text

The text-based localization experiment applies AMFMN to multiscale slices of large RS scenes and aggregates their similarities into a filtered probability map. Representative examples show effective localization, although some probability remains on visually related background regions.

  • Multiscale localization: Large RS images are segmented at multiple scales and offsets, and each slice is matched with the query text to preserve targets across varying sizes.The stated slice size is 256 × 256 pixels, with repeated cutting using changed sizes and shifted vertices.
  • Localization pipeline: AMFMN similarities across slices are averaged at pixel level into a probability map, then median filtering removes impulse noise for the final location result.The pipeline uses multilevel similarity aggregation followed by median filtering.
  • Localization results: The method successfully focuses on queried regions such as a rectangular house beside a white roof and six tennis courts adjacent to trees.The tennis-court example still assigns some probability to lawn and trees, while the house example focuses on the right region.
  • Scope and comparison: AMFMN can locate RS images and detect adjacency among multiple objects, but its accuracy remains below that of specialized tasks such as target detection and road extraction.The paper characterizes text-image retrieval as a higher-level task and notes that its accuracy has not reached the ideal level.
  • Efficiency: AMFMN inference time is reported at roughly the same level as MTFN and CAMP, while SCAN is the most time-consuming method.VSE++ is simpler, whereas the proposed method is compared as a baseline for evaluation and inference time.
Loading 2204.09868v1…