PaperlayerView exact revision on arXiv

Source-linked AI summary

RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data

Yang Zhan, Zhitong Xiong, Yuan Yuan

arXiv:2210.12634v1cs.CV

TL;DR

The paper addresses the under-explored problem of localizing language-referred objects in remote-sensing imagery. It constructs RSVGD, benchmarks existing methods, and proposes MLCM; natural-image methods achieve only acceptable results, while MLCM targets RS scale variation and clutter.

  • Problem

    Object-level visual grounding for remote-sensing images remains under-explored despite related language-guided RS tasks.

  • Method

    The paper constructs RSVGD from DIOR with automatic generation and manual verification, and proposes transformer-based MLCM using multi-scale visual and multi-granularity textual features.

  • Results

    Natural-image visual grounding methods obtain only acceptable results on RSVGD, while the proposed method outperforms other works in the reported comparison.

  • Takeaways & Limitations

    RSVGD, benchmark analyses, and MLCM provide a basis for advancing visual grounding research on remote-sensing data.

Abstract

from arXiv · show

In this paper, we introduce the task of visual grounding for remote sensing data (RSVG). RSVG aims to localize the referred objects in remote sensing (RS) images with the guidance of natural language. To retrieve rich information from RS imagery using natural language, many research tasks, like RS image visual question answering, RS image captioning, and RS image-text retrieval have been investigated a lot. However, the object-level visual grounding on RS images is still under-explored. Thus, in this work, we propose to construct the dataset and explore deep learning models for the RSVG task. Specifically, our contributions can be summarized as follows. 1) We build the new large-scale benchmark dataset of RSVG, termed RSVGD, to fully advance the research of RSVG. This new dataset includes image/expression/box triplets for training and evaluating visual grounding models. 2) We benchmark extensive state-of-the-art (SOTA) natural image visual grounding methods on the constructed RSVGD dataset, and some insightful analyses are provided based on the results. 3) A novel transformer-based Multi-Level Cross-Modal feature learning (MLCM) module is proposed. Remotely-sensed images are usually with large scale variations and cluttered backgrounds. To deal with the scale-variation problem, the MLCM module takes advantage of multi-scale visual features and multi-granularity textual embeddings to learn more discriminative representations. To cope with the cluttered background problem, MLCM adaptively filters irrelevant noise and enhances salient features. In this way, our proposed model can incorporate more effective multi-level and multi-modal features to boost performance. Furthermore, this work also provides useful insights for developing better RSVG models. The dataset and code will be publicly available at https://github.com/ZhanYang-nwpu/RSVG-pytorch.

I. INTRODUCTION

The paper introduces RSVG, which localizes objects in remote-sensing images from natural-language expressions, and develops a dataset and transformer-based approach for the task.

  • RSVG remains under-explored despite extensive research on language-guided RS captioning, retrieval, and visual question answering.
  • RSVG localizes a referred object in an RS image from a phrase or sentence, producing its bounding box.
  • The overall framework performs multi-modal encoding, multi-level cross-modal learning, multi-modal fusion, and localization end to end.
  • RSVGD is a benchmark constructed from DIOR using automatic image-query generation with manual assistance, containing 38,320 pairs and 17,402 images.
  • The paper benchmarks two-stage, one-stage, and transformer-based natural-image grounding methods on RSVGD, finding that transfer yields only acceptable results.
  • MLCM addresses RS scale variation and cluttered backgrounds by learning from multi-scale visual and multi-granularity textual features while filtering irrelevant noise.

A. Two-stage Visual Grounding Methods

This section reviews visual grounding methods and describes how RSVGD expressions are generated from sampled DIOR objects and their attributes.

  • Two-stage Visual Grounding Methods: Two-stage grounding methods extract candidate-region features and score language-region correspondence, often using contextual comparison, attention, or subject-relation-object parsing.
  • One-stage Visual Grounding Methods: One-stage methods avoid multi-stage error accumulation and are more computation-efficient, using CNN/LSTM features, multimodal pooling, and guided attention.
  • Transformer-based Visual Grounding Methods: Transformer-based methods use end-to-end attention architectures such as VGTR and TransVG to model expression-guided visual features and multimodal fusion.
  • A. RSVGD: a new dataset for RSVG: RSVGD samples DIOR boxes after removing annotation and size errors and limits each category to five objects per image to reduce ambiguous references.
  • A. RSVGD: a new dataset for RSVG: The dataset extracts object and relational attributes, including category, color, geometry, absolute location, relative location, and relative size.
  • A. RSVGD: a new dataset for RSVG: Templates generate phrase expressions from object attributes and sentence expressions from relationships between two objects, with uniqueness checks guiding selection.
  • A. RSVGD: a new dataset for RSVG: Worker verification corrects attribute errors and ambiguous expressions arising from complex backgrounds, bounding-box regions, and location judgments.

B. Data analysis

RSVGD contains 38,320 language expressions across 17,402 RS images and 20 object categories. Its statistical analysis characterizes object, attribute, relationship, expression-length, and bounding-box distributions.

  • 38,320 language expressions span 17,402 RS images and 20 object categories, with an average expression length of 7.47 and a vocabulary of 100.
  • Vehicle is the most frequent category, harbor is the least frequent, and the other 18 categories each comprise 2%–10% of the dataset.
  • Most query expressions contain two attributes, followed by four, while expressions with five or six attributes are uncommon.
  • RSVGD commonly uses color, size, and absolute-position attributes, with upper-right and left relative-location relations among the most frequent relationships.
  • The dataset’s bounding-box areas are mainly within 20% of the RS image, and word-cloud sizes represent term frequency.

IV. METHODS

The proposed transformer-based RSVG framework combines multimodal encoding, MLCM refinement, multimodal fusion, and localization. MLCM integrates multi-scale visual and multi-granularity textual features while filtering noise and modeling salient-region dependencies.

  • Overview: The transformer-based framework has four components: a multimodal encoder, MLCM module, multimodal fusion module, and localization module.
  • Multimodal Encoder: The CNN backbone uses a truncated ResNet-50 plus six convolution layers to extract multi-scale visual features while preserving information about differently sized objects.
  • Multimodal Encoder: BERT supplies word-level and sentence-level textual embeddings, capturing local semantic and global sentence-context information.
  • Multi-level Cross-modal Fusion: MLCM refines visual features by selecting and aggregating valid information from multi-level, multi-modal inputs through cross-modal learning and self-attention.
  • Multimodal Fusion: The multimodal fusion module combines projected visual and textual tokens with a learnable token before a six-layer fusion transformer.
  • Localization: The localization module maps the fused learnable-token representation through an MLP to a four-dimensional bounding-box coordinate.

C. Loss

Training combines smooth L1 and generalized IoU losses for bounding-box regression. GIoU addresses the large scale variation of RS objects because it is not affected by target-box scale.

  • The framework applies smooth L1 loss and generalized IoU loss to the four-dimensional bounding-box coordinates.
  • GIoU is included because smooth L1 can produce scale-dependent values for large and small target boxes, whereas GIoU is not affected by scale.
  • Ground-truth coordinates are normalized according to image size, and λ balances the smooth L1 and GIoU losses.

V. EXPERIMENTS

The experiments evaluate RSVG using multiple IoU-threshold precision metrics, mean IoU, and cumulative IoU, and compare the proposed method with state-of-the-art approaches and ablations.

  • Main experiments: The experiments compare the proposed method with state-of-the-art visual grounding approaches on RSVG.
  • Ablation and qualitative analysis: Ablation experiments assess the effectiveness of the proposed MLCM module, alongside qualitative result analysis.

A. Evaluation metrics for RSVG

RSVG evaluation uses IoU-threshold precision alongside mean IoU and cumulative IoU to assess predicted bounding boxes.

  • A predicted bounding box is counted as correct when its IoU with the ground truth exceeds the selected threshold.Precision is reported at thresholds 0.5, 0.6, 0.7, 0.8, and 0.9 as Pr@0.5 through Pr@0.9.
  • The evaluation additionally reports mean IoU and cumulative IoU (cumIoU).For mean IoU, It and Ut denote intersection and union areas for image-query pair t, while M is dataset size.

B. Implementation Details

The implementation randomly partitions RSVGD, standardizes images and language dimensions, and initializes visual and textual components from pretrained models.

  • RSVGD is split into training, validation, and test sets using 40%, 10%, and 50% of expressions and corresponding images.
  • Images are resized to 640×640, language is limited to Nt = 40 tokens, and feature dimension is set to D = 256.
  • ResNet-50 and MLCM use DETR pretraining, while BERTbase is initialized with pretrained BERT weights.

C. Remote Sensing Image Visual Grounding Results

On RSVGD, the proposed transformer-based approach outperforms compared natural-image grounding methods, while its multi-level feature interaction addresses limitations of alternative architectures.

  • The proposed method outperforms the compared state-of-the-art natural-image visual grounding methods on RSVGD.The comparison includes two-stage, one-stage, and transformer-based methods.
  • The proposed transformer structure enables fuller interaction between visual and textual information than manually designed fusion mechanisms.
  • Except at Pr@0.9, the proposed method is much higher than one-stage methods on the other reported metrics.FAOA achieves the best accuracy at Pr@0.9, but remains deficient at smaller thresholds.
  • VLTVG improves visual distinctiveness by focusing on text-related regions but remains insufficient because it ignores multi-level modality information.

D. Ablation Study

The ablation study shows that complete multi-level cross-modal learning improves RSVG performance and filters clutter-induced attention noise more effectively than partial feature configurations.

  • Ablation results: 72.41% Pr@0.5 is achieved without multi-level cross-modal feature learning, while complete MLCM improves performance by 4.37%.
  • Ablation results: Using only multi-scale visual features drops performance by 6.63%, whereas adding only multi-granularity textual embeddings improves it by 0.37%.
  • Ablation results: Adding sentence-level and word-level textual embeddings individually drops performance by 4.95% and 1.51%, respectively.
  • Attention analysis: Multi-scale-only attention maps contain substantial dark regions in background or non-target areas, indicating introduced noise from cluttered RS imagery.
  • Attention analysis: Multi-granularity textual embeddings greatly filter attention-map noise, while complete MLCM adaptively filters irrelevant noise and enhances salient features.

E. Qualitative Results

Qualitative results show that MLCM can localize described objects, including small-scale targets, while producing interpretable attention; failures remain under clutter, ambiguous expressions, and model limitations.

  • Qualitative grounding results: MLCM accurately localizes objects described with specific attributes and generates attention corresponding to the target’s shape and location.Visualizations include bridges, overpasses, and vehicles, with attention enhanced in their corresponding regions.
  • Small-scale targets: MLCM improves small-scale object grounding by combining multiscale visual features with fine-granularity textual embeddings.This interaction precisely enhances visual features associated with small-scale objects.
  • Attention behavior: Wrong attention regions mainly involve same-category objects or objects related to the target, but word-level embeddings help align modalities for accurate grounding.The transformer-based multimodal fusion module is described as avoiding the impact of these irrelevant attention regions.
  • Failure cases: Failure cases arise from cluttered backgrounds, ambiguous or incomplete expressions, and instances where the model fails despite salient objects and clear attributes.These cases indicate that the approach still has shortcomings requiring further research.

VI. CONCLUSION

The paper introduces RSVG and the RSVGD benchmark, evaluates natural-image grounding methods, and proposes MLCM for remote-sensing scale variation and clutter. The authors report better performance than existing natural-image methods while identifying continued need for RSVG research.

  • Conclusion: The paper introduces RSVG and constructs RSVGD, a large-scale dataset for grounding natural-language expressions in remote-sensing imagery.RSVGD is obtained from DIOR using automatic generation with manual verification.
  • Conclusion: Natural-image visual grounding methods achieve only acceptable results on RSVGD, motivating methods tailored to remote-sensing imagery.The benchmark covers extensive state-of-the-art natural-image methods and analyzes their results.
  • Conclusion: MLCM addresses cluttered backgrounds and scale variation by adapting to multi-scale inputs and integrating multi-level, multi-modal features.The module learns visual attention relevant to the query.
  • Conclusion: The proposed approach achieves better performance than existing natural-image visual grounding methods on the constructed RSVGD benchmark.The conclusion presents this comparison as evidence of the approach’s superiority.
  • Conclusion: Future RSVG research should further consider the distinctive characteristics of remote-sensing images.
Loading 2210.12634v1…