RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data

• RSVG remains under-explored despite extensive research on language-guided RS captioning, retrieval, and visual question answering.Source: §I · p. 1
• RSVG localizes a referred object in an RS image from a phrase or sentence, producing its bounding box.Source: §I · p. 1
• The overall framework performs multi-modal encoding, multi-level cross-modal learning, multi-modal fusion, and localization end to end.Source: Figure 1 · p. 1
• RSVGD is a benchmark constructed from DIOR using automatic image-query generation with manual assistance, containing 38,320 pairs and 17,402 images.Source: §I · pp. 1–2
• The paper benchmarks two-stage, one-stage, and transformer-based natural-image grounding methods on RSVGD, finding that transfer yields only acceptable results.Source: §I · p. 2
• MLCM addresses RS scale variation and cluttered backgrounds by learning from multi-scale visual and multi-granularity textual features while filtering irrelevant noise.Source: §I · p. 2

Two-stage Visual Grounding Methods

• Two-stage grounding methods extract candidate-region features and score language-region correspondence, often using contextual comparison, attention, or subject-relation-object parsing.Source: A. Two-stage Visual Grounding Methods · p. 2

One-stage Visual Grounding Methods

• One-stage methods avoid multi-stage error accumulation and are more computation-efficient, using CNN/LSTM features, multimodal pooling, and guided attention.Source: B. One-stage Visual Grounding Methods · p. 3

Transformer-based Visual Grounding Methods

• Transformer-based methods use end-to-end attention architectures such as VGTR and TransVG to model expression-guided visual features and multimodal fusion.Source: C. Transformer-based Visual Grounding… · p. 3

A. RSVGD: a new dataset for RSVG

• RSVGD samples DIOR boxes after removing annotation and size errors and limits each category to five objects per image to reduce ambiguous references.Source: A. RSVGD: a new dataset for RSVG · p. 3
• The dataset extracts object and relational attributes, including category, color, geometry, absolute location, relative location, and relative size.Source: A. RSVGD: a new dataset for RSVG · pp. 3–4
• Templates generate phrase expressions from object attributes and sentence expressions from relationships between two objects, with uniqueness checks guiding selection.Source: A. RSVGD: a new dataset for RSVG · p. 4
• Worker verification corrects attribute errors and ambiguous expressions arising from complex backgrounds, bounding-box regions, and location judgments.Source: A. RSVGD: a new dataset for RSVG · p. 4

• 38,320 language expressions span 17,402 RS images and 20 object categories, with an average expression length of 7.47 and a vocabulary of 100.Source: B. Data analysis · p. 4
• Vehicle is the most frequent category, harbor is the least frequent, and the other 18 categories each comprise 2%–10% of the dataset.Source: B. Data analysis · p. 4
• Most query expressions contain two attributes, followed by four, while expressions with five or six attributes are uncommon.Source: B. Data analysis · p. 4, Figure 3 · p. 5
• RSVGD commonly uses color, size, and absolute-position attributes, with upper-right and left relative-location relations among the most frequent relationships.Source: B. Data analysis · p. 5, Figure 3 · p. 5
• The dataset’s bounding-box areas are mainly within 20% of the RS image, and word-cloud sizes represent term frequency.Source: B. Data analysis · p. 5, Figure 4 · p. 5

Overview

• The transformer-based framework has four components: a multimodal encoder, MLCM module, multimodal fusion module, and localization module.Source: §IV · p. 5, Figure 5 · p. 6

Multimodal Encoder

• The CNN backbone uses a truncated ResNet-50 plus six convolution layers to extract multi-scale visual features while preserving information about differently sized objects.Source: B. Multi-level Cross-modal Fusion · p. 6
• BERT supplies word-level and sentence-level textual embeddings, capturing local semantic and global sentence-context information.Source: B. Multi-level Cross-modal Fusion · p. 6

Multi-level Cross-modal Fusion

• MLCM refines visual features by selecting and aggregating valid information from multi-level, multi-modal inputs through cross-modal learning and self-attention.Source: A. Overview · p. 5, B. Multi-level Cross-modal Fusion · p. 6

Multimodal Fusion

• The multimodal fusion module combines projected visual and textual tokens with a learnable token before a six-layer fusion transformer.Source: B. Multi-level Cross-modal Fusion · p. 7

Localization

• The localization module maps the fused learnable-token representation through an MLP to a four-dimensional bounding-box coordinate.Source: B. Multi-level Cross-modal Fusion · p. 7

• The framework applies smooth L1 loss and generalized IoU loss to the four-dimensional bounding-box coordinates.Source: C. Loss · p. 7
• GIoU is included because smooth L1 can produce scale-dependent values for large and small target boxes, whereas GIoU is not affected by scale.Source: C. Loss · p. 7
• Ground-truth coordinates are normalized according to image size, and λ balances the smooth L1 and GIoU losses.Source: C. Loss · p. 7

Main experiments

• The experiments compare the proposed method with state-of-the-art visual grounding approaches on RSVG.Source: §V · p. 7

Ablation and qualitative analysis

• Ablation experiments assess the effectiveness of the proposed MLCM module, alongside qualitative result analysis.Source: §V · p. 7

• A predicted bounding box is counted as correct when its IoU with the ground truth exceeds the selected threshold.Precision is reported at thresholds 0.5, 0.6, 0.7, 0.8, and 0.9 as Pr@0.5 through Pr@0.9.Source: A. Evaluation metrics for RSVG · p. 8
• The evaluation additionally reports mean IoU and cumulative IoU (cumIoU).For mean IoU, It and Ut denote intersection and union areas for image-query pair t, while M is dataset size.Source: A. Evaluation metrics for RSVG · p. 8

• RSVGD is split into training, validation, and test sets using 40%, 10%, and 50% of expressions and corresponding images.Source: B. Implementation Details · p. 8
• Images are resized to 640×640, language is limited to Nt = 40 tokens, and feature dimension is set to D = 256.Source: B. Implementation Details · p. 8
• ResNet-50 and MLCM use DETR pretraining, while BERTbase is initialized with pretrained BERT weights.Source: B. Implementation Details · p. 8

• The proposed method outperforms the compared state-of-the-art natural-image visual grounding methods on RSVGD.The comparison includes two-stage, one-stage, and transformer-based methods.Source: C. Remote Sensing Image Visual… · p. 8
• The proposed transformer structure enables fuller interaction between visual and textual information than manually designed fusion mechanisms.Source: C. Remote Sensing Image Visual… · p. 8
• Except at Pr@0.9, the proposed method is much higher than one-stage methods on the other reported metrics.FAOA achieves the best accuracy at Pr@0.9, but remains deficient at smaller thresholds.Source: C. Remote Sensing Image Visual… · p. 8
• VLTVG improves visual distinctiveness by focusing on text-related regions but remains insufficient because it ignores multi-level modality information.Source: C. Remote Sensing Image Visual… · p. 8

Ablation results

• 72.41% Pr@0.5 is achieved without multi-level cross-modal feature learning, while complete MLCM improves performance by 4.37%.Source: D. Ablation Study · p. 8
• Using only multi-scale visual features drops performance by 6.63%, whereas adding only multi-granularity textual embeddings improves it by 0.37%.Source: D. Ablation Study · p. 8
• Adding sentence-level and word-level textual embeddings individually drops performance by 4.95% and 1.51%, respectively.Source: D. Ablation Study · p. 8

Attention analysis

• Multi-scale-only attention maps contain substantial dark regions in background or non-target areas, indicating introduced noise from cluttered RS imagery.Source: D. Ablation Study · p. 8
• Multi-granularity textual embeddings greatly filter attention-map noise, while complete MLCM adaptively filters irrelevant noise and enhances salient features.Source: D. Ablation Study · p. 9

Qualitative grounding results

• MLCM accurately localizes objects described with specific attributes and generates attention corresponding to the target’s shape and location.Visualizations include bridges, overpasses, and vehicles, with attention enhanced in their corresponding regions.Source: E. Qualitative Results · p. 9

Small-scale targets

• MLCM improves small-scale object grounding by combining multiscale visual features with fine-granularity textual embeddings.This interaction precisely enhances visual features associated with small-scale objects.Source: E. Qualitative Results · p. 9

Attention behavior

• Wrong attention regions mainly involve same-category objects or objects related to the target, but word-level embeddings help align modalities for accurate grounding.The transformer-based multimodal fusion module is described as avoiding the impact of these irrelevant attention regions.Source: E. Qualitative Results · pp. 9–10

Failure cases

• Failure cases arise from cluttered backgrounds, ambiguous or incomplete expressions, and instances where the model fails despite salient objects and clear attributes.These cases indicate that the approach still has shortcomings requiring further research.Source: E. Qualitative Results · p. 10

Conclusion

• The paper introduces RSVG and constructs RSVGD, a large-scale dataset for grounding natural-language expressions in remote-sensing imagery.RSVGD is obtained from DIOR using automatic generation with manual verification.Source: §VI · pp. 10–11
• Natural-image visual grounding methods achieve only acceptable results on RSVGD, motivating methods tailored to remote-sensing imagery.The benchmark covers extensive state-of-the-art natural-image methods and analyzes their results.Source: §VI · p. 11
• MLCM addresses cluttered backgrounds and scale variation by adapting to multi-scale inputs and integrating multi-level, multi-modal features.The module learns visual attention relevant to the query.Source: §VI · p. 11
• The proposed approach achieves better performance than existing natural-image visual grounding methods on the constructed RSVGD benchmark.The conclusion presents this comparison as evidence of the approach’s superiority.Source: §VI · p. 11
• Future RSVG research should further consider the distinctive characteristics of remote-sensing images.Source: §VI · p. 11