PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing

Yuxi Li, Lu Si, Xia Liao, Yidong Shen, Zheng Wei, Jun Zhang, Xin Yuan

arXiv:2610.33551v1cs.CV

TL;DR

Remote-sensing referring segmentation must align language with high-resolution imagery, while existing approaches and datasets offer limited joint cross-scale reasoning and structured evaluation of complex references. The paper proposes CSSAM and introduces M3RRSIS-D, reporting competitive performance across two public benchmarks and its new dataset, particularly at strict IoU thresholds.

  • Problem

    Existing methods offer limited joint modeling of cross-scale visual hierarchies and linguistic dependencies, and datasets rarely explicitly cover multi-target, multi-hop, and multi-granularity references.

    Source: §1 · pp. 2–3

  • Method

    CSSAM combines cross-scale multimodal fusion and query-dependent sparse attention, alongside M3RRSIS-D’s explicit multi-target, multi-hop, and multi-granularity annotations.

    Source: §1 · p. 4

  • Results

    CSSAM achieves competitive segmentation performance on M3RRSIS-D and two public RRSIS benchmarks, particularly under strict IoU thresholds.

    Source: §6 · p. 27

  • Takeaways & Limitations

    M3RRSIS-D provides a structured basis for evaluating fine-grained reasoning in future RRSIS studies.

    Source: §6 · p. 27

    The study is limited by the geographic and sensor diversity of its source imagery, and failure cases include missed small target groups, incorrect spatial relations, and inconsistent grounding across expression granularities.

    Source: §6 · p. 27

Abstract

from arXiv · show

High-resolution remote sensing (RS) imagery provides fine spatial detail over large areas, making it essential for fine-grained scene understanding. Referring Remote Sensing Image Segmentation (RRSIS) aims to segment regions described by natural language, yet many existing methods rely on sequential alignment--fusion pipelines with limited joint modeling of cross-scale visual hierarchies and linguistic dependencies. Moreover, dense attention mechanisms struggle with the sparsity and wide spatial dispersion of RS targets, while current datasets offer limited structured evaluation of multi-target references, multi-hop relational reasoning, and multi-granularity descriptions, limiting both model development and evaluation. To address these challenges, we propose a Cross-Scale Sparse Attention Model (CSSAM), which aligns multiscale visual features with complex referring expressions through two key modules. The Cross-scale Modality-aligned Fusion Module (CMFM) performs unified cross-scale cross-modal fusion, while the Dynamic Multimodal Sparse Hybrid Attention (DMSHA) selectively attends to informative visual--textual tokens, reducing dense visual interactions. Furthermore, we introduce M$^{3}$RRSIS-D, a RRSIS dataset explicitly designed around three core reasoning dimensions: multi-target (M$^{1}$), reference-dependent multi-hop (M$^{2}$), and multi-granularity (M$^{3}$) expressions. Extensive experiments on M$^{3}$RRSIS-D and two public RRSIS benchmarks demonstrate that CSSAM achieves competitive performance across accuracy, fine-grained reasoning, and high-IoU localization metrics, validating its effectiveness for complex semantic reasoning in realistic RS scenarios. Code, dataset, and model weights will be released upon acceptance.

1. Introduction

RRSIS uses language to select image regions, but remote-sensing methods and datasets leave cross-scale linguistic-visual reasoning and complex reference types insufficiently addressed. The paper introduces CSSAM and M3RRSIS-D to address these method and evaluation gaps.

1. Introduction

  • CSSAM jointly models cross-scale visual context and linguistic dependencies while reducing dense attention interactions for sparse, dispersed remote-sensing targets.The paper identifies both needs as challenges for RRSIS in high-resolution imagery.Source: §1 · p. 2
  • Existing RRSIS benchmarks rarely explicitly annotate multi-target, reference-dependent multi-hop, and multi-granularity expressions.These expressions involve multiple segmentation targets, relational grounding through another object, or descriptions at different levels of detail.Source: §1 · pp. 3–4
  • CSSAM integrates cross-scale multimodal fusion with query-dependent sparse attention for remote-sensing referring segmentation.CMFM combines bidirectional cross-modal attention with text self-attention and multiscale deformable visual self-attention; DMSHA attends to sampled visual keys and values together with text tokens.Source: §1 · p. 4
  • M3RRSIS-D explicitly covers multi-target, reference-dependent multi-hop, and multi-granularity expressions to assess performance across reasoning types.The dataset is described as a benchmark aligned with real-world remote-sensing applications.Source: §1 · p. 4
  • CMFM bridges text and multiscale visual features through cross- and self-attention, while DMSHA samples cross-scale visual key-value pairs and attends jointly to them and text tokens.DMSHA sampling locations depend on both the visual query and the referring expression.Source: §1 · pp. 4–5

2. Related work

Prior remote-sensing referring-segmentation work explores multimodal fusion and feature refinement, while sparse-attention work largely targets unimodal long-text modeling. CSSAM distinguishes itself through language-conditioned sampling across visual scales.

2. Related work

  • Earlier referring-segmentation approaches include CNN-RNN alignment, recurrent multiscale refinement, cross-modal attention, and progressively structured reasoning.The cited examples range from Hu et al. and Li et al. to CMSA, CMPC, and CrossVLT.Source: §2 · p. 5
  • Recent remote-sensing methods use language-guided feature enhancement, multiscale interaction, expression decomposition, and pretrained vision-language models.Examples include LGCE, RMSIN, FIANet, and RSRefSeg.Source: §2 · p. 5
  • CSSAM predicts language-conditioned sampling offsets for each visual query and attends jointly to sampled cross-scale visual features and text tokens.The paper contrasts this with SBANet and DiffRIS, which use different alignment and reasoning approaches.Source: §2 · p. 6
  • Sparse-attention strategies include KV compression, blockwise management, sampling or hashing, and integrated dynamic sparsity, but mainly target unimodal long-text modeling.The paper identifies limited adaptation to high-resolution remote-sensing imagery with sparse, dispersed targets.Source: §2 · p. 6
  • CSSAM focuses on spatial, language-guided sampling within one image, unlike entropy-based region selection or temporal consistency learning.The comparison distinguishes its focus from EntroFormer and CTVIS.Source: §2 · pp. 6–7
  • Existing remote-sensing referring-segmentation datasets include RefSegRS and the larger-scale RRSIS-D, alongside datasets for detection and segmentation.The section also surveys RS datasets such as VisDrone, DIOR-R, and LoveDA.Source: §2 · p. 7

3. M3RRSIS-D

M3RRSIS-D is a high-resolution remote-sensing referring-segmentation benchmark built from diverse imagery and manually annotated expressions. Its protocol and statistics describe three complementary semantic dimensions and their overlap.

3.1. Dataset Sources and Annotation Pipeline

  • The dataset draws on varied remote-sensing sources, and its expressions and target polygons are manually annotated and independently verified.Sources were selected to span imaging conditions, spatial resolutions, environments, and object categories.Source: §3 · p. 7, §3.1 · p. 8

3.2. Annotation Protocol

  • Group A authors expressions and polygon masks, while Group B independently localizes targets using only the text.The protocol involves 36 Group-A annotators and 7 Group-B annotators and lasted approximately 100 days.Source: §3.2 · p. 9
  • M1 covers multiple segmentation targets, M2 grounds targets through spatial relations to reference objects, and M3 groups descriptions of the same mask at up to five detail levels.M3 levels add attributes and relational context; a description may satisfy multiple dimensions.Source: §3.2 · pp. 10–11

3.3. Dataset Statistics

  • M3RRSIS-D contains 5,603 high-resolution images and 66,305 referring expressions across training and validation subsets.The section presents the dataset as covering multi-instance targets, relational reasoning, and multiple semantic-granularity levels.Source: §3.3 · p. 11
  • The dimensions are complementary rather than mutually exclusive, and M3RRSIS-D averages 11.83 expressions per image with mostly small masks and a few large regions.M1, M2, and M3 include 7,723, 23,357, and 51,976 expressions, respectively; their pairwise and three-way intersections are also reported.Source: §3.3 · p. 11

3.4. Distinctive Advantages over Existing RRSIS Datasets

  • Table 1 contrasts explicit semantic-dimension annotation with properties that may be present but are not annotated in existing datasets.M3RRSIS-D explicitly annotates multi-target, multi-hop, and multi-granularity dimensions.Source: §3.4 · p. 12, Table 1 · p. 12

4. Cross-Scale Sparse Attention Model

CSSAM maps an image and query text to a binary mask using cross-scale multimodal fusion, dynamically sampled sparse attention, a fusion gate, and a segmentation head. Its modules combine information across visual scales while limiting dense visual interactions.

4.1. Model Architecture

  • CSSAM encodes image and text inputs, aligns their features across scales, applies DMSHA, merges the module outputs with a fusion gate, and decodes fused features into a binary mask.The architecture uses visual and textual encoders, CMFM, DMSHA, a fusion gate, and a segmentation head.Source: §4.1 · p. 12

4.2. CMFM Module

  • CMFM applies bidirectional cross-modal attention, modality-specific self-attention, and multi-scale visual self-attention to exchange information across visual levels.The module stacks 12 layers with independent parameters.Source: §4.2 · pp. 13–14

4.3. DMSHA Module

  • DMSHA predicts query-dependent sampling offsets, gathers sparse visual evidence across scales, and attends jointly to that evidence and text tokens.Each head samples S points per level, yielding LS visual samples per query; DMSHA follows CMFM and its output is gated with CMFM features before decoding.Source: §4.3 · pp. 14–16

4.4. Fusion Gate

  • The fusion gate adaptively merges fusion-enhanced and DMSHA sparse-enhanced features at each scale.A spatial attention gate is used for the per-scale merge.Source: §4.4 · p. 16

4.5. Segmentation Head

  • A U-Net-like convolutional head progressively upsamples and merges features from low to high resolution, then applies a 1 × 1 convolution and sigmoid to produce the binary mask.Convolutional blocks refine the features during the upsample-and-merge process.Source: §4.5 · p. 17

4.6. Complexity and Memory Analysis of DMSHA Module

  • DMSHA attention-score memory is O(M(LS + T)), versus O(M(M + T)) for dense joint attention, with the visual-only interaction term reduced from O(M^2) to O(MLS).This analysis covers DMSHA attention, not the backbones, CMFM, or segmentation head.Source: §4.6 · p. 17

4.7. Loss Function

  • The training objective combines cross-entropy for pixel-wise classification with Dice loss for prediction–ground-truth mask overlap.The default loss weight is λ = 0.1.Source: §4.7 · p. 18

5. Experiments and Evaluations

Experiments compare CSSAM with prior methods, assess performance across reasoning dimensions, and ablate model choices. CSSAM performs strongly on M3RRSIS-D and two public benchmarks, with trade-offs in memory, speed, and some metric-specific rankings.

5.1. Implementation Details

  • Experiments use frozen Swin-B and BERTbase-uncased encoders initialized from Grounding DINO-B, train remaining modules for 20 epochs, and report mIoU, oIoU, and P@τ.Images and masks are resized to 800 × 800; P@τ is reported for τ=0.5, 0.6, 0.7, 0.8, and 0.9.Source: §5.1 · pp. 18–19

5.2. Comparison with State-of-the-Art Methods on M3RRSIS-D

  • 1.60 and 2.79 percentage points: CSSAM exceeds the strongest baselines in oIoU and mIoU, respectively, and improves across all IoU thresholds.The comparison is on M3RRSIS-D; Table 2 presents the benchmark results.Source: §5.2 · p. 19, Table 2 · p. 20

5.3. Fine-Grained Evaluation by Reasoning Dimension

  • Across M1–M3, CSSAM leads in mIoU and all reported precision thresholds and in oIoU for M2 and M3; LAVT leads in M1 oIoU.Across granularity levels L1–L5, CSSAM has the highest mIoU at every level, but LAVT, RSRefSeg, and FIANet each lead CSSAM on some oIoU comparisons.Source: §5.3 · p. 21, Table 3 · p. 22

5.4. Ablation Studies

  • Both CMFM and DMSHA improve performance individually and together achieve the best results; removing deformable visual attention lowers mIoU from 75.42% to 68.97%.The tested λ values of 0 and 0.5 yield 72.80% and 72.78% mIoU, respectively, versus 75.42% at λ=0.1; non-uniform allocations yield 73.44%–74.04%, below the uniform allocation’s 75.42%.Source: §5.4 · pp. 22–23, Table 4 · p. 24

5.3. Fine-Grained Evaluation by Reasoning Dimension

  • 75.42% mIoU and 82.45% oIoU: CSSAM’s validation results have a median per-query IoU of 86.30%, while 6.58% of queries fall below 10%.Reported failures include missed small target groups, a reversed spatial relation, and inconsistent grounding after adding a fine-grained descriptor.Source: §5.3 · p. 21, Figure 8 · p. 23, Figure 8b

5.4. Ablation Studies

  • The Transformer segmentation head achieves 72.72% mIoU and 80.59% oIoU, compared with 75.42% and 82.45% for the convolutional head.Both head variants are trained for 20 epochs.Source: §5.4 · p. 24
  • Dynamic sampling scores highest; increasing S from 4 to 10 gains 0.67 mIoU but raises memory from 17.25 to 23.55 GB and cuts throughput from 7.67 to 6.09 queries/s; dense attention runs out of memory.The authors use S=4 to balance accuracy, memory demand, and speed.Source: Table 5 · p. 25, §5.4 · p. 25

5.5. Additional Evaluation on Public RRSIS Benchmarks

  • On RRSIS-D, CSSAM exceeds SBANet by 5.45 mIoU points and leads on every reported metric except oIoU; on RefSegRS, it improves mIoU over HTVG by 4.43 points.On RefSegRS, FIANet and HTVG are marginally better at P@0.5 and oIoU, respectively, while CSSAM achieves substantially higher P@0.8 and P@0.9.Source: Table 6 · p. 26, Table 7 · p. 26, §5.5 · p. 27

6. Conclusion

The paper presents CSSAM and M3RRSIS-D for remote-sensing referring segmentation and reports competitive results on the new and two public benchmarks. The study is bounded by the geographic and sensor diversity of its source imagery.

6. Conclusion

  • CSSAM combines cross-scale multimodal alignment with dynamically sampled sparse attention, while M3RRSIS-D covers multi-target, multi-hop relational, and multi-granularity expressions.The benchmark provides a structured basis for evaluating fine-grained reasoning in future RRSIS studies.Source: §6 · p. 27
  • Experiments on M3RRSIS-D and two public RRSIS benchmarks show competitive segmentation performance, particularly under strict IoU thresholds.Source: §6 · p. 27
  • The study is limited by the geographic and sensor diversity of its source imagery; broader multi-sensor scenarios and lighter implementations are future work.Source: §6 · p. 27

CRediT authorship contribution statement

The statement assigns research and writing responsibilities across the authors, reports no known competing interests, and describes limited use of generative AI tools. Dataset, code, and trained weights are to become public upon acceptance.

CRediT authorship contribution statement

  • Yuxi Li and Lu Si contributed methodology, original-draft writing, review and editing, software, and data curation; Xia Liao led supervision, conceptualization, resources, project administration, and review.Yidong Shen and Zheng Wei contributed software, validation, and data curation; Jun Zhang contributed supervision, resources, and review; Xin Yuan contributed conceptualization, supervision, resources, and review.Source: CRediT authorship contribution statement · p. 28

Declaration of competing interest

preparation process

  • Generative AI tools were used for spelling and grammar checking, language polishing, and readability, but not to generate research ideas, data, or results.The authors state that they reviewed and edited the content after using the tools.Source: preparation process · p. 28

Data and Code Availability

  • The dataset, source code, and trained weights are stated to become publicly available upon acceptance; the repository is currently private.Source: Data and Code Availability · p. 28

Related papers

Loading 2610.33551v1…