PaperlayerView exact revision on arXiv

Source-linked AI summary

RRSIS: Referring Remote Sensing Image Segmentation

Zhenghang Yuan, Lichao Mou, Yuansheng Hua, Xiao Xiang Zhu

arXiv:2306.08625v2cs.CV

TL;DR

Remote-sensing referring image segmentation had received little research attention, despite its potential for practical image interpretation. The paper introduces the task, builds RefSegRS, and proposes LGCE; experiments report that existing methods often miss small and scattered objects, while LGCE improves performance on challenging scenes.

  • Problem

    Referring image segmentation is extensively studied for natural images but has received almost no attention for remote-sensing imagery, where users could specify objects for targeted image analysis.

    Source: §I · pp. 1–2

  • Method

    The paper creates the RefSegRS dataset and proposes LGCE, which uses language guidance to adaptively enhance multi-scale visual features by integrating deep and shallow features.

    Source: §I · p. 2, C. Language-guided Cross-scale… · p. 6

  • Results

    Existing referring-segmentation methods show limited efficacy on small and scattered remote-sensing objects, while LGCE’s effectiveness on such objects is verified by quantitative and qualitative experiments.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    The dataset, benchmark analysis, and LGCE provide insights into designing RRSIS models for remote-sensing images.

    Source: §VI · p. 10

Abstract

from arXiv · show

Localizing desired objects from remote sensing images is of great use in practical applications. Referring image segmentation, which aims at segmenting out the objects to which a given expression refers, has been extensively studied in natural images. However, almost no research attention is given to this task of remote sensing imagery. Considering its potential for real-world applications, in this paper, we introduce referring remote sensing image segmentation (RRSIS) to fill in this gap and make some insightful explorations. Specifically, we create a new dataset, called RefSegRS, for this task, enabling us to evaluate different methods. Afterward, we benchmark referring image segmentation methods of natural images on the RefSegRS dataset and find that these models show limited efficacy in detecting small and scattered objects. To alleviate this issue, we propose a language-guided cross-scale enhancement (LGCE) module that utilizes linguistic features to adaptively enhance multi-scale visual features by integrating both deep and shallow features. The proposed dataset, benchmarking results, and the designed LGCE module provide insights into the design of a better RRSIS model. We will make our dataset and code publicly available.

I. INTRODUCTION

The paper introduces referring remote sensing image segmentation (RRSIS), a language-guided task for producing pixel-level masks of desired objects, and creates RefSegRS to evaluate it. Existing natural-image models often miss small and scattered remote-sensing objects; the paper proposes LGCE to improve their segmentation.

I. INTRODUCTION

  • RRSIS uses a language expression and remote-sensing image to produce a pixel-level mask of desired objects.The authors note that the task had received almost no research attention in remote sensing, despite extensive study in natural images.Source: §I · p. 2
  • Natural-image referring segmentation models often miss small and scattered remote-sensing objects when applied directly to RefSegRS.The authors attribute this difficulty to remote-sensing objects occupying fewer pixels and appearing small and scattered.Source: §I · p. 2
  • LAVT outperforms existing CNN-based methods but still struggles with small and scattered remote-sensing objects.The authors propose LGCE to address this remaining difficulty.Source: §I · p. 2
  • RefSegRS contains 4,420 image-language-label triplets with objects spanning a wide range of scales.The dataset was created for the newly introduced RRSIS task.Source: §I · p. 2
  • The paper evaluates referring-image segmentation methods originally designed for natural images on RefSegRS and analyzes their quantitative and qualitative results.The evaluation is intended to provide insights into RRSIS.Source: §I · p. 2
  • LGCE is designed to improve segmentation of small and scattered objects by using language guidance to integrate deep and shallow visual features.Deep features are abstract and expressive, while shallow features have higher spatial resolution but lower semantic level.Source: §I · p. 2

II. RELATED WORK

The related work reviews referring image segmentation in natural images and language-related tasks in remote sensing. It also describes remote-sensing visual grounding, which localizes language-mentioned objects with bounding boxes.

II. RELATED WORK

  • The review covers referring-image segmentation in natural images and remote-sensing combinations with language, including visual grounding, VQA, and image captioning.These topics frame the paper’s related-work scope.Source: §II · p. 2

A. Referring Image Segmentation on Natural Images

  • Referring-image segmentation commonly encodes visual and linguistic features, fuses them, and predicts masks; methods vary in their encoders and fusion mechanisms.LAVT uses BERT and a Transformer network, including pixel-word attention to align visual representations with language features.Source: A. Referring Image Segmentation on… · pp. 2–3

B. Visual Grounding in Remote Sensing

  • Remote-sensing visual grounding localizes language-mentioned objects with bounding boxes, and existing research includes datasets and methods for aerial-scene challenges.Described approaches address numerical geospatial relations, background clutter, and scale variation.Source: B. Visual Grounding in Remote Sensing · p. 3

C. Visual Question Answering for Remote Sensing Data

The section surveys remote-sensing VQA and image captioning before describing RefSegRS construction. Its expressions combine object categories with attributes and spatial relationships, and “parking area” is the most frequent object expression.

C. Visual Question Answering for Remote Sensing Data

  • Remote-sensing VQA research spans datasets and multimodal methods for answering image-based questions, including attention, bilinear fusion, Transformers, and spatial reasoning.One described approach injects visual context into a language model; another extends VQA to change detection.Source: C. Visual Question Answering for… · p. 3

D. Remote Sensing Image Captioning

  • Remote-sensing image captioning research uses CNN-RNN and encoder-decoder systems, attention methods, and Transformers to generate textual image descriptions.The reviewed work also includes a Transformer approach addressing limited data and a model leveraging multi-scale decoder features.Source: D. Remote Sensing Image Captioning · p. 3

A. Image Collection

  • RefSegRS images come from SkyScapes tiles cropped into 1,200 × 1,200-pixel windows at a 600-pixel stride, then downsampled to 512 × 512 pixels.The source tiles contain 20 labeled categories and have 13 cm spatial resolution.Source: A. Image Collection · p. 3

B. Referring Expression Generation

  • Referring expressions use object categories, attributes, and spatial relationships, including adjacency and containment.The dataset includes templates combining categories with attributes, spatial relations, or both.Source: B. Referring Expression Generation · pp. 3–4
  • “Parking area” is the most frequently recurring object in the generated expressions.The dataset also includes expressions specifying categories, attributes, and spatial relationships.Source: B. Referring Expression Generation · pp. 3–4

C. Mask Generation

Mask generation maps referring expressions to binary masks using SkyScapes pixel labels, with additional filtering for spatial relationships.

C. Mask Generation

  • SkyScapes pixel-wise labels provide the source for binary ground-truth masks tied to referring expressions.Pixels inside and outside a category are set to 1 and 0, respectively.Source: C. Mask Generation · pp. 4–5
  • Expressions connect to original labels through identity or inclusion relationships.For example, “road marking” is identical to “lane marking,” while “light-duty vehicle” includes “car” and “van.”Source: C. Mask Generation · p. 4
  • For spatial expressions, connected instances are filtered against the stated relationships, buffered for adjacency judgments, and gathered into final masks.Source: C. Mask Generation · p. 5

D. Dataset Statistics

RefSegRS contains 4,420 image-language-label triplets across 285 scenes, and the methodology builds on LAVT while adding language-guided feature enhancement.

D. Dataset Statistics

  • 4,420 image-language-label triplets span 285 scenes, split into 2,172 training, 431 validation, and 1,817 test expressions without scene overlap.Source: D. Dataset Statistics · p. 5

IV. METHODOLOGY

  • The RRSIS framework uses visual and language encoders, multimodal fusion, and a segmentation head to predict masks.The visual and linguistic features are fused before the segmentation head produces the mask.Source: §IV · p. 5
  • The method adopts Transformer-based LAVT as its baseline and adds LGCE to improve identification of small and scattered remote-sensing objects.Source: §IV · p. 5

A. Overall Architecture

The architecture uses LAVT with Swin Transformer visual features, BERT language embeddings, and language-visual fusion. Its LGCE module addresses the need to combine deep and shallow features using language guidance.

A. Overall Architecture

  • The model adopts LAVT, using Swin Transformer for visual features and BERT for language embeddings.LAVT’s PWAM and language gate fuse language and visual features.Source: A. Overall Architecture · pp. 5–6
  • LGCE integrates deep and shallow features by leveraging language guidance.Source: Figure 5 · p. 6
  • Because expressions can signal target scale, the method replaces simple residual feature combination with language-guided multi-scale fusion for remote-sensing images.The paper contrasts expressions about cars or road markings, which tend to refer to small objects, with building expressions, which generally relate to large objects.Source: A. Overall Architecture · p. 6

B. Vision Transformer

  • ViT converts image patches into tokens, adds positional embeddings, and processes them with Transformer encoders.Source: B. Vision Transformer · p. 6
  • Each Transformer block applies multi-head self-attention and layer normalization, followed by an MLP, with residual connections.Source: B. Vision Transformer · p. 6

C. Language-guided Cross-scale Enhancement Module

LGCE uses language guidance to adaptively enhance multi-scale visual features by combining deep and shallow representations. It fuses language with features at two resolutions, then applies cross-scale attention before decoding.

C. Language-guided Cross-scale Enhancement Module

  • LGCE combines high-resolution shallow features and low-resolution deep features, using language guidance to enhance multi-scale visual representations.The module takes V3 and V4 as inputs; V4 has higher abstraction and lower spatial resolution.Source: C. Language-guided Cross-scale… · p. 6
  • Mean language features are projected to match each scale and concatenated with the corresponding visual features before Transformer processing.The concatenated representations contain both language and visual features.Source: C. Language-guided Cross-scale… · p. 6
  • Language features serve as intermediary tokens for cross-scale fusion, where attention layers combine features with different spatial resolutions.The resulting features are split, concatenated, and passed to subsequent decoding layers for mask prediction.Source: C. Language-guided Cross-scale… · p. 7

V. EXPERIMENTS

The experiments use RefSegRS, a remote-sensing referring segmentation dataset, and evaluate established natural-image methods with standard segmentation metrics. Five methods are included in the benchmark.

A. Dataset and Evaluation Metrics

  • RefSegRS contains 4,420 image-language-label triplets, split into 2,172 training, 431 validation, and 1,817 test samples.The evaluation metrics are oIoU, mIoU, and precision at thresholds from 0.5 to 0.9.Source: A. Dataset and Evaluation Metrics · p. 7
  • oIoU weights larger objects more, mIoU weights small and large objects equally, and threshold precision measures the share of samples exceeding a specified IoU.Source: A. Dataset and Evaluation Metrics · p. 7

B. Competing Methods

  • The benchmark evaluates LSTM-CNN, ConvLSTM, CMSA, BRINet, and LAVT, all originally designed for natural images.Source: B. Competing Methods · p. 7

C. Implementation Details

The implementation uses separate training configurations for CNN-based and Transformer-based models. The section also includes ablation and computational-complexity result tables.

C. Implementation Details

  • CNN-based methods use TensorFlow and train for 400,000 iterations with batch size 1 and Adam at learning rate 0.00025.Training hyperparameters are held consistent across these models for a fair comparison.Source: C. Implementation Details · pp. 7–8
  • The experiments include ablation results and a separate computational-complexity comparison on RefSegRS.Source: Table III · p. 8, Table IV · p. 8
  • LAVT and the proposed method use PyTorch, batch size 8, AdamW at learning rate 0.00005, weight decay 0.01, and Swin window size 12.Both train for 40 epochs, with the best checkpoint selected on validation performance.Source: C. Implementation Details · p. 8

D. Comparisons with Existing Methods

On RefSegRS, the proposed method improves over LAVT on all reported metrics, with gains concentrated in threshold precision and mIoU rather than oIoU. Transformer-based methods also outperform CNN-based methods most clearly on stricter threshold precision.

1) Comparison with CNN-based methods:

  • 73.75% Pr@0.5 for the proposed method versus 31.21% for ConvLSTM shows a large Transformer–CNN gap at this threshold.The authors report that Transformer-based methods greatly outperform CNN-based methods on Pr@0.5 and Pr@0.6.Source: 1) Comparison with CNN-based methods · p. 8
  • Most RefSegRS samples have foreground proportions below 5%, and CNN-based methods usually perform poorly on these small, scattered-object images.For cluttered vehicle examples, the authors identify limited deep-feature spatial detail and weak relationship modeling as CNN challenges.Source: 1) Comparison with CNN-based methods · p. 8
  • The CNN–Transformer performance gap is much smaller on oIoU and mIoU than on threshold precision.Source: 1) Comparison with CNN-based methods · p. 8

2) Comparison with LAVT:

  • The proposed method outperforms LAVT on all evaluation metrics, with an approximate 2% mIoU increase and no notable oIoU enhancement.The authors attribute the limited oIoU gain to LAVT's strong large-object detection; Fig. 8 shows improved identification of small and scattered objects.Source: 2) Comparison with LAVT · p. 8

E. Ablation Study and Discussion

The ablations support LGCE’s language-guided cross-scale design, while the conclusion reports improved results on challenging scenes and describes RefSegRS, a dataset for RRSIS.

E. Ablation Study and Discussion

  • LGCE fuses multi-scale visual features under linguistic guidance, with language guidance and cross-scale fusion as its two core components.The ablations test both components.Source: E. Ablation Study and Discussion · pp. 8, 10

VI. CONCLUSION

  • More than 5% IoU improvement over the baseline is reported for challenging scenes with cluttered and small objects.Fig. 6 shows example segmentation results and their IoU scores across methods.Source: Figure 6 · p. 9, §VI · p. 10

E. Ablation Study and Discussion

  • Removing language guidance reduces performance, removing cross-scale fusion lowers Pr@0.6, Pr@0.7, Pr@0.8, and Pr@0.9, and the full model outperforms LAVT.The authors report that the full model sets a new state of the art on RefSegRS.Source: E. Ablation Study and Discussion · p. 10
  • The full model adds only marginal computational cost over LAVT, while removing cross-scale fusion reduces cost but also declines performance.The model without language guidance has the same architecture, computational cost, and parameter count as the full model.Source: E. Ablation Study and Discussion · p. 10

VI. CONCLUSION

  • RefSegRS contains 4,420 image-language-label triplets across 285 scenes, created from SkyScapes tiles with template-generated expressions and manual filtering.The tiles are 1200 × 1200 pixels, cropped with a stride of 600 pixels.Source: §VI · p. 10

Related papers

Loading 2306.08625v2…