PaperlayerView exact revision on arXiv

Source-linked AI summary

Salient Object Detection in Optical Remote Sensing Images Driven by Transformer

Gongyang Li, Zhen Bai, Zhi Liu, Xinpeng Zhang, Haibin Ling

arXiv:2309.08206v1cs.CV

TL;DR

CNN-based ORSI-SOD methods are limited by local receptive fields and can struggle with varied object orientations and incomplete detections. GeleNet combines a transformer backbone with local and cross-level feature enhancement. The paper reports that it consistently outperforms compared methods on three datasets.

  • Problem

    CNN-based ORSI-SOD approaches use limited-receptive-field features, and reported examples show orientation insensitivity, incomplete detection, and missed salient objects.

    Source: §I · pp. 1–2

  • Method

    GeleNet uses a transformer backbone for global features, D-SWSAM and SWSAM for local enhancement, and KTM for cross-level contextual interactions.

    Source: §I · p. 2

  • Results

    GeleNet consistently outperforms compared methods on three datasets.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper presents global feature extraction followed by local and cross-level enhancement as an ORSI-SOD approach for varied orientations and complete salient objects.

    Source: §I · p. 2

Abstract

from arXiv · show

Existing methods for Salient Object Detection in Optical Remote Sensing Images (ORSI-SOD) mainly adopt Convolutional Neural Networks (CNNs) as the backbone, such as VGG and ResNet. Since CNNs can only extract features within certain receptive fields, most ORSI-SOD methods generally follow the local-to-contextual paradigm. In this paper, we propose a novel Global Extraction Local Exploration Network (GeleNet) for ORSI-SOD following the global-to-local paradigm. Specifically, GeleNet first adopts a transformer backbone to generate four-level feature embeddings with global long-range dependencies. Then, GeleNet employs a Direction-aware Shuffle Weighted Spatial Attention Module (D-SWSAM) and its simplified version (SWSAM) to enhance local interactions, and a Knowledge Transfer Module (KTM) to further enhance cross-level contextual interactions. D-SWSAM comprehensively perceives the orientation information in the lowest-level features through directional convolutions to adapt to various orientations of salient objects in ORSIs, and effectively enhances the details of salient objects with an improved attention mechanism. SWSAM discards the direction-aware part of D-SWSAM to focus on localizing salient objects in the highest-level features. KTM models the contextual correlation knowledge of two middle-level features of different scales based on the self-attention mechanism, and transfers the knowledge to the raw features to generate more discriminative features. Finally, a saliency predictor is used to generate the saliency map based on the outputs of the above three modules. Extensive experiments on three public datasets demonstrate that the proposed GeleNet outperforms relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/GeleNet.

I. INTRODUCTION

Existing CNN-based ORSI-SOD approaches use local feature extraction followed by local or contextual processing, while GeleNet adopts a global-to-local paradigm. It combines transformer features with local enhancement and cross-level contextual interaction.

I. INTRODUCTION

  • Existing paradigms have limitations in scenes with varied object orientations, scales, and categories, and example methods show orientation insensitivity, incomplete detection, and missed salient objects.The paper attributes the limitations to ignored contextual information in local-to-local methods and the limited receptive fields of convolution-based contextual processing.Source: §I · pp. 1–2
  • The proposed network combines global transformer features with local and cross-level interactions to generate saliency maps with accurate orientations and complete objects.The introduction reports that GeleNet consistently outperforms compared methods on three datasets.Source: §I · p. 2
  • GeleNet replaces the CNN backbone with a transformer to establish global relationships before locally enhancing features for ORSI-SOD.The paper describes this as a global-to-local paradigm, contrasting with the local-to-contextual paradigm followed by most CNN-based methods.Source: §I · p. 2
  • D-SWSAM uses directional convolutions to handle varied object orientations and enhance details in the lowest-level features.The paper assigns SWSAM to highest-level features to locate salient objects.Source: §I · p. 2
  • KTM models contextual interactions between two middle-level features and transfers their correlation knowledge to raw features for more discriminative representations.It models product and sum combinations of the features.Source: §I · p. 2

II. RELATED WORK

The related work reviews CNN-based ORSI-SOD paradigms, transformer use in salient object detection, and attention mechanisms. It positions GeleNet as a transformer-driven ORSI-SOD method with new feature-interaction modules.

A. Salient Object Detection in Optical Remote Sensing Images

  • CNN-based ORSI-SOD methods use either single-level local processing or cross-level contextual aggregation, while the latter still relies on convolution-based interactions.The paper proposes global long-range modeling followed by local and contextual enhancement as an alternative.Source: A. Salient Object Detection in Optical… · pp. 2–3

B. Salient Object Detection with Transformer

  • Transformer-based SOD has used transformers as feature extractors, decoder modules, or pure transformer architectures, and GeleNet is presented as the first transformer-driven ORSI-SOD method.GeleNet uses PVT as its backbone to extract long-range dependency features from optical remote sensing images.Source: B. Salient Object Detection with… · p. 3

C. Attention Mechanism

  • Prior SGE and shuffle attention ignore consistency across sub-feature attention, while GeleNet proposes SWSAM, D-SWSAM, and self-attention-based KTM for feature enhancement and contextual knowledge transfer.SWSAM is designed to improve channel interactions in global features; D-SWSAM adds directional convolutions for varied object orientations.Source: C. Attention Mechanism · p. 3

III. PROPOSED METHOD

The proposed-method section is organized around GeleNet's network overview, its feature-interaction modules, and saliency prediction with the loss function.

III. PROPOSED METHOD

  • Section III-A presents the network overview of the proposed transformer-driven GeleNet.Source: §III · p. 3
  • Sections III-B and III-C introduce D-SWSAM and KTM, respectively.Source: §III · p. 3
  • Section III-D describes the saliency predictor and loss function.Source: §III · p. 3

A. Network Overview

GeleNet uses a transformer feature extractor and three feature-enhancement modules before a saliency predictor produces the saliency map. The modules process low-, middle-, and high-level features for orientation, contextual, and location information.

A. Network Overview

  • PVT extracts four-level global feature embeddings, and GeleNet routes corresponding features through D-SWSAM, KTM, and SWSAM before saliency prediction.The pipeline comprises a feature extractor, three interaction or enhancement modules, and a saliency predictor.Source: A. Network Overview · p. 3, Figure 2 · p. 4
  • D-SWSAM applies horizontal, vertical, leading-diagonal, and reverse-diagonal convolutions to lowest-level features to capture orientation information and enhance details.Source: Figure 2 · p. 4, A. Network Overview · p. 4
  • KTM processes two middle-level features to model contextual correlation knowledge and transfer it to raw features, producing more discriminative features.The caption specifies product and sum combinations of the middle-level features.Source: Figure 2 · p. 4, A. Network Overview · p. 4
  • SWSAM enhances salient-object location in highest-level features, and the saliency predictor uses the module outputs to generate a saliency map.The predictor is described as a variant of the partial decoder.Source: Figure 2 · p. 4, A. Network Overview · p. 4

B. Direction-aware Shuffle Weighted Spatial Attention Module

D-SWSAM and SWSAM complement transformer features with local spatial enhancement tailored to different feature levels. D-SWSAM addresses orientation and detail at the lowest level, while SWSAM emphasizes location at the highest level.

B. Direction-aware Shuffle Weighted Spatial Attention Module

  • D-SWSAM processes lowest-level features with four directional convolutions to extract orientation information for salient objects with varied orientations.The parallel directions are horizontal, vertical, leading diagonal, and reverse diagonal.Source: B. Direction-aware Shuffle Weighted… · pp. 4–5
  • Channel shuffling disperses orientation information across four groups, enabling each sub-feature to contain information from all four directions.The sub-features receive separate spatial attention maps to capture local salient regions.Source: B. Direction-aware Shuffle Weighted… · p. 5
  • SWSAM fuses the sub-feature attention maps with learnable weights to produce a comprehensive spatial attention map for consistent detail enhancement.The weights are initialized to 0.25 and converge to learned values.Source: B. Direction-aware Shuffle Weighted… · p. 5
  • At the highest level, SWSAM omits directional convolutions because those features mainly encode location, while retaining channel shuffling to increase channel interaction.Its output feature has dimensions 32×11×11.Source: B. Direction-aware Shuffle Weighted… · p. 5

C. Knowledge Transfer Module

KTM enhances contextual interactions between two middle-level features by modeling correlations between their product and sum, then transferring the resulting knowledge to the raw features. Its output combines information from both feature combinations and is more contextual than the local-enhancement outputs.

C. Knowledge Transfer Module

  • KTM models contextual correlations between the product and sum of two middle-level features, whose information is complementary for identifying and elaborating salient objects.The product captures information shared by both features; the sum captures information in both without omission.Source: C. Knowledge Transfer Module · p. 5
  • Self-attention models pixel-to-pixel dependencies between co-existing significant information in the product and comprehensive information in the sum, helping avoid missed salient regions or objects.Source: C. Knowledge Transfer Module · p. 6
  • KTM transfers the modeled knowledge to features derived from the middle-level inputs, then fuses the transferred and raw features through residual connections.The fused features are integrated by element-wise summation and convolution.Source: C. Knowledge Transfer Module · p. 6
  • The KTM output inherits properties of both feature combinations and is more contextual than D-SWSAM and SWSAM outputs.This contextual feature is intended to combine with the local-enhancement features for salient-object inference.Source: C. Knowledge Transfer Module · p. 6

D. Saliency Predictor

The saliency predictor combines outputs from D-SWSAM, KTM, and SWSAM to generate a saliency map. It modifies the partial decoder's upsampling rate to accommodate the input feature resolutions.

D. Saliency Predictor

  • The saliency predictor uses a modified partial decoder to combine SWSAM, KTM, and D-SWSAM features at 11×11, 22×22, and 88×88 resolutions.Each input feature has 32 channels.Source: D. Saliency Predictor · p. 6
  • The predictor generates an initial 88×88 saliency map and upsamples it 4× to produce a final 352×352 map.Source: D. Saliency Predictor · p. 6
  • Training uses a hybrid loss comprising intersection-over-union loss and binary cross-entropy against the 352×352 ground-truth mask.The supplied passage identifies the loss components and ground truth.Source: D. Saliency Predictor · p. 6

A. Experimental Setup

Experiments use train/test splits from three public ORSI-SOD datasets, with a common training setup and several established evaluation metrics. Each dataset is used for both training and testing on its respective split.

A. Experimental Setup

  • Experiments use ORSSD (600 train, 200 test), EORSSD (1,400 train, 600 test), and ORSI-4199 (2,000 train, 2,199 test).The authors train and test GeleNet on each dataset's respective split.Source: A. Experimental Setup · p. 6
  • GeleNet uses a pretrained PVT-v2-b2 backbone, 352×352 inputs, rotation-based augmentation, and Adam training for 45 epochs with batch size 8.The base learning rate is 1e−4 and decays by a factor of ten every 30 epochs.Source: A. Experimental Setup · p. 6
  • Evaluation uses S-measure, F-measure, E-measure, mean absolute error, precision-recall curves, and F-measure curves.Source: A. Experimental Setup · p. 6

B. Comparison with State-of-the-arts

GeleNet is compared with state-of-the-art methods on EORSSD, ORSSD, and ORSI-4199 using quantitative and visual evaluations. Ours-PVT performs strongly across the datasets, while visual comparisons highlight its handling of challenging scenes.

Quantitative Comparison on the EORSSD and ORSSD Datasets

  • Ours-PVT outperforms all compared methods on EORSSD and ORSSD except for Sα, Emaxξ, and M on EORSSD.On ORSSD, it exceeds the second-best method on Sα (0.9469 vs. 0.9437), Fmaxξ (0.9860 vs. 0.9810), and M (0.0079 vs. 0.0087).Source: Table I · p. 7, B. Comparison with State-of-the-arts · p. 7

Quantitative Comparison

  • Ours-SwinT ranks first on seven of eight metrics and second on one on ORSSD, and outperforms all compared methods on all eight ORSI-4199 metrics.On ORSI-4199, it also achieves the best Emeanξ and M results, even compared with Ours-PVT.Source: B. Comparison with State-of-the-arts · pp. 7, 9
  • Ours-VGG and Ours-Res perform less strongly than the transformer-backed variants and are comparable to ERPNet, EMFINet, and CorrNet.The paper notes that its modules are designed specifically for transformer global features.Source: B. Comparison with State-of-the-arts · pp. 7, 9

Visual Comparison

  • In visual comparisons, Ours-PVT accurately handles varied orientations, multiple salient objects, fine structures, and low-contrast scenes where other methods miss objects or highlight background.The comparisons attribute orientation handling to D-SWSAM, localization to SWSAM, and low-contrast discrimination to PVT’s global modeling and the proposed modules’ local enhancement.Source: Figure 4 · p. 8, B. Comparison with State-of-the-arts · p. 9

Quantitative Comparison on the ORSI-4199 Dataset

Quantitative Comparison

  • Transformer-based methods perform near the top of their respective NSI-SOD and ORSI-SOD groups across three datasets; specialized ORSI-SOD methods generally outperform NSI-SOD methods.The authors identify these comparisons as motivation for further development of specialized ORSI-SOD solutions.Source: B. Comparison with State-of-the-arts · p. 9

C. Ablation Studies

Ablations assess the three modules, their component choices, and alternative attention mechanisms. The full model improves over the baseline, while the reported variants support distinct roles for D-SWSAM, KTM, and SWSAM.

Effectiveness of Each Component in D-SWSAM

  • Removing either directional convolution or SWSAM reduces D-SWSAM accuracy, with the larger drop occurring when SWSAM is removed.The ablation therefore identifies both components as necessary, with SWSAM the more important of the two in D-SWSAM.Source: Table IV · p. 10, C. Ablation Studies · p. 10

Effectiveness of Each Component in SWSAM

  • Removing channel shuffle or learnable attention weights makes SWSAM variants among the worst-performing ablations.Shuffle supports four-direction sub-features in D-SWSAM and channel communication in high-level features; learned weights account for differences among spatial attention maps.Source: Table IV · p. 10, C. Ablation Studies · pp. 10–11

Individual Contribution of Three Modules

  • With all three modules working together, the full model significantly outperforms “Baseline” by 1.27% in Sα, 2.06% in Fmaxβ, and 1.16% in Emaxξ.Source: C. Ablation Studies · p. 10
  • Using the same module at both the lowest and highest feature levels performs worse than assigning different enhancements to those levels.The visual comparisons show D-SWSAM aiding orientation handling and SWSAM supporting complete localization; the full model’s maps are described as visually indistinguishable from ground truth.Source: C. Ablation Studies · p. 10, Figure 5 · p. 10

Comparison with Classic Attention Mechanisms

  • SWSAM outperforms traditional spatial attention and SGE in the reported comparisons, with adaptive aggregation producing a comprehensive attention map for consistent enhancement.The visual examples describe background highlighting by traditional spatial attention and irrelevant-object highlighting by SGE.Source: Table V · p. 10, C. Ablation Studies · p. 11

Rationality of the Way of Modeling Knowledge in KTM

  • KTM performs worse when it models contextual knowledge from only the sum or only the product of the middle-level features.The authors describe the product and sum as complementary and report that modeling both is more conducive to inferring salient objects.Source: C. Ablation Studies · p. 10

V. CONCLUSION

GeleNet combines global transformer features with local enhancement and cross-level contextual interactions for ORSI-SOD. The authors report that its modules and overall performance are effective.

V. CONCLUSION

  • GeleNet is presented as the first transformer-driven ORSI-SOD solution, following a global-to-local paradigm while incorporating cross-level contextual interactions.It uses PVT for global features, SWSAM and D-SWSAM for local enhancement, and KTM for cross-level interactions.Source: §V · p. 11
  • D-SWSAM uses directional convolutions to perceive orientation in the lowest-level features, while SWSAM enhances location in the highest-level features.KTM uses self-attention to model complementary information between the product and sum of two middle-level features and generate discriminative features.Source: §V · p. 11
  • Extensive comparisons and ablation studies are reported to demonstrate GeleNet’s superiority and the effectiveness of its three proposed modules.D-SWSAM and SWSAM are also described as plug-and-play modules for related tasks,, –.Source: §V · p. 11

Related papers

Loading 2309.08206v1…