PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images

Gongyang Li, Zhi Liu, Weisi Lin, Haibin Ling

arXiv:2112.01932v1cs.CVeess.IV

TL;DR

Salient object detection in optical remote sensing images remains challenging because these images differ from natural-scene images in scales, illumination, and imaging orientations. MCCNet combines foreground, edge, background, and global image-level content in a multi-scale encoder-decoder network. On two benchmark datasets, it outperformed 23 state-of-the-art methods.

  • Problem

    Optical RSI-SOD remains challenging because optical remote sensing images differ from natural-scene images in scales, illumination, and imaging orientations.

    Source: §I · p. 1

  • Method

    MCCNet uses a multi-scale encoder-decoder network whose MCCM complements foreground, edge, background, and global image-level content.

    Source: §I · p. 2

  • Results

    On two benchmark RSI-SOD datasets, MCCNet outperformed 23 state-of-the-art methods under various evaluation metrics.

    Source: §I · p. 2

  • Takeaways & Limitations

    MCCNet runs at 95 fps on a single GPU while applying multi-content complementation at multiple feature scales for RSI-SOD.

    Source: §I · p. 2

Abstract

from arXiv · show

In the computer vision community, great progresses have been achieved in salient object detection from natural scene images (NSI-SOD); by contrast, salient object detection in optical remote sensing images (RSI-SOD) remains to be a challenging emerging topic. The unique characteristics of optical RSIs, such as scales, illuminations and imaging orientations, bring significant differences between NSI-SOD and RSI-SOD. In this paper, we propose a novel Multi-Content Complementation Network (MCCNet) to explore the complementarity of multiple content for RSI-SOD. Specifically, MCCNet is based on the general encoder-decoder architecture, and contains a novel key component named Multi-Content Complementation Module (MCCM), which bridges the encoder and the decoder. In MCCM, we consider multiple types of features that are critical to RSI-SOD, including foreground features, edge features, background features, and global image-level features, and exploit the content complementarity between them to highlight salient regions over various scales in RSI features through the attention mechanism. Besides, we comprehensively introduce pixel-level, map-level and metric-aware losses in the training phase. Extensive experiments on two popular datasets demonstrate that the proposed MCCNet outperforms 23 state-of-the-art methods, including both NSI-SOD and RSI-SOD methods. The code and results of our method are available at https://github.com/MathLee/MCCNet.

I. INTRODUCTION

RSI-SOD remains challenging because optical remote sensing images differ from natural-scene images in object types, scales, illumination, orientations, and backgrounds. MCCNet addresses this setting by combining complementary image content at multiple feature scales, and experiments report that it outperforms 23 state-of-the-art methods on two benchmark datasets.

I. INTRODUCTION

  • Optical RSIs differ from natural-scene images in object types, scales, illumination, imaging orientations, and backgrounds, making direct transfer of NSI-SOD methods potentially inappropriate.The task also includes scenes with no salient object.Source: §I · p. 1
  • MCCM complements foreground, edge, and background cues with global image-level content to represent both local detail and overall scene information.The authors describe this combination as supporting salient-region perception and discrimination of cluttered backgrounds.Source: §I · p. 2
  • MCCNet embeds MCCM at multiple feature scales in an encoder-decoder network and reports an inference speed of 95 fps on a single GPU.Its progressive inference is intended to accommodate varied salient-region scales and object types.Source: §I · p. 2
  • On two benchmark RSI-SOD datasets, MCCNet outperforms 23 state-of-the-art methods under various evaluation metrics.The experiments also verify the effectiveness of MCCM.Source: §I · p. 2

II. RELATED WORK

  • The paper is organized to review NSI-SOD and RSI-SOD, present MCCNet, describe experiments and ablations, and conclude.The introduction identifies these as Sections II through V.Source: §I · p. 2, §II · p. 2

A. Salient Object Detection in Natural Scene Images

Prior saliency-detection work spans traditional and CNN-based methods, with CNN approaches bringing multi-scale features, edge cues, attention, and complementary losses. RSI-SOD methods adapt such ideas to optical imagery, while the paper identifies a gap in jointly complementing foreground, edge, background, and global content.

A. Salient Object Detection in Natural Scene Images

  • Traditional NSI-SOD approaches used visual attention, region merging, ranking, segmentation cues, and foreground-background diffusion, though the authors say their performance was not impressive.The review presents these methods as sources of ideas for later work.Source: A. Salient Object Detection in Natural… · p. 2
  • CNN-based NSI-SOD advanced beyond traditional methods through deep supervision, multi-scale fusion, edge guidance, attention, background cues, and complementary losses.The review also notes global-context aggregation and recurrent mechanisms.Source: A. Salient Object Detection in Natural… · p. 2

B. Salient Object Detection in Optical Remote Sensing Images

  • RSI-SOD is broader than detection or extraction of specific targets because it covers varied optical-image scenes and objects while accounting for human-attended saliency.The review contrasts this general task with airport, ship, oil-tank, building, and residential-area tasks.Source: B. Salient Object Detection in Optical… · p. 3
  • CNN-based RSI-SOD methods use multi-resolution inputs or feature fusion, edge supervision, and specialized architectures to address scale variation and object boundaries.The review also describes weakly supervised RSI-SOD using pseudo labels and progressive saliency refinement.Source: B. Salient Object Detection in Optical… · p. 3
  • The paper identifies independently used foreground, edge, and background cues as a gap, and proposes complementing them with global content across five feature scales from one input.It characterizes this design as more comprehensive than and, and more efficient than and.Source: B. Salient Object Detection in Optical… · p. 3

III. PROPOSED METHOD

The proposed-method section is organized around the MCCNet overview, the MCCM, and the comprehensive loss function.

III. PROPOSED METHOD

  • Section III-A presents the MCCNet network overview.The section outline identifies this as the first method component.Source: §III · p. 3
  • Section III-B elaborates the Multi-Content Complementation Module.MCCM is the named module in the section outline.Source: §III · p. 3
  • Section III-C clarifies the comprehensive loss function.The section outline presents this as the method's training component.Source: §III · p. 3

A. Network Overview

MCCNet uses a VGG-16 encoder and five MCCM components within an encoder-decoder architecture. The MCCMs process features at multiple scales before a progressive decoder restores resolution and predicts salient objects.

A. Network Overview

  • VGG-16 supplies basic features from the 256×256×3 optical RSI through five convolution blocks after the final four original layers are removed.The five remaining blocks are denoted E_t, with t indexing blocks 1 through 5.Source: A. Network Overview · p. 3, Figure 2 · p. 4
  • Features from the encoder are fed to five corresponding MCCMs, which derive foreground, edge, background, and global image-level features and model their complementarity across scales.The framework uses multi-scale MCCM outputs for salient-object inference.Source: A. Network Overview · pp. 3–4, Figure 2 · p. 4
  • Each decoder block receives pixel-level BCE, map-level IoU, and metric-aware F-m loss supervision during training.The framework caption describes this as comprehensive supervision at each decoder block.Source: Figure 2 · p. 4, A. Network Overview · p. 4
  • A five-block decoder progressively restores resolution from MCCM features, with deconvolution between blocks, to infer salient objects.Decoder block structures correspond to the encoder-block pattern described in the passage.Source: A. Network Overview · p. 4

B. Multi-Content Complementation Module

MCCM addresses complex RSI scenes by complementing foreground, edge, background, and global image-level content. It combines these cues at multiple feature levels to support salient-region perception across varied scenes and objects.

B. Multi-Content Complementation Module

  • Foreground and edge maps are extracted in parallel; the foreground map is adaptive, while the edge map is learned with edge-ground-truth supervision.Their element-wise sum forms a foreground-edge map that multiplies features to highlight salient regions.Source: B. Multi-Content Complementation Module · p. 5
  • The reverse-attention background map defines background as 1 and foreground as -1, highlighting non-salient regions with features opposite to foreground-edge features.The paper says the two feature types are merged at channel level to extract complementary information.Source: B. Multi-Content Complementation Module · p. 5
  • Global image-level content captures the source features’ overall tone: pooled and smoothed features are upsampled, then converted by spatial attention into a global map.The reconstruction loses detail, but is described as reflecting the overall tone of the source features.Source: B. Multi-Content Complementation Module · p. 5
  • MCCM adaptively concatenates and convolves the content features, then adds a short connection to retain original content in its output.The output combines foreground, edge, background, global image-level, and original content.Source: B. Multi-Content Complementation Module · p. 6
  • MCCM aggregates foreground, edge, background, global image-level, and original content, with modules at five feature levels for complex optical RSI scenes.The module is designed to help the network adapt to changing objects and scenes.Source: B. Multi-Content Complementation Module · pp. 5–6
  • In the E3 visualization, the foreground-edge combination highlights salient regions across object sizes and topologies, while the background map makes non-salient regions obvious.The global image-level map reflects the source RSI’s basic tone.Source: B. Multi-Content Complementation Module · p. 6

C. Comprehensive Loss Function

MCCNet is trained with deep supervision and a composite loss that combines pixel-level, map-level, metric-aware, and edge supervision. The intermediate saliency maps are supervised at different sizes.

C. Comprehensive Loss Function

  • The training objective combines pixel-level BCE, map-level IoU, and metric-aware F-m loss, alongside edge supervision.The edge loss uses the edge ground truth, and the total loss aggregates the training losses.Source: C. Comprehensive Loss Function · p. 6
  • Deep supervision monitors intermediate saliency maps of different sizes to encourage features to learn characteristics of salient regions at different scales.The paper uses the widely adopted deep-supervision scheme during training.Source: C. Comprehensive Loss Function · p. 6
  • The edge map is learned by minimizing an edge loss, with the edge ground truth generated in the same way as in.The edge loss is defined using binary cross-entropy between the upsampled edge map and edge ground truth.Source: C. Comprehensive Loss Function · p. 6

A. Experimental Protocol

The experiments use two public RSI-SOD datasets, specified train/test splits, and a common training and evaluation protocol.

Datasets

  • ORSSD uses 600 images for training and 200 for testing; EORSSD uses 1,400 for training and 600 for testing.ORSSD contains 800 annotated images, while EORSSD contains 2,000 annotated images and more complex, challenging scenes.Source: A. Experimental Protocol · p. 6

Implementation Details

  • Training resizes images to 256×256, applies flipping and rotation, and uses Adam with batch size 8 and initial learning rate 1e−4.The encoder is initialized with pretrained VGG-16; training runs for 39 epochs on EORSSD and 34 on ORSSD.Source: A. Experimental Protocol · p. 6

Evaluation Metrics

  • Evaluation includes structural similarity, weighted precision–recall balance, pixel/global match information, and PR curves over thresholds from 0 to 255.The table compares 23 methods on EORSSD and ORSSD; arrows indicate whether larger or smaller scores are better.Source: A. Experimental Protocol · pp. 6–7, Table I · p. 7

B. Comparison with State-of-the-art Methods

MCCNet performs strongly against methods from four SOD categories on both RSI-SOD datasets, with a particularly consistent comparison against recent CNN-based RSI-SOD methods. The section also reports computational efficiency and visual comparisons across challenging scenes.

Quantitative Comparison

  • MCCNet beats DAFNet on all eight ORSSD metrics and six EORSSD metrics, but trails it on EORSSD Fmaxβ and M; it also outperforms EMFINet on both datasets.The reported ORSSD examples are Sα: 0.9437 (Ours) v.s. 0.9191 (DAFNet) and M: 0.0087 (Ours) v.s. 0.0113 (DAFNet).Source: B. Comparison with State-of-the-art… · p. 9

Computational Complexity Comparison

  • 95 fps inference speed is reported for MCCNet; its 67.65M parameters and 112.8M FLOPs are below EMFINet’s 107.26M and 480.9M, respectively.The paper describes MCCNet’s parameter and FLOP counts as midstream overall and says its speed is friendly to practical applications.Source: B. Comparison with State-of-the-art… · p. 9

Ablation Study

  • The ablation table evaluates individual MCCM content contributions against an encoder–decoder baseline with skip connections.FG, EG, BG, and GIC denote foreground, edge, background, and global image-level content; the best result in each column is bold.Source: Table II · p. 9

Visual Comparison

  • Across scenes with tiny or multiple objects, shadows, interference, low contrast, and complex topology, MCCNet maps locate salient objects and outline fine details.The comparison includes representative airplane, car, building, river, and pool scenes.Source: Figure 6 · p. 8, B. Comparison with State-of-the-art… · p. 9

C. Ablation Studies

The ablations assess the contributions of MCCM’s content components, original-content merging, and composite training losses. Results support the full module and comprehensive loss design on both datasets.

1) The individual contribution of each content in MCCM

  • FG activates network potential; adding EG and BG improves performance further, while GIC raises Fmaxβ from 0.8879 to 0.8904 on EORSSD and 0.9096 to 0.9155 on ORSSD.Source: C. Ablation Studies · pp. 9–10
  • The full MCCM improves Baseline by 2.17% and 1.70% on Fmaxβ, and by 2.77% and 1.92% on Fmaxξ, on EORSSD and ORSSD, respectively.Source: C. Ablation Studies · p. 10
  • Comparisons across nine content variants indicate that each content contributes; BG is defined jointly by FG and EG, and omitting it adversely affects performance.Source: C. Ablation Studies · p. 10, Figure 7 · p. 10

2) The necessity of merging the original content in MCCM

3) The effectiveness of our comprehensive loss function

  • On ORSSD, the combined BCE, IoU, and F-m losses improve Fmaxξ by 0.0158 over single BCE; IoU or F-m assistance yields gains of about 0.0018–0.0127.Single BCE reaches Fmaxξ 0.9728 on ORSSD; the combined losses achieve the best performance without additional parameters.Source: C. Ablation Studies · p. 10, Table IV · p. 10

V. CONCLUSION

The conclusion presents MCCNet as an RSI-SOD approach that combines multiple complementary contents across feature scales. Experiments support its effectiveness, and the reported inference speed is 95 fps.

V. CONCLUSION

  • MCCM models complementarity among foreground, edge, background, and global image-level content for optical RSI data.Source: §V · p. 10
  • MCCM is applied at five feature sizes to address variable object scales, types, and quantities, with a comprehensive loss function used in training.Source: §V · p. 11
  • Experiments on two public RSI-SOD datasets demonstrate MCCNet’s and MCCM’s effectiveness, and MCCNet runs at 95 fps.Source: §V · p. 11
Loading 2112.01932v1…