PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Attention Fluid Network for Salient Object Detection in Optical Remote Sensing Images

Qijian Zhang, Runmin Cong, Chongyi Li, Ming-Ming Cheng, Yuming Fang, Xiaochun Cao, Yao Zhao, Sam Kwong

arXiv:2011.13144v1cs.CV

TL;DR

Optical RSI salient object detection remains challenging because remote-sensing scenes exhibit complex backgrounds, scale variation, and complex object structures. The paper proposes DAFNet, which combines global context-aware attention with dense cross-level attention flow, and constructs a 2,000-image annotated benchmark. Experiments and ablations demonstrate the effectiveness of the architecture, while the paper reports consistent superiority over 15 state-of-the-art competitors.

  • Problem

    Optical RSI SOD has limited research and must handle complicated backgrounds, scale variation, complex structures, and a small existing dataset.

  • Method

    DAFNet is an end-to-end encoder-decoder network using GCA to model long-range dependencies and DAF to propagate shallow attention cues into deeper layers.

  • Results

    DAFNet consistently outperforms 15 state-of-the-art competitors, while ablations show F-measure increasing from 0.8391 to 0.8504 with GFA.

  • Takeaways & Limitations

    The work provides an attention-guided optical RSI SOD architecture together with a larger, more challenging benchmark containing 2,000 pixel-wise annotated images.

Abstract

from arXiv · show

Despite the remarkable advances in visual saliency analysis for natural scene images (NSIs), salient object detection (SOD) for optical remote sensing images (RSIs) still remains an open and challenging problem. In this paper, we propose an end-to-end Dense Attention Fluid Network (DAFNet) for SOD in optical RSIs. A Global Context-aware Attention (GCA) module is proposed to adaptively capture long-range semantic context relationships, and is further embedded in a Dense Attention Fluid (DAF) structure that enables shallow attention cues flow into deep layers to guide the generation of high-level feature attention maps. Specifically, the GCA module is composed of two key components, where the global feature aggregation module achieves mutual reinforcement of salient feature embeddings from any two spatial locations, and the cascaded pyramid attention module tackles the scale variation issue by building up a cascaded pyramid framework to progressively refine the attention map in a coarse-to-fine manner. In addition, we construct a new and challenging optical RSI dataset for SOD that contains 2,000 images with pixel-wise saliency annotations, which is currently the largest publicly available benchmark. Extensive experiments demonstrate that our proposed DAFNet significantly outperforms the existing state-of-the-art SOD competitors. https://github.com/rmcong/DAFNet_TIP20

I. INTRODUCTION

Optical RSI salient object detection is difficult because scenes contain clutter, scale variation, complex object structures, and limited training data. DAFNet addresses these challenges with dense attention guidance, global context modeling, and a larger benchmark dataset.

  • Challenges: Optical RSIs contain complicated backgrounds, diverse noise, large object-scale variation, and arbitrary orientations that make NSI saliency detectors less reliable.These differences arise from overhead acquisition and flexible sensor-object distances.
  • Challenges: Background interference and redundancy cause existing methods to produce false negatives and false positives when locating salient objects.The paper illustrates failures from cluttered backgrounds and imaging shadows.
  • Contributions: DAFNet propagates shallow attention cues into deeper attention modules to guide high-level attention-map generation.The dense attention fluid establishes explicit connections among attention levels.
  • Challenges: Complex object structures and long narrow topologies can yield incomplete or broken saliency predictions because distant spatial positions have inconsistent features.Examples include incompletely detected buildings and fragmented rivers.
  • Contributions: The EORSSD benchmark contains 2,000 images with pixel-wise annotations and covers more challenging scene types, object attributes, and real-world circumstances.It extends the 800-image ORSSD dataset and is described as publicly available.
  • Contributions: Its GCA mechanism models global-context semantic relationships and addresses scale variation through a cascaded pyramid attention framework.The method is designed to capture long-range context while progressively refining multi-scale attention.

II. RELATED WORK

Prior work spans natural-scene saliency models and a small number of optical RSI methods, while DAFNet introduces an attention-guided encoder-decoder architecture for this setting. Its design combines global-context attention, dense cross-level attention flow, and progressive feature decoding.

  • SOD background: Natural-scene SOD research includes bottom-up models using handcrafted features or visual priors and top-down supervised models using pixel-wise labels.These approaches are described as stimulus-driven and task-driven, respectively.
  • Optical RSI SOD: Only three prior works focus on optical RSI SOD, including sparse-representation, low-rank feature-fusion, and early deep-learning approaches.The paper contrasts this limited body of work with the larger NSI research literature.
  • Applications: Optical RSI SOD also supports downstream tasks such as ROI extraction, building extraction, airport detection, oil-tank detection, and ship detection.The paper presents SOD as an auxiliary component in these remote-sensing applications.
  • DAFNet: DAFNet uses an encoder-decoder architecture whose dense attention fluid lets each GCA unit receive prior attentive results and enhance raw features through residual connections.The decoder progressively fuses feature levels to generate saliency and salient-edge maps.
  • DAFNet: The framework can use different backbones; VGG16 is used illustratively to provide five sequential convolutional feature levels.The described backbone removes the final fully connected layers and truncates the first max-pooling layer.
  • DAFNet: The architecture includes attention-fluid encoding and progressive feature decoding, with top-down fusion, channel alignment, bottleneck integration, and saliency prediction.The top-most side outputs are excluded from deep supervision because their low resolution hinders detailed object depiction.

1) Global Context-aware Attention Mechanism:

The GCA mechanism captures long-range semantic dependencies and addresses object-scale variation through global feature aggregation and cascaded pyramid attention. Its coarse-to-fine refinement progressively combines contextual features and attention cues to generate detailed saliency maps.

  • Global Context-aware Attention: GCA explicitly models long-range semantic dependencies among all spatial locations for more discriminative saliency features.It comprises global feature aggregation and cascaded pyramid attention.
  • Global Feature Aggregation: Global feature aggregation evaluates mutual influence between embeddings at any two positions and aggregates local features into global contextual descriptors.The resulting feature representation is intended to maintain consistency across whole salient regions.
  • Global Feature Aggregation: The global context map weights spatial relationships using cosine-distance feature similarity and Gaussian-weighted column aggregation.This encodes relative spatial impact between embedded feature positions.
  • Feature Recalibration: Feature recalibration models channel inter-dependencies using average- and max-pooled descriptors before producing a refined feature map.The channel reweighing vector is broadcast across spatial dimensions and multiplied with the contextual feature map.
  • Cascaded Pyramid Attention: Cascaded pyramid attention progressively refines features and attentive cues from coarse to fine instead of independently summing attention maps across resolutions.Coarse cues from lower-resolution features are applied and propagated to higher-resolution features for more accurate details.

2) Dense Attention Fluid Structure:

The Dense Attention Fluid structure connects attention maps across hierarchical convolutional stages. Shallow attention cues are propagated into deeper layers as guidance for high-level attention generation.

  • Hierarchical Feature Guidance: The structure uses hierarchical features whose shallow representations emphasize edges and textures while deeper representations capture high-level semantics.The design uses their complementary attention-selection strategies.
  • Dense Attention Fluid Structure: Dense Attention Fluid propagates low-level attentive cues into deeper attention units to guide and refine high-level attention maps.This connects attention information across hierarchical feature representations.
  • Attention Information Flow: The attention fluid combines sequential connections and feed-forward skip connections among attention maps from different convolutional stages.Down-sampled shallow attention maps are concatenated with the current stage’s attention representation.
  • Feature Enhancement: The updated attention map generates the final stage feature through a residual connection, producing enhanced side features across convolutional stages.The enhanced side features are obtained sequentially from the five convolution stages.

C. Progressive Feature Decoding

The progressive decoder fuses deep and shallow features across resolutions to produce side outputs, saliency maps, and salient edge maps. The final network output is the saliency map from the top decoding level.

  • Progressive Feature Decoding: Progressive decoding fuses deep features with shallow features to generate multiple side outputs at different resolutions.Top-down fusion aligns adjacent side features through up-sampling and 1 × 1 convolution.
  • Prediction: Mask and edge prediction layers transform decoded 3D features into 2D saliency and salient edge maps, with scores mapped into [0, 1].The final DAFNet output is the saliency map predicted at the top decoding level.

D. Loss Function

The loss design combines hierarchical saliency supervision with salient-edge supervision at multiple decoding stages. Class-balanced terms address pixel imbalance, while edge supervision targets fine-grained contours.

  • Supervision Strategy: Hierarchical optimization applies deep supervision to side outputs at different convolution stages for robust saliency representations.Edge supervision is additionally introduced to capture fine-grained saliency patterns and object contours.
  • Saliency Supervision: Class-balanced binary cross-entropy supervises predicted saliency maps using positive- and background-pixel balancing factors.The factors are computed from the number of positive pixels and total pixels.
  • Salient Edge Supervision: Salient-edge supervision derives two-pixel-thick edge targets from ground-truth boundaries using Canny extraction followed by 2 × 2 maximum filtering.The predicted salient edge map is trained with a similarly balanced edge loss.
  • Overall Loss: The overall loss uses three side-supervision levels and weights saliency and edge losses while ignoring the roughest lowest-resolution predictions.Full-resolution stage 1 and deeper stages 2 and 3 contribute predictions.

A. Overview

EORSSD extends the small ORSSD benchmark into a larger, publicly available dataset designed to represent more diverse and challenging optical remote-sensing scenes. Its challenges include multiple and small salient objects, varied scenarios, imaging interference, and specific circumstances.

  • Dataset construction: EORSSD expands ORSSD from 800 images to 2,000 images with corresponding pixel-wise saliency annotations.The original ORSSD contains 600 training and 200 testing images; EORSSD adds 1,200 images.
  • Dataset construction: EORSSD covers more complicated scene types, diverse object attributes, and comprehensive real-world circumstances.The dataset includes challenging cases involving clouds, deserts, varied object categories, and practical imaging conditions.
  • Dataset challenges: Multiple salient objects occur frequently, with scenes containing more than two targets accounting for 36.5% of the dataset.Examples include eleven ships and seven airplanes in individual images.
  • Dataset challenges: Salient-object size varies substantially, creating a challenging small-object detection problem in optical remote-sensing imagery.Examples include vehicles and aircraft that appear as small targets in the dataset.
  • Dataset challenges: The dataset includes varied imaging interference and specific circumstances such as distortion, cloud occlusion, illumination changes, over-exposure, and partial visibility.These cases are intended to increase diversity and practical difficulty.
  • Availability: EORSSD is publicly available, but Google Earth acquisition and redistribution policies restrict the dataset to academic use.The project website provides access to the dataset under this usage boundary.

V. EXPERIMENTS

The experiments evaluate salient-object detection using threshold-based precision-recall analysis and complementary F-measure, MAE, and S-measure metrics. Training uses augmented EORSSD data with a fixed implementation and optimization protocol.

  • Evaluation metrics: Performance is evaluated with P-R curves, F-measure, MAE score, and S-measure.These metrics assess thresholded precision-recall behavior, precision-recall balance, map error, and structural similarity.
  • Evaluation metrics: P-R curves are generated by thresholding saliency maps from 0 to 255 and comparing resulting binary masks with ground truth.Curves closer to (1, 1) indicate better performance.
  • Evaluation metrics: MAE measures the average absolute difference between the continuous saliency map and ground truth, where smaller values are better.The calculation averages pixel-wise errors over the testing image dimensions.
  • Evaluation metrics: S-measure combines region and object similarity with equal weighting, using α = 0.5.Higher S-measure values indicate better structural similarity.
  • Network training: Training uses 1,400 EORSSD images, augmentation to 11,200 image pairs, 60 ADAM epochs, and batch size 8.Augmentation combines flipping and rotation; images are uniformly resized to 128 × 128.

C. Comparison with State-of-the-art Methods

DAFNet is compared with fifteen state-of-the-art methods on ORSSD and EORSSD through qualitative visualizations, P-R curves, and quantitative measures. It shows stronger localization, structural integrity, challenging-case handling, and overall metric performance, including with VGG16 and Res2Net-50 backbones.

  • Comparison setup: Fifteen competing SOD methods span unsupervised NSI methods, retrained deep-learning NSI methods, and optical-RSI methods.Comparisons are conducted on the testing subsets of both ORSSD and EORSSD.
  • Qualitative Comparison: DAFNet more accurately localizes salient regions and suppresses background interference, including for river scenes.The qualitative comparison reports that competing methods often miss valid regions or detect only parts of the salient object.
  • Qualitative Comparison: DAFNet preserves structural details and complete object content better than competing methods in scenes with thin river tributaries and multiple aircraft.Other methods may lose aircraft wings or tails and fail to detect all salient aircraft.
  • Qualitative Comparison: DAFNet remains competitive for small and multiple objects, accurately detecting four cars and a small aircraft in challenging examples.Other methods nearly miss some objects, incompletely detect the aircraft, or retain background regions incorrectly.
  • Qualitative Comparison: DAFNet suppresses shadow regions and retains more accurate boundaries under shadows and drastic lighting changes.The comparison specifically contrasts its results with EGNet and other methods.
  • Quantitative Comparison: DAFNet-V's P-R curve is higher than all comparison methods, while the Res2Net-50 version achieves the highest precision across both testing datasets.The Res2Net-50 result is reported with a remarkable margin over other methods.
  • Quantitative Comparison: DAFNet achieves the best performance under all reported quantitative measures and a large gain over other methods.For reference, EGNet's F-measure reaches 0.8438 on ORSSD and 0.8060 on EORSSD, while DAFNet is reported as best overall.

3) Attribute-based Study:

DAFNet outperforms LVNet across category-, quantity-, and size-based conditions, while ablations show progressively stronger results as GCA, CPA, and DAF are added. Visualizations indicate that GCA suppresses background interference and DAF improves object completeness through multi-level attention fusion.

  • Attribute-based Study: DAFNet achieves uniformly larger F-measure than LVNet across category-, quantity-, and size-based testing conditions.Performance remains superior for difficult aircraft, road, tiny-object, and very-large-object cases.
  • Attribute-based Study: Detection becomes more difficult as salient-object count increases, but DAFNet still greatly outperforms LVNet.
  • Attribute-based Study: DAFNet remains especially superior for very tiny objects (≤1%) and very large objects (≥40%), although tiny-object detection still needs improvement.
  • Ablation Study: Visual ablations show GCA suppressing background interference and DAF combining low- and high-level attention to recover more integral object structures.
  • Ablation Study: Adding GFA improves F-measure from 0.8391 to 0.8504 and S-measure from 0.8432 to 0.8661, with gains of 1.3% and 2.7%, respectively.
  • Ablation Study: CPA adds relative gains of 2.8% in F-measure and 1.1% in S-measure while reducing MAE by 15.3% relative to GFA.

E. Failure Cases

The paper identifies persistent DAFNet failures on tiny objects and objects whose appearance resembles surrounding context. These limitations are linked to input-size constraints and the difficulty of learning abstract semantic distinctions.

  • Failure Cases: Even in very challenging examples, DAFNet does not always obtain perfect results.
  • Failure Cases: Tiny salient objects can disappear during down-scaling because the method uses a relatively small input size under limited GPU resources.The paper suggests larger inputs or more memory-efficient global-context-aware attention as possible directions.
  • Failure Cases: DAFNet can miss or incompletely detect thin structures, such as a ship and a thin line in challenging examples.
  • Failure Cases: Objects with high appearance similarity to surrounding context can be missed when contextual texture and color overwhelm their distinctive information.The paper identifies better learning strategies for abstract semantic understanding as a direction for addressing this issue.
Loading 2011.13144v1…