PaperlayerView exact revision on arXiv

Source-linked AI summary

Adjacent Context Coordination Network for Salient Object Detection in Optical Remote Sensing Images

Gongyang Li, Zhi Liu, Dan Zeng, Weisi Lin, Haibin Ling

arXiv:2203.13664v1cs.CV

TL;DR

Optical remote sensing images differ from natural scenes in resolutions, object types, and scales, making direct migration of natural-scene saliency methods unsatisfactory. ACCoNet coordinates adjacent encoder features and enriches decoder context for RSI-SOD, outperforming previous methods on public benchmarks.

  • Problem

    Differences between natural scenes and optical remote sensing images in resolutions, object types, and object scales make directly migrated NSI-SOD methods often perform unsatisfactorily.

  • Method

    ACCoNet coordinates current, previous, and subsequent encoder features through ACCoM and uses bifurcation-aggregation decoder blocks to capture multiscale contextual information.

  • Results

    ACCoNet outperforms previous methods on public benchmarks, while dilated-convolution bifurcations improve max Fβ to 88.37% on EORSSD and 91.49% on ORSSD.

  • Takeaways & Limitations

    Exploring contextual information in both encoder and decoder improves salient-object representation for variable object scales, shapes, quantities, and challenging RSI scenes.

Abstract

from arXiv · show

Salient object detection (SOD) in optical remote sensing images (RSIs), or RSI-SOD, is an emerging topic in understanding optical RSIs. However, due to the difference between optical RSIs and natural scene images (NSIs), directly applying NSI-SOD methods to optical RSIs fails to achieve satisfactory results. In this paper, we propose a novel Adjacent Context Coordination Network (ACCoNet) to explore the coordination of adjacent features in an encoder-decoder architecture for RSI-SOD. Specifically, ACCoNet consists of three parts: an encoder, Adjacent Context Coordination Modules (ACCoMs), and a decoder. As the key component of ACCoNet, ACCoM activates the salient regions of output features of the encoder and transmits them to the decoder. ACCoM contains a local branch and two adjacent branches to coordinate the multi-level features simultaneously. The local branch highlights the salient regions in an adaptive way, while the adjacent branches introduce global information of adjacent levels to enhance salient regions. Additionally, to extend the capabilities of the classic decoder block (i.e., several cascaded convolutional layers), we extend it with two bifurcations and propose a Bifurcation-Aggregation Block to capture the contextual information in the decoder. Extensive experiments on two benchmark datasets demonstrate that the proposed ACCoNet outperforms 22 state-of-the-art methods under nine evaluation metrics, and runs up to 81 fps on a single NVIDIA Titan X GPU. The code and results of our method are available at https://github.com/MathLee/ACCoNet.

I. INTRODUCTION

RSI-SOD differs from NSI-SOD because optical remote sensing images contain distinct resolutions, object types, scales, and challenging scenes. ACCoNet addresses these differences by coordinating adjacent features and enriching decoder context.

  • Optical RSIs differ from natural images in shooting devices, scenes, view orientations, resolutions, object types, and object scales.
  • Directly transferring NSI-SOD methods to RSI-SOD often performs unsatisfactorily, with GateNet incompletely highlighting salient objects.
  • Existing RSI-SOD methods have insufficient feature-interaction coverage and may lose sharp boundaries and fine details.
  • ACCoNet coordinates current, previous, and subsequent features to expand interaction coverage and capture contextual information for varied object types and sizes.
  • ACCoM uses one local branch and two adjacent branches, while BAB extends decoder blocks with bifurcations to capture multi-scale context.
  • The paper presents ACCoNet as an encoder-decoder network designed to improve RSI-SOD through adjacent-feature coordination and contextual-information capture.

II. RELATED WORK

Related work spans traditional and CNN-based SOD for natural and optical remote sensing images. RSI-SOD methods adapt feature processing to optical imagery, but task-specific methods may not generalize across varied scenes.

  • NSI-SOD: NSI-SOD research includes unsupervised, semi-supervised, and supervised traditional methods based on handcrafted features.
  • NSI-SOD: CNN-based NSI-SOD methods improve detection accuracy through multi-level, multi-scale, suppression, labeling, and edge-aware feature strategies.
  • RSI-SOD: RSI-SOD methods modify handcrafted features or CNN feature-processing strategies because optical RSIs differ essentially from natural images.
  • RSI-SOD: Traditional RSI-SOD methods use color information, global cues, background priors, sparse representation, saliency analysis, and feature fusion.
  • RSI-SOD: CNN-based RSI-SOD includes multi-resolution architectures designed to infer salient objects from optical remote sensing features.
  • Task scope: Methods developed for specific optical-RSI scenes may perform well there but generalize poorly across diverse scenes.
  • Task scope: Optical RSI-SOD remains challenging because salient objects have complex geometry, variable sizes, uncertain quantities, occlusion, shadows, and abnormal illumination.

III. METHODOLOGY

ACCoNet uses an encoder-decoder architecture with five-scale encoding, adjacent context coordination, and bifurcation-aggregation decoding. These components combine local, cross-scale, and multi-scale contextual information for salient-object inference.

  • Network overview: ACCoNet comprises an encoder, several ACCoMs, and a decoder containing BABs.
  • Encoder network: The encoder extracts feature maps from five VGG-16 blocks, providing multiple scales for subsequent processing.
  • Adjacent Context Coordination Module: ACCoM coordinates current, previous, and subsequent feature levels because high-level features provide semantics while low-level features provide fine details.
  • Adjacent Context Coordination Module: Its local branch adaptively modulates current features, while two adjacent branches use spatial attention to provide aligned cross-scale information.
  • Decoder network: The decoder infers salient objects, but classic cascaded blocks depend strongly on encoder features and are not sensitive to optical-RSI scenes.
  • Bifurcation-Aggregation Block: BABs replace plain cascade structure with bifurcation-aggregation processing to capture multi-scale contextual information in the decoder.

B. Adjacent Context Coordination Module

The Adjacent Context Coordination Module (ACCoM) connects encoder and decoder features by combining local enhancement with adjacent-level assistance. Its coordinated branches activate salient regions and make them more obvious in the output features.

  • Module structure: ACCoM connects the encoder and decoder through local and adjacent branches that coordinate current, previous, and subsequent features.ACCoM-2, ACCoM-3, and ACCoM-4 use three branches, while ACCoM-1 and ACCoM-5 use two.
  • Local branch: The local branch applies four parallel dilated convolutions with different dilation rates to the current encoder feature.The dilated convolutions traverse regions with different sizes before adaptive feature modulation.
  • Local branch: Channel and spatial attention adaptively purify the summarized current feature and select valuable contents for the local branch output.Channel attention uses spatial-wise global max pooling, while spatial attention uses channel-wise global max pooling.
  • Adjacent branches: The adjacent branches align previous and subsequent features with the current feature, contributing fine-detail and object-location information.The previous-to-current branch uses 2× downsampling, whereas the subsequent-to-current branch uses 2× bilinear upsampling.
  • Branches integration: With all branches working together, ACCoM-3 comprehensively coordinates features and makes salient objects more obvious in its output.The feature visualization reports accurate activation of each salient region when the branches operate jointly.

C. Bifurcation-Aggregation Block

The Bifurcation-Aggregation Block (BAB) extends the decoder with two dilated-convolution bifurcations and aggregates their outputs to capture contextual cues. It processes current ACCoM features and the previous BAB output to infer salient-object masks.

  • Block role: BAB is the decoder’s basic unit, processing current ACCoM features and the previous BAB output before inferring salient-object masks.The block is applied through the decoder as a cascaded processing unit.
  • Bifurcations: Two bifurcations use dilated convolutions to expand the receptive field and capture contextual cues from BAB features.Different BABs use different dilation rates because their feature resolutions differ.
  • Aggregation: The two bifurcation outputs are concatenated and convolved with the original third cascaded-convolution feature to form the aggregated representation.The table specifies kernel size, channel numbers, dilation rate, and output-feature size for the bifurcations.
  • Purpose: The BAB design is intended to adapt decoder inference to changes in salient-object shape, size, and quantity in optical remote sensing images.This extends the contextual scanning capability beyond a plain cascade of convolutional layers.

D. Loss Function

ACCoNet uses deep supervision with hybrid pixel-level and map-level losses to train intermediate and final saliency predictions. The authors state that this stabilizes training and improves detection accuracy.

  • Deep supervision: Deep supervision attaches pixel-level supervision to every decoder block, generating intermediate and final saliency maps.Each map is produced after a BAB by an additional convolutional layer.
  • Hybrid loss: The total loss combines binary cross-entropy loss with intersection-over-union loss for complementary pixel-level and map-level enhancement.The ground truth is used for both loss components.
  • Effect: The deep-supervision strategy with hybrid losses stabilizes ACCoNet training and improves detection accuracy.

A. Experimental Protocol

ACCoNet is evaluated on two public RSI-SOD datasets using nine metrics, with comparisons spanning 22 state-of-the-art methods and reported processing speed.

  • Datasets: The evaluation uses ORSSD and EORSSD, containing 800 and 2,000 optical RSIs respectively, with pixel-wise annotations and separate training/testing splits.ORSSD uses 600 training and 200 testing images; EORSSD uses 1,400 training and 600 testing images.
  • Implementation: Training and testing resize inputs to 256 × 256 and initialize the encoder with pretrained VGG-16 parameters on an NVIDIA Titan X GPU.Newly added layers use normal-distribution initialization, and the initial learning rate is 1e−4.
  • Evaluation metrics: Nine metrics assess structural similarity, F-measure, E-measure, mean absolute error, and precision-recall performance.The metrics include S-measure, maximum, mean, and adaptive F-measure; maximum, mean, and adaptive E-measure; MAE; and the PR curve.
  • Reporting: Table II reports the nine performance measures together with frames per second, using upward or downward arrows to indicate preferred directions.The table marks the top three results and reports FPS for the compared methods.

B. Comparison with State-of-the-arts

ACCoNet is compared qualitatively and quantitatively with representative NSI-SOD and RSI-SOD methods on challenging optical-remote-sensing scenes. It achieves leading benchmark performance and fast inference while addressing tiny, multiple, irregular, and shadowed objects.

  • Quantitative comparison: On EORSSD, ACCoNet ranks first in four of eight tabulated metrics and second in the other four, performing best overall among compared methods.It surpasses EMFINet by 1.17% on max Fβ, while remaining marginally lower on adaptive Fβ and adaptive Eξ.
  • Quantitative comparison: On ORSSD, ACCoNet outperforms every compared method on all nine quantitative metrics, including 0.0088 MAE.Its gains over the second-best method reach 1.89% on adaptive Fβ, 1.47% on max Fβ, and 1.15% on mean Fβ.
  • Overall comparison: Across the two datasets, the authors identify ACCoNet as the best RSI-SOD method and report specialized RSI-SOD methods outperforming same-period NSI-SOD methods.The authors use this comparison to argue that specialized RSI-SOD development is necessary and urgent.
  • Visual comparison: ACCoNet more accurately highlights objects under shadows and handles tiny-object scenes where several traditional and NSI-SOD methods detect wrong or no objects.The visual comparisons specifically emphasize tiny objects and multiple tiny objects.
  • Visual comparison: In challenging multiple-object scenes, ACCoNet locates all objects finely without redundant regions while other methods miss objects or detect extra regions.The comparison includes representative CNN-based RSI-SOD, CNN-based NSI-SOD, and traditional methods.
  • Ablation study: Ablation examples show that ACCoM and BAB can each complete missing objects, while BAB and ACCoM suppress progressively more background errors.The baseline is the basic encoder-decoder network, and the complete ACCoNet combines both components.
  • Speed comparison: 81 fps makes ACCoNet the fastest of 16 compared methods, exceeding the second-best EFMINet at 25 fps by more than three times.The authors characterize the method as combining detection accuracy with fast processing speed.

C. Ablation Studies

Ablations show that ACCoM and BAB both improve RSI-SOD, while local and adjacent ACCoM branches provide complementary benefits and dilated convolutions strengthen decoder context capture.

  • The overall contributions of ACCoM and BAB: ACCoNet improves the EORSSD baseline by 1.95% on max Fβ, 0.0017 on M, and 1.80% on max Eξ when ACCoM and BAB cooperate.On ORSSD, the corresponding improvements are 3.17%, 0.0050, and 2.30%.
  • The effectiveness of two types of branches in ACCoM: On ORSSD, removing the local branch reduces max Fβ from 91.49% to 90.29%, M from 0.0088 to 0.0113, and max Eξ from 97.96% to 96.91%.The same degradation trend is observed on EORSSD.
  • The effectiveness of two types of branches in ACCoM: On ORSSD, removing adjacent branches reduces max Fβ from 91.49% to 90.72%, M from 0.0088 to 0.0108, and max Eξ from 97.96% to 97.39%.The same trend is observed on EORSSD.
  • The effectiveness of two types of branches in ACCoM: The local branch supports multiple or large salient objects, whereas adjacent branches preserve fine details through cross-level contextual complementarity.Without the local branch, objects may be missed; without adjacent branches, details such as occluded regions, airplane tails, and slender rivers are damaged.
  • The rationality of the dilated convolution based bifurcations in BAB: On EORSSD, direct-connection bifurcations achieve 88.31% max Fβ versus 88.37% for the original BAB, while normal convolutions achieve 88.34%.On ORSSD, the corresponding values are 91.36%, 91.49%, and 91.44%; the authors attribute the advantage to varied receptive fields for contextual information.

4. The complementarity between BCE and IoU in loss

The ablation study finds that BCE and IoU losses provide complementary supervision, and combining them improves ACCoNet performance and salient-object completeness.

  • The complementarity between BCE and IoU in loss: Combining BCE and IoU losses outperforms training with either BCE alone or IoU alone.BCE supplies pixel-level supervision, whereas IoU supplies map-level supervision.
  • The complementarity between BCE and IoU in loss: The composite loss helps preserve salient-object completeness by training the network from pixel-level and map-level perspectives.The paper identifies this combined loss as common in salient object detection.

V. CONCLUSION

The paper proposes ACCoNet for RSI-SOD by coordinating adjacent encoder features and capturing multiscale decoder context. Experiments report superiority over 22 relevant methods and significant contributions from both proposed modules.

  • V. CONCLUSION: ACCoNet uses ACCoM to coordinate current, previous, and subsequent features for salient-region activation.ACCoM explores adjacent information in the encoder.
  • V. CONCLUSION: BAB captures multiscale contents in the decoder, while deep supervision with hybrid losses stabilizes network training.Both ACCoM and BAB learn contextual information for salient-object representation.
  • V. CONCLUSION: ACCoNet is reported superior to 22 relevant state-of-the-art methods, and both ACCoM and BAB contribute significantly to performance.The conclusion cites quantitative, visual, speed, and ablation comparisons.
Loading 2203.13664v1…