RRNet: Relational Reasoning Network with Parallel Multi-scale Attention for Salient Object Detection in Optical Remote Sensing Images

Runmin Cong; Yumo Zhang; Leyuan Fang; Jun Li; Yao Zhao; Sam Kwong

Source-linked AI summary

RRNet: Relational Reasoning Network with Parallel Multi-scale Attention for Salient Object Detection in Optical Remote Sensing Images

Runmin Cong, Yumo Zhang, Leyuan Fang, Jun Li, Yao Zhao, Sam Kwong

arXiv:2110.14223v2Oct 27, 2021cs.CV

TL;DR

Optical RSI SOD remains challenged by complex backgrounds, scale variation, and incomplete detections. RRNet combines spatial-channel relational reasoning with parallel multi-scale attention, and experiments on two datasets report superior performance to existing state-of-the-art detectors. The method nevertheless retains failure cases involving shadows, high-contrast nonsalient objects, and incomplete targets.

Problem
Existing optical RSI SOD methods can miss or incompletely detect scattered or large salient objects in complex, scale-varying scenes.
Method
RRNet combines high-level spatial and channel relational reasoning with low-level parallel multi-scale attention to model semantic relations and restore details.
Results
RRNet outperforms state-of-the-art salient object detectors on two optical RSI datasets, while ablations show gains from its proposed modules.
Takeaways & Limitations
Relational reasoning improves salient-object completeness, while parallel multi-scale attention supports detail recovery and handling of object scale variation.
Takeaways & Limitations
The method still struggles with complex shadows, high-contrast nonsalient objects, and completely detecting some salient objects.

Abstract

from arXiv · show

Salient object detection (SOD) for optical remote sensing images (RSIs) aims at locating and extracting visually distinctive objects/regions from the optical RSIs. Despite some saliency models were proposed to solve the intrinsic problem of optical RSIs (such as complex background and scale-variant objects), the accuracy and completeness are still unsatisfactory. To this end, we propose a relational reasoning network with parallel multi-scale attention for SOD in optical RSIs in this paper. The relational reasoning module that integrates the spatial and the channel dimensions is designed to infer the semantic relationship by utilizing high-level encoder features, thereby promoting the generation of more complete detection results. The parallel multi-scale attention module is proposed to effectively restore the detail information and address the scale variation of salient objects by using the low-level features refined by multi-scale attention. Extensive experiments on two datasets demonstrate that our proposed RRNet outperforms the existing state-of-the-art SOD competitors both qualitatively and quantitatively.

I. INTRODUCTION

Optical RSI SOD is difficult because salient objects vary greatly in scale, appear in random positions, and may be scattered or partially detected. RRNet addresses these challenges through relational reasoning and parallel multi-scale attention to improve completeness and detail restoration.

Optical RSIs contain objects ranging from automobiles occupying several pixels to rivers spanning the image, with scale differences also caused by imaging height.
Existing optical RSI SOD methods commonly miss or incompletely detect scattered objects and large salient regions.
Parallel multi-scale attention suppresses unimportant low-level features while restoring details through attention maps derived from multiple receptive fields.
RRNet combines a relational reasoning encoder with a multi-scale attention decoder for end-to-end optical RSI SOD.
Joint spatial and channel relational reasoning models semantic relations to promote complete salient-object generation.

B. Deep Learning based SOD Models in optical RSIs

Deep-learning SOD models for optical RSIs developed specialized encoder-decoder approaches to address scale variation and generate saliency maps from hierarchical features. RRNet uses relational reasoning in its encoder and multi-scale attention in its decoder.

LVNet was introduced with the ORSSD dataset, containing 600 training and 200 testing images, to support optical RSI SOD research.
RRNet is an encoder-decoder network whose relation-reasoning encoder models semantic relations and whose attention decoder restores multi-scale details.
The backbone uses five sequential convolutional blocks, obtaining low-level features from the first two and high-level features from the last three.
Relational reasoning modules follow each high-level convolutional stage, while the final decoding features are passed through additional convolutions to predict the saliency map.

B. Relational Reasoning Encoder

The relational reasoning encoder models dependencies in both spatial and channel dimensions using high-level features. It constructs graphs and learns data-dependent relations to improve the integrity of salient objects and regions.

Optical RSI SOD requires relational reasoning because objects can be numerous, differently scaled, widely separated, and embedded in redundant backgrounds.
The encoder decouples relational reasoning into spatial and channel modules applied successively to high-level features from stages 3, 4, and 5.
1) Graph Construction:: Spatial relational reasoning connects any two spatial locations, helping model long-range objects such as rivers spanning large image regions.
1) Graph Construction:: The spatial graph uses HW vertices from reshaped feature maps, with each vertex represented by a channel feature.
1) Graph Construction:: Channel graph reasoning models semantic relationships between channels by representing each channel with its spatial feature.
The adjacency matrix measures pairwise vertex similarity, while a data-dependent graph Laplacian dynamically captures semantic relations between vertex features.

2) Graph Reasoning:

The graph reasoning stage applies relational reasoning to encoder features using a data-dependent graph formulation. Reasoning over spatial and channel graphs improves the integrity of multiple salient objects and individual targets.

2) Graph Reasoning:: Graph reasoning encodes features using a graph convolution with a trainable weight matrix, graph Laplacian, and ReLU activation.The graph Laplacian is data-dependent and dynamically captures semantic relations between vertex features.
2) Graph Reasoning:: Spatial reasoning produces one relational feature representation, while channel reasoning produces another.The two graph choices model relationships in spatial and channel dimensions.
2) Graph Reasoning:: After graph reasoning, the integrity between multiple salient objects and within the same salient target is significantly improved.The paper reports this improvement from feature visualizations before and after relational reasoning.

C. Multi-scale Attention Decoder

The multi-scale attention decoder fuses decoder and encoder features while refining shallow features with attention. This design uses complementary encoder information to restore details across varying object scales.

C. Multi-scale Attention Decoder: The decoder gradually fuses feature maps from different levels to generate saliency-related features.Upsampled decoder features are integrated with corresponding encoder features through a fusion unit.
C. Multi-scale Attention Decoder: Parallel multi-scale attention is computed in the attention-map dimension through two complementary feature-generation strategies.One strategy uses different receptive fields on input features; the other computes attention over extracted multi-scale features.
C. Multi-scale Attention Decoder: The resulting multi-scale attention maps refine upsampled decoder features using low-level encoder information.The low-level features provide higher spatial resolution and detail information for decoding.

1) Multi-scale attention on the single-scale features:

The single-scale attention stream builds spatial attention maps from pooled input features using multiple convolutional receptive fields. Their aggregation produces a multi-scale attention map that highlights or suppresses spatial regions.

1) Multi-scale attention on the single-scale features:: Average-pooling and max-pooling convert the input features into a two-channel descriptor for spatial attention.The pooled maps are concatenated along the channel axis.
1) Multi-scale attention on the single-scale features:: The PMA module uses spatial attention and convolutions with different kernel sizes to construct its multi-scale attention processing.The figure identifies “Att” as spatial attention and “Conv” as convolution.
1) Multi-scale attention on the single-scale features:: Convolutions with 3 × 3, 5 × 5, and 7 × 7 filters transform the descriptor into three 2D spatial attention maps.These maps encode where to emphasize or suppress features.
1) Multi-scale attention on the single-scale features:: The three attention maps are aggregated by element-wise summation to produce the final multi-scale attention map.The equations define the aggregation across different receptive fields.
1) Multi-scale attention on the single-scale features:: The single-scale stream emphasizes different attention information under different receptive fields.This stream applies multi-scale convolutions directly to the input features before attention-map fusion.

2) Multi-scale attention on the multi-scale features:

The multi-scale-feature stream combines attention generated from features at multiple scales with decoder–encoder fusion. Attention from shallower features highlights deep features to support multi-scale detail restoration.

2) Multi-scale attention on the multi-scale features:: Attention generated from single-scale and multi-scale features is combined into the final attention map.The two attention sources correspond to the complementary streams of the parallel design.
2) Multi-scale attention on the multi-scale features:: The decoder fuses upsampled deeper features with corresponding shallower features by concatenation and convolution.The fused representation is treated as a deeper feature for subsequent merging with shallow features.
2) Multi-scale attention on the multi-scale features:: Multi-scale attention generated from shallower features highlights deep features during fusion to restore multi-scale details.The fusion uses the complementary spatial detail in shallow features and semantic information in deeper features.
2) Multi-scale attention on the multi-scale features:: The fusion operation uses element-wise multiplication with channelwise broadcasting and 2× spatial up-sampling.These operations are specified in the decoder fusion formulation.

D. Loss Function

The network derives its final saliency map from the top decoding level and supervises prediction with class-balanced binary cross-entropy.

Class-balanced binary cross-entropy supervises saliency prediction.The weighting balances salient and background pixel contributions.
The loss uses p and q to balance salient and background pixels based on image size and positive-label pixel count.p = (B − Bm)/B and q = Bm/B.

IV. EXPERIMENTS

Experiments evaluate RRNet on ORSSD and EORSSD using standard saliency metrics, visual comparisons, and comparisons with thirteen SOD methods. The reported results show strong performance across datasets and challenging remote-sensing scenes.

A. Evaluation Metrics: RRNet is evaluated with P-R curves, Fβ, MAE, E-measure, and S-measure.Higher Fβ, E-measure, and S-measure are better, while lower MAE is better.
B. Implementation Details: Experiments use ORSSD and EORSSD, with 600/200 and 1,400/600 training/testing images, respectively.
C. Comparison Methods: RRNet is compared with thirteen SOD methods, including optical-RSI-specific and natural-scene models.
D. Qualitative Comparison: Visual comparisons report more accurate and complete detection in complex scenes through global semantic relations and multi-scale detail restoration.
D. Qualitative Comparison: RRNet is reported to occupy a higher position on the P-R curves than the compared methods on both datasets.
D. Qualitative Comparison: The visual comparisons highlight improved detail detection, boundary quality, background suppression, and multi-object detection.

2) Quantitative Comparison:

Quantitative comparisons and ablations assess RRNet against competing methods and isolate the contributions of PMA and relational reasoning. The reported results favor the complete model across datasets and variants.

Quantitative Comparison: RRNet achieves the best reported S-measure, E-measure, and F-measure rankings among the compared methods.
Quantitative Comparison: 0.8% E-measure, 1.9% MAE, and 1.0% S-measure gains are reported over the second-best method on ORSSD.
Quantitative Comparison: 0.5% S-measure and 0.7% F-measure are the minimum reported gains on EORSSD.
Ablation Study: The full model combining spatial and channel relational reasoning more accurately locates salient objects and suppresses backgrounds than the partial variants.
Ablation Study: Adding PMA improves F-measure from 0.8302 to 0.8819 and S-measure from 0.8695 to 0.9021.The reported percentage gains are 3.7% and 6.2%, respectively.
Ablation Study: Spatial relational reasoning further improves performance, with a reported 13.3% gain in MAE score.
Further Validation: Replacing relational reasoning with a Non-local module changes MAE from 0.0076 to 0.0093, a reported 22.3% drop.
Further Validation: Removing either PMA branch reduces F-measure from 0.9119 to 0.9100 or by 0.9%, supporting complementary branch contributions.

D. Discussion

RRNet improves semantic reasoning and detail restoration for optical RSI salient-object detection, but challenging cases remain unresolved. The authors identify performance and input-spectrum limitations as directions for future improvement.

Failure Cases: Challenging examples still produce imperfect detection results, including errors involving shadows, high-contrast nonsalient objects, and object-background similarity.These failure modes can cause false positives or incomplete salient-object detection.
Future Work: The method’s MAE achieves only the second-best performance among the competitors.
Future Work: Near-infrared or other spectral channels may provide richer information than RGB input, motivating a large-scale multispectral SOD benchmark.The authors propose constructing such a benchmark to improve prediction ability.
RRNet models relationships across spatial and channel dimensions while restoring details through parallel multi-scale attention.The relational reasoning module targets semantic completeness, while the attention module addresses detail recovery across object scales.