PaperlayerView exact revision on arXiv

Source-linked AI summary

Nested Network with Two-Stream Pyramid for Salient Object Detection in Optical Remote Sensing Images

Chongyi Li, Runmin Cong, Junhui Hou, Sanyi Zhang, Yue Qian, Sam Kwong

arXiv:1906.08462v1cs.CV

TL;DR

Optical RSI saliency detection is difficult because overhead imagery contains varied objects, scales, orientations, and cluttered backgrounds. The paper proposes LV-Net, combining two-stream pyramids with nested encoder-decoder connections, and reports superior qualitative and quantitative performance on a new 800-image benchmark.

  • Problem

    Optical RSI saliency detection lacks sufficient training data and architectures suited to diverse scales, multiple salient objects, and complicated backgrounds.

  • Method

    LV-Net is an end-to-end network combining a two-stream pyramid module with nested encoder-decoder connections for optical RSI salient object detection.

  • Results

    LV-Net significantly outperforms state-of-the-art methods qualitatively and quantitatively on the constructed optical RSI benchmark.

  • Takeaways & Limitations

    The paper provides a specialized optical RSI saliency method and a publicly available benchmark with 800 images and pixel-wise annotation.

Abstract

from arXiv · show

Arising from the various object types and scales, diverse imaging orientations, and cluttered backgrounds in optical remote sensing image (RSI), it is difficult to directly extend the success of salient object detection for nature scene image to the optical RSI. In this paper, we propose an end-to-end deep network called LV-Net based on the shape of network architecture, which detects salient objects from optical RSIs in a purely data-driven fashion. The proposed LV-Net consists of two key modules, i.e., a two-stream pyramid module (L-shaped module) and an encoder-decoder module with nested connections (V-shaped module). Specifically, the L-shaped module extracts a set of complementary information hierarchically by using a two-stream pyramid structure, which is beneficial to perceiving the diverse scales and local details of salient objects. The V-shaped module gradually integrates encoder detail features with decoder semantic features through nested connections, which aims at suppressing the cluttered backgrounds and highlighting the salient objects. In addition, we construct the first publicly available optical RSI dataset for salient object detection, including 800 images with varying spatial resolutions, diverse saliency types, and pixel-wise ground truth. Experiments on this benchmark dataset demonstrate that the proposed method outperforms the state-of-the-art salient object detection methods both qualitatively and quantitatively.

I. INTRODUCTION

Optical RSI salient object detection is challenging because overhead imagery contains varied object orientations, scales, types, cluttered backgrounds, and occasional scenes without salient regions. The paper introduces LV-Net and a challenging 800-image pixel-wise annotated dataset for this setting.

  • Optical RSI salient object detection differs from ordinary object detection by producing pixel-level saliency probability maps rather than bounding boxes for all objects.
  • Overhead optical RSIs contain objects with diverse orientations, scales, and types, alongside cluttered backgrounds and shadow noise.
  • Existing nature-scene saliency methods are difficult to transfer directly to optical RSIs, while some RSI methods overlook their unique characteristics.
  • LV-Net is an end-to-end optical RSI detector combining a two-stream pyramid module with an encoder-decoder module using nested connections.
  • The two-stream pyramid learns complementary features for scale variability and local details, while nested connections suppress cluttered backgrounds and highlight salient objects.
  • The constructed benchmark contains 800 optical RSI images with corresponding pixel-wise ground truth, and LV-Net achieves the best performance against fourteen state-of-the-art methods.

II. RELATED WORK

Related work spans bottom-up and top-down saliency models, but optical RSI saliency remains comparatively underdeveloped. The paper addresses limited data and architectures that do not adequately handle diverse scales and cluttered backgrounds.

  • Bottom-up saliency models are stimulus-driven and explore low-level visual features such as contrast, background, and compactness.
  • Top-down saliency models are task-driven and use supervised learning with ground truth, with deep learning enabling effective feature extraction.
  • Only a small amount of work targets optical RSI saliency, and many proposed methods instead support ROI extraction or generalized object detection.
  • Optical RSI saliency is difficult because existing nature-scene methods inadequately address multiple salient objects, diverse scales, complicated backgrounds, and insufficient training data.
  • The paper responds with a deep learning method specially designed for optical RSIs and the first publicly available optical RSI salient-object dataset.

III. PROPOSED METHOD

LV-Net combines a two-stream pyramid with a nested encoder-decoder to retain details, capture multiscale semantics, and integrate both feature types for saliency prediction.

  • Framework: LV-Net takes an optical RSI as input and outputs its saliency map through a two-stream pyramid and a nested encoder-decoder module.
  • Two-Stream Pyramid Module: The input pyramid progressively downsamples the image, while a multiscale convolution unit extracts feature representations from each scaled input.
  • Two-Stream Pyramid Module: Concatenating multiresolution inputs with multiscale features forms complementary detail and semantic streams.
  • Encoder-Decoder Module with Nested Connections: Nested connections gradually integrate encoder detail features with decoder semantic features and use deep supervision to predict salient regions.

C. Encoder-Decoder Module with Nested Connections

The V-shaped module is a deeply supervised encoder-decoder network that uses nested connections to integrate encoder details with decoder semantics and produce the final saliency map.

  • C. Encoder-Decoder Module with Nested Connections: Nested connections link encoder and decoder pathways under deep supervision to select discriminative saliency features and reduce cluttered-background interference.The design addresses the complementary weaknesses of high-level decoder features and low-level encoder features.
  • C. Encoder-Decoder Module with Nested Connections: Encoder features are formed across down-sampling stages, while decoder units combine features with up-sampled outputs through concatenation.The notation {·} denotes feature concatenation rather than element-wise addition.
  • C. Encoder-Decoder Module with Nested Connections: The final unit CU(0,4) outputs F(0,4), which serves as the final saliency map.Except for CU(0,4), convolution units output 64 × 2^i feature maps, where i indexes the encoder down-sampling layer.

D. Loss Function

The loss function learns the saliency prediction mapping with sigmoid cross-entropy, modified by clipping predicted scores to avoid divergence at boundary values.

  • D. Loss Function: Sigmoid cross-entropy loss is minimized to learn the saliency prediction mapping from true labels and predicted scores.The labels 0 and 1 denote non-salient and salient regions, while y and z denote the true label and predicted score.
  • D. Loss Function: The loss is rewritten with clipped predictions because the original formulation can approach infinity when z equals 0 or 1.Images without salient objects can produce this boundary condition in optical RSI data.
  • D. Loss Function: Fclip clips values to the interval from ρ to µ, with ρ set to 1e−15 and µ set to 1 −1e−15.Values below ρ are raised to ρ, and values above µ are lowered to µ.

IV. EXPERIMENTS

The experiments evaluate the constructed benchmark, training and implementation settings, comparisons with state-of-the-art methods, and ablations of the proposed components.

  • IV. EXPERIMENTS: The experiments introduce the benchmark dataset, evaluation metrics, training strategies, and implementation details.These elements establish the evaluation and training protocol for the method.
  • IV. EXPERIMENTS: Comparisons with state-of-the-art methods are used to demonstrate the proposed LV-Net’s advantage.The study also includes ablations to verify component roles and analyze parameter effects.
  • IV. EXPERIMENTS: Ablation studies analyze the role of each component and the effects of parameter settings.The ablations complement the benchmark comparisons by examining design choices.

A. Benchmark Dataset

The ORSSD benchmark contains 800 optical remote sensing images with manually produced pixel-wise saliency annotations and challenging variation in resolution, backgrounds, and salient-object types.

  • A. Benchmark Dataset: Each image has a manually produced pixel-wise annotation of salient objects.Five people independently marked salient objects, and objects receiving four marks were selected as ground truth.
  • A. Benchmark Dataset: Sample images are paired with pixel-wise annotations, with optical RSIs shown in the first row and annotations in the second row.The figure illustrates the constructed ORSSD dataset and its corresponding ground truth format.
  • A. Benchmark Dataset: The dataset includes varying spatial resolutions, including 1264×987, 800×600, and 256×256 images.This resolution diversity contributes to the dataset’s stated challenge.
  • A. Benchmark Dataset: ORSSD covers cluttered backgrounds and diverse salient objects, including shadows, trees, buildings, airplanes, ships, cars, rivers, ponds, bridges, stadiums, and beaches.These variations are identified as sources of dataset difficulty.

B. Evaluation Metrics

The evaluation uses PR curves, F-measure, MAE, and S-measure to assess salient-object detection performance against pixel-wise ground truth.

  • Evaluation Metrics: PR curves evaluate precision-recall combinations obtained by thresholding saliency maps against ground truth.Precision forms the vertical axis, recall the horizontal axis, and curves nearer (1, 1) indicate better performance.
  • Evaluation Metrics: F-measure is the weighted harmonic mean of precision and recall, with β^2 set to 0.3 to emphasize precision.Higher Fβ values indicate better comprehensive performance.
  • Evaluation Metrics: MAE measures the difference between the continuous saliency map and ground truth, where lower scores indicate greater similarity.
  • Evaluation Metrics: S-measure evaluates structural similarity between the saliency map and ground truth using region and object similarity.The weighting parameter α is set to 0.5, and larger values indicate better structural similarity.

C. Training Strategies and Implementation Details

LV-Net is trained and evaluated on an ORSSD split with augmentation and fixed implementation settings, then compared visually and quantitatively against fourteen state-of-the-art methods.

  • Training Strategies and Implementation Details: 600 ORSSD images were used for training and 200 for testing, with flipping and rotation producing 4,800 training image pairs.Training samples were resized to 128 × 128 because of limited memory.
  • Comparison Setup: The comparison includes seven unsupervised, four deep learning-based, and three optical-RSI saliency methods.Results were generated from source code or provided by the authors.
  • Comparison Setup: Visual comparisons cover six optical RSIs containing ships, cars, airplanes, playgrounds, and islands.The study reports visual results in Fig. 6 and quantitative evaluations in Fig. 7 and Table II.
  • Comparison Results: LV-Net achieves higher precision and recall than other methods, with its PR curve substantially above the alternatives.Table II compares Precision, Recall, F-measure, MAE, and S-measure; reported gains over the second-best method include 3.4% precision, 3.8% recall, and 4.9% F-measure.
  • Efficiency Results: LV-Net ranks second smallest in model size among the compared deep learning methods.Table III reports average running time in seconds per image and model size in MB; the authors conclude that LV-Net is effective and efficient.

E. Module Analysis

Ablation studies show that LV-Net’s performance depends on both its two-stream pyramid and nested connections, while brute-force skips and simply enlarging feature maps are less effective.

  • Quantitative Ablation Results: LV-Net outperforms the tested ablation variants across all evaluation metrics, supporting the complete network architecture.
  • Two-Stream Pyramid: Removing either the input pyramid or feature pyramid worsens performance, indicating that complementary detail and semantic features are effective together.
  • Module Contributions: The complete LV-Net learns more competitive representations than variants without the L-shaped module or nested connections.
  • Nested Connections: Brute-force skip connections improve boundary completeness but introduce background noise, whereas nested connections support clearer and sharper salient-object boundaries.
  • Architecture and Capacity: The V-Net backbone does not filter complicated backgrounds effectively, while doubling its feature maps does not improve performance.
  • Conclusion: The ablations attribute competitive performance to both the two-stream pyramid and nested connections rather than merely larger hyper-parameters.

F. Parameter Analysis

The parameter analysis evaluates output-feature counts, scale counts, and multi-scale convolution choices in LV-Net, showing that richer settings improve saliency-detection performance.

  • Parameter settings: The study varies output-feature counts, scale counts, and convolution scale settings to assess their effects on LV-Net.Comparisons include reduced feature widths, four- and three-scale variants, and a single-scale convolution variant.
  • Output features: F-measure and S-measure decrease while MAE increases when the number of output features is reduced.The reduced-feature variants are LV-Net-16-32 and LV-Net-8-16.
  • Number of scales: Performance decreases as the number of scales is reduced from five to four or three.The analysis attributes the decline to fewer useful multi-scale features and less comprehensive representations.
  • Convolution scale: Multi-scale convolutional layers in the M-CU improve LV-Net compared with the single-scale LV-Net-S-CU.The single-scale variant replaces the multi-scale operations with three 3 × 3 convolutions.

V. CONCLUSION

The paper concludes that LV-Net detects salient objects in optical RSIs while suppressing cluttered backgrounds, and that its components and parameter settings are effective. Future work targets sharper boundaries, stronger spatial consistency, expanded data, and broader saliency tasks.

  • Conclusion: LV-Net accurately locates salient objects with diverse scales and effectively suppresses cluttered backgrounds in optical RSIs.The conclusion attributes these capabilities to the two-stream pyramid module and nested connections.
  • Conclusion: The constructed optical RSI dataset provides pixel-wise annotation for salient object detection.The dataset is presented alongside the proposed method as a contribution.
  • Conclusion: Experiments show that LV-Net significantly outperforms state-of-the-art methods qualitatively and quantitatively.Module analysis and parameter discussion also support the effectiveness of the designed components and settings.
  • Future work: Future work will add a context module to improve edge sharpness and spatial consistency and extend the ORSSD dataset.The authors also plan to extend co-saliency detection to optical RSIs.
Loading 1906.08462v1…