PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Salient Object Detection in Optical Remote Sensing Images via Feature Correlation

Gongyang Li, Zhi Liu, Zhen Bai, Weisi Lin, and Haibin Ling

arXiv:2201.08049v1cs.CV

TL;DR

Existing ORSI-SOD methods prioritize detection accuracy while neglecting parameter and computational complexity, motivating a lightweight solution. CorrNet lightens feature extraction and modules, uses correlation-guided coarse-to-fine refinement, and achieves competitive or better performance with 4.09M parameters and 100fps inference.

  • Problem

    Existing ORSI-SOD methods neglect parameter and computational complexity despite high detection accuracy, creating a need to balance accuracy, parameters, and computational cost.

  • Method

    CorrNet lightens the VGG-based backbone and other modules, then generates a coarse saliency map through cross-layer feature correlation before refining it with low-level features and Dense Lightweight Refinement Blocks.

  • Results

    CorrNet achieves competitive or better performance than previously proposed large CNN-based methods while using 4.09M parameters and reaching 100fps inference on EORSSD.

  • Takeaways & Limitations

    CorrNet demonstrates a lightweight ORSI-SOD framework that maintains competitive or better performance while reducing model resource demands.

Abstract

from arXiv · show

Salient object detection in optical remote sensing images (ORSI-SOD) has been widely explored for understanding ORSIs. However, previous methods focus mainly on improving the detection accuracy while neglecting the cost in memory and computation, which may hinder their real-world applications. In this paper, we propose a novel lightweight ORSI-SOD solution, named CorrNet, to address these issues. In CorrNet, we first lighten the backbone (VGG-16) and build a lightweight subnet for feature extraction. Then, following the coarse-to-fine strategy, we generate an initial coarse saliency map from high-level semantic features in a Correlation Module (CorrM). The coarse saliency map serves as the location guidance for low-level features. In CorrM, we mine the object location information between high-level semantic features through the cross-layer correlation operation. Finally, based on low-level detailed features, we refine the coarse saliency map in the refinement subnet equipped with Dense Lightweight Refinement Blocks, and produce the final fine saliency map. By reducing the parameters and computations of each component, CorrNet ends up having only 4.09M parameters and running with 21.09G FLOPs. Experimental results on two public datasets demonstrate that our lightweight CorrNet achieves competitive or even better performance compared with 26 state-of-the-art methods (including 16 large CNN-based methods and 2 lightweight methods), and meanwhile enjoys the clear memory and run time efficiency. The code and results of our method are available at https://github.com/MathLee/CorrNet.

I. INTRODUCTION

ORSI-SOD methods achieve strong accuracy but often incur substantial parameter and computational costs. CorrNet addresses this gap with a lightweight coarse-to-fine design using cross-layer correlation and refinement modules.

  • Motivation: 141.06M parameters and 16 fps characterize PA-KRN, while EMFINet uses 107.26M parameters and runs at 25 fps.These examples motivate a solution balancing accuracy, parameter count, and inference speed.
  • Lightweight CorrNet: 4.09M parameters are used by CorrNet, which lightens the VGG backbone and replaces regular convolutions with depthwise separable convolutions.The depthwise separable design reduces regular-convolution parameters by about 90%.
  • Coarse-to-fine design: CorrNet generates a coarse saliency map from high-level semantic correlation and uses it to guide low-level feature refinement.The refinement subnet employs Dense Lightweight Refinement Blocks to produce the final fine map.
  • Reported performance: 86.20% mean F-measure, 4.09M parameters, and 100fps are reported for CorrNet on EORSSD.The reported comparison combines accuracy, model size, and inference speed.
  • Evaluation: Evaluation against 26 state-of-the-art methods on two ORSI-SOD datasets finds CorrNet competitive with or better than previously proposed large CNN-based methods.The compared methods include large CNN-based and lightweight methods.

A. Lightweight Methods for NSI-SOD

Prior lightweight SOD work targets natural scenes, whereas ORSI-SOD methods generally emphasize accuracy while neglecting deployment costs. CorrNet introduces a lightweight ORSI-SOD framework to balance accuracy, parameters, and computational complexity.

  • Lightweight NSI-SOD: Lightweight NSI-SOD targets solutions suitable for edge computing devices, but lightweight ORSI-SOD remains largely unexplored.The paper positions CorrNet as the first lightweight ORSI-SOD solution it identifies.
  • Traditional ORSI-SOD: Traditional ORSI-SOD methods rely on hand-crafted features and are not efficient enough for practical applications because of computational and memory burdens.The stated burden limits the efficiency of these methods for deployment.
  • CNN-based ORSI-SOD: CNN-based ORSI-SOD methods achieve high detection accuracy on ORSSD and EORSSD but neglect parameter and computational complexity.The paper identifies this imbalance as a barrier to practical-system deployment.
  • CorrNet direction: CorrNet balances accuracy, parameters, and computational complexity by implementing all components in a lightweight manner.Its design combines a lightened VGG-16 backbone with lightweight modules.

A. Network Overview

CorrNet uses a lightweight feature-extraction subnet, a Correlation Module, and a refinement subnet in a coarse-to-fine pipeline. High-level features locate salient objects, while enhanced low-level features refine their boundaries and details.

  • Network components: CorrNet consists of a lightweight feature extraction subnet, a Correlation Module, and a refinement subnet.The feature extraction subnet includes a general Feature Enhancement Module, while the refinement subnet includes Dense Lightweight Refinement Blocks.
  • Feature extraction: The modified VGG-16 feature extractor contains five convolution blocks, with the last two using depthwise-separable convolutions.The subnet is named LFE-VGG and outputs features at multiple levels.
  • Feature enhancement: Channel and spatial attention enhance low-level features before refinement.The Feature Enhancement Module is applied to the first extracted feature groups.
  • Correlation Module: The Correlation Module models cross-layer correlation between high-level semantic features to generate the initial coarse saliency map S4.The map mines object-location information from high-level semantic context.
  • Refinement: The coarse map modulates intermediate features, and three Dense Lightweight Refinement Blocks produce the final fine saliency map S1.This implements the coarse-to-fine transition from object localization to detailed refinement.

B. Lightweight Feature Extraction Subnet

The lightweight feature-extraction design replaces parameter-heavy VGG components with LFE-VGG and uses CorrM to derive efficient location guidance from high-level feature correlations.

  • Lightweight Feature Extraction Subnet: The modified vanilla VGG remains unsuitable as a lightweight backbone because its last two convolution blocks contain 12.98M parameters, about 88.18% of all parameters.
  • Lightweight Feature Extraction Subnet: 11.50M parameters are removed from vanilla VGG, leaving LFE-VGG with 3.22M parameters.E4 decreases from 5.90M to 0.67M parameters, while E5 decreases from 7.08M to 0.81M.
  • Correlation Module: CorrM computes cross-layer correlations between continuous high-level semantic features to identify salient-region locations.The operator reshapes features, applies a learnable weight matrix, and uses matrix multiplication with row- and column-wise softmax normalization.
  • Correlation Module: The correlation operator uses few parameters and low computational cost while retaining strong salient-object localization capability.The correlated features have size 128×32×32, limiting the matrix-based operator's computational cost.
  • Correlation Module: CorrM combines a polishing gate with coarse-map generation and feature modulation to provide location information for subsequent refinement.The coarse saliency map is used to modulate low-level features, while the gate filters redundant correlated information.

D. Dense Lightweight Refinement Block

DLRB replaces cascaded regular convolutions with a dense, lightweight refinement structure that captures multiscale information for challenging ORSI objects.

  • D. Dense Lightweight Refinement Block: DLRB addresses multiple-object and small-object scenarios where cascaded refinement is suboptimal for ORSIs.The cascaded structure is described as inadequate for capturing multiscale information.
  • D. Dense Lightweight Refinement Block: Each DLRB uses three dilated DSConvs with dilation rates {2,4,6} and three 1×1 convolution layers.Dilated DSConvs enlarge the receptive field, while 1×1 convolutions merge the captured features.
  • D. Dense Lightweight Refinement Block: The dense DLRB structure combines multiscale information to strengthen feature representation during salient-object refinement.The paper states that this supports carving salient objects in ORSIs and leads to good performance.

E. Loss Function

CorrNet is trained with a combined BCE and IoU loss, alongside deep supervision of intermediate, coarse, and fine saliency maps.

  • E. Loss Function: CorrNet combines BCE loss and IoU loss into a comprehensive training objective.The loss design follows previous salient-object-detection methods.
  • E. Loss Function: Deep supervision trains two intermediate refinement-subnet maps together with the coarse and fine saliency maps.The intermediate and fine saliency maps are generated by 1×1 convolution layers.
  • E. Loss Function: The ground truth and the BCE and IoU components define the terms used in the total loss.G denotes ground truth, while ℓ_bce and ℓ_iou denote BCE and IoU loss, respectively.

A. Implementation Details and Evaluation Metrics

CorrNet is evaluated on ORSSD and EORSSD against 26 state-of-the-art methods using quantitative, computational-complexity, PR-curve, and visual comparisons. It achieves strong accuracy while offering substantially faster and smaller inference than large CNN-based alternatives.

  • Implementation Details: CorrNet is trained on 600 ORSSD and 1,400 EORSSD images, with 200 and 600 images reserved for testing, respectively.Augmentation produces 4,800 ORSSD and 11,200 EORSSD training pairs.
  • Evaluation Metrics: The evaluation compares CorrNet with 26 state-of-the-art methods spanning traditional, CNN-based, ORSI-SOD, NSI-SOD, and lightweight categories.The comparison includes eight traditional methods, sixteen CNN-based methods, and two lightweight methods.
  • Computational Complexity: 100 fps makes CorrNet 2.1× faster than SARNet at 47 fps, while its parameter count and FLOPs are smaller than those of CNN-based methods.Against CSNet and SAMNet, CorrNet is slightly inferior on the three computational-complexity metrics.
  • Quantitative Comparison: Its PR curve is closest to the upper-right region on both datasets, indicating competitive performance across precision-recall operating points.The PR plots distinguish the top five methods by color and show the remaining methods in gray.
  • Quantitative Comparison: CorrNet ranks first in five of eight EORSSD metrics and outperforms every compared method on all eight ORSSD metrics.On EORSSD, it also has 4× faster inference, 26× fewer parameters, and 23× fewer FLOPs than EMFINet.
  • Visual Comparison: Visual comparisons show CorrNet locating salient objects and preserving details across low-contrast, multiple-object, large-object, and cluttered-background scenes.Competing methods sometimes miss objects, omit bridge ends, lose details, or include confusing background regions.

C. Ablation Studies

Ablation studies examine CorrNet’s lightweight backbone, coarse-to-fine design, modules, correlation and gating operations, and dilation rates. The results support the efficiency of LFE-VGG and the contributions of the refinement and correlation components.

  • Coarse-to-Fine Strategy: The coarse-to-fine variants show generally incremental performance from the initial coarse map through the final fine map.The study evaluates S4, S3, S2, and S1 to assess progressive saliency refinement.
  • Individual Module Contributions: The full CorrNet improves the primitive Baseline by 1.43%, 2.30%, 1.61%, and 0.0030 on Sα, Fmax, Fξ, and M, respectively.The variants isolate FEM, DLRB, and CorrM before combining them in the full model.
  • Individual Module Contributions: DLRB and CorrM each improve Fξ over Baseline+FEM, while their cooperation produces the largest reported gains of 1.87% and 1.55%.These results support complementary contributions from refinement and correlation modules.
  • CorrM Components: Removing cross-layer correlation lowers Sα and Fβ to 92.32% and 86.90%, while removing gates yields 92.53% and 87.28%.The variants indicate that both correlation and polishing gates contribute to CorrM’s localization behavior.
  • DLRB Dilation Rates: Dilation rates {2,4,6} outperform no dilation and {1,3,5}, whereas expanding them to {3,5,7} degrades performance.The authors associate moderately larger receptive fields with complementary multiscale information and excessively large fields with poorer variable-scale capture.

D. Discussion

CorrNet remains constrained for deployment despite being lightweight relative to most CNN-based methods. Its main practical boundaries are model size on edge devices and real-time execution on CPUs.

  • CorrNet’s model size remains too large for edge-computing devices because it is based on GPU execution.
  • Despite reduced parameters and computations compared with most CNN-based methods, CorrNet is still difficult to run in real time on CPUs.
  • Future work targets a smaller lightweight backbone and model pruning to remove redundant layers and accelerate inference.

V. CONCLUSION

CorrNet is a lightweight ORSI-SOD framework that combines efficient feature extraction with coarse-to-fine saliency refinement. Across two ORSI-SOD datasets, it achieves competitive or better performance with substantially reduced computational cost and 100 fps inference.

  • CorrNet uses a lightened VGG backbone, DSConvs, a Correlation Module, and Dense Lightweight Refinement Blocks in a coarse-to-fine framework.
  • 4.09M parameters and 21.09G FLOPs enable CorrNet to achieve competitive or better performance than large CNN-based methods on two ORSI-SOD datasets.
  • 100 fps inference demonstrates CorrNet’s runtime efficiency alongside its reduced parameter and computation costs.
  • CorrNet’s success is attributed to cross-layer correlation, DSConvs, and lightweight refinement blocks that preserve effective feature representation with few parameters.
Loading 2201.08049v1…