PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Salient Object Detection in Optical Remote-Sensing Images via Semantic Matching and Edge Alignment

Gongyang Li, Zhi Liu, Xinpeng Zhang, Weisi Lin

arXiv:2301.02778v2cs.CV

TL;DR

ORSI-SOD methods often carry excessive parameters and computational cost, limiting portability and practical use. SeaNet addresses this with lightweight semantic matching and edge alignment, and reports competitive accuracy with 2.76M parameters and 1.7G FLOPs for 288×288 inputs.

  • Problem

    CNN-based ORSI-SOD methods often have large parameter counts and computational costs, while practical applications require more lightweight and portable models.

  • Method

    SeaNet combines MobileNet-V2 feature extraction with DSMM for high-level semantic localization, ESAM for low-level edge enhancement, and a lightweight decoder.

  • Results

    2.76M parameters and 1.7G FLOPs for 288×288 inputs accompany competitive detection accuracy against lightweight methods and competitive computational complexity against conventional methods.

  • Takeaways & Limitations

    SeaNet provides a lightweight ORSI-SOD solution whose semantic and edge modules support accurate saliency inference at reduced computational cost.

Abstract

from arXiv · show

Recently, relying on convolutional neural networks (CNNs), many methods for salient object detection in optical remote sensing images (ORSI-SOD) are proposed. However, most methods ignore the huge parameters and computational cost brought by CNNs, and only a few pay attention to the portability and mobility. To facilitate practical applications, in this paper, we propose a novel lightweight network for ORSI-SOD based on semantic matching and edge alignment, termed SeaNet. Specifically, SeaNet includes a lightweight MobileNet-V2 for feature extraction, a dynamic semantic matching module (DSMM) for high-level features, an edge self-alignment module (ESAM) for low-level features, and a portable decoder for inference. First, the high-level features are compressed into semantic kernels. Then, semantic kernels are used to activate salient object locations in two groups of high-level features through dynamic convolution operations in DSMM. Meanwhile, in ESAM, cross-scale edge information extracted from two groups of low-level features is self-aligned through L2 loss and used for detail enhancement. Finally, starting from the highest-level features, the decoder infers salient objects based on the accurate locations and fine details contained in the outputs of the two modules. Extensive experiments on two public datasets demonstrate that our lightweight SeaNet not only outperforms most state-of-the-art lightweight methods but also yields comparable accuracy with state-of-the-art conventional methods, while having only 2.76M parameters and running with 1.7G FLOPs for 288x288 inputs. Our code and results are available at https://github.com/MathLee/SeaNet.

I. INTRODUCTION

SeaNet addresses the computational burden of CNN-based ORSI-SOD by separating high-level semantic localization from low-level detail enhancement in a lightweight encoder-decoder network. It targets lower complexity than CorrNet while maintaining competitive detection performance.

  • Motivation: CNN-based SOD methods can achieve strong accuracy but often impose large parameter counts and computational costs.This motivates lightweight approaches for practical ORSI-SOD applications.
  • Approach: SeaNet explores high-level and low-level features with different strategies for semantic matching and edge alignment.The design uses high-level features for salient-object localization and low-level features for detail preservation.
  • Efficiency and positioning: 2.76M parameters and 1.7G FLOPs for a 288×288 image make SeaNet substantially lightweight.The paper reports competitive detection accuracy against lightweight methods and competitive computational complexity against conventional methods.
  • Approach: DSMM compresses semantic information into kernels and matches them with high-level features to perceive salient-object locations.It also uses channel-wise correlation to enhance channel interactions.
  • Approach: ESAM aligns cross-scale edge information from low-level features through L2 loss and uses the corrected edges for detail enhancement.This is intended to correct edge errors while retaining fine object details.

II. RELATED WORK

Prior ORSI-SOD methods address varied object sizes, complex backgrounds, and boundary preservation, but commonly remain computationally heavy. Lightweight methods reduce parameters, yet existing ORSI-SOD solutions retain substantial computation or lack feature-processing specificity.

  • CNN-based SOD: Conventional NSI-SOD strategies inspire ORSI-SOD research but often do not handle ORSI-specific properties and usually prioritize accuracy over computational complexity.The paper cites deep supervision, edge assistance, and differential feature processing as influential strategies.
  • CNN-based ORSI-SOD: Specialized ORSI-SOD methods use multi-resolution inputs, foreground-background-edge integration, and related strategies to handle varied object sizes and complex scenes.These methods achieve satisfactory performance but generally use many parameters and high computational cost.
  • Lightweight SOD: Lightweight NSI-SOD backbones learn multi-level and multi-scale information but directly infer salient objects without further level-specific feature enhancement.They were developed for natural scenes rather than optical remote-sensing images.
  • Positioning of SeaNet: SeaNet extends lightweight feature extraction with DSMM for semantic matching and ESAM for self-aligned cross-scale edge correction.The modules are designed to process extracted features at different levels rather than directly decoding them.
  • Lightweight ORSI-SOD: CorrNet reduces parameters to 4.09M but still requires 21.1G FLOPs for lightweight ORSI-SOD.SeaNet adopts MobileNet-V2 and lightweight modules to address this computational-cost issue.

III. PROPOSED METHOD

SeaNet uses a MobileNet-V2 encoder, semantic matching for high-level localization, edge self-alignment for low-level detail enhancement, and a lightweight decoder for progressive saliency inference.

  • Network overview: SeaNet follows an encoder-decoder architecture containing MobileNet-V2, SKC, DSMM, ESAM, and a lightweight decoder.Semantic matching activates salient-object locations before edge alignment enhances details.
  • Encoder: MobileNet-V2 produces five-level features that are divided into high-level and low-level groups for different processing strategies.The encoder uses a truncated MobileNet-V2 and organizes features into five blocks.
  • High-level processing: SKC compresses the highest-level semantic features into two kernels, which DSMM applies to two groups of high-level features through dynamic depthwise convolutions.Channel-wise correlation further enhances feature interactions.
  • Low-level processing: ESAM extracts low-level edge information, corrects cross-scale edge errors through self-alignment, and uses the corrected edges for feature enhancement.The module also includes channel-wise correlation.
  • Decoder: The decoder progressively combines the highest-level features with DSMM and ESAM outputs to recover resolution and generate the final saliency map.Saliency heads after decoder blocks support deep supervision and final output generation.

B. Dynamic Semantic Matching Module

DSMM activates salient-object locations from high-level features through lightweight spatial semantic matching and channel-wise correlation. Semantic kernels generated by SKC drive dynamic depthwise convolutions, while channel dependencies provide additional feature interaction.

  • Overview: DSMM uses high-level features to activate salient regions with limited parameters and computational cost.It combines spatial semantic matching with channel-wise correlation for localization.
  • Semantic Knowledge Compression: SKC compresses high-level semantic information into two semantic kernels, k3 ∈R32×5×5 and k4 ∈R96×5×5.Parallel DSconv and adaptive average pooling layers perform channel and resolution compression.
  • Spatial Semantic Matching: Semantic kernels drive dynamic depthwise convolutions with dilation rates {1, 2, 3} to perceive salient objects across multiple receptive fields.DDconv performs dynamic convolution depthwise, reducing parameters while supporting objects with variable sizes and multiple objects.
  • Spatial Semantic Matching: Outputs from three dilated DDconv layers are summed element-wise and fused with pointwise convolution to combine features from different receptive fields.This design further reduces parameters and facilitates multi-receptive-field feature interaction.
  • Channel-wise Correlation: Channel-wise correlation models dependencies between feature channels and transfers the learned dependencies to spatially enhanced features.Row-wise and column-wise softmax normalize the affinity matrix before feature integration through a short connection and DSconv.

C. Edge Self-Alignment Module

ESAM extracts edge information from low-level features and self-aligns cross-scale edges for detail enhancement. The lightweight module uses corrected edges to preserve complex salient-object shapes while adding channel-wise interaction.

  • Overview: ESAM explores low-level texture and detail features to preserve the complex shapes of salient objects without edge supervision.Its lightweight design targets efficient edge-based detail enhancement.
  • Edge-based Enhancement Unit with Self-Alignment: Low-level inputs are aligned by DSconv and upsampling before pooling-subtraction extracts two groups of edge features.The aligned features provide inputs to the edge-based enhancement units.
  • Edge-based Enhancement Unit with Self-Alignment: L2 loss adaptively corrects errors between cross-scale edge features when edge supervision is unavailable.The self-alignment mechanism aims to produce accurate and consistent edge information.
  • Edge-based Enhancement Unit with Self-Alignment: Corrected edge features enhance aligned low-level features in EEUs through spatial feature modulation.The edge-enhanced EEU outputs are then used for further channel interaction.
  • Channel-wise Correlation: Channel-wise correlation is applied to EEU outputs after spatial enhancement, producing the final ESAM features.ESAM thereby saves parameters and computational cost while supporting accurate highlighting of complex geometry and topology.

D. Decoder and Loss Function

SeaNet decodes the location and detail features from DSMM and ESAM into saliency maps, using deep supervision during training. Its objective combines saliency and edge-alignment losses.

  • Decoder: The lightweight decoder uses three blocks and produces saliency maps at 36×36, 144×144, and 288×288 resolutions.The first two maps support deep supervision, while the 288×288 map is the final output.
  • Decoder: Each decoder block contains two DSconv layers, upsampling, and another DSconv layer.SalHeads after the blocks use dropout and 1×1 convolution to generate the saliency maps.
  • Loss Function: SeaNet jointly trains with binary cross-entropy and intersection-over-union losses for saliency supervision.The total objective also includes an edge-alignment loss weighted separately from saliency loss.

A. Experimental Setup

Experiments evaluate SeaNet on ORSSD and EORSSD using saliency, error, and computational-complexity metrics. Training uses PyTorch on an NVIDIA Titan X GPU with batch size 1 for speed reporting.

  • Datasets: ORSSD contains 800 images, split into 600 training and 200 test images.The dataset provides corresponding pixel-level annotations.
  • Datasets: EORSSD contains 2000 images, split into 1400 training and 600 test images.It adds 1200 images to ORSSD.
  • Evaluation Metrics: Evaluation uses S-measure, F-measure, E-measure, and mean absolute error, with the first three higher-is-better and the last lower-is-better.Maximum, mean, and adaptive F-measure and E-measure are reported.
  • Evaluation Metrics: Computational evaluation reports inference speed, parameter amount in millions, and FLOPs in giga units.Speed uses batch size 1 without I/O time; parameters and FLOPs are lower-is-better.
  • Training Protocol: Training is conducted with PyTorch on a computer equipped with an NVIDIA Titan X GPU with 12GB memory.

B. Performance Analysis

SeaNet balances accuracy and efficiency across conventional and lightweight SOD comparisons, while qualitative results show strong localization and detail preservation in challenging ORSI scenes.

  • Comparison with Conventional SOD Methods: SeaNet achieves comparable accuracy to conventional ORSI-SOD methods with substantially lower computational complexity.Against EMFINet, it reports similar Emaxξ values while using 24.5× fewer parameters, 282.8× fewer FLOPs, and 3.6× faster inference.
  • Comparison with Lightweight SOD Methods: SeaNet outperforms lightweight NSI-SOD methods by 3.50%∼8.44% in Sα, 4.87%∼13.76% in Fmeanξ, and 0.0037∼0.0120 in M across two datasets.Its main remaining weakness in this comparison is that it has no parameter or FLOP advantage, although inference speed is better.
  • Comparison with Lightweight SOD Methods: 1.3× fewer parameters and 12.4× fewer FLOPs than CorrNet, with comparable inference speed and only slightly different accuracy.SeaNet records Eadpξ 0.9670 versus 0.9721 on ORSSD and M 0.0073 versus 0.0083 on EORSSD.
  • Overall Results: SeaNet achieves one first place, two second places, and five third places among quantitative evaluation metrics.The results indicate a balance between effectiveness and efficiency.
  • Qualitative Comparison: Qualitatively, SeaNet accurately highlights multiple objects, complete large-object boundaries, and salient objects in chaotic backgrounds.The reported improvements are associated with DSMM's multi-scale semantic matching and ESAM's edge-based detail enhancement.

C. Ablation Studies

Ablations show that DSMM and ESAM contribute complementary localization and detail capabilities, while their lightweight design remains adaptable across backbones.

  • Contribution of Two Lightweight Modules: Removing DSMM changes M from 0.0073 to 0.0093, whereas removing ESAM changes it to 0.0084.The authors attribute the larger DSMM effect to degraded salient-object localization; ESAM primarily affects object details.
  • Contribution of Two Lightweight Modules: DSMM and SKC together add 0.01M parameters and 0.10G FLOPs, while ESAM adds 0.06M parameters and 0.07G FLOPs.Their cooperation with MobileNet-V2 supports performance without many parameters or much computational cost.
  • Backbone Embedding: SeaNet-SAM and SeaNet-HVP retain comparable performance to SeaNet while demonstrating adaptation to different feature-extraction backbones.The variants embed the two modules and decoder into SAMNet and HVPNet.
  • Backbone Embedding: SeaNet-SAM uses 1.51M parameters and 1.4G FLOPs, while HVPNet uses 1.23M parameters and 1.1G FLOPs; SeaNet-SAM nevertheless outperforms HVPNet by a large margin.The comparison supports the reported advantage of the SeaNet modules when complexity is comparable.
  • Effectiveness of DSMM: Removing DSMM spatial matching produces the worst performance among the tested DSMM variants by truncating object-localization capability.The ablation also evaluates dilation and channel-wise correlation components.
  • Effectiveness of ESAM: Removing ESAM edge units changes Fmeanβ from 0.8519 to 0.8441, while removing edge alignment degrades all four reported metrics.The authors describe edge alignment as improving edge-information accuracy without increasing parameters or FLOPs.

V. CONCLUSION

SeaNet treats high- and low-level features differently through lightweight semantic matching and edge alignment modules. Their cooperation enables fast salient-object inference, with experiments supporting the network's effectiveness and efficiency.

  • Conclusion: DSMM explores salient-object locations through spatial semantic matching, while ESAM aligns low-level edge information for detail enhancement.The conclusion presents these modules as the complementary treatment of high- and low-level features.
  • Conclusion: The cooperation of DSMM and ESAM enables accurate salient-object inference in the decoder at a fast speed.Performance analysis and ablation studies support the reported effectiveness and efficiency against state-of-the-art methods.
Loading 2301.02778v2…