PaperlayerView exact revision on arXiv

Source-linked AI summary

LSKNet: A Foundation Lightweight Backbone for Remote Sensing

Yuxuan Li, Xiang Li, Yimian Dai, Qibin Hou, Li Liu, Yongxiang Liu, Ming-Ming Cheng, Jian Yang

arXiv:2403.11735v6cs.CVcs.LG

TL;DR

Remote sensing recognition can require broad and object-dependent context, a prior that relatively few efficient foundation models explicitly exploit. The paper proposes lightweight LSKNet, which dynamically adjusts large receptive fields through spatial selective kernels, and reports state-of-the-art performance across remote sensing benchmarks.

  • Problem

    Remote sensing studies have relatively rarely considered strong remote-sensing priors, even though objects may require wide and varying contextual information for recognition.

  • Method

    LSKNet is a lightweight backbone that dynamically selects and spatially aggregates features from decomposed large kernels to adjust its receptive field.

  • Results

    LSKNet achieves state-of-the-art performance across remote sensing classification, detection, semantic segmentation, and change detection benchmarks, including 46.93% and 47.87% mAP for LSKNet-T and LSKNet-S on FAIR1M-v1.0.

  • Takeaways & Limitations

    The analysis validates the importance of the identified remote-sensing priors and the effectiveness of LSKNet for downstream remote sensing tasks.

  • Takeaways & Limitations

    Feature ensembles and ViT-based backbones can be complex, computationally inefficient, or resource-intensive for some practical applications.

Abstract

from arXiv · show

Remote sensing images pose distinct challenges for downstream tasks due to their inherent complexity. While a considerable amount of research has been dedicated to remote sensing classification, object detection and semantic segmentation, most of these studies have overlooked the valuable prior knowledge embedded within remote sensing scenarios. Such prior knowledge can be useful because remote sensing objects may be mistakenly recognized without referencing a sufficiently long-range context, which can vary for different objects. This paper considers these priors and proposes a lightweight Large Selective Kernel Network (LSKNet) backbone. LSKNet can dynamically adjust its large spatial receptive field to better model the ranging context of various objects in remote sensing scenarios. To our knowledge, large and selective kernel mechanisms have not been previously explored in remote sensing images. Without bells and whistles, our lightweight LSKNet sets new state-of-the-art scores on standard remote sensing classification, object detection and semantic segmentation benchmarks. Our comprehensive analysis further validated the significance of the identified priors and the effectiveness of LSKNet. The code is available at https://github.com/zcablii/LSKNet.

1 Introduction

Remote sensing recognition often depends on broad, object-specific context that conventional approaches may overlook. LSKNet addresses this gap with a lightweight backbone that dynamically adjusts its receptive field and achieves state-of-the-art results across remote sensing tasks.

  • Remote sensing images combine high resolution, random orientation, large intraclass variation, multiscale scenes, and dense small objects.
  • Recognition may depend more on surrounding context than appearance, because context provides clues about objects’ shape, orientation, and characteristics.
  • Different object types require different contextual ranges, from relatively limited context for soccer fields to extremely large receptive fields for tree-obscured intersections.
  • LSKNet dynamically modulates the backbone receptive field using input-dependent spatial selection over sequential large depth-wise kernels.
  • LSKNet achieves state-of-the-art performance across classification, detection, semantic segmentation, and change detection benchmarks spanning 14 public datasets.

2 Related Work

Related work has explored feature ensembles, transformers, attention, kernel selection, and large receptive fields, but remote sensing priors and spatially adaptive large kernels remain underexamined. LSKNet differs by selecting decomposed large kernels across the spatial rather than channel dimension.

  • Remote Sensing: Feature ensembles and ViT-based backbones can improve remote sensing performance but may be complex, computationally inefficient, or resource-intensive.
  • Remote Sensing Object Detection: Existing remote sensing detection methods address rotation variance but do not consider the strong prior information presented in aerial images.
  • Remote Sensing Semantic Segmentation: Attention and multiscale fusion methods improve segmentation, while large receptive-field semantics are important for combining fine-grained details with coarse-grained information.
  • Large Kernel Networks: Large-kernel networks have used kernels from 7×7 to 51×51, but their significance had not been examined in remote sensing before this work.
  • Attention/Selective Mechanism: Unlike SKNet’s channel-wise selection, LSKNet adaptively aggregates decomposed large-kernel features spatially to model target-specific spatial variance.

3 Methods

LSKNet is built from repeated LSK Blocks that combine large-kernel receptive-field expansion with spatially selective kernel aggregation. The design dynamically adapts contextual processing to different objects while reducing the cost of large convolutions.

  • 3.1 LSKNet Architecture: LSKNet repeats LSK Blocks, each containing Large Kernel Selection and Feed-forward Network residual sub-blocks.The feed-forward sub-block performs channel mixing and feature refinement.
  • 3.2 Large Kernel Convolutions: The large-kernel stage decomposes a large convolution into sequential depth-wise convolutions with growing kernels and increasing dilation.This produces multiple receptive fields for later adaptive selection.
  • 3.2 Large Kernel Convolutions: Increasing kernel size and dilation rapidly expand receptive fields while bounding dilation prevents gaps between feature maps.Two or three decomposed convolutions provide theoretical receptive fields of 23 and 29, respectively.
  • 3.2 Large Kernel Convolutions: Sequential decomposition reduces parameters relative to standard large kernels at the same theoretical receptive field.The decomposition also supplies features with varied large receptive fields for kernel selection.
  • 3.3 Spatial Kernel Selection: Spatial selection concatenates multi-receptive-field features, pools them by channel-wise average and maximum operations, and generates spatial attention maps.Sigmoid masks weight the decomposed-kernel features before convolutional fusion into S.
  • 3.3 Spatial Kernel Selection: The LSK module multiplies the input feature X element-wise by the fused attention feature S to produce its output.This mechanism adaptively collects different large receptive fields for different objects.

4 Experiments

Experiments evaluate LSKNet on remote sensing scene classification, object detection, and semantic segmentation across 11 datasets using specified pretraining protocols.

  • 4 Experiments: Experiments cover scene classification, object detection, and semantic segmentation across a total of 11 datasets.The main results use 300-epoch ImageNet-1K backbone pretraining, while scene classification uses 300-epoch millionAID pretraining.

4.1 Scene Classification

Scene-classification experiments use standard UCM, AID, and NWPU benchmarks with multiple dataset sizes and evaluation splits. LSKNet-T and LSKNet-S achieve competitive performance without feature-ensemble tricks.

  • 4.1 Scene Classification: Scene-classification experiments use the standard UCM, AID, and NWPU datasets and evaluate five benchmarks: UCM-82, AID-28, AID-55, NWPU-19, and NWPU-28.These benchmarks follow mainstream remote sensing classification practice.
  • 4.1 Scene Classification: UCM contains 2,100 images across 21 categories, while AID contains 10,000 images across 30 categories.UCM images are 256 × 256 and AID images are 600 × 600.
  • 4.1 Scene Classification: NWPU contains 31,500 images across 45 categories, with 700 images per category and 256 × 256 image size.The reported NWPU benchmarks are NWPU-19 and NWPU-28.
  • 4.1 Scene Classification: Without feature ensembles, lightweight LSKNet-T and LSKNet-S deliver competitive performance across multiple scene-classification datasets.The paper compares them with 22 other state-of-the-art methods.

4.2 Oriented Object Detection and SAR Object Detection

LSKNet is evaluated on four demanding oriented and SAR object-detection datasets, achieving strong results across RGB and grayscale modalities. Its reported performance includes state-of-the-art mAP on DOTA-v1.0, HRSC2016, and FAIR1M-v1.0, alongside improved SAR detection.

  • Datasets: The evaluation covers HRSC2016, DOTA-v1.0, FAIR1M-v1.0, and SAR-Aircraft, spanning oriented RGB detection and grayscale SAR detection.The datasets include diverse object categories and challenging remote-sensing scenes.
  • DOTA-v1.0: 81.37%, 81.64%, and 81.85% mAP are achieved by LSKNet-T, LSKNet-S, and LSKNet-S* on DOTA-v1.0, respectively.LSKNet-S reaches 18.1 FPS on 1024x1024 images using one RTX3090 GPU.
  • FAIR1M-v1.0: 46.93% and 47.87% mAP are achieved by LSKNet-T and LSKNet-S on FAIR1M-v1.0, respectively, surpassing the compared models.FAIR1M-v1.0 contains more than 1 million instances across five categories and 37 object subcategories.
  • SAR-Aircraft: LSKNets provide a significant performance improvement for SAR object detection compared with five state-of-the-art backbones under Cascade Mask RCNN and RetinaNet.SAR-Aircraft is grayscale and contains seven aircraft categories.
  • Efficiency comparison: ViTDet has 4.0x the FLOPs and 4.9x the parameters of LSKNet-T yet performs poorly on DOTA-v1.0 object detection.The comparison reports computational complexity, model size, and detection performance together.

4.3 Semantic Segmentaion

LSKNet is evaluated for semantic segmentation on five standard remote-sensing datasets using UNetFormer or SegFormer frameworks. LSKNet-T and LSKNet-S surpass other compared methods across the datasets in most major metrics.

  • Datasets: The segmentation evaluation uses Potsdam, Vaihingen, LoveDA, UAVid, and GID, covering high-resolution, complex, multiscale, UAV, and land-cover imagery.The datasets vary in image resolution, semantic categories, and scene complexity.
  • Evaluation setup: LSKNets are integrated within UNetFormer for Potsdam, Vaihingen, LoveDA, and UAVid, while GID comparisons use the SegFormer framework.The framework choices follow reported performance and availability considerations.
  • Results: LSKNet-T and LSKNet-S surpass other state-of-the-art methods across all five segmentation datasets in most major metrics.The comparison includes 14, 16, 13, 16, and 6 competing models for Potsdam, Vaihingen, LoveDA, UAVid, and GID, respectively.

4.4 Change Detection

The paper evaluates LSKNet for change detection on LEVIR-CD and S2Looking using bi-temporal remote-sensing image pairs. LSKNet-T and LSKNet-S surpass the compared methods on both datasets in F1 and IoU.

  • Datasets: Change detection is evaluated on LEVIR-CD and S2Looking, which provide bi-temporal 1024x1024 image pairs with binary-change annotations.LEVIR-CD contains 637 pairs, while S2Looking contains 5,000 pairs captured by optical satellites worldwide.
  • Evaluation setup: LSKNet-T and LSKNet-S are built within the Changer framework and compared with 17 recent high-performance models.The framework is selected for its reported performance and open-source availability.
  • Results: LSKNet-T and LSKNet-S surpass all compared state-of-the-art methods across both datasets in the main F1 and IoU metrics.The reported comparison covers LEVIR-CD and S2Looking jointly.

4.5 Ablation Study

The ablation studies examine LSKNet’s kernel decomposition, receptive-field size, spatial selection, pooling choices, framework generality, and comparisons with other large-kernel or selective-attention backbones. Results identify design settings that balance speed and accuracy and show lightweight performance advantages across tasks.

  • Large Kernel Decomposition: Decomposing the large kernel into two depth-wise kernels provides the best tradeoff between inference speed and mAP when the theoretical receptive field is 29.The configuration is evaluated through FPS and mAP.
  • Kernel Receptive Field Size: A receptive-field size of approximately 23 is most effective, while excessively small or large receptive fields can hinder LSKNet performance.This finding follows the two-kernel decomposition setting.
  • Selection mechanism: LSKNet selects information across the spatial dimension using sequentially decomposed large kernels, unlike SKNet’s parallel decomposition and channel-wise selection.The paper motivates spatial selection by the spatial variance of different remote-sensing targets.
  • Pooling Layers in Spatial Selection: Using both maximum and average pooling in spatial selection achieves the best performance without sacrificing inference speed.The pooling choices are evaluated in an ablation study of the LSK module.
  • Detection Framework Generality: LSKNet-T significantly improves detection over ResNet-18 across multiple detection frameworks while using 38% of its parameters and 50% fewer FLOPs.The frameworks include ORCNN, RoI Transformer, S2A-Net, and R3Det.
  • Comparison with Other Large Kernel/Selective Attention Backbones: Under similar or lower model-size and complexity budgets, LSKNet outperforms other compared backbones on DOTA-v1.0 detection, Vaihingen segmentation, and LEVIR-CD change detection.ViTDet is reported to perform poorly despite having the largest model size and computational complexity among the compared models.

5 Analysis

The analysis shows that LSKNet adapts contextual range by object category and network depth, while its visualizations and comparisons support improved handling of challenging remote-sensing detections.

  • Detection Results Visualization: The analysis evaluates LSKNet specifically on object detection because instance-level information is important for understanding the model’s overall behaviour.The supplied comparison tables report LSKNet results against other backbones and against ResNet-18 under similar detection settings.
  • Detection Results Visualization: Detection visualizations indicate that LSKNet captures relevant context, whereas ResNet captures limited context and ViTDet captures large but coarse spatial information.The comparison is presented for hard cases involving small and crowded objects.
  • Relative Context Range for Different Objects: LSKNet’s expected selective receptive-field range varies across object categories, measured by Rc as the ratio of expected selective receptive-field area to ground-truth box area.The receptive-field visualization is derived from spatial selection activation in the trained LSKNet model.
  • Relative Context Range for Different Objects: Bridge requires substantial additional context because it resembles roads and needs clues about whether it is surrounded by water, while Court categories have Rc below 0.1.Roundabout also has a relatively high Rc of 0.57.
  • Kernel Selection Behaviour: LSKNet-T selects broader context for Bridge than Roundabout and for Roundabout than Soccer-ball-field, matching their reported contextual requirements.The normalized kernel-selection difference is about 30% higher for Bridge than Roundabout and about 70% higher for Roundabout than Soccer-ball-field.
  • Kernel Selection Behaviour: The average kernel-selection difference decreases from 0.78 in first-layer blocks to 0.40 in second and third blocks and 0.33 in last-layer blocks.This pattern indicates greater use of large receptive fields in shallow layers and smaller kernels at higher levels.

6 Conclusion

The paper proposes LSKNet, a lightweight backbone for remote-sensing tasks that adapts a large spatial receptive field to diverse object contexts. Experiments report state-of-the-art performance on competitive remote-sensing benchmarks, and the analysis supports the model’s effectiveness.

  • 6 Conclusion: LSKNet is a lightweight backbone for scene classification, object detection, and semantic segmentation in remote-sensing images.It is designed to accommodate wider and adaptable contextual understanding.
  • 6 Conclusion: LSKNet adapts its large spatial receptive field to capture diverse contextual nuances across remote-sensing object types.The conclusion describes this adaptation as leveraging inherent characteristics of remote-sensing images.
  • 6 Conclusion: Extensive experiments report state-of-the-art performance for the lightweight model on competitive remote-sensing benchmarks.The paper’s comprehensive analysis is reported to validate the model’s effectiveness and significance.
Loading 2403.11735v6…