PaperlayerView exact revision on arXiv

Source-linked AI summary

LSNet: See Large, Focus Small

Ao Wang, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding

arXiv:2503.23135v1cs.CV

TL;DR

Existing CNNs, ViTs, and lightweight models face efficiency and effectiveness challenges in token mixing under limited computational budgets. The paper introduces LS convolution and LSNet using large-kernel perception with small-kernel aggregation, and reports superior performance-efficiency trade-offs across vision tasks.

  • Problem

    Existing lightweight networks commonly use self-attention or convolution for token mixing, limiting the efficiency or effectiveness of perception and aggregation under constrained computation.

  • Method

    LS convolution combines large-kernel static perception with small-kernel dynamic aggregation, and LSNet builds lightweight models from this operation.

  • Results

    LSNet achieves superior performance and efficiency compared with existing lightweight models across various vision tasks.

  • Takeaways & Limitations

    LSNet offers a lightweight network family with reported state-of-the-art performance-efficiency trade-offs across a broad range of vision tasks.

  • Takeaways & Limitations

    The paper notes that potential malicious use of the models is a concern, while related mitigation discussions are beyond its scope.

Abstract

from arXiv · show

Vision network designs, including Convolutional Neural Networks and Vision Transformers, have significantly advanced the field of computer vision. Yet, their complex computations pose challenges for practical deployments, particularly in real-time applications. To tackle this issue, researchers have explored various lightweight and efficient network designs. However, existing lightweight models predominantly leverage self-attention mechanisms and convolutions for token mixing. This dependence brings limitations in effectiveness and efficiency in the perception and aggregation processes of lightweight networks, hindering the balance between performance and efficiency under limited computational budgets. In this paper, we draw inspiration from the dynamic heteroscale vision ability inherent in the efficient human vision system and propose a ``See Large, Focus Small'' strategy for lightweight vision network design. We introduce LS (\textbf{L}arge-\textbf{S}mall) convolution, which combines large-kernel perception and small-kernel aggregation. It can efficiently capture a wide range of perceptual information and achieve precise feature aggregation for dynamic and complex visual representations, thus enabling proficient processing of visual information. Based on LS convolution, we present LSNet, a new family of lightweight models. Extensive experiments demonstrate that LSNet achieves superior performance and efficiency over existing lightweight networks in various vision tasks. Codes and models are available at https://github.com/jameslahm/lsnet.

1. Introduction

Existing vision networks and lightweight models face a difficult balance between computational efficiency, perception, aggregation, and representational effectiveness. LSNet addresses this with a “See Large, Focus Small” strategy implemented through LS convolution.

  • CNNs and ViTs advance computer vision but remain computationally expensive for practical and real-time deployment.
  • Lightweight models commonly rely on self-attention or convolution for token mixing, creating trade-offs between architectural expressiveness and inference speed.
  • Self-attention and convolution use different perception and aggregation processes, motivating more effective and efficient token mixing under limited computational costs.
  • Human vision motivates first capturing broad scene context through peripheral vision and then focusing on specific elements through central vision.
  • LS convolution combines large-kernel static perception with small-kernel dynamic aggregation to emulate this heteroscale strategy.
  • LSNet builds lightweight models from LS convolution and reports superior performance and efficiency over existing lightweight models across various vision tasks.

2. Related Work

Related work develops efficient CNNs, ViTs, and token-mixing mechanisms for practical vision models. These approaches improve efficiency through architectural changes, hybrid designs, or adaptive feature aggregation.

  • Efficient CNNs use designs such as depth-wise separable convolutions, inverted residual blocks, linear bottlenecks, and channel shuffling.
  • Lightweight ViTs improve deployment efficiency through architectures including hybrid combinations of MobileNet blocks and multi-head self-attention.
  • Efficient token-mixing research explores adaptive convolutional weighting, per-example kernel routing, and other mechanisms for spatial information exchange.
  • The related approaches frame LS convolution as an alternative token-mixing design alongside self-attention and convolution.

3. Methodology

The methodology separates broad contextual perception from focused feature aggregation. LS convolution uses large-kernel perception to generate context-adaptive weights and small-kernel dynamic convolution to aggregate related local features, forming the basis of LSNet.

  • Revisiting Self-Attention and Convolution: Token mixing can be expressed as perception of contextual relationships followed by aggregation of features from other tokens.
  • Revisiting Self-Attention and Convolution: Self-attention performs global perception and weighted aggregation but incurs redundant attention, excessive aggregation, and increased complexity at larger contextual scopes.
  • Revisiting Self-Attention and Convolution: Convolution uses a K × K neighborhood with fixed relative-position weights, limiting perception range and adaptation to token-specific context.
  • LS (Large-Small) Convolution: LS convolution uses different contextual scopes for perception and aggregation, enabling large-field context capture with small-field adaptive feature fusion.
  • LS (Large-Small) Convolution: Large-Kernel Perception projects features to a lower channel dimension, applies large-kernel depth-wise convolution, and generates context-adaptive aggregation weights.
  • LS (Large-Small) Convolution: Small-Kernel Aggregation divides channels into groups and uses generated weights for grouped dynamic convolution over K_S × K_S neighborhoods.
  • LS (Large-Small) Convolution: The proposed LS convolution outperforms simple large- and small-kernel combinations in reported comparisons.

L 2 ), and O(HWCK2

LSNet provides variants for different computational budgets and uses a staged architecture that concentrates more blocks in later, lower-resolution stages. Its design combines efficient LS blocks with a final attention-based stage.

  • Computational Complexity: The reported computational formulation has linear complexity with respect to input resolution.
  • LS Block: LS blocks use LS convolution for token mixing, skip connections for optimization, depth-wise convolution and SE for additional local inductive bias, and FFNs for channel mixing.
  • Architecture: LSNet uses overlapping patch embedding, convolutional downsampling, LS blocks in the top three stages, and an MSA block in the final low-resolution stage.
  • Model Variants: LSNet-T, LSNet-S, and LSNet-B use 0.3G, 0.5G, and 1.3G FLOPs, respectively.
  • Architecture: The architecture places more blocks in later stages because processing early high-resolution stages is more time consuming.

4. Experiments

Experiments across classification, downstream vision tasks, robustness benchmarks, ablations, and architectural transfers show that LSNet combines strong accuracy with computational efficiency. The analyses support contributions from both large-kernel perception and small-kernel aggregation.

  • Image Classification: LSNet achieves state-of-the-art ImageNet-1K classification across computational costs, with favorable accuracy–inference-speed trade-offs.LSNet-B surpasses AFFNet by 0.5% top-1 accuracy with nearly 3× faster inference speed.
  • Object Detection and Instance Segmentation: LSNet-T exceeds StarNet-S1 by 0.6 AP and 1.3 AP50 in RetinaNet with notably less computational cost.LSNet-B also surpasses PoolFormer-S12 and PVT-Tiny by 3.0 AP and 2.5 AP, respectively.
  • Semantic Segmentation: LSNet consistently outperforms competitors on ADE20K semantic segmentation across model scales and computational costs.LSNet-T gains 1.6 mIoU over VAN-B0 and 2.9 mIoU over PVTv2-B0; LSNet-B gains 1.6 and 2.0 mIoUs over SwiftFormer-L1 and FastViT-SA24.
  • Robustness Evaluation: LSNet shows strong robustness and domain generalization across ImageNet-C, ImageNet-A, ImageNet-R, and ImageNet-Sketch.Compared with UniRepLKNet-A, LSNet-B reduces mCE by 1.3 and improves top-1 accuracy by 1.2%, 1.5%, and 1.5% on the other three benchmarks.
  • Model Analyses: Removing LS convolution reduces top-1 accuracy by 2.3% while adding only 0.02G FLOPs, and replacing it with other token mixers lowers accuracy.LS convolution also surpasses CondConv and DY-Conv by 1.8% and 1.6% top-1 accuracy, respectively.
  • Model Analyses: Large-kernel perception and small-kernel aggregation each materially improve LS convolution, while adaptive aggregation achieves its best accuracy–cost trade-off at KS = 3.Removing large-kernel perception lowers top-1 accuracy by 1.1%, whereas removing small-kernel aggregation lowers it by 1.5% relative to the complete design.
  • Model Analyses: Replacing 3×3 convolutions or self-attention with LS convolution improves ResNet50 and DeiT-T top-1 accuracy by 1.9% and 0.8%, respectively.This transfer uses LS convolution directly in the two architectures under the same 300-epoch training settings.

5. Conclusion

LSNet is a lightweight vision-network family built around the “See Large, Focus Small” strategy and LS convolution. The paper reports strong performance-efficiency trade-offs across diverse vision tasks.

  • LSNet integrates the “See Large, Focus Small” strategy inspired by the human vision system.
  • LS convolution combines large-kernel perception with small-kernel aggregation for efficient and accurate visual-information processing.
  • Extensive experiments show state-of-the-art performance and efficiency trade-offs across diverse tasks.

A.1. Implementation Details

The paper specifies task-specific training procedures and analyzes LS convolution against alternative token-mixing and convolution designs. These details cover optimization settings, architectural variants, and computational comparisons.

  • Implementation Details: ImageNet-1K models use 224×224 inputs, 300 training epochs, AdamW, cosine scheduling, and a total batch size of 2048.The initial learning rate is 4×10−3, with mixup, RandAugment, CutMix, and random erasing used for augmentation.
  • Implementation Details: COCO-2017 models use 12 epochs, batch size 16, 1333×800 resolution, AdamW, and pretrained ImageNet-1K backbones.The learning rate starts at 2×10−4 and decays at the 8-th and 11-th epochs.
  • Implementation Details: ADE20K models use 40K AdamW iterations, batch size 32, 512×512 training resolution, poly scheduling, and pretrained ImageNet-1K backbones.Single-scale validation results are reported, with poly-schedule power 0.9 and initial learning rate 2×10−4.
  • Implementation Details: ImageNet-C, ImageNet-A, ImageNet-R, and ImageNet-Sketch are used for robustness evaluation.The benchmarks cover algorithmic corruptions, naturally occurring misclassified examples, natural renditions, and sketch images.
  • Implementation Details: LSNet variants differ in their numbers of blocks and channels across stages.
  • Implementation Details: LS convolution uses adaptable contextual modeling, unlike simple large-plus-small convolution combinations and other dynamic alternatives.The comparison discusses fixed kernel weights, limited context adaptability, and differing token- or example-dependent perception.

C.1. Analyses for LS Convolution

Visualization analyses examine LS convolution’s receptive fields, aggregation weights, feature maps, and downstream qualitative behavior. They show broad context capture alongside focused, semantically relevant detail processing.

  • Effective Receptive Field: LS convolution produces broad peripheral perception and central focusing, whereas RepMixer and CGA show unnatural effective-receptive-field patterns.Removing LKP yields a smaller receptive field than the complete LS convolution.
  • Aggregation Weights: Aggregation weights in LS convolution correlate with semantically relevant regions and become more precise with LKP.The visualization attributes improved contextual information and visual-pattern capture to integrating LKP.
  • Feature Maps: LKP captures a broad scene view, while SKA further captures subtle features and detailed patterns.The feature-map analysis presents this as the complementary behavior of the two LS-convolution components.
  • Aggregation Weights: SKA strengthens semantically relevant visual regions and helps capture discriminative image patterns.The aggregation-weight analysis describes favorable interpretability for these spatial emphases.
  • Component Effects: LKP and SKA help LS convolution grasp critical visual information under limited computational costs.The passage links these components with improved efficiency and effectiveness in visual processing.
  • Downstream Qualitative Results: Qualitative COCO results show precise detection and instance segmentation of instances in diverse images.
  • Downstream Qualitative Results: Qualitative ADE20K results show high-quality semantic segmentation masks compared with the ground-truth-mask presentation.
  • Component Effects: LS convolution supports effective and efficient perception and aggregation processes.

C.2. Analyses for Downstream Tasks

LSNet is evaluated in downstream frameworks for detection, instance segmentation, and semantic segmentation. The qualitative results show precise predictions and high-quality masks across diverse images.

  • Downstream Tasks: LSNet is integrated into Mask-RCNN for object detection and instance segmentation.
  • Downstream Tasks: LSNet is integrated into Semantic FPN for semantic segmentation.
  • Downstream Tasks: The qualitative results show precise instance detection and segmentation alongside high-quality semantic segmentation masks.

D. Contribution, Limitation, and Impact

The paper proposes the “See Large, Focus Small” strategy, LS convolution, and LSNet for lightweight vision networks. It reports broad performance–efficiency trade-offs while limiting evaluation to selected scenarios and pretraining settings.

  • Contribution: “See Large, Focus Small” uses broad perceptual context with focused feature aggregation for efficient lightweight network design.The strategy is inspired by the human vision system and targets detailed visual understanding.
  • Contribution: LS convolution combines large-kernel perception with small-kernel aggregation to process visual information effectively and efficiently.It is introduced as a novel operation for modeling visual features in lightweight models.
  • Contribution: LSNet is a family of lightweight vision networks built on LS convolution, with reported state-of-the-art performance–efficiency trade-offs across vision tasks.The paper describes extensive experiments comparing LSNet with other lightweight networks.
  • Limitation: The study does not extend LSNet to visual-language or unsupervised-learning scenarios because of limited computational resources.These applications are identified as future exploration areas.
  • Limitation: Large-scale pretraining, including on ImageNet-21K, is not investigated for the same resource-related reason.The limitation concerns the paper’s evaluated pretraining scope.
  • Impact: The authors identify potential malicious use of the models but provide no further discussion because it is beyond the paper’s scope.The study reports no direct negative social impact identified by the authors.
Loading 2503.23135v1…