PaperlayerView exact revision on arXiv

Source-linked AI summary

Scale-Aware Modulation Meet Transformer

Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, Lianwen Jin

arXiv:2307.08579v2cs.CV

TL;DR

Vision Transformers face high-resolution self-attention costs, while hybrid architectures struggle to model the transition from local to global dependencies. SMT combines scale-aware convolutional modulation with an evolutionary hybrid network, and it reports superior performance across classification, detection, and segmentation benchmarks with fewer parameters and FLOPs.

  • Problem

    High-resolution self-attention is computationally expensive, and existing hybrid CNN-Transformer architectures do not adequately model the transition from local to global dependencies.

  • Method

    SMT combines Scale-Aware Modulation using Multi-Head Mixed Convolution and lightweight Scale-Aware Aggregation with an evolutionary hybrid network that transitions from local to global modeling.

  • Results

    SMT consistently outperforms state-of-the-art models across classification, object detection, instance segmentation, and semantic segmentation while using fewer parameters and FLOPs.

  • Takeaways & Limitations

    SMT provides a hybrid ConvNet–vision Transformer backbone for diverse visual tasks with reported accuracy and efficiency advantages over compared models.

Abstract

from arXiv · show

This paper presents a new vision Transformer, Scale-Aware Modulation Transformer (SMT), that can handle various downstream tasks efficiently by combining the convolutional network and vision Transformer. The proposed Scale-Aware Modulation (SAM) in the SMT includes two primary novel designs. Firstly, we introduce the Multi-Head Mixed Convolution (MHMC) module, which can capture multi-scale features and expand the receptive field. Secondly, we propose the Scale-Aware Aggregation (SAA) module, which is lightweight but effective, enabling information fusion across different heads. By leveraging these two modules, convolutional modulation is further enhanced. Furthermore, in contrast to prior works that utilized modulations throughout all stages to build an attention-free network, we propose an Evolutionary Hybrid Network (EHN), which can effectively simulate the shift from capturing local to global dependencies as the network becomes deeper, resulting in superior performance. Extensive experiments demonstrate that SMT significantly outperforms existing state-of-the-art models across a wide range of visual tasks. Specifically, SMT with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs can achieve 82.2% and 84.3% top-1 accuracy on ImageNet-1K, respectively. After pretrained on ImageNet-22K in 224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with resolution 224^2 and 384^2, respectively. For object detection with Mask R-CNN, the SMT base trained with 1x and 3x schedule outperforms the Swin Transformer counterpart by 4.2 and 1.3 mAP on COCO, respectively. For semantic segmentation with UPerNet, the SMT base test at single- and multi-scale surpasses Swin by 2.0 and 1.1 mIoU respectively on the ADE20K.

1. Introduction

SMT addresses the cost and modeling limitations of vision Transformers by combining scale-aware convolutional modulation with an evolutionary hybrid architecture that transitions from local to global dependencies. Across classification, detection, and segmentation, the paper reports strong performance with efficient computation.

  • Motivation: High-resolution vision Transformers face quadratic self-attention costs, while convolutional networks primarily model local regions.This motivates more efficient architectures that balance local modeling and global interactions.
  • Results: 82.2% and 84.3% top-1 accuracy are achieved on ImageNet-1K by SMT tiny and base models, respectively.The reported results are shown against recent state-of-the-art vision Transformers and convolutional networks.
  • Results: SMT consistently outperforms state-of-the-art models on COCO and ADE20K for object detection, instance segmentation, and semantic segmentation.The evaluation covers classification, detection, and segmentation benchmarks while using fewer parameters and lower computational costs than the compared vision Transformers.
  • Method: SMT introduces Scale-Aware Modulation with Multi-Head Mixed Convolution and lightweight Scale-Aware Aggregation to integrate multi-scale contexts and adaptively modulate tokens.The two components are designed to enhance convolutional modulation and support more precise predictions.
  • Method: SMT’s evolutionary hybrid network models the transition from local to global dependencies as network depth increases.The design responds to limitations of simplistic convolution–Transformer stacking strategies.

2. Related Work

Related work spans vision Transformers, convolutional networks, and hybrid CNN-Transformer architectures. Existing approaches improve efficiency or combine local and global modeling, but current hybrid networks do not adequately model range-dependency transitions.

  • Vision Transformers: Vision Transformers adapt Transformer self-attention to images but face quadratic complexity and difficulty representing image 2D structure.These challenges are especially relevant for high-resolution vision tasks.
  • Convolutional Networks: Convolutional networks provide efficient local modeling, with later designs adding multiple paths, attention mechanisms, neural architecture search, or hierarchical structures.The cited CNN developments include Inception, ResNeXt, Res2Net, MixNet, EfficientNets, MobileNetV3, and ConvNeXt.
  • Hybrid Networks: Figure 2 presents SMT’s architecture and its Mix Block, which stacks Scale-Aware Modulation and multi-head self-attention blocks successively.SAM denotes scale-aware modulation, while MSA denotes multi-head self-attention.
  • Hybrid Networks: Hybrid CNN-Transformer models combine convolutions for local features with Transformers for long-range dependencies and have shown effectiveness in recognition and downstream applications.Examples include CvT, CMT, MobileViT, EdgeNeXt, MobileFormer, and EfficientFormer.
  • Hybrid Networks: Existing hybrid networks lack the ability to model transitions between dependency ranges, making further performance improvement challenging.This limitation motivates the evolutionary hybrid network proposed by SMT.

3. Method

SMT combines scale-aware convolutional modulation with strategically placed self-attention to model multi-scale features efficiently and transition from local to global dependencies. Its SAM uses MHMC and SAA, while EHN places SAM and MSA blocks across stages to balance computational cost and dependency range.

  • Overall architecture: SMT uses four stages with downsampling rates {4, 8, 16, 32}, applying SAM in the top two stages and sequential SAM–MSA blocks in the penultimate stage.This hybrid design models the transition from local to global dependency capture rather than constructing an attention-free network.
  • Scale-Aware Modulation: MHMC splits channels into heads and applies depth-wise separable convolutions with monotonically increasing kernel sizes to capture multi-scale spatial features.Different heads use distinct kernels, while larger kernels expand the receptive field and reduce parameter and computational costs through depth-wise separability.
  • Scale-Aware Modulation: Multi-head convolution captures shallow local details while preserving target-object shape and semantic information more effectively than single-head convolution at greater depth.Visualizations show foreground localization and background filtering in early stages, with deeper representations retaining overall object structure.
  • Scale-Aware Modulation: SAA groups channels across heads, performs lightweight within-group up-down fusion, and applies point-wise convolution for cross-group information aggregation.The grouping strategy enhances multi-scale feature diversity while limiting computation and enabling global information exchange.
  • Scale-Aware Modulation: SAA strengthens semantically relevant low-frequency signals and focuses modulation on essential object regions, improving feature representation for recognition.Visualizations highlight features such as eyes, head, and body after aggregation, compared with convolution maps before aggregation.
  • Evolutionary Hybrid Network: EHN restricts MSA blocks to the penultimate stage and uses two hybrid stacking strategies to reduce computational burden while simulating local-to-global dependency evolution.The relative receptive field shows an early adaptation period followed by a smooth upward trend as depth increases.

4. Experiments

SMT is evaluated across image classification, object detection, instance segmentation, semantic segmentation, and component ablations. Across these experiments, SMT reports strong accuracy and efficiency, while MHMC, SAA, and evolutionary hybrid stacking each contribute measurable gains.

  • ImageNet-1K classification: 82.2% top-1 accuracy is achieved by the tiny SMT on ImageNet-1K, surpassing PVTv2-b1 and Shunted-T by 3.5% and 2.4%, respectively.
  • ImageNet-22K pretraining: 88.1% top-1 accuracy is achieved by SMT-L after ImageNet-22K pretraining, exceeding InternImage-XL by 0.1% with 80.5M versus 335M parameters and 54.6G versus 163G FLOPs.
  • Object detection and instance segmentation: 5.6 and 4.2 box mAP gains over Swin Transformer are reported for small and base SMT models under the 1× Mask R-CNN schedule.
  • Semantic segmentation: 1.5 and 0.9 mIoU gains over Swin-B are reported for SMT-B at single- and multi-scale ADE20K evaluation, with over 50% lower model size.
  • Ablation study: The proposed SAA outperforms alternative aggregation modules while using few parameters and low computational cost, with larger benefits as model size increases.
  • Ablation study: 0.3% higher performance is achieved by sequentially stacking one scale-aware modulation block and one multi-head self-attention block compared with the alternative hybrid strategy.

5. Conclusion

The paper introduces SMT, a hybrid ConvNet and vision Transformer backbone that combines scale-aware modulation with an evolutionary hybrid network. Experiments show efficacy across diverse downstream vision tasks with fewer parameters and FLOPs than comparable models.

  • SMT combines Multi-Head Mixed Convolution and lightweight Scale-Aware Aggregation within Scale-Aware Modulation to integrate multi-scale contexts and adaptively modulate tokens.
  • The evolutionary hybrid network models the transition from local to global dependencies as network depth increases.
  • SMT achieves comparable or better performance than well-designed ConvNets and vision Transformers across downstream tasks with fewer parameters and FLOPs.

A. Detailed Architecture Specifications

SMT uses a four-stage architecture with convolutional and attention-based components arranged to progress from local to global modeling. Its architecture specifications define the stem, convolution heads, attention heads, and aggregation expansion ratio.

  • Stem: The stem uses two stride-2 convolution layers to produce a token sequence with length H/4 × W/4.
  • Scale-aware modulation: The multi-head mixed convolution module with four heads uses 3 × 3, 5 × 5, 7 × 7, and 9 × 9 convolutions.
  • Scale-aware modulation: The architecture denotes multi-head self-attention with eight heads and scale-aware aggregation with twice the expansion ratio.

B. Detailed Experimental Settings

The experiments use standardized training configurations for classification, detection, instance segmentation, and semantic segmentation, with additional comparisons of evolutionary hybrid stacking. Figure 8 visualizes penultimate-stage modulation for variants with and without EHN.

  • Image classification: ImageNet-1K models are trained for 300 epochs at 224 × 224 resolution with AdamW, weight decay 0.05, and standardized augmentation and regularization.
  • Image classification: SMT-L pretraining uses 90 epochs with batch size 4096, followed by 30 epochs of ImageNet-1K fine-tuning.
  • Object detection and instance segmentation: COCO experiments use Mask R-CNN, Cascade Mask R-CNN, RetinaNet, Sparse R-CNN, ATSS, and DINO with ImageNet-1K-pretrained backbones.
  • Evolutionary hybrid network: Figure 8 contrasts penultimate-stage modulation from stacking SAM blocks exclusively with an evolutionary hybrid strategy stacking one SAM block and one MSA block successively.

C. More Experiments

The experiments scale SMT across model sizes and show strong ImageNet-1K performance at both compact and large configurations.

  • These results demonstrate that SMT scales to models of varying sizes for ImageNet-1K classification.
  • SMT-M achieves 78.4% top-1 accuracy with 6.5M parameters and 1.3 GFLOPs.
  • SMT-L reaches 84.6% top-1 accuracy while outperforming similarly sized and computationally comparable state-of-the-art networks.

D. Additional Network Analysis

Additional analysis examines how SMT’s modulation evolves across layers and reports its computational and accuracy characteristics through backbone comparisons.

  • Evolutionary SMT and general SMT are compared using learned scale-aware modulation value maps in SMT-T variants.
  • Table 14 compares different backbones on ImageNet-1K classification.
  • General SMT emphasizes local details in shallow layers and semantic information in deeper layers.
  • The layer-specific design aligns each block with its computational characteristics, leading to enhanced performance across diverse visual tasks.

E. Additional Visual Examples

Supplementary visualizations show how scale-aware modulation responds to image content across different stages.

  • Scale-aware modulation strengthens semantically relevant low-frequency signals and localizes the most discriminative image regions.
  • Figure 9 visualizes modulation value maps at the top three stages.
Loading 2307.08579v2…