PaperlayerView exact revision on arXiv

Source-linked AI summary

MogaNet: Multi-order Gated Aggregation Network

Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, Stan Z. Li

arXiv:2211.03295v4cs.CVcs.AI

TL;DR

Modern ConvNets can be efficient, but multi-order analysis indicates that they underuse expressive interactions, creating a representation bottleneck. MogaNet introduces compact gated spatial and channel aggregation to adaptively encode these interactions, achieving competitive performance and efficiency across vision benchmarks.

  • Problem

    Modern ConvNets do not effectively encode expressive multi-order interactions, despite increased kernel sizes.

  • Method

    MogaNet uses compact spatial and channel aggregation modules that combine locality perception, gated context aggregation, and adaptive channel-wise reallocation.

  • Results

    MogaNet shows competitive performance and parameter efficiency across image classification, detection, segmentation, and other vision tasks, including 80.0% ImageNet-1K accuracy for MogaNet-T.

  • Takeaways & Limitations

    MogaNet provides a scalable pure ConvNet design with favorable accuracy–computational efficiency trade-offs across vision benchmarks.

Abstract

from arXiv · show

By contextualizing the kernel as global as possible, Modern ConvNets have shown great potential in computer vision tasks. However, recent progress on multi-order game-theoretic interaction within deep neural networks (DNNs) reveals the representation bottleneck of modern ConvNets, where the expressive interactions have not been effectively encoded with the increased kernel size. To tackle this challenge, we propose a new family of modern ConvNets, dubbed MogaNet, for discriminative visual representation learning in pure ConvNet-based models with favorable complexity-performance trade-offs. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adaptively. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks, including COCO object detection, ADE20K semantic segmentation, 2D&3D human pose estimation, and video prediction. Notably, MogaNet hits 80.0% and 87.8% accuracy with 5.2M and 181M parameters on ImageNet-1K, outperforming ParC-Net and ConvNeXt-L, while saving 59% FLOPs and 17M parameters, respectively. The source code is available at https://github.com/Westlake-AI/MogaNet.

1 INTRODUCTION

Modern ConvNets offer efficient alternatives to ViTs, but their representations may underuse expressive multi-order interactions. MogaNet addresses this with gated spatial and channel aggregation, achieving competitive performance across scales and vision tasks.

  • ViTs provide global-range feature interaction, but quadratic self-attention limits efficiency and high-resolution applications.
  • Modern ConvNets remain competitive with well-tuned ViTs across a wide range of vision benchmarks.
  • Multi-order analysis indicates that modern ConvNets underexploit representation capacity, especially expressive interactions beyond simple local concepts.
  • MogaNet combines locality perception, gated context aggregation, and adaptive channel-wise reallocation in compact spatial and channel aggregation blocks.
  • 80.0% top-1 accuracy is achieved by MogaNet-T on ImageNet-1K, outperforming ParC-Net-S by 1.0% with 2.04G lower FLOPs.
  • 2.3% APb separates MogaNet from Swin-L on COCO detection, with fewer parameters and lower computational budget.

2 RELATED WORK

The related work traces a shift from ViTs toward efficient token-mixing and modern ConvNets. MogaNet follows this framework while separating spatial and channel propagation and modeling context aggregation adaptively.

  • ViTs achieve strong vision results but are described as over-parameterized and dependent on large-scale pre-training compared with ConvNets.
  • MogaNet adopts a hierarchical four-stage architecture whose stages contain embedding stems and Moga Blocks with spatial and channel aggregation blocks.
  • Vision architectures differ in token-mixing strategies including relative position encoding, local window shifting, and MLP layers.
  • Modern ConvNets combine an embedding stem, spatial mixing block, and channel mixing block, with the stem reducing resolution and redundancy.
  • Stage modules are decoupled into SMixer and CMixer for spatial-wise and channel-wise information propagation.
  • Context aggregation adaptively weights positions through an aggregation branch and reweights embedded features from a context branch.

3 MULTI-ORDER GAME-THEORETIC INTERACTION FOR DEEP ARCHITECTURE DESIGN

Multi-order interaction analysis characterizes how DNNs encode relationships across context scales. Existing architectures favor low- or high-order interactions while suppressing expressive middle-order interactions, motivating architecture design around this imbalance.

  • I(m)(i, j) measures average interaction complexity between patch pairs across contexts containing m patches.
  • The interaction order m represents the scale of context involved in game-theoretic interactions between image patches.
  • Low-order interactions tend to encode common local textures, whereas high-order interactions tend to memorize rare outlier patterns.
  • Existing DNNs favor excessively low- or high-order interactions while suppressing more expressive and versatile middle-order interactions.
  • ViTs may encode high-order interactions through global self-attention but lack middle-order interactions because locality priors are absent.

4 METHODOLOGY

MogaNet combines multi-order spatial gated aggregation with channel reallocation in a scalable pure ConvNet architecture. Its modules adaptively extract and contextualize discriminative features while maintaining favorable computational costs.

  • Architecture: MogaNet uses a four-stage architecture with embedding stems, Moga Blocks, and task-specific output heads for classification or dense prediction.Stages operate at progressively reduced resolutions, and each Moga Block contains spatial and channel aggregation blocks.
  • Spatial aggregation: The spatial aggregation block decomposes features, applies parallel depth-wise convolutions with dilation ratios 1, 2, and 3, and gates their aggregated multi-order contexts.These branches target low-, middle-, and high-order interactions, while feature decomposition reweights complementary interactions.
  • Spatial aggregation: MogaNet captures more middle-order interactions with spatial aggregation blocks at similar parameters and FLOPs to ConvNeXt's DW7×7 convolution.This design avoids the cost of self-attention while producing discriminative multi-order representations.
  • Channel aggregation: The channel aggregation block reduces and reallocates channel-wise information, addressing redundancy that makes vanilla channel MLPs parameter-inefficient.The channel projection and GELU-based reallocation target complementary channel interactions.
  • Channel aggregation: With MLP ratio r = 4, channel aggregation improves the baseline by 0.6% with 0.04M extra parameters and 0.01G extra FLOPs.It matches the baseline performance obtained with r = 8 while using a smaller expansion ratio.
  • Scalability: MogaNet scales across six model sizes by stacking different numbers of spatial and channel aggregation blocks at each stage.The variants are designed with parameter counts similar to RegNet variants.

5 EXPERIMENTS

MogaNet is evaluated across ImageNet classification and downstream detection, segmentation, pose, and video-prediction tasks, showing competitive accuracy with efficient parameter and computation use. Ablations and activation maps support the contribution of its spatial and channel aggregation modules.

  • Experimental scope: MogaNet is evaluated on image classification, object detection, instance and semantic segmentation, pose estimation, and video prediction.Experiments use ImageNet, COCO, ADE20K, and MMNIST benchmarks.
  • ImageNet classification: 80.0% top-1 accuracy is achieved by MogaNet-T on ImageNet-1K at 2562 resolution, outperforming ParC-Net-S by 1.0 point.The result uses refined training settings.
  • ImageNet classification: 83.4% top-1 accuracy is achieved by MogaNet-S, exceeding Swin-T and ConvNeXt-T by 2.1 and 1.2 points, respectively.MogaNet-XL reaches 87.8% after ImageNet-21K pre-training and ImageNet-1K fine-tuning.
  • Dense prediction tasks: 42.6 APb is achieved by Mask R-CNN with MogaNet-T, exceeding Swin-T by 0.4 with 48% fewer parameters and 27% fewer FLOPs.Cascade Mask R-CNN with MogaNet-XL reaches 56.2 APb, exceeding ConvNeXt-L and RepLKNet-31L.
  • Pose and video tasks: MogaNet variants improve 2D pose estimation by at least 0.9 AP and improve SimVP video prediction by 6.58 MSE over its baseline.The evaluations cover COCO pose estimation and MMNIST video prediction.
  • Ablation and analysis: Removing proposed spatial or channel modules reduces performance, while interaction distributions and Grad-CAM maps support learned middle-order interactions and semantically gathered activations.The ablation evidence attributes contributions to the Moga and CA modules.

6 CONCLUSION

The paper concludes that MogaNet uses multi-order interaction to build efficient modern ConvNets with strong accuracy across model scales and vision benchmarks. Its architecture combines compact spatial and channel aggregation modules across configurable four-stage variants.

  • Conclusion: MogaNet introduces a compact Moga Block and channel aggregation module to emphasize overlooked interactions across spatial and channel spaces.The design is built upon the modern ConvNet framework.
  • Conclusion: MogaNet demonstrates accuracy and computational efficiency compared with popular ConvNets, ViTs, and hybrid architectures across vision benchmarks.The conclusion presents multi-order interaction as a useful perspective for representation learning and efficient architecture design.
  • Architecture scalability: MogaNet scales across X-Tiny, Tiny, Small, Base, Large, and X-Large variants by changing the numbers of spatial and channel aggregation blocks at each stage.The variants use architecture configurations detailed in Table A1.
  • Architecture scalability: The four-stage architecture uses hierarchical stages containing embedding stems and repeated SMixer and CMixer blocks with PreNorm and residual connections.Features retain stage-wise shapes, while CMixer expands channels through an inverted bottleneck.

A.2 EXPERIMENTAL SETTINGS FOR IMAGENET

The experiments use standard or task-specific training protocols to evaluate MogaNet on ImageNet and downstream benchmarks. ImageNet-21K pre-training is followed by ImageNet-1K fine-tuning for the largest model setting.

  • ImageNet settings: ImageNet classification experiments use OpenMixup and timm codebases on eight NVIDIA A100 GPUs.The settings follow DeiT and RSB A2 procedures for fair comparison.
  • ImageNet settings: MogaNet models are trained from scratch on ImageNet-1K for 300 epochs with AdamW, batch size 1024, learning rate 1 × 10−3, and weight decay 0.05.The default input resolution is 2242, while lightweight experiments use 2562 resolution.
  • Downstream settings: COCO detection and instance segmentation use RetinaNet, Mask R-CNN, and Cascade Mask R-CNN with ImageNet-pretrained backbones.Training uses AdamW and 1× or 3× schedules, with multi-scale training for Cascade Mask R-CNN.
  • Downstream settings: ADE20K semantic segmentation uses Semantic FPN and UperNet, while COCO 2D pose estimation uses Top-Down SimpleBaseline.The corresponding protocols use AdamW or Adam with task-specific schedules.
  • Downstream settings: Video prediction uses SimVP on Moving MNIST, where ten frames predict the next ten frames and models are trained from scratch with Adam.Evaluation uses MSE, MAE, and SSIM.

B EMPIRICAL EXPERIMENT RESULTS

The empirical analysis interprets model representations through multi-order interaction and examines MogaNet’s interaction distributions, ablations, activation maps, module choices, and computational cost. Results indicate that MogaNet’s design captures contextual interactions while retaining practical throughput.

  • Multi-order interaction: Multi-order interaction measures collaboration between image patches under contexts of different sizes, ranging from low-order local interactions to high-order global interactions.The interaction order m denotes the scale of context involved in the computation.
  • Multi-order interaction: J(m) summarizes the distribution of interaction orders encoded by a DNN, with its area under the curve normalized to one.The analysis computes J(m) on 14 × 14 patch grids for ImageNet-1K models.
  • Activation analysis: MogaNet activation maps are more semantically gathered than those of previous ConvNets and resemble attention architectures such as Swin.The comparison includes Transformer and ConvNet baselines on ImageNet-1K.
  • Module analysis: SiLU performs best for both gating and context branches, while parallel dilated depth-wise convolutions efficiently learn multi-order contextual information.The multi-order DWConv analysis uses dilation ratios 1, 2, and 3.
  • MogaNet analysis: MogaNet and its channel aggregation module contribute to learning middle-order interactions, according to the interaction distributions and module ablations.The ablations remove spatial and channel aggregation components from the MogaNet-S baseline.
  • Efficiency analysis: MogaNet has throughput similar to Swin Transformer while achieving better performance than Swin and ConvNet, although ConvNeXt is faster in part because MogaNet adds channel splitting and GAP operations.The comparison is reported across MogaNet variants on ImageNet-1K.

C.4 ABLATION OF NORMALIZATION LAYERS

MogaNet uses BatchNorm as its default normalization, while precise BN provides slight additional gains, particularly for larger models using EMA.

  • BatchNorm remains MogaNet’s default because combining convolution and BN within each stage is useful for pure ConvNets.
  • Precise BN improves performance by around 0.1%, especially for MogaNet-S/B with EMA.
  • The normalization ablation is summarized in Table A6.

C.5 REFINED TRAINING SETTINGS FOR LIGHTWEIGHT MODELS

Refined training settings improve MogaNet-XT/T on ImageNet-1K by changing the learning rate and augmentation strategy while keeping other settings unchanged.

  • +0.4∼0.6% is gained by MogaNet-XT/T with a larger learning rate and 3-Augment.
  • The refined recipe adjusts learning rate and augmentation strategies for faster convergence without changing other settings.
  • The advanced lightweight-model training recipe is reported in Table A7.
  • MogaNet at 2242 resolution outperforms DeiT-T by 1.2% with only 3M parameters.

D.1 FAST TRAINING ON IMAGENET-1K

Under 100- and 300-epoch ImageNet-1K settings, MogaNet delivers strong performance and faster convergence against similarly sized architectures.

  • The 100-epoch setting uses RSB A3, while the 300-epoch comparison uses RSB A2 for classical CNNs and original settings for Transformers or modern ConvNets.
  • MogaNet-XT/T/S/B achieves the best performance among similarly sized models under both 100- and 300-epoch training.
  • MogaNet-T surpasses EfficientNet-B0 and DeiT-T by 2.4% and 8.7%, respectively, after 100 epochs.
  • MogaNet-S outperforms Swin-T by 3.4% and ConvNeXt-T by 2.3% with similar parameters.

D.2 DETECTION AND SEGMENTATION RESULTS ON COCO

MogaNet achieves competitive or state-of-the-art detection, instance-segmentation, and semantic-segmentation results across COCO and ADE20K, including lightweight and large model scales.

  • COCO detection and instance segmentation: 45.8/47.7/48.7 APb is achieved by MogaNet-T/S/B with RetinaNet on COCO under a 1× schedule.
  • COCO detection and instance segmentation: MogaNet-T improves Swin-T by 0.4 APb and exceeds PVT-S by 1.3 APm with Mask R-CNN using around two-thirds as many parameters.
  • COCO detection and instance segmentation: MogaNet-L/XL obtain 53.3/56.2 APb and 46.1/48.8 APm with Cascade Mask R-CNN under 3× multi-scaling training.
  • ADE20K semantic segmentation: 49.2/50.1 mIoUss is achieved by MogaNet-S/B on ADE20K, exceeding HorNet-T by 1.1 and SLaK-S by 0.7.

D.4 2D HUMAN POSE ESTIMATION RESULTS ON COCO

MogaNet achieves competitive or state-of-the-art 2D human pose estimation on COCO across two input resolutions. Its larger variants outperform Swin-B/L while using nearly half the parameters and computation.

  • MogaNet variants achieve competitive or state-of-the-art performance on COCO 2D human keypoint estimation at two resolutions.The evaluation uses Top-Down SimpleBaseline.
  • 74.9/75.3 AP is achieved at 256×192, while 76.4/77.3 AP is achieved at 384×288 by MogaNet-S/B.
  • MogaNet-S/B outperform Swin-B/L by 2.0/1.0 and 1.5/1.0 AP at the two resolutions.
  • The MogaNet-S/B comparisons use nearly half the parameters and computation budgets of the corresponding Swin models.
  • MogaNet-XT/T significantly improve existing lightweight models while using similar parameters and FLOPs.
Loading 2211.03295v4…