PaperlayerView exact revision on arXiv

Source-linked AI summary

OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels

Meng Lou, Yizhou Yu

arXiv:2502.20087v3cs.CV

TL;DR

Modern ConvNets lack explicit top-down semantic guidance because their hierarchical processing proceeds only from lower- to higher-level features. OverLoCK addresses this gap with a three-branch DDS architecture and ContMix dynamic convolution, achieving superior results across classification, detection, and segmentation benchmarks.

  • Problem

    Modern hierarchical ConvNets lack explicit top-down semantic guidance, while existing top-down-attention methods can be incompatible with modern backbones or computationally costly recurrent designs.

  • Method

    OverLoCK combines Base-Net, Overview-Net, and Focus-Net through DDS, and uses ContMix to inject global context into spatially varying convolution kernels.

  • Results

    OverLoCK delivers leading performance across vision tasks; for example, OverLoCK-T reaches 84.2% Top-1 accuracy and OverLoCK-S improves semantic-segmentation mIoU over MogaNet-B by 1.2%.

  • Takeaways & Limitations

    The results support a pure ConvNet backbone that combines top-down semantic guidance with long-range dependency modeling and strong local inductive biases.

Abstract

from arXiv · show

Top-down attention plays a crucial role in the human vision system, wherein the brain initially obtains a rough overview of a scene to discover salient cues (i.e., overview first), followed by a more careful finer-grained examination (i.e., look closely next). However, modern ConvNets remain confined to a pyramid structure that successively downsamples the feature map for receptive field expansion, neglecting this crucial biomimetic principle. We present OverLoCK, the first pure ConvNet backbone architecture that explicitly incorporates a top-down attention mechanism. Unlike pyramid backbone networks, our design features a branched architecture with three synergistic sub-networks: 1) a Base-Net that encodes low/mid-level features; 2) a lightweight Overview-Net that generates dynamic top-down attention through coarse global context modeling (i.e., overview first); and 3) a robust Focus-Net that performs finer-grained perception guided by top-down attention (i.e., look closely next). To fully unleash the power of top-down attention, we further propose a novel context-mixing dynamic convolution (ContMix) that effectively models long-range dependencies while preserving inherent local inductive biases even when the input resolution increases, addressing critical limitations in existing convolutions. Our OverLoCK exhibits a notable performance improvement over existing methods. For instance, OverLoCK-T achieves a Top-1 accuracy of 84.2%, significantly surpassing ConvNeXt-B while using only around one-third of the FLOPs/parameters. On object detection, our OverLoCK-S clearly surpasses MogaNet-B by 1% in AP^b. On semantic segmentation, our OverLoCK-T remarkably improves UniRepLKNet-T by 1.7% in mIoU. Code is publicly available at https://github.com/LMMMEng/OverLoCK.

1. Introduction

OverLoCK addresses the absence of explicit top-down semantic guidance in conventional hierarchical backbones with a branched ConvNet design and context-mixing dynamic convolution. It reports stronger vision-task performance while modeling long-range dependencies and preserving local inductive biases.

  • Existing top-down-attention models are often incompatible with modern backbones or rely on recurrent operations that add computational overhead.
  • Top-down attention uses feedback signals to guide perception toward meaningful regions, unlike conventional hierarchical backbones that rely only on preceding layers.
  • DDS decomposes OverLoCK into Base-Net, Overview-Net, and Focus-Net, progressing from low/mid-level encoding to coarse context formation and guided fine-grained representation.
  • ContMix generates spatially varying dynamic kernels from affinities between input tokens and context-region centers, allowing kernels to carry global information.
  • 84.2% Top-1 accuracy is achieved by OverLoCK-T on ImageNet-1K, outperforming UniRepLKNet-T by 1% and VMamba-T by 1.6%.
  • OverLoCK-S surpasses MogaNet-B by 1.2% in mIoU on semantic segmentation and PeLK-S by 1.4% in APb on object detection.

2. Related Work

Related work spans ConvNet evolution, dynamic convolutions, and biomimetic vision models. These lines of research motivate OverLoCK’s combination of convolutional representation, input-dependent kernels, and top-down attention.

  • ConvNets evolved from stacked small kernels and skip-connected architectures toward increasingly large kernels that mimic self-attention.
  • Dynamic convolutions improve ConvNets with input-dependent filters, including spatially varying weights for individual feature-map pixels.
  • Biomimetic vision models draw on mechanisms such as peripheral perception and top-down attention to develop vision backbones and task-specific methods.

3. Methodology

OverLoCK’s Deep-stage Decomposition Strategy splits processing into Base-Net, Overview-Net, and Focus-Net, using a coarse context prior to guide detailed refinement. Its ContMix module converts global context relationships into spatially varying dynamic kernels for long-range modeling with local inductive biases.

  • Deep-stage Decomposition: DDS uses Base-Net for mid-level features, Overview-Net for a lightweight semantic overview, and Focus-Net for guided high-level refinement.The Overview-Net output becomes a context prior fused into Focus-Net, which progressively refines features while enlarging the receptive field.
  • Deep-stage Decomposition: During ImageNet-1K pre-training, Overview-Net and Focus-Net each use a classifier head with the same classification loss.Auxiliary supervision for Overview-Net is removed during downstream transfer because it has already learned high-level representations and would make dense prediction training time-consuming.
  • Network Building Blocks: Focus-Net uses Dynamic Blocks containing residual 3×3 DWConv, GDSA, and ConvFFN, with ContMix serving as GDSA’s token mixer.A gated mechanism is additionally used to eliminate contextual noise.
  • Context Flow: Context flow fuses the context prior and feature map at each Focus-Net block, providing feature-level and weight-level guidance while updating the prior during forward propagation.Feature-level guidance uses a dynamic gate, while weight-level guidance injects the context prior into dynamic convolutions.
  • Dynamic Convolution with Context-Mixing: ContMix transforms feature tokens and pooled region centers into grouped affinity matrices, then maps affinity rows into input-dependent spatially varying K × K kernels.The kernels are normalized with softmax, and channels within each group share the corresponding dynamic kernel; the implementation sets S to 7 for linear-time complexity.
  • Network Architecture: OverLoCK provides four variants—XT, T, S, and B—whose model sizes are controlled by channels, blocks, kernel sizes, and groups.The configurations determine the scale and capacity of the Base-Net, Overview-Net, and Focus-Net components.

4. Experiments

Experiments evaluate OverLoCK across classification, detection, segmentation, and ablations, showing strong accuracy and downstream-task performance alongside analyses of its components and receptive-field behavior.

  • Evaluation setup: OverLoCK is evaluated on image classification, object detection, instance segmentation, and semantic segmentation using established datasets and comparable training settings.Experiments use ImageNet-1K, COCO 2017, and ADE20K with standard backbone pretraining and task-specific frameworks.
  • Image classification: 84.2% Top-1 accuracy is achieved by OverLoCK-T, improving upon MogaNet-S and PeLK-T by 0.8% and 1.6%, respectively.OverLoCK-S improves upon BiFormer-B and UniRepLKNet-S by 0.5% and 0.9% in Top-1 accuracy, while OverLoCK-B reaches 85.1%.
  • Object detection and instance segmentation: OverLoCK-S surpasses BiFormer-B and MogaNet-B by 0.8% and 1.5% in APb with Mask R-CNN under the 1× schedule.With Cascade Mask R-CNN, it improves upon PeLK-S and UniRepLKNet-S by 1.4% and 0.6% in APb.
  • Analysis: OverLoCK captures long-range dependencies at large resolutions while retaining strong local inductive bias, supporting its performance on downstream vision tasks.The paper reports larger effective receptive fields and more reasonable feature responses than competing methods.
  • Semantic segmentation: OverLoCK-T outperforms MogaNet-S and UniRepLKNet-T by 1.1% and 1.7% in mIoU on ADE20K semantic segmentation.It also surpasses VMamba-T by 2.3% in mIoU, with the advantage observed in both Small and Base models.
  • Ablation studies: The ablation roadmap improves from PlainNet at 76.3%/38.8% Top-1/mIoU to the Baseline at 78.5%/41.1%, then to the DDS Model at 79.0%/41.6%.Adding dynamic blocks, auxiliary supervision, initial context priors, and gating ultimately yields 80.8%/43.8% for OverLoCK-XT.

5. Conclusion

The paper concludes with DDS and ContMix as complementary components for a pure ConvNet backbone that injects semantic context and models long-range dependencies while preserving inductive biases.

  • Contributions: DDS injects semantically meaningful contexts into intermediate network stages through a biomimetic decomposition mechanism.The mechanism is designed to provide top-down semantic guidance within the backbone.
  • Contributions: ContMix captures long-range dependencies while preserving strong inductive biases in dynamic convolution.Together, DDS and ContMix form the OverLoCK pure ConvNet vision backbone.
  • Conclusion: OverLoCK achieves clearly superior performance compared with strong baselines.The conclusion attributes the backbone’s results to integrating DDS with context-mixing dynamic convolution.

A. More Ablation Studies

Ablations show that top-down guidance, dynamic convolution, kernel sizing, stage allocation, channel reduction, and auxiliary supervision each affect OverLoCK performance. The DDS-based design and ContMix both contribute measurable advantages over hierarchical or static alternatives.

  • Impact of Kernel Sizes: The kernel configuration {[17, 15, 13], [7], [13] [7]} achieves the best performance on both image classification and semantic segmentation.Further enlarging the kernels provides no additional improvement.
  • Impact of Stage Ratio: A 1:2 stage ratio allocates more blocks to Focus-Net for extracting robust contextual information.The study compares this default with 1:1 and 1:3 settings.
  • Impact of Channel Reduction Factor: The Channel Reduction Factor controls context-prior channels and therefore influences Overview-Net’s guidance capability.The default uses a 1×1 convolution to reduce Overview-Net output channels by a factor of 4 before concatenation with Base-Net output.
  • Impact of Auxiliary Loss: Auxiliary loss improves classification accuracy when applied to Overview-Net.The experiments vary its weight while keeping the compared architectures consistent.
  • DDS and ContMix Ablations: Removing Overview-Net causes a noticeable performance drop, while the resulting hierarchical model still outperforms the fully static baseline.This supports contributions from both DDS-based top-down guidance and dynamic convolution.
  • DDS and ContMix Ablations: Replacing the designated inputs for ContMix affinity computation with fused features produces a marginal performance decline.The comparison evaluates alternative components within the ContMix framework.

B.1. Large Resolution Evaluation

At 384×384 input resolution, OverLoCK-B maintains strong classification performance after fine-tuning from 224×224 pretraining. It surpasses MaxViT-B while using substantially fewer parameters and improves over PeLK-B.

  • High-Resolution Classification: OverLoCK-B surpasses MaxViT-B by 0.5% in Top-1 accuracy at 384×384 resolution.The model is pretrained at 224×224 and fine-tuned at 384×384 for 30 epochs.
  • High-Resolution Classification: OverLoCK-B reduces parameter count by over one-third compared with MaxViT-B under high-resolution input conditions.The passage also reports significant improvements over PeLK-B.

B.2. Robustness Evaluation

OverLoCK demonstrates robustness across ImageNet out-of-distribution benchmarks, outperforming representative ConvNets, Vision Transformers, and Vision Mamba models. Its gains over MogaNet-L are especially pronounced on OOD datasets.

  • ImageNet OOD Robustness: OverLoCK-B improves over MogaNet-L by 0.4% in Top-1 accuracy on ImageNet-1K.This in-distribution gain is accompanied by larger improvements across four OOD benchmarks.
  • ImageNet OOD Robustness: OverLoCK-B gains 1.4% on ImageNet-V2, 6.7% on ImageNet-A, 6.3% on ImageNet-R, and 6.8% on ImageNet-Sketch over MogaNet-L.The evaluation covers ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch.

C. Speed Analysis

OverLoCK offers favorable speed–accuracy trade-offs across comparisons with ConvNets, Transformers, and Mamba-based models. It can achieve higher accuracy and throughput than larger or competing architectures.

  • Throughput and Accuracy: OverLoCK-T improves throughput by about 200 imgs/s over ConvNeXt-B, achieves better performance, and uses around one-third of its FLOPs.The comparison emphasizes simultaneous speed and accuracy improvements.
  • Throughput and Accuracy: OverLoCK-S surpasses MogaNet-B by over 100 imgs/s while increasing Top-1 accuracy from 84.3% to 84.8%.The reported comparison combines a throughput advantage with a 0.5 percentage-point accuracy increase.
  • Context-Guided Perception: Overview-Net first localizes objects coarsely, after which Focus-Net produces more accurate object locations and shapes under top-down guidance.The behavior is visualized with class activation maps.

D.2. Effective Receptive Field Analysis

OverLoCK-T’s Effective Receptive Field is evaluated against similarly complex models using ImageNet-1K validation images. The visualization indicates both global responses and strong local sensitivity.

  • The analysis compares OverLoCK-T’s Effective Receptive Field with representative models of comparable complexity.The visualizations use over 300 randomly sampled 224×224 images from the ImageNet-1K validation set.
  • OverLoCK-T produces global responses in the Effective Receptive Field visualization.
  • OverLoCK-T also exhibits significant local sensitivity in the Effective Receptive Field visualization.
Loading 2502.20087v3…