Focal Modulation Networks

Jianwei Yang; Chunyuan Li; Xiyang Dai; Lu Yuan; Jianfeng Gao

Source-linked AI summary

Focal Modulation Networks

Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao

arXiv:2203.11926v3Mar 22, 2022cs.CVcs.AIcs.LG

TL;DR

Self-attention provides input-dependent long-range interactions but requires heavy query-key interactions and query-value aggregation. The paper replaces it with Focal Modulation, which hierarchically contextualizes features, selectively aggregates contexts, and modulates each query. FocalNets outperform state-of-the-art self-attention counterparts across classification, detection, and segmentation at similar computational costs.

Problem
The paper asks whether input-dependent long-range interactions can be modeled more efficiently than self-attention's heavy query-key interactions and aggregations.
Method
Focal Modulation hierarchically gathers short- to long-range contexts, uses gated aggregation to form a query-dependent modulator, and injects it through element-wise modulation.
Results
FocalNets consistently outperform state-of-the-art self-attention counterparts on image classification, object detection, and segmentation with comparable computational costs.
Takeaways & Limitations
Focal Modulation provides an attention-free mechanism for visual token interactions that transfers effectively across classification, detection, and segmentation tasks.
Takeaways & Limitations
The paper leaves application to NLP and cross-modulation for multimodal learning as areas requiring further study.

Abstract

from arXiv · show

We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation mechanism for modeling token interactions in vision. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (ii) gated aggregation to selectively gather contexts for each query token based on its content, and (iii) element-wise modulation or affine transformation to inject the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational costs on the tasks of image classification, object detection, and segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K in 224 resolution, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 and 384, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1\times outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3\times schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and Mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. Using huge FocalNet and DINO, we achieved 64.3 and 64.4 mAP on COCO minival and test-dev, respectively, establishing new SoTA on top of much larger attention-based models like Swinv2-G and BEIT-3. Code and checkpoints are available at https://github.com/microsoft/FocalNet.

1 Introduction

The paper asks whether input-dependent long-range token interactions can be modeled more efficiently than self-attention, and proposes Focal Modulation as an alternative. FocalNets replace self-attention and outperform comparable-cost attention-based counterparts across vision tasks.

Motivation and contribution: Focal Modulation decouples context aggregation from individual queries, making token interactions lightweight while retaining input-dependent interactions.It replaces self-attention in FocalNets.
Empirical results: FocalNets consistently outperform state-of-the-art self-attention counterparts on image classification, object detection, and segmentation at comparable costs.The introduction reports gains across all three task families.
Empirical results: 82.3% and 83.9% top-1 accuracy are achieved by tiny and base FocalNets on ImageNet-1K, respectively.With ImageNet-22K pretraining, the reported accuracies are 86.5% at 224 resolution and 87.3% at 384 resolution.
Empirical results: FocalNet base reaches 49.0 box mAP with Mask R-CNN under a 1× schedule, exceeding Swin's 48.5 box mAP under a 3× schedule.The comparison is reported for COCO object detection.

2 Related Work

The related work situates visual token interaction across self-attention, MLP architectures, and convolutional networks. It also frames Focal Modulation alongside prior attention designs through a side-by-side comparison with self-attention.

Self-attention: Self-attention enabled input-dependent token interactions in vision and supported applications including classification, detection, and segmentation.The cited figure comparison places self-attention and Focal Modulation side by side.
MLP architectures: Visual MLPs include global-mixing methods using spatial projections and local-mixing methods using nearby-token operations.Mix-Shift-MLP combines local and global interactions in a manner related to focal attention.
Convolutions: ConvNet research includes efficiency-focused architectures and methods that incorporate global context or dynamic operations.Examples include MobileNet, ShuffleNet, EfficientNet, non-local networks, and Involution.

3 Focal Modulation Network

Focal Modulation replaces self-attention with shared hierarchical context aggregation, content-dependent gating, and lightweight query modulation. Its design captures multiple context ranges while reducing the expensive query-token interactions of self-attention.

From Self-Attention to Focal Modulation: Focal Modulation aggregates contexts before query interaction, amortizing computation through a shared operator instead of query-specific attention scores.The query then interacts with the aggregated feature using a lightweight operator rather than token-to-token attention.
From Self-Attention to Focal Modulation: The mechanism applies element-wise modulation between a projected query and a context-derived modulator.The context aggregation function m(·) produces the modulator used in the query transformation.
Hierarchical Contextualization: Hierarchical contextualization uses stacked depth-wise convolutions to produce feature maps with progressively broader receptive fields.Depth-wise convolution is learnable, structure-aware, and cheaper than regular convolution; global average pooling adds whole-input context.
Gated Aggregation: Gated aggregation uses spatial- and level-aware weights to select how much context each query gathers from different focal levels.Small objects can favor fine-grained low-level context, while uniform backgrounds can use broader higher-level context.
Modulator Visualization: The resulting modulator exhibits object-focused responses, suggesting that FocalNets localize discriminative regions without class guidance.The paper presents this emergent localization as an interpretive property of the proposed mechanism.

4 Experiment

FocalNets consistently outperform self-attention counterparts across image classification, detection, and segmentation at comparable computational costs. Ablations indicate that hierarchical multi-scale aggregation, content-dependent gating, global context, and multiplicative modulation each contribute to performance.

Image Classification: 0.9% higher Top-1 accuracy than Swin-Tiny, while FocalNets outperform Focal Transformers across model sizes and run faster.The comparisons use similar model size, FLOPs, and throughput for Swin; FocalNets avoid time-consuming operations such as rolling and unfolding.
Detection and Segmentation: 2.2, 1.5, and 1.9 box-mAP gains over Swin are obtained by tiny, small, and base FocalNets under a 1× schedule.FocalNet-T/B with 1× training reach 45.9/48.8, rivaling Swin-T/B at 46.0/48.5 trained with 3× schedules.
Detection and Segmentation: 2.1 and 1.4 mIoU improvements over Swin are achieved by base FocalNets with single-scale and multi-scale UPerNet segmentation.FocalNets also outperform Focal Transformers while using substantially fewer FLOPs.
Network Inspection: 6.7% and 1.8% accuracy drops result from using only global pooling or replacing depth-wise convolution with average pooling.Multi-scale or sliding-window self-attention variants also increase latency or cost more while performing worse than FocalNet.
Network Inspection: 0.4% accuracy drops follow removal of lower focal levels or gating, while global pooling removal causes a 0.3% drop.Increasing focal levels eventually plateaus, but hierarchical aggregation remains important because a single larger receptive field underperforms multi-level aggregation by 0.4%.

5 Conclusion

The paper introduces Focal Modulation as an attention-free mechanism for input-dependent visual token interactions and builds FocalNets around it. Experiments show strong performance across vision tasks, while cross-domain and multimodal extensions remain open.

Focal Modulation gathers short- to long-range contexts hierarchically, adaptively aggregates them into a modulator, and applies element-wise modulation to each query.
FocalNets replace self-attention and consistently outperform state-of-the-art attention counterparts with similar time and memory costs across classification, detection, and semantic segmentation.
The method achieves new state-of-the-art COCO object-detection performance with fewer parameters and less pretraining data than prior work.
Applying Focal Modulation to other domains such as NLP and to cross-modulation for multimodal learning requires further study.

A More Implementation Details

The implementation constructs FocalNet variants by matching Swin’s stage layouts and hidden dimensions while replacing self-attention with Focal Modulation. Model configurations vary mainly in focal levels and starting kernel size, which determine receptive fields.

FocalNet variants use the same stage layouts and hidden dimensions as Swin, replacing its self-attention modules with Focal Modulation modules.
FocalNet configurations differ in focal levels and starting kernel size, while SRF and LRF denote small and large receptive fields.
The configuration table reports the effective receptive field at the top focal level for each stage.

A.2 Training settings for ImageNet-1K

The ImageNet training settings specify shared augmentation, regularization, optimization, learning-rate, weight-decay, and gradient-clipping choices, with the accompanying tables organizing the experimental protocols.

Training follows Swin’s augmentations, including Random Augmentation, Mixup, CutMix, and Random Erasing.
Label Smoothing and DropPath provide model regularization during training.
Optimization uses AdamW with cosine learning-rate scheduling, a 10^-3 initial learning rate after warmup, weight decay 0.05, and gradient clipping norm 5.0.
The appendix separately tabulates ImageNet-1K training settings and ImageNet-22K pretraining followed by ImageNet-1K finetuning.

A.3 Training settings for ImageNet-22K

ImageNet-22K pretraining uses FocalNet-B and FocalNet-L at 224 × 224, followed by ImageNet-1K finetuning with specified optimization and stability settings.

FocalNet-B and FocalNet-L are pretrained for 90 epochs at 224 × 224 with batch size 4096.
Pretraining uses a 10^-3 initial learning rate after five warmup epochs and stochastic-depth drop rates of 0.2.
LayerScale with initial value 10^-4 is applied to all layers for training stability.
After pretraining, the models are finetuned on ImageNet-1K for 30 epochs with initial learning rate 3 × 10^-5.

A.4 Training settings for Object365

FocalNets are pretrained on Object365 using the DINO settings, with a large-scale training set and standard high-resolution inputs.

FocalNets+DINO are pretrained on Object365, which contains around 1.7M training images and 80k validation images.The authors merge 75k validation images into training and retain the remainder for evaluation.
Pretraining runs for 26 epochs with learning rate 1e−4, reduced tenfold after epoch 24.
The pretraining uses a standard image resolution of 800 × 1333.

B Downstream Tasks

The downstream-task study examines kernel-size effects for FocalNet-T on Mask R-CNN, balancing detection performance against training cost.

Effect of kernel size: Moderate first-level kernels, from 5 through 11, slightly outperform kernels that are too small or too large for object detection.Kernel size 3 may miss long-range dependencies, whereas sizes 13 and 15 may miss detailed local context.
Effect of kernel size: Figure 8 varies the first-level kernel size from 3 to 15 and reports box and mask mAP for Mask R-CNN 1× training.
Effect of kernel size: Increasing the kernel size gradually increases Mask R-CNN training memory and time.The measurements use 16 V100 GPUs with batch size 16.
Effect of kernel size: The authors set the first-level kernel size to kℓ=1 = 9 for object-detection finetuning as a performance-cost trade-off.

B.1.2 Results with deeper and thinner FocalNets

Additional experiments study deeper, thinner FocalNets for detection and segmentation, alongside comparisons with multi-scale contextual architectures.

Semantic segmentation: For FocalNet-T semantic segmentation, increasing depth does not bring extra improvement.
Semantic segmentation: Deeper versions outperform shallow versions for larger models, particularly FocalNet-B, in semantic segmentation.
Additional backbone results: Deeper, thinner FocalNets are evaluated as alternative backbones for Mask R-CNN 1× and semantic segmentation.The additional Mask R-CNN results are marked in gray rows, while semantic-segmentation results use UPerNet on ADE20K.
Kernel visualization: Figures 10 and 11 visualize learned kernels at three focal levels and four stages for FocalNet-T and FocalNet-B.Only the last layer of each stage is shown.
Comparison with MPViT: FocalNets achieve better performance than MPViT at comparable cost in semantic segmentation.The authors associate this comparison with hierarchical and gated contextualization covering large receptive fields for high-resolution dense prediction.

C Additional Model Interpretation

The interpretation analysis examines learned convolutional kernels, gating values, and modulators, while noting a training-data bias concern for real-world deployment.

Convolutional kernel patterns: Earlier stages emphasize local regions, whereas later stages use more global context for final prediction.The analysis links this progression to increasingly scattered weights at higher focal levels.
Gating function: Gating values emphasize object textures at level 1, object boundaries at level 2, and whole objects at level 3.
Gating function: At level 4, foreground regions show less interest in global context, while background regions show more.Across the first three levels and the global context, foreground tokens gather more local information and background tokens more global information.
Modulator interpretation: FocalNet modulators localize discriminative image regions without class guidance from CAM or Grad-CAM.The paper presents this as an interpretability property of the modulation mechanism.
Social impact: The work warns that large-scale webly-crawled training data may amplify offensive or biased content in models and predictions.The authors recommend sanity checks before training and deployment in realistic applications.