PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear-Time Global Visual Modeling without Explicit Attention

Ruize He, Dongchen Han, Gao Huang

arXiv:2605.01711v2cs.CV

TL;DR

Transformers’ global modeling is tied to quadratic explicit attention, motivating whether dynamic parameterization can replace it at linear complexity. The paper reframes attention as an input-conditioned dynamic MLP, evaluates dynamic parameter strategies in vision models, and finds competitive global modeling with lower computational overhead.

  • Problem

    Whether dynamic parameterization can replace explicit attention while preserving global sequence modeling with linear complexity remains an open question.

  • Method

    The paper reframes attention as an input-conditioned dynamic MLP and evaluates multiple dynamic parameter prediction strategies within vision models.

  • Results

    Vision models using dynamic parameterization achieve competitive global receptive fields and representational power comparable to Vision Transformers with significantly lower computational overhead.

  • Takeaways & Limitations

    Dynamic parameters can effectively substitute explicit attention for global sequence modeling in vision models while maintaining linear complexity.

  • Takeaways & Limitations

    The evaluation is restricted to vision tasks, so generalization of dynamic parameterization to other domains remains unclear.

Abstract

from arXiv · show

Existing research largely attributes the global sequence modeling capability of Transformers to the explicit computation of attention weights, a process that inherently incurs quadratic computational complexity. In this work, we offer a novel perspective: we demonstrate that attention can be mathematically reframed as a Multi-Layer Perceptron (MLP) equipped with dynamically predicted parameters. Through this lens, we explain attention's global modeling power not as explicit token-wise aggregation, but as an implicit process where dynamically generated parameters act as a compressed representation of the global context. Inspired by this insight, we investigate a fundamental question: can we achieve Transformer-level sequence global modeling entirely through dynamic parameterization while maintaining linear complexity, effectively replacing explicit attention? To explore this, we design various dynamic parameter prediction strategies and integrate them into standard network layers. Extensive empirical studies on vision models demonstrate that dynamic parameterization can indeed serve as a highly effective, linear-complexity alternative to explicit attention, opening new pathways for efficient sequence modeling. Code is available at https://github.com/LeapLabTHU/WeightFormer.

1 Introduction

The paper reframes attention as a dynamically parameterized MLP whose generated weights implicitly compress global context, then investigates replacing explicit attention with dynamic parameterization at linear complexity. Experiments in vision models validate this approach as an effective, lower-overhead substitute for attention-based global modeling.

  • Attention as Dynamic MLP: Attention can be mathematically reframed as a dynamic MLP, with K^T and V serving as dynamically generated layer weights and Softmax as the activation.The parameters are conditioned on the input rather than static.
  • Implicit Global Modeling: Global sequence modeling emerges implicitly because dynamically predicted parameters compactly represent the input’s global context and integrate long-range dependencies without explicit attention weights.This perspective replaces explicit token-to-token routing with forwarding through a dynamically parameterized network.
  • Research Question: The central question is whether dynamic parameters can completely replace attention while preserving global sequence modeling with linear rather than quadratic complexity.The proposed direction decouples implicit global modeling from attention’s quadratic matrix multiplications.
  • Dynamic Parameter Strategies: The authors design lightweight dynamic parameter predictors, including linear and depthwise convolution layers, conditioned on global sequence context to avoid the N × N token interaction matrix.These strategies generate network weights dynamically while preserving linear computational complexity with respect to sequence length.
  • Empirical Validation: Experiments show that vision models using only dynamic parameterization achieve competitive global receptive fields and representational power comparable to Vision Transformers with significantly lower computational overhead.The study evaluates feasibility in prototype vision models rather than optimizing a new state-of-the-art architecture for specific benchmarks.

2 Related Work

Related work has largely centered on attention as the dominant mechanism for Transformer global modeling and on making attention computation more efficient. This work instead frames global modeling as dynamic parameterization, with input-conditioned parameters implicitly encoding global information without explicit attention-weight computation.

  • Attention and Global Modeling Paradigms: Attention dominates Transformer global modeling by explicitly computing pairwise token similarities to reweight value representations.
  • Attention and Global Modeling Paradigms: Prior efficiency research modifies the attention-matrix computation to reduce the cost of explicit attention.
  • Dynamic Networks and Connections to Attention: The proposed perspective interprets attention as a dynamic parameterized MLP whose input-conditioned parameters compress global information and enable implicit global modeling.
  • Dynamic Networks and Connections to Attention: Dynamic neural networks adapt parameters to inputs, with prior vision work exploring dynamic filters, conditional convolutions, weight modulation, and dynamic depthwise operations.
  • Dynamic Networks and Connections to Attention: Unlike treating attention weights as dynamic kernels, the approach identifies K and V as the core dynamic parameters of an MLP-like structure.

3 Attention as Dynamic Parameterized MLP

The section reframes attention as a dynamic parameterized MLP whose parameters are predicted from the entire input, rather than as explicit token-wise weighting. This perspective explains attention’s quadratic scaling and motivates dynamic parameter prediction as a replacement for explicit attention in global modeling.

  • Standard Attention: Standard attention explicitly computes a dense N×N affinity matrix A and uses it to recombine value representations.Aij represents the pairwise similarity of token j to token i.
  • Dynamic MLP Interpretation: Attention can instead be viewed as a dynamic parameterized MLP whose parameters are generated from the input sequence.K^⊤ and V serve as dynamically generated MLP parameters because K = XW_K and V = XW_V.
  • Dynamic MLP Interpretation: Global context is compressed into dynamically predicted parameters, enabling implicit integration of global dependencies without constructing an attention weight matrix.Each head can be conceptualized as an independent MLP operating on the global sequence, with compression from the full channel dimension to the head dimension.
  • Complexity Explanation: The dynamic MLP’s effective width scales with token count N because K^⊤ ∈ R^d×N and V ∈ R^N×d, explaining attention’s quadratic computational and memory complexity.The dynamic network therefore grows with sequence length.
  • Beyond Attention: Explicit attention weights are not essential for global modeling, which can instead be achieved through dynamic parameter prediction and implicit computation.This perspective motivates a broader design space for efficient architectures.
  • Beyond Attention: The authors test dynamic parameterization as a complete replacement for explicit attention by generating globally conditioned weights for linear and depthwise convolution layers.This extends the approach to convolutional networks for implicit global sequence modeling.

4 Dynamic Weight Prediction

This section replaces explicit token interactions with input-conditioned linear and depthwise-convolution weights predicted from fixed-size global representations. Pooling and correlation strategies preserve linear complexity, while experiments identify effective predictors and show global receptive fields.

  • Motivation: Dynamic parameters compress global information into input-conditioned linear and depthwise-convolution weights, enabling global modeling through forward passes without explicit token-to-token interactions.The approach adapts convolutional networks using dynamically generated parameters conditioned on global context.
  • Global Representation: Fixed-size spatial compression makes parameter generation independent of input resolution, preserving linear complexity for input feature matrix X ∈ R^N×d.The global representation ϕ(X) is constructed independently of sequence length N.
  • Global Representation: The method explores pooling, which produces an M × d descriptor, and correlation, which distills global second-order statistics through X^⊤X ∈ R^d×d.Unlike attention’s channel-wise compression, these strategies focus on spatial compression.
  • Linear Layers: Conditioning linear-layer updates on global input implicitly integrates global context into channel transformations despite the layers performing no token-wise mixing.For X ∈ R^N×d, a dynamic update ∆W(X) ∈ R^d×d modulates a learnable static weight W0.
  • Linear Layers: 76.4% top-1 accuracy was achieved by Bilateral Activation on the first linear layer, providing the best accuracy-efficiency trade-off among evaluated linear strategies.Pooling before correlation reduced FLOPs and slightly improved accuracy; the cited result is reported on ImageNet-1K.
  • Depthwise Convolutions: Spatially Adaptive depthwise prediction offered the best practical trade-off, whereas Amp-Dir and Conv gained only 0.2% while increasing parameters and FLOPs and reducing throughput.Spatially adaptive prediction preserves spatial structure during kernel generation.
  • Global Modeling: All dynamic variants developed expansive effective receptive fields covering the entire input, unlike the localized static baseline, confirming global modeling from input-conditioned parameterization.The resulting behavior provides Transformer-like global reasoning through conditioning on global statistics.

5 WeightFormer: Dynamic Weights For Linear-Time Global Visual Modeling

WeightFormer uses sparse dynamic parameterization to replace selected static layers while preserving strictly linear time and memory complexity. Experiments show competitive or improved vision performance, with substantial efficiency gains at high resolution and benefits across discriminative and generative tasks.

  • Architecture: WeightFormer inserts dynamic parameterization every third block, leaving the remaining blocks static to balance modeling capacity and computational efficiency.The architecture uses a sparse distribution strategy rather than applying dynamic layers uniformly.
  • Architecture: Each dynamic block combines Spatially Adaptive Prediction for depthwise convolution with Bilateral Activation applied only to the MLP’s first linear layer.The second linear layer and all static blocks remain unchanged.
  • Downstream Vision Tasks: WeightFormer-T improves COCO box/mask AP from 44.4/38.1 to 45.0/38.3 while reducing total FLOPs from 594G to 566G.The comparison uses ViTDet as the basic framework and reports consistent but modest detection and segmentation improvements over DeiT.
  • Downstream Vision Tasks: WeightFormer-T reaches 40.7 mIoU with 12M params and 38G FLOPs, while WeightFormer-S reaches 45.6 mIoU and surpasses DeiT-S by 1.6 points.WeightFormer-T exceeds DeiT-T’s 39.2 mIoU, and WeightFormer-S uses 27G versus 35G backbone FLOPs.
  • Efficiency Analysis: At 1248×1248 with 6,084 tokens, WeightFormer achieves 7.7× higher throughput and 91% memory reduction compared to DeiT.Its linear complexity enables scaling to high-resolution inputs.
  • Ablation on Dynamic Block Frequency: Increasing the number of dynamic blocks raises theoretical capacity and computational cost, but excessive replacement causes severe underfitting and optimization challenges.Performance gains are not monotonic as dynamic block frequency increases.

6 Conclusion

The paper reframes attention as a dynamic MLP with input-conditioned parameters and proposes WeightFormer to replace explicit attention. Vision-task results indicate competitive performance with improved efficiency, while generalization and the properties and optimization of dynamic parameterization remain open questions.

  • Attention is revisited as a dynamic MLP with input-conditioned parameters, motivating dynamic parameterization as a replacement for explicit attention.
  • WeightFormer is proposed from this perspective as the paper’s dynamic-parameterization approach.
  • Results on vision tasks show competitive performance with improved efficiency.
  • The evaluation is limited to vision tasks, leaving generalization to other domains unclear.
  • The expressivity, inductive biases, and optimization challenges of dynamic parameterization remain insufficiently understood.

A Appendix · A.1 Dynamic Weight Strength Analysis

The appendix analyzes dynamic updates relative to static parameters across network depth. It finds that dynamic linear layers maintain a relative strength close to 1 throughout the model, while dynamic depthwise convolution follows a distinct depth-dependent pattern.

  • A.1 Dynamic Weight Strength Analysis: The analysis measures dynamic-update strength relative to static parameters at each dynamic layer.The ratio is defined using the static weight W0 and predicted dynamic update ΔW.
  • A.1 Dynamic Weight Strength Analysis: The static weight is denoted W0, while ΔW denotes the predicted dynamic update.These quantities form the basis of the relative-strength analysis.
  • A.1 Dynamic Weight Strength Analysis: The ratio r reflects the contribution of input-conditioned parameters.It quantifies how strongly dynamic parameters contribute relative to static parameters.
  • A.1 Dynamic Weight Strength Analysis: The relative strength of dynamic parameters exhibits a clear depth-dependent pattern.Figure 6 reports dynamic weight strength across depth.
  • A.1 Dynamic Weight Strength Analysis: r remains close to 1 across all depths for dynamic linear layers.This indicates consistent modulation of channel-mixing transformations by input-conditioned updates.
  • A.1 Dynamic Weight Strength Analysis: Dynamic depthwise convolution exhibits a contrasting depth-dependent strength pattern.The supplied passage contrasts this behavior with the nearly constant ratio of dynamic linear layers.
Loading 2605.01711v2…