PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer Finetuning Benefits from Non-Smooth Components

Ambroise Odonnat, Laetitia Chapel, Romain Tavenard, Ievgen Redko

arXiv:2602.06883v3cs.LGcs.CVstat.ML

TL;DR

The role of transformer smoothness in transfer learning remains poorly understood. This paper analyzes component plasticity through theory and extensive finetuning experiments, finding that high-plasticity attention and feedforward modules yield better and more stable performance.

  • Problem

    The role of transformer smoothness in transfer learning remains poorly understood, despite distribution shifts requiring models to adapt their weights.

    Source: §1 · p. 1

  • Method

    The paper ranks vision transformer components by plasticity, defined as their average rate of change, using theoretical bounds and empirical analysis.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    High-plasticity attention modules and feedforward layers consistently produce better and more stable finetuning performance across diverse classification benchmarks, optimizers, initializations, and learning rates.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Prioritizing non-smooth attention and feedforward components provides guidance for selecting modules during transformer finetuning.

    Source: §1 · p. 2, §7 · p. 9

    The study does not analyze how tailored optimization, such as adaptive learning rates or schedulers, affects each module’s performance.

    Source: §7 · p. 9

Abstract

from arXiv · show

The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their \emph{plasticity}. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments -- over $1,000$ finetuning runs on large-scale vision transformers -- showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on transformers' functional properties. The code is available at https://github.com/ambroiseodt/vit-plasticity.

1. Introduction

The paper studies transformer-component plasticity—the average rate of change, with higher values indicating lower smoothness—as a guide for efficient vision-transformer finetuning. Theory and extensive experiments show that attention and feedforward modules, which have the highest plasticity, consistently deliver better adaptation performance.

Approach

  • A systematic component-wise study finetunes each module type in isolation to connect intrinsic component properties with adaptation performance while avoiding confounders and combinatorial computational cost.The study focuses on normalization layers, multihead attention modules, and feedforward layers in vision transformers.Source: §1 · p. 2

Plasticity measure

  • The paper formalizes module plasticity as an average rate of change that measures how outputs respond to input variations, with high plasticity corresponding to low smoothness.This measure is intended to quantify component adaptability during finetuning.Source: §1 · p. 2

Theoretical analysis

  • The paper derives upper bounds that theoretically rank transformer components by plasticity and uses experiments on large-scale ViTs to support the ranking and its finetuning implications.The broader motivation is that distribution shifts between pretraining and downstream data require updating model weights, while smoothness can limit component adaptability.Source: §1 · pp. 1–2

Plasticity ranking

  • Pretrained ViTs consistently rank attention highest in plasticity, followed by the two feedforward layers, feedforward-preceding LayerNorm, and attention-preceding LayerNorm; the ranking also holds for DINOv3 and GPT2.The ranking is supported by empirical validation of the theoretical analysis.Source: §1 · p. 2

Finetuning benefits

  • High plasticity in attention and feedforward modules yields higher and more stable finetuning performance across classification benchmarks, optimizers, initializations, and learning rates.The experiments use exhaustive large-scale ViT finetuning runs with both SGD and Adam.Source: §1 · p. 2, Figure 2 · p. 2

2. Background

This section defines neural-network smoothness through concepts such as Lipschitz continuity and introduces the Vision Transformer architecture and its main components.

Neural network smoothness

  • Smoothness concerns a function’s differentiability, Lipschitz continuity, or robustness to input perturbations, with the Lipschitz constant quantifying output change relative to input change.A function is Lipschitz continuous when a finite constant bounds its output distance by the input distance; the smallest such constant is Lip(f).Source: §2 · p. 3

Vision transformers

  • A Vision Transformer splits images into patches, embeds them as token sequences, prepends a CLS token, and classifies from the final CLS embedding.Its architecture is a succession of transformer encoders containing LayerNorms, multi-head attention, and feedforward linear layers.Source: §2 · p. 3

Transformer components

  • The transformer components operate on token sequences through LayerNorm, multi-head self-attention with row-wise softmax, and GeLU-based feedforward layers.LayerNorm acts independently on each token, attention combines multiple heads, and the feedforward module applies two linear transformations around a GeLU.Source: §2 · p. 3

3. Vision transformer plasticity

This section defines plasticity as the average sensitivity of transformer components to input changes, linking high plasticity to low smoothness and potentially better finetuning adaptation. It motivates prioritizing non-smooth components because they may permit larger parameter gradients and faster learning.

Motivation

  • The motivation is that smoothness, although useful for generalization, stability, and robustness, can constrain capacity and adaptability to new tasks.This motivates identifying components whose Lipschitz constants may be too small for effective finetuning.Source: §3 · p. 3

Plasticity measure

  • Plasticity measures a component’s average rate of output change over pairs of token sequences, capturing its ability to adapt to input perturbations.The measure is introduced as a formal notion of a module’s ability to adapt its output when inputs change.Source: §3 · p. 3
  • Plasticity lower-bounds the Lipschitz constant and distinguishes smooth contraction from non-smooth amplification: P(f)<1 denotes low plasticity, whereas P(f)>1 denotes high plasticity.The Lipschitz constant gives worst-case control, while plasticity captures average behavior over input sequences.Source: §3 · p. 4

Connection to finetuning

  • High plasticity implies high input sensitivity, looser smoothness constraints, and potentially larger parameter gradients during finetuning.The section connects input-gradient and parameter-gradient norms through Lipschitz regularization, arguing that excessive smoothness regularization can hinder optimization.Source: §3 · p. 4

Expected benefits of plasticity

  • The expected benefit is faster and better downstream adaptation because plastic components may carry more information about downstream data than smooth components.The authors explicitly caution that plasticity is not expected to have a linear relationship with downstream performance.Source: §3 · p. 4

4. Theoretical analysis

Theoretical upper bounds rank transformer components by plasticity, with multi-head attention highest, feedforward layers intermediate, and LayerNorms lowest. The analysis also derives a tighter attention bound under image-energy assumptions and extends several results to decoder-only language models.

Attention bound

  • Multi-head attention requires bounded token sequences because self-attention is not globally Lipschitz, and its bound grows effectively as r^2√n under practical conditions.The bounded-token setting is argued to hold in practice, and the bound is tight in sequence length n.Source: §4 · p. 5

Tighter attention bound

  • An image-total-energy assumption yields a tighter attention bound with √n growth by controlling the Frobenius norm of token sequences.Although mean-field limits can produce sequence-length-independent bounds, they are unsuitable for vision transformers whose sequence lengths are usually below 10^3.Source: §4 · p. 5

Component bounds

  • LayerNorm and feedforward bounds imply tighter control for normalization than linear layers, while feedforward plasticity is bounded by the weight matrix’s spectral norm.The LayerNorm bound assumes tokens at each position share mean and standard deviation with a positive minimum standard deviation.Source: §4 · pp. 4–5

Theoretical ranking

  • The derived ranking is MHA → FC1 ≈ FC2 → LN2 ≈ LN1, with attention having the loosest plasticity control and LayerNorm the tightest.Numerical evaluation on an 86M pretrained ViT with sequence length n = 197 and 12 attention heads confirms this ordering.Source: §4 · p. 5

Extension to large language models

  • The bounds from Propositions 1–3 also apply to decoder-only language models because masked self-attention satisfies the required attention bound.The paper states that ViTs and LLMs exhibit similar plasticity patterns.Source: §4 · p. 5

5. Experiments

Experiments across 11 benchmarks show that transformer components with higher plasticity—especially attention and feedforward layers—produce better, more stable finetuning. Plasticity empirically follows the theoretical module ranking and remains predictive across model scales, optimizers, architectures, and data regimes.

5.1. Plasticity analysis

  • Pretrained transformers follow the empirical plasticity ranking MHA → FC1 → FC2 → LN2 → LN1, with attention and feedforward layers non-smooth and LayerNorms rigid.High-plasticity components have P(f) > 1, whereas LayerNorms have P(f) < 1; the ranking is consistent across benchmarks.Source: §5.1 · pp. 6–7
  • The plasticity pattern generalizes beyond supervised ViTs to DINOv3 and GPT2, while prioritizing high-plasticity components remains superior in the 1,000-sample VTAB regime.The attention module, especially MHA, outperforms low-plasticity LayerNorms in the low-data setting.Source: §5.1 · p. 7, §5.2 · p. 9

5.2. Benefits of plasticity for finetuning

  • Higher-plasticity attention and feedforward components achieve better finetuning accuracy than LayerNorms across 11 benchmarks, especially on Cifar100, Clipart, and Sketch.The performance ordering aligns with the plasticity ranking, and the comparison extends to larger ViT scales on Clipart.Source: Table 1 · p. 7, §5.2 · p. 7
  • High-plasticity components provide more stable finetuning across learning rates and initializations, with the attention module showing the smallest variability.The same benefits of higher accuracy and stability occur with Adam, using learning rates rescaled by 10^-2.Source: §5.2 · p. 8, Table 2 · p. 9
  • Higher plasticity facilitates optimization, yielding larger gradient norms, steeper loss descent, and better generalization across benchmarks, learning rates, and seeds.The optimization patterns are illustrated on Sketch and remain consistent across the reported experimental conditions.Source: §5.2 · p. 8, Figure 5 · p. 8

6. Related work

Prior work largely treats smoothness as beneficial across deep-learning objectives and practices, whereas this work studies low-smoothness transformer components as advantageous for finetuning. It also differs from Lipschitz-estimation and PEFT research by analyzing each transformer module’s adaptability component-wise.

Smoothness

  • This work identifies low-smoothness transformer components and demonstrates benefits from non-smooth components for finetuning without promoting smoothness during learning.Smoothness is commonly encouraged through weight decay, dropout, and early stopping, and studied in generalization, stability, generative modeling, robustness, and privacy.Source: §6 · p. 9

Lipschitz constant estimation

  • Transformer Lipschitz estimation is difficult because self-attention is nonlinear and vanilla attention is not globally Lipschitz, motivating specialized bounds and related proof techniques.Existing bounds are often loose, though tighter bounds arise for sequences of bounded tokens; the paper’s Section 4 techniques are akin to Lipschitz-bound derivations.Source: §6 · p. 9

Low-data finetuning

  • Table 3 evaluates component ordering by decreasing plasticity under the VTAB protocol using only 1,000 samples per dataset, highlighting consistent benefits in low-data regimes.The table marks the best component performance in bold and non-smooth components in gray.Source: Table 3 · p. 9

Parameter-efficient finetuning

  • Component-wise analysis complements selective, additive, and reparameterization PEFT methods by theoretically examining each transformer module’s adaptability rather than tuning module types together.Related PEFT approaches include selective finetuning, inserted adapters, and LoRa-style weight reparameterization.Source: §6 · p. 9

7. Discussion · Appendix · A. Extended related work

The paper concludes that prioritizing high-plasticity attention and feedforward components improves finetuning, while identifying tailored optimization and decoder-only models as future directions. Its appendices provide proofs, implementation details, code and carbon accounting, extended experiments, and related-work context on smoothness, scaling, Lipschitz estimation, and parameter-efficient finetuning.

7. Discussion

  • Finetuning non-smooth, high-plasticity attention modules and feedforward layers consistently delivers better and more stable performance.The authors present plasticity analysis as guidance for selecting components during adaptation and as a novel perspective on transformer smoothness.Source: §7 · p. 9
  • Tailored optimization, including adaptive learning rates and schedulers, is proposed for future study, alongside extending plasticity analysis to decoder-only models and LLMs.The authors describe the methodology and theoretical insights as naturally extending to decoder-only architectures.Source: §7 · p. 9

Appendix

  • The appendix roadmap covers extended related work, theoretical proofs, open-source code and carbon footprint, full implementation details, and extensive additional experiments.These materials are organized across Appendices A–E.Source: Appendix · p. 19

A. Extended related work

  • Prior smoothness research links Lipschitz constants and spectral norms to generalization, training stability, generative modeling, adversarial robustness, and differential privacy.Regularization, weight decay, dropout, and early stopping are presented as practices that encourage neural-network smoothness.Source: A. Extended related work · p. 20
  • At scale, gradient-norm controls and spectral-norm normalization address training instabilities and loss spikes in large models, including through QK-Norm, hyperspherical representations, Muon, and MuonClip.The cited approaches target stable optimization of large language models and related architectures.Source: A. Extended related work · p. 20
  • Lipschitz estimation is difficult for feedforward networks and self-attention, although tighter results exist for bounded-token attention and some attention variants are globally Lipschitz.The passage contrasts known tight constants for linear and activation layers with NP-hard feedforward estimation and nonlinear attention analysis.Source: A. Extended related work · p. 20
  • Parameter-efficient finetuning comprises selective, additive, prompt, reparameterization, and hybrid methods, while tool use, Toolformer [Schick et al., 2023], and RAG [Lewis et al., 2020] shift adaptation beyond static weight updates.The related work distinguishes parameter-reduction families and notes the rise of dynamic, context-aware language-model agents.Source: A. Extended related work · p. 20

B. Proofs … C.1. Open-source code

The appendices prove the plasticity bounds using matrix-norm properties, LayerNorm and linear-layer inequalities, and self-attention Lipschitz arguments under bounded-token assumptions. The accompanying open-source implementation supports reuse and extension of the study.

B. Proofs

  • The proofs rely on standard Frobenius and spectral norm properties, including submultiplicativity, spectral-norm bounds, and invariance under transposition.These properties are established through matrix-product inequalities and singular-value arguments.Source: B. Proofs · pp. 21–22

B.1. Proof of Proposition 1

  • The Proposition 1 proof upper-bounds LayerNorm plasticity by expressing its scaling parameters through a diagonal matrix and applying the norm lemma.The expectation over distinct token-sequence pairs then yields the rate-of-change bound under sequence-independent token standard deviations.Source: §B.1 · pp. 22–23

B.2. Proof of Proposition 2

  • The Proposition 2 proof derives linear-layer bounds directly from the matrix-norm lemma for both the d×4d and 4d×d weight matrices.The bounds are extended from sampled sequence pairs to expected rates of change.Source: §B.2 · p. 23

B.3. Proof of Proposition 3

  • The Proposition 3 proof applies a Lipschitz bound for self-attention on bounded token sequences and converts the supremum rate bound into an expected plasticity bound.The argument applies the bound separately to each attention head using its query, key, and value weights.Source: §B.3 · p. 24

B.4. Proof of Proposition 4

  • The Proposition 4 proof obtains bounded token norms from finite image energy and the embedding spectral norm, then bounds self-attention changes using softmax and matrix-norm inequalities.The image-energy assumption places token sequences in the bounded domain required by the preceding self-attention argument.Source: §B.4 · pp. 24–25

C.1. Open-source code

  • The authors publicly release the code and findings, with a library designed for adapting all or part of the implementation to other use cases.They specifically identify extending the study to large language models as a possible future application.Source: §C.1 · p. 27

C.2. Carbon footprint · D. Implementation details · D.1. Vision transformers

The paper reports the environmental cost of its large-scale experiments and specifies the implementation and ViT architectures used. The experiments required about 1,000 finetuning runs, while the models followed standard ViT designs and included three pretrained model sizes.

C.2. Carbon footprint

  • Around 1,000 finetuning runs required 3,700 GPU hours, potentially costing up to $40,000 and producing roughly 259 kgCO2eq in France.The emissions estimate is equivalent to a Paris–Madrid Boeing 737 round trip and is lower than in coal- and gas-dependent regions.Source: §C.2 · p. 27
  • The carbon estimate used the ML Impact calculator presented in Lacoste et al. (2019).Source: §C.2 · p. 27

D. Implementation details

D.1. Vision transformers

  • ViT inputs are patchified, linearly embedded with a prepended CLS token and positional embeddings, then processed by successive transformer encoders (Vaswani et al., 2017).Each block contains pre-LayerNorm, multihead self-attention, a two-layer GeLU MLP with hidden dimension 4d, and a residual connection (Hendrycks & Gimpel, 2016; Ba et al., 2016).Source: §D.1 · p. 27
  • Experiments use 86M-, 307M-, and 632M-parameter ViTs with patch sizes 16, 16, and 14, respectively, pretrained on ImageNet22k.The code follows the original ViT implementation and uses a convolutional image-embedding layer, as in the HuggingFace implementation.Source: §D.1 · p. 27
  • Table 4 reports the ViT variants’ patch sizes, sequence lengths, layer counts, attention heads, embedding dimensions, FFN dimensions, parameters, and pretrained-weight links.Source: Table 4 · p. 27, §D.1 · p. 27
  • Figure 6 displays the ViT-Base implementation with a ten-class classification head built using the modular library vitef.Source: §D.1 · p. 27, Figure 6 · p. 28

D.2. Data preprocessing … E.1.1. THEORETICAL PLASTICITY RANKING

The appendix specifies preprocessing, plasticity computation, and isolated-component finetuning protocols across diverse benchmarks, then validates the theoretical plasticity ranking on ViT-Base. The attention module has the largest theoretical upper bound, followed by feedforward layers and LayerNorm components, while empirical plasticity preserves this ordering at lower magnitude.

D.2. Data preprocessing

  • Experiments cover 11 classification benchmarks, including Cifar10, Cifar100, Cifar10-C variants, DomainNet domains, Flowers102, and Pets.The benchmark collection includes corruption, domain-shift, and standard image-classification datasets.Source: §D.2 · p. 28
  • Training images receive random cropping, 224×224 resizing, and horizontal flips, while evaluation images are resized, center-cropped, and ImageNet-normalized.Datasets without predefined splits use deterministic 80%-20% training-test partitions to prevent data contamination.Source: §D.2 · p. 28

D.3. Plasticity setup

  • Plasticity is computed from pretrained-model token sequences across pretraining and downstream distributions without assuming a predefined shift category.GPU forward passes over N batches of size b store outputs on the CPU; FC2 inputs are lifted from R^d to R^4d by zero-padding.Source: §D.3 · p. 29

D.4. Finetuning setup

  • Across ViT-Base and ViT-Large, each transformer component is finetuned in isolation by freezing all other weights, with full finetuning as the All baseline.The five isolated configurations are LN1, MHA, LN2, FC1, and FC2; the classification head is randomly initialized.Source: §D.4 · p. 29, Table 5 · p. 29
  • Finetuning uses SGD with momentum 0.9, cosine decay, 2000-step warmup, batch size 512, gradient clipping at norm 1, four learning rates, and three seeds.Final test accuracy is measured at the checkpoint with the best validation accuracy, using a 20% validation split.Source: §D.4 · p. 29

E. Additional experiments

  • The additional-experiments material reports detailed results for the paper’s figures, further experiments related to Sections 4 and 5.1, and the project’s carbon footprint.These sections provide reproducibility-oriented supplementary analyses rather than introducing a separate primary result.Source: E. Additional experiments · p. 30, §E.1 · p. 30

E.1.1. THEORETICAL PLASTICITY RANKING

  • The attention module has the largest theoretical plasticity upper bound, followed by FC1, FC2, LN2, and LN1; its bound is several orders of magnitude larger than the others.Fig. 7 evaluates ViT-Base with sequence length n = 197, H = 12, and reference radius r = 19.4 from Cifar10.Source: §E.1.1 · p. 30
  • Empirical plasticity follows the same component ranking as the theoretical bounds but with lower and more similar magnitudes, especially for attention.The discrepancy is attributed to the difficulty of bounding the self-attention Lipschitz constant and the gap between Lipschitz constants and average rates of change.Source: §E.1.1 · p. 30

E.1.2. PLASTICITY EXPERIMENTS OF ALL BENCHMARKS

Across additional benchmarks, plasticity consistently follows the theoretical ranking: attention is highest, feedforward layers follow, and LayerNorms are most rigid. The same pattern appears in ViT-Base and ViT-Huge, with higher attention plasticity for the longer-sequence model.

All benchmarks

  • Across additional datasets, attention has the highest plasticity, followed by the first and second feedforward linear layers, while LayerNorms remain below 1.The results extend the Section 5.1 analysis and are reported in Figures 8–17.Source: §E.1.2 · p. 30
  • The observed component ranking aligns with the theoretical analysis across the additional benchmark experiments.The reported ordering is attention, first feedforward layer, second feedforward layer, then more rigid LayerNorms.Source: §E.1.2 · pp. 30–31

Model scaling

  • ViT-Huge shows higher attention plasticity than ViT-Base, consistent with the predicted sequence-length effect.The figure captions attribute this validation to ViT-Huge having a larger sequence length than ViT-Base.Source: Figure 8 · p. 31

E.1.3. PLASTICITY EXPERIMENTS ON DINOV3 AND GPT2 … E.2.2. EXTENSION TO VIT-LARGE (307M)

Across DINOv3 and GPT2, attention and feedforward layers remain the most plastic while LayerNorms are most rigid. Across 11 benchmarks and in ViT-Large, higher-plasticity trainable components produce better finetuning performance.

E.1.3. PLASTICITY EXPERIMENTS ON DINOV3 AND GPT2

  • DINOv3 experiments use Cifar10 because ImageNet22k belongs to its pretraining data, while GPT2 experiments use AGNews and WikiText-103 as downstream proxies.AGNews approximates WebText’s news-oriented distribution, and WikiText-103 is enabled by WebText’s explicit removal of Wikipedia documents.Source: §E.1.3 · p. 37
  • In DINOv3 and GPT2, attention has the highest plasticity, followed by the feedforward layers, whereas LayerNorms are most rigid.Figure 18 shows LayerNorm plasticity below 1 and confirms the same theoretical ranking across ViT-Base, DINOv3-7B, and GPT2.Source: Figure 18 · p. 37, §E.1.3 · p. 37
  • Figure 18 shows that plasticity distributions across DINOv3 and GPT2 follow the theoretical ranking established for transformer components.The observed layerwise patterns are consistent with those previously found in supervised ViTs.Source: Figure 18 · p. 37, §E.1.3 · p. 37

E.2. Finetuning analysis

  • The finetuning analysis reports additional results, figures, and experiments related to the paper’s main finetuning study.These supplementary analyses provide the basis for the performance comparisons and visualizations in the following subsections.Source: §E.2 · p. 38

E.2.1. PERFORMANCE COMPARISON

  • Table 8 evaluates trainable transformer-component configurations on 11 diverse image-classification benchmarks using best test top-1 accuracy over dataset-specific learning-rate grids.Entries report means and standard deviations over three seeds, with full finetuning and linear probing included as reference configurations.Source: Table 8 · p. 38, §E.2.1 · p. 38

E.2.2. EXTENSION TO VIT-LARGE (307M)

  • ViT-Large finetuning reaches similar conclusions, showing consistent benefits from selecting higher-plasticity components.The extension evaluates ViT-Large with 307M parameters on Clipart; Table 9 reports mean and standard deviation over learning rates.Source: §E.2.2 · p. 38, Table 9 · p. 38

E.2.1. PERFORMANCE COMPARISON

  • Higher plasticity yields better performance across transformer-component configurations on 11 image-classification benchmarks.Figure 19 compares average top-1 accuracy across 11 benchmarks, while Table 8 reports full finetuning results over three seeds.Source: §E.2.1 · p. 38, Table 8 · p. 38, Figure 19 · p. 39

E.2.3. ROBUSTNESS ANALYSIS

Across 11 benchmarks, plastic transformer components produce more stable finetuning performance across learning rates and seeds, with attention modules consistently performing better and showing small variation.

Robustness analysis

  • Across 11 benchmarks, plastic components yield more stable finetuning performance across learning rates and three seeds, while attention modules consistently perform better with small variation.The comparison is relative to network initialization and dataloaders and orders transformer components by decreasing plasticity.Source: Figure 20 · p. 40, §E.2.3 · p. 40

E.2.4. GRADIENT NORM ANALYSIS · E.2.5. EXTENSION TO ADAM

Across benchmarks, higher-plasticity components show stronger optimization dynamics and faster validation-loss reduction, while Adam preserves and even amplifies the finetuning advantage and stability of non-smooth components. Gradient-norm ordering tracks plasticity across optimizers, especially for attention and feedforward layers.

E.2.4. GRADIENT NORM ANALYSIS

  • Across all benchmarks, higher-plasticity components have larger gradient norms, steeper validation-loss descent, and faster, better convergence, particularly on challenging datasets.The reported benefits are especially salient on Cifar100, Clipart, and Sketch, and are also illustrated on Cifar10 and Cifar100.Source: §E.2.4 · p. 41, Figure 21 · p. 41, Figure 24 · p. 43
  • LayerNorms exhibit lower gradient norms and less steep loss descent, with the gap worsening at low learning rates, indicating greater sensitivity to learning-rate choice than more plastic components.This pattern is described as evidence of LayerNorms’ lower plasticity relative to other transformer components.Source: §E.2.4 · p. 41
  • At high learning rates, high-plasticity components can show sharply increasing gradient norms before decreasing as optimization escapes pretraining minima and converges to a flat local minimum.The transient sharp-region behavior is particularly salient for attention on Clipart.Source: §E.2.4 · p. 41
  • On Pet and Flowers102, pretrained models’ good linear-probing performance yields lower gradient norms overall, but plastic components still descend more steeply at low learning rates.These datasets therefore qualify the general pattern by showing smaller absolute gradient norms while retaining the plasticity ordering.Source: Figure 48 · p. 55, Figure 51 · p. 56

E.2.5. EXTENSION TO ADAM

  • Adam consistently gives non-smooth components higher and more stable finetuning accuracy than smoother modules, with the stability advantage more salient than under SGD.Table 10 compares average top-1 accuracy and standard deviation across learning rates for Adam and SGD; accuracy is slightly lower with Adam overall.Source: Table 10 · p. 58, §E.2.5 · p. 58
  • With Adam, performance across learning rates is better and more stable for plastic components, while larger gradient norms and improved generalization preserve the SGD dynamics.The Sketch comparison and additional Adam training-dynamics figures show the same ordering across learning rates.Source: Figure 54 · p. 58, §E.2.5 · p. 58
  • Adam’s component-wise training dynamics on Cifar100, Motion Blur, Clipart, and Sketch reproduce the pattern that higher plasticity produces larger gradient norms, steeper loss descent, and better downstream performance.The figures report this pattern across the listed datasets using increasing learning rates.Source: Figure 55 · p. 59, Figure 56 · p. 59, Figure 57 · p. 60

Related papers

Loading 2602.06883v3…