PaperlayerView exact revision on arXiv

Source-linked AI summary

Stronger Normalization-Free Transformers

Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, Zhuang Liu

arXiv:2512.10938v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Normalization layers stabilize Transformers but depend on activation statistics, motivating a systematic search for stronger statistics-free point-wise replacements. The paper analyzes four function properties, searches candidate designs, and introduces Derf. Derf consistently outperforms normalization methods and DyT across diverse modalities and tasks, with gains attributed mainly to stronger generalization.

  • Problem

    A comprehensive analysis of point-wise function designs that could surpass normalization layers was missing, although Dynamic Tanh had shown comparable empirical performance.

  • Method

    The paper analyzes zero-centeredness, boundedness, center sensitivity, and monotonicity, then searches transformed point-wise functions and introduces learnable Dynamic erf, or Derf.

  • Results

    Derf consistently outperforms normalization-based methods and DyT across diverse modalities, tasks, architectures, and training settings.

  • Takeaways & Limitations

    Well-designed point-wise functions can replace normalization layers in many Transformer architectures while delivering stronger empirical performance.

Abstract

from arXiv · show

Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce $\mathrm{Derf}(x) = \mathrm{erf}(αx + s)$, where $\mathrm{erf}(x)$ is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including visual recognition and generation, speech representation, and DNA sequence modeling. Our analysis also suggests that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.

1 Introduction

The paper studies how point-wise function properties affect normalization-free Transformer training and searches for a function that surpasses normalization methods. It identifies Derf and evaluates it across modalities, tasks, and training paradigms.

  • 1 Introduction: Normalization layers stabilize training and accelerate convergence but introduce activation-statistics overhead and can be sensitive to batch size.
  • 1 Introduction: The study independently examines zero-centeredness, boundedness, center sensitivity, and monotonicity to determine how point-wise functions influence training dynamics and final performance.The analysis aims to derive a design principle for normalization-free Transformers.
  • 1 Introduction: Guided by the property analysis, the authors search candidate point-wise functions and identify Dynamic erf as a simple, highly performant design.Derf augments the S-shaped, Gaussian-related erf function with learnable parameters.
  • 1 Introduction: Derf consistently surpasses LayerNorm and Dynamic Tanh across vision, language, speech, and DNA tasks spanning classification, generation, and sequence modeling.The evaluation covers supervised and self-supervised training settings.
  • 1 Introduction: Derf's superior performance is associated with stronger generalization rather than enhanced fitting capacity, because it has higher evaluation-mode training loss than normalization-based models.

2 Background

This background contrasts statistics-dependent normalization with point-wise alternatives and frames the missing systematic analysis of their design space. It motivates identifying properties and functions that can outperform normalization layers.

  • 2 Background: Normalization methods center and scale grouped activations using activation statistics, with their main differences arising from how those groups are formed.LayerNorm computes statistics along the channel dimension independently for each token.
  • 2 Background: Four properties—zero-centeredness, boundedness, center sensitivity, and monotonicity—collectively characterize point-wise functional behavior and influence training dynamics.Figure 2 distinguishes functions satisfying each property from those violating it using blue and red curves.
  • 2 Background: LayerNorm is well-suited to Transformers because it computes per-token normalization while token activations can have diverse statistics.
  • 2 Background: Point-wise functions provide statistics-free alternatives by applying the same parametric mapping independently to each activation.
  • 2 Background: A comprehensive analysis of the design space for statistics-free point-wise operators was missing despite Dynamic Tanh's normalization-level empirical performance.

3 Function Property Analysis

Controlled experiments identify zero-centeredness, boundedness, center sensitivity, and monotonicity as important properties for stable and effective point-wise replacements of normalization layers. Across these analyses, deviations from these properties degrade performance or cause training failure.

  • 3.1 Zero-centeredness: Zero-centeredness is required for stable convergence: horizontal shifts degrade performance as |λ_horiz| grows, while vertical shifts decline with |λ_vert| and fail at |λ| ≥2.Performance remains comparable to the zero-centered baseline for |λ_horiz| ≤0.5.
  • 3.2 Boundedness: Clipping unbounded functions improves performance, while making bounded functions more unbounded lowers accuracy, supporting boundedness as a useful replacement property.The clipped versions outperform unbounded baselines across tested values, whereas unbounded variants perform slightly below bounded baselines.
  • 3.2 Boundedness: Point-wise functions have an upper growth-rate limit: logquad(x) is the fastest tested function that converges, whereas faster-growing linear(x) and power23(x) diverge early.Rapid growth fails to suppress variance effectively and leads to large initial gradient norms.
  • 3.3 Center Sensitivity: Center sensitivity improves results near the origin: performance is best without a flat region, degrades as λ increases, and diverges when λ ≥3.0.The degradation becomes more obvious once λ exceeds 1.0.
  • 3.4 Monotonicity: Both increasing and decreasing monotonic functions train stably and achieve high accuracy, whereas hump-shaped and oscillatory functions perform worse.The comparison controls output ranges to isolate monotonicity effects.

4 Function Search

The search retains point-wise functions satisfying desirable properties and evaluates them across ViT and DiT using accuracy and FID. Among candidates, transformed erf(x) performs best and surpasses normalization baselines.

  • The candidate set is built by retaining functions that are near zero-centered, bounded, center-sensitive, and monotonic.
  • The empirical search evaluates candidates on ViT and DiT trained on ImageNet-1K, using top-1 accuracy and FID respectively.
  • Each candidate is instantiated with learnable input scaling and shifting, alongside affine parameters analogous to normalization layers.
  • Transformed erf(x) is the best-performing candidate, consistently surpassing other point-wise functions and baseline normalization layers.

5 Dynamic erf (Derf)

The paper introduces Derf, a learnable erf-based point-wise layer that replaces normalization layers throughout Transformer architectures. Its additional scalar parameters control input scaling and shifting, while affine parameters remain per-channel.

  • Derf is Dynamic erf, an erf-based point-wise layer with learnable scalar scale and shift parameters.
  • Derf replaces pre-attention, pre-FFN, and final normalization layers one-to-one throughout Transformer architectures.
  • Derf initializes its affine parameters like standard normalization and sets the scale and shift scalars to 0.5 and 0, respectively.

6 Experiments

Derf is evaluated across vision, image generation, speech, DNA, and language models, consistently outperforming normalization baselines and DyT in most reported settings. Its gains coincide with stronger generalization rather than greater fitting capacity than normalization layers.

  • Across tested architectures, Derf consistently achieves stronger performance than baseline normalization methods and DyT.
  • Vision Transformers: Derf achieves higher ImageNet top-1 accuracy than LayerNorm and DyT across ViT model sizes.
  • Diffusion Transformers: Derf achieves lower ImageNet FID than LayerNorm and DyT across Diffusion Transformer model sizes.
  • Speech models: Derf yields lower LibriSpeech validation loss than LayerNorm and DyT across two wav2vec 2.0 model sizes.
  • DNA models: Derf surpasses normalization layers and DyT on averaged GenomicBenchmarks accuracy across DNA-model subtasks.
  • Language models: Derf matches LayerNorm while outperforming DyT on GPT-2 validation loss.
  • Fitting capacity and generalization: Evaluation-mode training losses follow Norm < Derf < DyT, indicating Derf and DyT have lower fitting capacity than normalization layers.
  • Fitting capacity and generalization: Derf’s consistent downstream gains are attributed primarily to stronger generalization, while also retaining greater fitting power than DyT.

7 Analysis

Ablations show that a learnable shift improves performance, while scalar and per-channel shift parameters perform similarly. Scaling tanh improves it modestly but does not match erf.

  • Shift parameter ablations: Adding the learnable shift s consistently improves training performance, although the magnitude varies across point-wise functions.
  • Shift parameter ablations: Scalar and per-channel vector versions of s produce nearly identical final performance, supporting the scalar choice for simplicity.
  • Approximation analysis: Scaling tanh by ε≈1.205 yields comparable or slightly better results than tanh, but remains below erf.

8 Related Work

Prior work explains normalization’s optimization benefits and motivates methods that achieve stable convergence without normalization, including point-wise alternatives.

  • Normalization stabilizes gradient flow, reduces initialization sensitivity, tunes learning rates, and smooths the loss landscape.
  • Normalization-free methods pursue stable convergence through tailored initialization, self-normalizing activations, weight normalization, and adaptive gradient clipping.

9 Conclusion

The paper argues that carefully designed point-wise functions can surpass normalization layers in Transformer models. It identifies four useful properties and presents Derf as the strongest-performing simple design across modalities and tasks.

  • Derf consistently outperforms normalization-based methods and DyT across a wide range of Transformer modalities and tasks.
  • Zero-centeredness, boundedness, center sensitivity, and monotonicity are identified as key properties for strong normalization-free Transformer performance.
  • Derf’s simplicity and empirical performance make it a compelling normalization replacement for many Transformer architectures.

A Property Analysis Details

The analysis evaluates how point-wise function properties affect fitting and performance, then searches broad function families constrained by those properties. It highlights monotonicity, boundedness, and construction choices as central design considerations.

  • Property Analysis Details: Vertical shifts consistently raise training loss, whereas a small horizontal shift nearly matches or slightly improves the unshifted function’s loss.
  • Property Analysis Details: Larger center sensitivity raises training loss, indicating that a wider flat zone directly limits fitting capacity.
  • Property Analysis Details: Monotonic functions achieve lower training loss than non-monotonic functions, matching the observed ImageNet-1K accuracy trends.
  • Property Analysis Details: The function search explores polynomial, rational, trigonometric, hyperbolic, cumulative-distribution, and transformed or clipped variants.
  • Property Analysis Details: Candidates are grouped into natural, transformed basic, clipped unbounded, and canonical ratio functions for comparison and visualization.
  • Property Analysis Details: Natural functions such as erf, tanh, and arctan satisfy zero-centeredness, boundedness, center sensitivity, and monotonicity, with ranges unified to [−1, 1].
  • Property Analysis Details: Transformed, clipped, and ratio-based constructions reshape candidate functions to enforce the target properties while preserving or producing saturating behavior.

C Experimental Settings

Experiments use established model implementations and task-specific training configurations across vision, diffusion, speech, DNA, and language models. Derf replaces normalization selectively or broadly while comparisons report best results under specified tuning procedures.

  • C Experimental Settings: ImageNet-1K ViT experiments use shared configurations for ViT-B and ViT-L, with modified momentum and stochastic depth for ViT-L.
  • C Experimental Settings: Diffusion Transformer experiments evaluate three learning rates for both LayerNorm and point-wise functions and report the best result.
  • C Experimental Settings: wav2vec 2.0 retains input-oriented normalization layers and otherwise uses the official Base and Large configurations.
  • C Experimental Settings: DNA evaluations follow official HyenaDNA and Caduceus implementations, replacing each model’s original normalization layer with Derf and averaging accuracy across tasks.
  • C Experimental Settings: GPT-2 experiments tune separate Derf and DyT α initializations for attention-following and other point-wise layers, reporting the best validation loss.

D Additional Results

Additional evaluations show that Derf generally outperforms normalization alternatives and DyT across vision, diffusion, speech, DNA, and language tasks, with runtime comparable to competing operators.

  • Vision Transformers: Derf achieves higher ImageNet-1K top-1 accuracy than all other evaluated methods across ViT model sizes.Table 22 reports comparisons against LayerNorm, DyT, RMSNorm, and GroupNorm.
  • Diffusion Transformers: Derf achieves lower FID than LayerNorm, DyT, and RMSNorm across DiT models, indicating better reported image-generation quality.Lower FID denotes better image-generation quality in this evaluation.
  • Speech models: Derf yields lower validation loss than other methods across two wav2vec 2.0 models.The comparison includes LayerNorm, DyT, and RMSNorm.
  • DNA models: Derf consistently outperforms alternative methods on averaged GenomicBenchmarks accuracy across HyenaDNA and Caduceus.The evaluated alternatives include LayerNorm, RMSNorm, and DyT.
  • Language models and efficiency: Derf matches LayerNorm and RMSNorm while outperforming DyT on GPT-2 validation loss, with runtime comparable to DyT and LayerNorm.The runtime comparison uses custom Triton kernels under a unified setup across practical hidden dimensions.
Loading 2512.10938v2…