Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers

1 Introduction

• Recentring Pre-LN main-branch vectors removes redundant mean information, allowing LayerNorm to be reduced to RMSNorm without changing model functionality.The paper identifies that these vectors are always normalized before use.Source: §1 · p. 2
• Pre-CRMSNorm applies lossless compression to zero-mean vectors, decompressing them before RMSNorm while preserving equivalent arithmetic functionality.CRMSNorm takes a vector in R^(d−1) and decompresses it to a zero-mean vector in R^d.Source: §1 · p. 2
• Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm are formally established as equivalent in both training and inference.A Pre-LN model can be converted to an equivalent Pre-(C)RMSNorm model for deployment or downstream adaptation with minimal conversion cost.Source: §1 · p. 2
• The paper argues that modest relative efficiency gains can yield substantial absolute savings for foundation models and complement other efficiency methods.It gives reducing ChatGPT inference cost by 1% as an example that may save $7,000 per day.Source: §1 · p. 2
• The paper presents the work as unifying LayerNorm and RMSNorm in pre-normalization Transformers through proven arithmetic equivalence.The proposed Pre-RMSNorm and Pre-CRMSNorm variants can be interchanged with the original Pre-LN without affecting functionality.Source: §1 · p. 2
• 1% −10% greater efficiency is reported for the proposed architectures than for Pre-LN in both training and inference.The gains require no fine-tuning or calibration.Source: §1 · p. 3

2 Background

• The paper frames normalization as a component of Transformers and introduces LayerNorm, RMSNorm, and an abstraction for Pre-LN Transformers.Source: §2 · p. 3

2.1 LayerNorm and RMSNorm

• LayerNorm recenters and rescales activations, whereas RMSNorm rescales by RMS; on zero-mean vectors, their outputs are equivalent.The paper limits its focus to computation rather than normalization optimization or expressivity.Source: §2.1 · p. 3

2.2 Normalization in Transformers

• Pre-LN remains mainstream, while Pre-RMSNorm is used in some large language models; the preferred normalization remains unclear and conversion between types is challenging.The paper notes RMSNorm's reported speed advantage with similar performance in large models.Source: §2.2 · pp. 3–4

2.3 Pre-LN Transformer

• A Pre-LN Transformer preprocesses inputs, applies a stack of residual blocks, then applies LayerNorm and postprocessing to the final representation.Each block adds its residual-branch output to the main-branch vector; its operators begin with LayerNorm followed by linear and block operations.Source: §2.3 · p. 4

3 Method

• Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm Transformers are claimed to be arithmetically equivalent.The method analyzes this equivalence and the variants’ computational efficiency.Source: §3 · p. 4

3.1 Pre-LN Transformer = Pre-RMSNorm Transformer

• Three changes—recenter initial and residual-branch outputs, then replace LayerNorm with RMSNorm—preserve function because zero-mean inputs make the normalizers equivalent.Recenter the initial vector and residual-branch outputs; then replace LayerNorm at residual-block beginnings and before postprocessing.Source: §3.1 · p. 5

3.2 Pre-RMSNorm Transformer = Pre-CRMSNorm Transformer

• CRMSNorm decompresses compressed zero-mean inputs before RMSNorm, while lossless compression represents zero-mean vectors in d dimensions using d−1 coordinates.The compression has a 1/d space-saving ratio; the proposed variant also simplifies input and output projection weights and biases.Source: §3.2 · pp. 5–6

3.3 Pre-LN Transformer = Pre-CRMSNorm Transformer

• Pre-LN Transformers with main-branch vectors in d dimensions are equivalent to Pre-CRMSNorm Transformers in d−1 dimensions.The paper presents this dimensional correspondence as further evidence of redundancy in Pre-LN Transformers.Source: §3.3 · p. 6

3.4 Training and Inference Efficiency

• Conversions among the variants require one-time parameter adjustments, without expensive fine-tuning or calibration.The paper compares this conversion process to operator fusion.Source: §3.4 · p. 6

3.4.1 Pre-RMSNorm Transformer

• Pre-RMSNorm removes LayerNorm overhead; inference can fold recentered linear parameters in advance, while training incurs a small parameter-adjustment cost.The parameter adjustment is relatively small and can be computed ahead of time; recentering during training is on the fly.Source: §3.4.1 · pp. 6–7

3.4.2 Pre-CRMSNorm Transformer

• CRMSNorm theoretically reduces model size, computation, communication, and memory by 1/d, but accelerators may handle d−1 vectors less efficiently than d-dimensional vectors.For inference, the paper suggests adding zero embeddings or decompressing to Pre-RMSNorm; for training, it suggests retaining dimension d.Source: §3.4.2 · p. 7

3.5 Training and Inference Equivalence

• Inference converts Pre-LN parameters to an equivalent Pre-RMSNorm model; training computes through that model but updates the original Pre-LN parameters.The paper states that this preserves the original Pre-LN training performance and stability.Source: §3.5 · pp. 7–8

4 Experiments

• The efficiency analysis focuses on the components of the method because the authors treat efficiency as accompanying the claimed equivalence.The cited prior work is said to have shown RMSNorm's efficiency advantage over LayerNorm [44] [26].Source: §4 · p. 8
• The experiments compare representative encoder-only ViT and decoder GPT-3 architectures, using replicated models, dummy data, and repeated timing measurements.The paper says it omits task metrics such as classification accuracy and perplexity because training and inference equivalence are guaranteed; ViT dropout requires explicit recentering for Pre-RMSNorm, whereas the cited DeiT settings disable dropout.Source: §4 · p. 8, §4.1 · p. 8

4.1 Experiments on ViT

• 1%–9% lower inference time is reported for Pre-RMSNorm than Pre-LN across ViT model sizes and mini-batch sizes.The measurements use a single A100 GPU; the left subfigure reports batch-size-1 latency.Source: §4.1 · p. 8
• Other precision, CPU, and JAX ViT experiments show an average 3.0% inference time reduction.The paper directs readers to Appendix D for further details.Source: §4.1 · p. 9
• 2.36% lower training time is reported for the Pre-RMSNorm ViT, with LayerNorm taking 12.6% of baseline time and added recentering and projection costs totaling 0.98%.RMSNorm takes 9.27% of total time in the Pre-RMSNorm variant.Source: §4.1 · p. 9
• Pre-CRMSNorm trains 1.74% faster than Pre-LN, but the tested GPUs could not efficiently use compressed Rd−1 vectors.Zero padding and decompression can make Pre-CRMSNorm less efficient than Pre-RMSNorm; cases that efficiently handle Rd−1 vectors can reach 10% time reduction.Source: §4.1 · p. 9

4.2 Experiments on GPT

• Replacing LayerNorm with (C)RMSNorm reduces inference time by up to 10% in small GPT-3 models, while normalization accounts for less than 1% of inference time in GPT-3 XL and larger models.Measurements use batch size 1 and sequence length 512; the normalization share bounds the speedup in the larger models.Source: §4.2 · p. 10
• With int8 matrix multiplication, LayerNorm accounts for 10% of inference time for GPT-3 XL and 2.7B, and Pre-RMSNorm reduces inference time by 4%.GPT-3 Small and Medium training time reductions are 1.5% and 1.8%, respectively.Source: §4.2 · p. 10

5 Conclusion

• Pre-RMSNorm maintains zero-mean main-branch vectors to simplify LayerNorm, while Pre-CRMSNorm losslessly compresses those vectors.The paper frames these variants as equivalent and more efficient alternatives to Pre-LN.Source: §5 · p. 10
• Equivalent Pre-(C)RMSNorm models can replace pretrained Pre-LN models with minimal overhead, improving training and inference efficiency.The authors state this can support more efficient deployment and higher-throughput training.Source: §5 · p. 10
• Practical efficiency gains require highly optimized (C)RMSNorm implementations to bridge the gap between theoretical and practical improvements.The conclusion also suggests exploring other architectures and integrating the method into machine-learning compilers.Source: §5 · p. 10

A Proof for Lemma 1.

• A linear transformation's output is decomposed into a zero-mean linear transform and a separate mean term, with the matrix columns and bias recentered.The output mean is represented by the scalar term k1^T Ax + µ(b), with k = 1/m yielding the output mean.Source: A Proof for Lemma 1 · p. 14

B Post-LN Transformers

• Post-LN can keep a zero-mean main branch and replace LayerNorm with RMSNorm after modifying the residual branch's output projection.Explicit recentering is unnecessary when LayerNorm's elementwise affine transformation is disabled; when enabled, the output is not guaranteed zero-mean and recentering is necessary.Source: B Post-LN Transformers · p. 14

C Revertible Conversions

• The conversions are reversible: Pre-CRMSNorm decompresses to zero-mean vectors before replacing CRMSNorm with RMSNorm, while the reverse Pre-RMSNorm-to-Pre-LN steps remove recentering and restore LayerNorm.The appendix also specifies restoring mean-related components of the linear layers and bias during conversion.Source: C Revertible Conversions · pp. 14–15

D.1 Implementation of Normalization

• The reported results use the implementations described by the authors, who provide JAX and PyTorch implementations in supplementary material.The statement identifies the implementation basis of the reported results.Source: §D.1 · p. 15
• JAX uses Flax LayerNorm and RMSNorm APIs, while PyTorch uses NVIDIA Apex implementations; CRMSNorm uses customized code.The authors also provide customized LayerNorm and RMSNorm implementations.Source: §D.1 · p. 15
• No normalization implementation dominates across all cases: PyTorch LayerNorm can be faster for small inference inputs but slower for large inputs and training than Apex.The comparison is sensitive to input size and whether the workload is inference or training.Source: §D.1 · p. 15

D.2 Extended Experiments in ViT

• Table 2 lists ViT architectures of different sizes, with the number in each model name indicating its patch size.Source: Table 2 · p. 15
• Table 3 reports normalized ViT inference time, averaged over 36 experiments spanning six models and six batch sizes.The batch sizes were 1, 4, 16, 64, 256, and 1024; measurements used an A100 with automatic mixed precision, with additional precision, platform, and framework settings tested.Source: Table 3 · p. 15, §D.2 · pp. 15–16

D.3 Numerical Issue

• Floating-point computation may not preserve theoretical arithmetic equivalence, and small discrepancies may accumulate and impair numerical stability in large models.The method cannot ensure an exactly zero-mean main branch numerically; the authors compare this issue to layer fusion, which can change numerical results while often having neutral task-performance impact.Source: §D.3 · p. 16
• The pretrained Pre-LN ViT-H/14 converted to Pre-(C)RMS variants showed no accuracy change on ImageNet validation.Replacing PyTorch’s LayerNorm implementation with apex had a larger impact on model performance.Source: §D.3 · p. 16

• Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm variants achieved similar training performance on ImageNet-1k and wikitext-103-raw-v1.The ViT-S/16 test-accuracy curves cover 800 epochs; GPT-2 Small training and evaluation loss curves were also remarkably similar.Source: §D.4 · p. 17