Source-linked AI summary
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu, Lei Huang
TL;DR
Normalization remains nonlinear at inference, so inserting it beside an activation may change a shallow MLP’s represented function. The paper uses width augmentation and reparameterization to approximate prescribed MLPs with normalized networks. It reports sharp worst-case width bounds and transfer of applicable universal-approximation results.
Problem
It is not well understood whether inserting input-dependent LN or RMSNorm preserves a function represented by a shallow MLP.
Source: §1 · pp. 1–2Method
Results
The N + 1 RMSNorm and N + 2 LN bounds are worst-case sharp, applicable MLP universality results transfer, and experiments recover predicted approximation rates.
Source: §9 · p. 9Takeaways & Limitations
Relative approximation transfers applicable classical density results to normalized architectures by recovering prescribed MLP functions.
Source: §2 · p. 3The analysis covers LN and RMSNorm on compact sets in one shallow composition, while behavior in high-dimensional or deep networks is not established by the scalar synthetic experiments.
Source: §9 · p. 9
Abstract
from arXiv · showhide
Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-$N$ one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width $N+1$, while every continuous nonconstant activation admits width $N+r$ with error $O_f(r^{-1})$. For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width $N+1$; general continuous activations admit width $N+r$ with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths $N+2$ and $N+2r$, respectively. We prove that $N+1$ for RMSNorm and $N+2$ for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.
1 INTRODUCTION
Normalization remains nonlinear at inference, leaving open whether inserting LN or RMSNorm preserves a function represented by a shallow MLP. The paper studies this relative-approximation question using width augmentation and constructions that handle LN’s centering.
1 INTRODUCTION
- The representational effect of normalization on an MLP’s existing function is less understood than normalization’s use as an expressivity source.LN and RMSNorm recompute statistics from each representation, so both remain nonlinear at inference.Source: §1 · p. 1
- Width augmentation moves bounded features farther from the origin, shrinking the angle they subtend and making RMSNorm’s input-dependent scaling nearly constant after rescaling.Figure 1 contrasts the uneven spacing from near-field normalization with the shorter, nearly flat arc in the far field.Source: §1 · p. 2, Figure 1 · p. 2
- For LN, zero-mean preactivations in PN networks or a zero-sum AN readout absorb centering before width augmentation stabilizes scaling.These are the corresponding LN constructions built from the RMSNorm mechanism.Source: §1 · p. 2
- The paper establishes worst-case-sharp width bounds, transfers applicable MLP universal-approximation theorems, and reports experiments recovering predicted rates.The introduction identifies N + 1 for RMSNorm and N + 2 for LN as the sharp bounds.Source: §1 · p. 2
- Table 1 organizes width and error guarantees as upper and lower bounds, with ✓, ×, and ✓/× denoting present, absent, or arbitrary affine parameters.A dash means no matching lower bound is claimed; matching entries are sharp.Source: §1 · p. 2, Table 1 · p. 2
2 RELATED WORK
Prior work studies ordinary-network approximation and normalization as a source of expressivity, while this paper asks whether normalization preserves a prescribed MLP function. Its approach uses relative approximation to connect that preservation question to classical universality.
2 RELATED WORK
- Classical shallow-network results establish universal approximation for sigmoidal and broader nonpolynomial activations, while other work studies rates, width limits, and residual-network universality.The cited works include Cybenko (1989), Hornik (1991), Leshno et al. (1993), Barron (1993), Park et al. (2021), Lu et al. (2017), and Lin & Jegelka (2018).Source: §2 · p. 3
- Prior LN theory uses centering and scaling to study finite-sample label fitting, VC-dimension lower bounds, and universal approximation in parallel LN networks.The cited studies are Ni et al. (2024) and Ni et al. (2025).Source: §2 · p. 3
- Unlike work using normalization as an expressivity primitive, this paper asks how much width recovers a function already represented by an activated MLP.Width augmentation stabilizes scaling, reparameterization absorbs centering, and classical density transfers through relative approximation.Source: §2 · p. 3
3 NOTATION AND PRELIMINARIES
The paper fixes a width-N one-hidden-layer MLP as the target and measures normalized-network approximation on a compact input set. Its RMSNorm preliminaries establish a one-coordinate construction for approximating linear maps on compact feature sets.
3 NOTATION AND PRELIMINARIES
- Ordinary italic letters denote scalars, bold lowercase letters vectors, and bold uppercase letters matrices.The all-ones column vector is written 1_d, and inputs and outputs lie in R^d_x and R^d_y unless stated otherwise.Source: §3 · p. 3
3.1 TARGET ONE-HIDDEN-LAYER MLPS
- The target class M_N(ϕ) consists of width-N one-hidden-layer MLPs with coordinate-wise activation ϕ, compared on compact K using the uniform norm.For each target, the paper fixes one width-N realization; constants indexed by the target may depend on that realization and K.Source: §3.1 · p. 3
3.2 RMS NORMALIZATION
- RMSNorm removes input-magnitude dependence, and one added coordinate suffices to approximate any linear map uniformly on a compact feature set.The construction lifts each feature h to [hᵀ, M]ᵀ, making the normalized map approach a linear map as M grows; the added coordinate receives zero readout weight.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
4 ACTIVATION-NORMALIZED RMS NETWORKS
RMS activation-normalized networks can recover a prescribed width-N MLP by augmenting width to stabilize normalization. One large attainable activation value supports the N + 1 construction, while repeated bounded values provide the general-activation alternative.
4 ACTIVATION-NORMALIZED RMS NETWORKS
- RMS-AN uses a shared affine and no-affine class because its affine scale can be absorbed into the readout weights.The common construction recovers the original features up to a uniformly vanishing scaling distortion, while added coordinates control normalization and receive zero target-readout weight.Source: §4 · p. 4
4.1 UNBOUNDED ACTIVATIONS: MAGNITUDE CONTROLS THE NORMALIZATION SCALE
- For continuous activations with unbounded range, the construction uses one attainable large value to obtain width N + 1.The added activated coordinate has zero readout weight, and the signal’s scaling factor converges uniformly to one; this route requires attainable values whose magnitude tends to infinity.Source: §4.1 · pp. 4–5
4.2 GENERAL ACTIVATIONS: MULTIPLICITY REPLACES MAGNITUDE
- For every continuous nonconstant activation, repeating a nonzero attainable value replaces unbounded magnitude and gives width N + r.The repeated coordinates increase the normalization scale while the target feature vector remains bounded.Source: §4.2 · p. 5
5 PREACTIVATION-NORMALIZED RMS NETWORKS
Preactivation normalization acts before the activation, so its scale cannot generally be absorbed into the output map. Width N + 1 suffices under affine parameters or positive homogeneity, while general continuous activations can be handled by increasing width.
5 PREACTIVATION-NORMALIZED RMS NETWORKS
- Affine and no-affine PN classes are treated separately because the affine RMSNorm scale acts before the nonlinear activation and generally cannot be absorbed into the output map.The section’s summary contrasts the width N + 1 conditions with width growth for general continuous activations.Source: §5 · p. 5, §5.4 · p. 6
5.1 PN NETWORKS WITH AFFINE PARAMETERS
- Affine PN networks approximate every prescribed width-N MLP at width N + 1 for continuous activations.Appending a constant coordinate makes the first N normalized preactivations converge uniformly to the originals as |M| grows; continuity carries this limit through the activation.Source: §5.1 · p. 5
5.2 A FINITE-WIDTH OBSTRUCTION WITHOUT AFFINE PARAMETERS
- For ψ_d(t) = max{0, t − d}, every width-d no-affine RMS-PN network is constant, although ordinary ψ_d-networks can be nonconstant.Thus no fixed width guarantees approximation uniformly over all continuous activations; normalization can erase an activation’s nonconstant regime.Source: §5.2 · p. 5
5.3 NO-AFFINE PN NETWORKS WITH POSITIVELY HOMOGENEOUS ACTIVATIONS
- Continuous degree-one positively homogeneous activations permit no-affine PN approximation at width N + 1.Positive homogeneity moves the common normalization factor through the activation, allowing an inverse rescaling of the second-layer weights to cancel it.Source: §5.3 · p. 6
5.4 NO-AFFINE PN NETWORKS WITH GENERAL CONTINUOUS ACTIVATIONS
- For general continuous activations, no-affine PN networks approximate the target with width N + r as r increases.Appending r constant coordinates makes the normalized preactivation converge uniformly to the original on the compact input set; the stated error bound applies for r ≥ r_f.Source: §5.4 · p. 6
6 ADDING CENTERING: FROM RMSNORM TO LAYERNORM
LayerNorm adds centering to RMSNorm’s input-dependent scaling. The constructions absorb centering through reparameterization and use width augmentation to make the remaining scale nearly constant.
6 ADDING CENTERING: FROM RMSNORM TO LAYERNORM
- A mean-zero two-coordinate lift makes LayerNorm equal RMSNorm on the augmented vector, giving width N + 2 and error O(M^-2).The added coordinates are chosen so their sum cancels the original coordinate sum; the same decoder is then used.Source: §6 · pp. 6–7
6.1 CENTERING THROUGH REPARAMETERIZATION
- For unbounded activations, arbitrary LN-AN targets can be approximated at width N + 2 by combining a zero-sum readout rewrite with one attainable large coordinate.The zero-sum rewrite uses one extra activated coordinate; appending the large coordinate then controls residual scaling while the readout centering aligns with LayerNorm’s centering.Source: §6.1 · p. 7
6.2 REMAINING LAYERNORM APPROXIMATION GUARANTEES
- LayerNorm retains the affine-PN and homogeneous-PN width N + 2 guarantees and the general-activation width N + 2r guarantees, with the stated rates unchanged.For general activations, symmetric replicated coordinate pairs handle centering; the finite-width obstruction also persists when RMSNorm is replaced by LayerNorm.Source: §6.2 · p. 8
7 WIDTH LOWER BOUNDS AND UNIVERSAL APPROXIMATION
The paper establishes worst-case width thresholds for RMSNorm and LayerNorm, then transfers applicable one-hidden-layer universal-approximation results to the normalized classes.
7.1 JOINT WORST-CASE WIDTH LOWER BOUNDS
- The identity target on the unit ball exposes the worst-case width thresholds: RMSNorm requires N + 1, while LayerNorm requires N + 2.Too-narrow normalized maps collapse a nontrivial input segment; every width d ≤ N + 1 LayerNorm map has error at least 1/2, making the first admissible width N + 2.Source: §7.1 · p. 8
7.2 UNIVERSAL APPROXIMATION TRANSFER
- If the union of ordinary width-N one-hidden-layer MLPs is dense, the corresponding normalized classes are dense under the stated activation conditions.For AN, the activation must be continuous and nonconstant; for affine and no-affine PN, continuity suffices.Source: §7.2 · p. 8
8 NUMERICAL ILLUSTRATION
The numerical experiments recover the predicted approximation-rate patterns for the three fixed-overhead constructions and report similar behavior in broader width and activation sweeps.
8 NUMERICAL ILLUSTRATION
- Experiments across other activations, target widths N ∈ {4, 8, 16, 32}, and the remaining width-growing constructions exhibit the same predicted behavior.The extended results include interquartile ranges and the shifted-ReLU width obstruction.Source: §8 · p. 9
- RMSNorm AN follows the M^-2 rate and LN AN the M^-1 rate, while affine and homogeneous PN show nearly overlapping LN and RMSNorm errors consistent with M^-2 rates.Figure 3 reports mean grid-based maximum absolute errors across six width-16 targets; its gray lines are reference slopes, not error bounds.Source: Figure 3 · p. 9, §8 · p. 9
9 CONCLUSION
Width augmentation makes normalization approximately linear on compact feature sets, supporting recovery of a prescribed shallow MLP; the conclusion also reports sharp width bounds and experiments recovering predicted rates. The proof passages give specific RMSNorm and LayerNorm constructions and a finite-width obstruction for a shifted-ReLU RMS-PN network.
9 CONCLUSION
- Width augmentation makes normalization approximately linear on compact feature sets; the conclusion reports worst-case-sharp widths N + 1 for RMSNorm and N + 2 for LayerNorm, with experiments recovering predicted rates.Applicable universal-approximation results for ordinary MLPs also transfer to the normalized architectures.Source: §9 · p. 9
C.1 PROOF OF THEOREM 5.1
- For affine RMS-PN, appending a constant coordinate makes the first N normalized preactivations converge uniformly to the original preactivations, with O(M^-2) scaling error.Continuity of the activation gives uniform function convergence; local Lipschitz continuity yields the stated rate.Source: §C.1 · p. 12
C.2 PROOF OF PROPOSITION 5.2 √
- Every width-d no-affine RMS-PN network with shifted-ReLU activation ψd is constant, although ordinary ψd-networks can represent nonconstant functions.The proof uses the bounded coordinates of RMSNorm outputs to show every activated coordinate is zero.Source: §C.2 · p. 13
C.3 PROOF OF THEOREM 5.3
- Positive homogeneity lets no-affine RMS-PN rescale before activation and compensate in the readout, recovering the target as the remaining factor approaches one.The factor converges at O(M^-2).Source: §C.3 · p. 13
C.4 PROOF OF THEOREM 5.4
- Appending r unit coordinates in the general RMS-PN construction yields uniform convergence on K; local Lipschitz continuity gives the stated O_f(r^-1) rate for sufficiently large r.The added coordinates have zero readout weight.Source: §C.4 · p. 13
D.1 LAYERNORM COUNTERPART OF THEOREM 3.1
- The two-coordinate LayerNorm linearization has uniform O(M^-2) error on the compact feature set.The construction uses an augmented vector that is already centered before normalization.Source: §D.1 · p. 13
D.2 PROOF OF THEOREM 6.1
The LayerNorm proof passages adapt the RMSNorm constructions by handling centering through readout or preactivation choices. They give one-coordinate and replicated-coordinate approximations for AN and PN networks, alongside a finite-width shifted-ReLU obstruction.
D.2 PROOF OF THEOREM 6.1
- For unbounded-range activations, a zero-sum readout cancels the mean subtracted by LayerNorm; the resulting AN construction has width N + 2 and can achieve any prescribed error.The target is first represented exactly with a zero-sum readout, then the one-coordinate approximation is applied.Source: §D.2 · p. 15
D.3 LAYERNORM COUNTERPART OF THEOREM 4.2
- For general continuous activations, the LayerNorm-AN construction has width N + 2r and error O_f(r^-1).It appends r copies each of two distinct activated values as constant coordinates.Source: §D.3 · p. 15
D.4 LAYERNORM COUNTERPART OF THEOREM 5.1
- In affine LayerNorm-PN, the first N normalized preactivations converge uniformly to the originals with O(M^-2) scaling error, yielding uniform function convergence.The two added activated coordinates receive zero readout weight.Source: §D.4 · p. 16
D.5 LAYERNORM COUNTERPART OF PROPOSITION 5.2 √
- Every finite-width no-affine LayerNorm-PN network with shifted-ReLU activation ψd is constant, while an ordinary ψd-network can represent nonconstant functions.The proof notes that the LayerNorm coordinates stay within the activation’s zero-output range.Source: §D.5 · p. 16
D.6 LAYERNORM COUNTERPART OF THEOREM 5.3
- For degree-one positive-homogeneous activations, rescaling through the activation and readout reduces the LayerNorm-PN output to a factor converging uniformly to one.The factor converges at O(M^-2).Source: §D.6 · p. 16
D.7 LAYERNORM COUNTERPART OF THEOREM 5.4
- For general continuous activations, the LayerNorm-PN construction has width N + 2r and converges uniformly; local Lipschitz continuity on compact sets gives O_f(r^-1).The zero readout block ignores the 2r augmentation coordinates after activation.Source: §D.7 · pp. 16–17
E.1 PROOF OF PROPOSITION 7.1
The proof establishes finite-width lower bounds by showing that normalized networks identify inputs along certain directions or rays. It then uses ordinary MLP density together with the approximation constructions to obtain the stated universality transfer.
E.1 PROOF OF PROPOSITION 7.1
- For RMSNorm, rank-deficient first layers give approximation error at least 1; otherwise inputs on a segment of diameter at least one share a normalized representation, giving error at least 1/2.The proof uses the same-ray invariance of RMSNorm outputs and handles a zero-variance endpoint by a limit.Source: §E.1 · p. 17
- For LayerNorm, the same-ray argument gives error at least 1/2 for widths d ≤ N + 1; rank deficiency gives a lower bound of 1.The proof applies the argument to centered affine images, which share an LN representation when they lie on the same nonzero ray.Source: §E.1 · p. 17
E.2 PROOF OF COROLLARY 7.2
- Ordinary one-hidden-layer MLP density combines with the normalized approximation constructions to transfer universality to the applicable RMSNorm and LayerNorm architectures.The proof chooses both approximation errors below ε/2 and applies the triangle inequality.Source: §E.2 · p. 17
F EXPERIMENTAL DETAILS
The experiments test predicted approximation rates across activation families and illustrate a finite-width obstruction for shifted-ReLU PN networks. They use explicit constructions and grid-based error measurements, with observed tail behavior broadly consistent with the predicted rates.
F EXPERIMENTAL DETAILS
- The extended sweep spans five regimes, widths 4, 8, 16, and 32, and six targets per width; pooled tail slopes summarize trends, not a uniform-in-width error bound.For each target–width pair, slopes are fitted over the final six log–log points; Table 2 reports medians and interquartile ranges over 24 pairs.Source: F EXPERIMENTAL DETAILS · p. 18, Table 2 · p. 18
F.1 EXTENDED ACTIVATION-FAMILY VERIFICATION
- Across the tested activations, extended-sweep tails broadly match the predicted rates: unbounded AN differs for LN and RMSNorm, general AN and PN approach r^-1, and affine and homogeneous PN approach M^-2.Some curves deviate at small augmentation values.Source: §F.1 · p. 18, Figure 4 · p. 19, Figure 5 · p. 19
F.2 SHIFTED-RELU FINITE-WIDTH BARRIER
- For shifted-ReLU PN, no-affine errors show d^-1-consistent decay after a necessary finite-width barrier, while fixed-width affine constructions show M^-2-consistent decay.The barrier is necessary, not the exact width at which the plotted constructions turn on.Source: §F.2 · p. 20, Figure 6 · p. 20
- Across 24 random targets, shifted-ReLU no-affine relative grid errors decrease approximately as d^-1 after the barrier, while fixed-width affine curves follow the M^-2 reference slope.Figure 7 reports median relative grid maximum errors and interquartile ranges across widths 4, 8, 16, and 32.Source: §F.2 · p. 20, Figure 7 · p. 21