PaperlayerView exact revision on arXiv

Source-linked AI summary

LayerNorm Induces Recency Bias in Transformer Decoders

Junu Kim, Xiao Liu, Zhenghao Lin, Lei Ji, Yeyun Gong, Edward Choi

arXiv:2509.21042v4cs.CLcs.LG

TL;DR

Prior work found that stacked causal self-attention alone biases attention toward earlier tokens, unlike the recency bias observed in full Transformer decoders. This paper analyzes architectural components and input embeddings, finding that LayerNorm with causal self-attention induces recency bias. It also identifies limits to the analysis, including unexamined interactions with RoPE and the unassessed relationship between recency bias and model performance.

  • Problem

    Prior work on stacked causal self-attention found an earlier-token bias, while full Transformer decoders show recency bias, and the causal mask's role in positional information remains less understood.

    Source: §1 · p. 1

  • Method

    The paper theoretically analyzes causal self-attention with LayerNorm and examines residual connections and input-embedding distributions, alongside empirical analysis of recency bias.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    LayerNorm combined with causal self-attention induces recency bias, and the analysis investigates how residual connections and anisotropic input embeddings affect it.

    Source: §1 · p. 1, §4 · p. 5

  • Takeaways & Limitations

    The findings provide theoretical insights for designing positional encodings and studying length generalization in Transformer decoders.

    Source: §4 · p. 5

    The study does not evaluate the relationship between recency bias and overall model performance.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in attention scores toward earlier tokens. However, this differs from the bias toward later tokens typically observed in Transformer decoders, known as recency bias. We address this discrepancy by analyzing the interaction between causal self-attention and other architectural components. We show that stacked causal self-attention layers combined with LayerNorm induce recency bias. Furthermore, we examine the effects of residual connections and the distribution of input token embeddings on this bias. Our results provide new theoretical insights into how positional information interacts with architectural components and suggest directions for improving positional encoding strategies.

1 Introduction

The role of the causal mask in supplying positional information is less understood than that of positional encodings. The paper examines whether LayerNorm and residual connections help explain why full Transformer decoders show recency bias despite earlier-token bias from stacked causal self-attention alone.

1 Introduction

  • Positional encodings and the causal mask supply positional information, but the causal mask's role remains less well understood.The paper notes that positional information is tied to performance and generalization to longer sequences.Source: §1 · p. 1
  • Prior work found that stacked causal self-attention layers bias attention toward earlier tokens, whereas full Transformer decoders show bias toward more recent tokens.The paper calls the latter phenomenon recency bias and identifies the discrepancy as motivating study of other architectural components.Source: §1 · p. 1
  • LayerNorm combined with stacked causal self-attention induces recency bias without positional encoding; the paper also examines residual connections and input-embedding distributions.The authors frame these findings as theoretical insights for positional-encoding and length-generalization research.Source: §1 · p. 1

2 Theoretical Analysis

The analysis defines recency bias as attention favoring closer keys for a fixed query, then studies how LayerNorm, residual connections, and input-embedding distributions affect second-layer attention scores. Under normal input embeddings and sufficiently large hidden size, LayerNorm induces recency bias; the stated results also hold with or without residual connections and across the modeled anisotropy.

2.1 Preliminaries

  • Recency bias means that for a fixed query, a closer key receives a higher attention score than a more distant key.The decoder layer maps the previous layer's token representations to the current layer's representations, with a causal mask preventing attention to future positions.Source: §2.1 · pp. 1–2

2.2 LayerNorm

  • With embeddings drawn from N(0, I_d/d) and d ≫ 1, second-layer attention scores exhibit recency bias when causal self-attention is stacked with LayerNorm.The analysis obtains a score increasing with key index for a fixed query; without LayerNorm, the second-layer scores do not exhibit recency bias.Source: §2.2 · pp. 2–3

2.3 Residual Connection

  • The second-layer attention scores exhibit recency bias regardless of whether the residual connection is present.The authors state that its effect on causal bias is also examined empirically.Source: §2.3 · p. 3

2.4 Distribution of Input Token Embeddings

  • In the modeled setting, second-layer attention scores exhibit recency bias regardless of input-token embedding anisotropy.The anisotropic embedding model combines a shared vector with independent Gaussian noise; the paper motivates this case by noting that pretrained decoder embeddings are typically anisotropic.Source: §2.4 · pp. 3–4

3 Empirical Analysis

The simulations show that LayerNorm’s effect on recency bias depends on the input embedding distribution and hidden size. Anisotropic inputs produce a clear second-layer bias, while residual connections make it less pronounced.

3 Empirical Analysis

  • Without LayerNorm, RP stays near 0.5, and second-layer scores are uniform off the diagonal, consistent with no recency bias.This contrasts with the earlier-token bias reported for stacked causal self-attention alone (Wu et al., 2025).Source: §3 · p. 4
  • At α = 0 and d = 64, off-diagonal scores are nearly uniform (RP = 0.5000), but at d = 16 a mild recency bias appears (RP = 0.5015).The paper attributes the dimension difference to the d term suppressing the effect of key position more strongly at larger dimensions.Source: §3 · p. 4
  • With anisotropic inputs (α = 0.5), second-layer attention scores increase with key position, with RP = 0.6382 for d = 16 and RP = 0.5544 for d = 64.The scores increase strictly for each fixed query.Source: §3 · p. 4
  • For fixed query position, attention scores factor into query- and key-dependent terms, with h(j) governing key-position variation and increasing rapidly when α ≠ 0.When α = 0, the d term dominates and h(j) grows slowly; Figure 2 visualizes h(j) for multiple α values.Source: §3 · p. 4, Figure 2 · p. 4
  • The observed bias is consistent with nearby query–key hidden states having high cosine similarity, which LayerNorm approximates through scaled ℓ2 normalization.The paper relates this account to Zuo et al. (2025).Source: §3 · p. 4
  • Residual connections weaken the layer-2 recency bias, with RP = 0.5931 for d = 16 and RP = 0.5457 for d = 64.The authors attribute this reduction to fewer shared components, which lowers off-diagonal attention scores.Source: §3 · p. 5

4 Conclusion

The paper analyzes how causal self-attention and architectural components interact to produce recency bias. It concludes that this bias is not relative positional encoding, and suggests studying causal-mask modifications as a mitigation direction.

4 Conclusion

  • Causal self-attention combined with LayerNorm induces recency bias in attention scores.The analysis also considers residual connections and anisotropic input embeddings.Source: §4 · p. 5
  • The theoretical framework extends to MHA, MQA, and GQA by replacing the model dimension with the per-head dimension d_h.The derivation preserves its structure and qualitative conclusions.Source: §4 · p. 5
  • The resulting bias is not relative positional encoding: it depends on query and key positions, not solely on their distance.The paper notes that this non-uniform within-sequence bias may adversely affect length generalization.Source: §4 · p. 5
  • Modifying the causal mask to counteract the uneven bias is presented as a promising mitigation direction.The findings are framed as insights for positional-encoding design and length-generalization research.Source: §4 · p. 5

Limitations

The study identifies unexamined architectural components, positional-encoding interactions, and model-performance outcomes as limitations.

Limitations

  • The study does not analyze feed-forward networks, other learnable parameters, or multi-head attention, and leaves RoPE interactions and links to overall model performance unevaluated.These boundaries limit the conclusions about complete modern decoder layers and practical model outcomes.Source: Limitations · p. 5

A.1 Proof of Theorem 1

The proof derives the second-layer attention score’s dependence on key position. For a fixed query, that score is strictly increasing in the key index.

A.1 Proof of Theorem 1

  • The derivation of Equation 5 establishes that S(2)_ij is strictly increasing in j for fixed i.The proof’s comparison for i > j uses that the numerator’s order in j exceeds the denominator’s.Source: §A.1 · p. 7

A.2 Proof of Proposition 1

The proof shows that the second-layer attention score without LayerNorm does not exhibit recency bias.

  • The proof proceeds in the same manner as the preceding proof.Source: §A.2 · p. 7
  • The argument applies the law of large numbers.Source: §A.2 · p. 7
  • Without LayerNorm, the second-layer attention score does not exhibit recency bias.Source: §A.2 · p. 8

A.3 Proof of Proposition 2 and 3

The analysis incorporates residual connections and anisotropic input embeddings, then shows that the second-layer attention score increases with key index regardless of either factor.

  • The analysis rewrites the model equation to include residual connections and considers anisotropic input embeddings.Source: §A.3 · p. 8
  • For d ≫ 1, the inner product in the second term is negligible relative to those in the first and third terms.Source: §A.3 · p. 8
  • The proof reduces the recency-bias condition to showing that the first-layer representation’s squared norm decreases with j.Source: §A.3 · p. 9
  • For 0 ≤ α < 1, both terms are strictly negative, supporting the decrease in the first-layer representation’s squared norm.Source: §A.3 · p. 10
  • S(2)_ij is strictly increasing in j for fixed i, exhibiting recency bias regardless of residual connections or input-embedding anisotropy.Source: §A.3 · p. 10

B Extended Results

The extended results visualize attention scores across additional layers, dimensions, and α values, with and without residual connections, and also plot h(j) against key index.

B Extended Results

  • Figure 3 visualizes attention scores for layers 3 and 4 at d = 16 and d = 64.Source: Figure 3 · p. 11
  • Figure 4 visualizes attention scores for layers 1 and 2 at d = 256 and d = 1024.Source: Figure 4 · p. 12
  • Figure 5 extends Figure 1 across multiple α values without residual connections.Source: Figure 5 · p. 13
  • Figure 6 extends Figure 1 across multiple α values with residual connections.Source: Figure 6 · p. 14
  • Figure 7 plots h(j) over key index j for multiple α values, including residual connections.Source: Figure 7 · p. 15

Related papers

Loading 2509.21042v4…