PaperlayerView exact revision on arXiv

Source-linked AI summary

A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

Shuyang Xiang

arXiv:2610.24133v1cs.AI

TL;DR

The paper asks how much prompt information remains in a transformer’s hidden state after generation. It measures prompt offsets under shared continuations, proves a conditional decay bound, and finds power-law forgetting that is faster in trained than untrained models.

  • Problem

    The paper asks whether hidden states still distinguish prompts after many generated tokens, and why that influence fades.

    Source: §1 · p. 2

  • Method

    It compares hidden states for different prompts followed by the same continuation, measuring the resulting prompt offset and its decay.

    Source: §1 · p. 2

  • Results

    Prompt offsets decay approximately as t^-α; trained models forget faster than random-weight baselines, and forgetting appears within the first billion Pythia training tokens.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The paper concludes that dilution is necessary for the measured forgetting and is sufficient in untrained networks and the linear relay model.

    Source: §10 · p. 29

    The theorem gives an asymptotic shape rather than numerical predictions, and its bound becomes informative only when t ≫ Pe^∆, beyond measured windows for trained networks.

    Source: §4.3 · p. 11

Abstract

from arXiv · show

We ask how much an autoregressive transformer retains of its prompt after generating $t$ tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law $t^{-\alpha}$, where $\alpha=1$ is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast ($\alpha\approx0.55$--$0.7$), untrained ones slowly ($\alpha\approx0.2$--$0.5$). We prove that this offset is $O\big(P(\log t)^{L-1}/t\big)$ for a prompt of length $P$ and depth $L$. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and \emph{relay} --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the \emph{conditional no-hair theorem}: given the text generated so far, the prompt's influence on the next token vanishes as $t$ grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards $1$ seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.

1 Introduction

The paper measures how prompt-specific information fades from a transformer’s hidden state and explains the decay through attention dilution and relay. It reports faster forgetting in trained models, proves an asymptotic bound under architectural conditions, and tests interventions that compensate for dilution.

1 Introduction

  • Training accelerates forgetting relative to random-weight models, while compensating dilution with a prompt logit bias removes or reverses forgetting.On Pythia, forgetting appears within the first billion training tokens, before attention sinks form; attention weights alone are a misleading proxy.Source: §1 · pp. 2–3
  • The prompt offset decays approximately as t^-α; trained Pythia follows a close-to-pure power law, while untrained networks show logarithmic corrections.The spread across continuations stays essentially constant, so a single exponent characterizes forgetting in most settings.Source: §1 · p. 2
  • The theorem bounds the offset for finite-depth pre-LN transformers with non-degenerate pre-norm activations, and uniform attention attains the bound asymptotically.The offset is bounded up to logarithmic factors, with at most one log factor per relay round.Source: §1 · p. 2
  • A direct-and-relay decomposition reproduces the measured decay exponent within 0.06 across layers and training stages.Direct influence comes from queries reading the prompt; relay comes from queries reading continuation tokens that already carry prompt information.Source: §1 · p. 2
  • Mild log t compensation increases instruction adherence at a small perplexity cost but induces instruction echoing.In multi-turn conversations, adherence trends higher, but no more than with a constant boost using the same mean bias.Source: §1 · p. 3

2 Related work

The paper connects prompt forgetting to work on instruction drift, information flow, and attention decay, while distinguishing hidden-state memory from output behavior. Its results also qualify explanations based on attention sinks or long-range statistics of text.

Instruction drift

  • The prompt’s share of uniform attention shrinks as 1/t, providing a quantitative geometric explanation for instruction drift [Li et al., 2024].The authors report that their causal experiment agrees with the proposed attention-decay cause.Source: §2 · p. 3

Attention sinks and over-mixing

  • The onset of forgetting precedes attention-sink formation by several billion tokens, so sinks cannot be its initial cause [Xiao et al., 2024, Gu et al., 2025].The cited work argues that sinks prevent over-mixing [Barbero et al., 2025].Source: §2 · p. 3

Information flow

  • The theorem quantifies loss of sensitivity to a fixed prompt as generation proceeds, and the linear relay model shows the t^-1 rate cannot be improved under uniform attention [Barbero et al., 2024].The path decomposition is motivated by norm-based attention analysis [Kobayashi et al., 2020], attention flow [Abnar and Zuidema, 2020], and rank collapse [Dong et al., 2021].Source: §2 · p. 3

Long-range statistics of text

  • An untrained network already exhibits power-law forgetting, with exponents unchanged to within 0.01 when continuation sentences are shuffled, ruling out corpus statistics as the source of that power law.In trained networks, the exponent depends on continuation text, but the ordering of axes and trained–untrained contrast do not.Source: §2 · p. 3

Prompt influence on generated text

  • The paper measures prompt influence in hidden states, enabling layer-wise tracking and separation of direct influence from relay, unlike work measuring output distributions [Marklová et al., 2026].Its question complements work on which sequences transformers can generate [Meyer et al., 2026].Source: §2 · p. 4

Power-law decay of attention

  • Attention weights are an unreliable proxy for prompt-specific information because sinks can attract weight without carrying that content.The hidden-state trace determines whether the prompt remains readable.Source: §2 · p. 4

Attention fading and length-dependent scaling

  • The dilution analysis predicts that length-scaled logits, including Scalable-Softmax, retain prompt information longer, but unbounded logits fall outside the lemma’s assumptions [Nakanishi, 2025].The cited method multiplies logits by s log n.Source: §2 · p. 4

3 Setup and measurement

The study isolates prompt influence by comparing hidden states along shared continuations, then measures its signal relative to continuation-induced variation. It estimates decay across prompt pairs, continuations, model families, and training checkpoints.

3 Setup and measurement

  • The setup asks what hidden-state difference remains after generation and how quickly it fades, requiring a controlled comparison, a measure of readability, and reliable decay-rate estimates.The paper distinguishes the network’s residual-stream interior from the generated-text exterior, which an observer can consult.Source: §3 · p. 4, §3.1 · p. 4

3.1 What to compare

  • Holding continuation text fixed makes any remaining hidden-state difference attributable to the prompt, and teacher-forcing supplies that shared text for measurement.Teacher-forcing is a measurement device; free generation is the setting where prompt influence can travel through text it has caused.Source: §3.1 · pp. 4–5

3.2 What to measure

  • The prompt offset is the difference between hidden states for two prompts followed by the same continuation; its signal is compared with hidden-state spread across continuations.The signal is the offset component common across continuations, while the noise is continuation-induced spread, measured in pooled first-window standard-deviation units.Source: §3.2 · p. 5
  • The signal and noise decay exponents are α and β, while decodability scales as t^-2(α-β); β is approximately zero almost everywhere.Late-time signal estimates correct for sampling bias; comparing signal with mean squared offset indicates how much decay reflects decorrelation rather than shrinking norm.Source: §3.2 · p. 5, §3.3 · p. 6

3.3 Estimating decay rates

  • Measurements use octave-spaced windows centered in time because end-labeling biases fitted exponents upward; hand-written-scene late-time exponents are biased low by a plug-in estimate.The study uses eight continuations per real-corpus pair and sixteen per scene; global and local exponents are estimated from log-time slopes.Source: §3.3 · pp. 5–6

3.4 Prompts and continuations

  • Prompt differences span word changes, paraphrases, genre rewrites, translations, and unrelated pairs, with shared human-written continuations for each prompt pair.Hand-written scenes use sixteen continuations each; real-corpus pairs use eight shared WikiText-103 continuations per pair.Source: §3.4 · p. 6, Hand-written scenes · pp. 6–7

3.5 Models

  • Pythia checkpoints track training in a fixed architecture, while Qwen2.5 tests a different architecture using trained and randomly initialized models.The study includes Pythia-160m at eight checkpoints, plus Pythia-410m, Pythia-1b, and Qwen2.5 models.Source: §3.5 · p. 7

4 A conditional no-hair theorem

This section bounds how prompt-dependent hidden-state offsets decay under a shared continuation in finite-depth pre-LN transformers, with direct prompt attention diluted by context growth and relay adding at most logarithmic factors. It also presents an exactly solvable model whose long-time local exponent approaches 1 and whose fits reproduce untrained Pythia curves and estimate layer gains.

4.1 Setting and notation

  • The proof considers L-layer pre-LN decoders and assumes pre-norm activations stay a positive distance from zero; empirical checks do not establish this condition for arbitrary contexts.The architecture scope excludes post-LN blocks, linear attention, and recurrent state.Source: §4.1 · pp. 7–8

4.2 Dilution

  • A continuation query’s attention to a prompt of length P is at most Pe^(2B)/(P + i), while the layer recursion includes both direct prompt influence and relay of earlier offsets.The relay term averages offsets at earlier positions; only the direct prompt contribution necessarily shrinks with distance i.Source: §4.2 · pp. 9–10

4.3 The conditional no-hair theorem

  • Under Assumption 1, the final-layer prompt offset is bounded by O(P(L + ln t)^(L−1)/t), with one possible logarithmic factor per relay round.The architecture-dependent constant may grow exponentially with depth, and the theorem fixes the asymptotic shape rather than finite-window fitted exponents.Source: §4.3 · pp. 10–11

4.4 The exterior carries the hair

  • For every shared history, the next-token distributions differ by at most a bound of order P(L + ln t)^(L−1)/t, but the result does not rule out cumulative prompt influence over long horizons.Over a short window, the next tokens are determined up to the accumulated stepwise bound by the generated text, whichever prompt produced it.Source: §4.4 · p. 11

4.5 An exactly solvable model

  • In the uniform-attention linear relay model, the bound’s l−1 logarithmic powers are attained and cannot be lowered; arbitrary layer gains also admit an exact solution.The model also supplies a parameter-free prediction for untrained networks.Source: §4.5 · pp. 12–13
  • In the solvable model, αloc approaches 1 as t grows, but at fixed t it does not increase with accumulated gain, so deeper layers can have lower finite-window exponents.As the observation window expands, the measured exponent of an untrained network creeps toward 1.Source: §4.5 · p. 14
  • Untrained Pythia matches the model’s first-block predictions, whereas untrained Qwen matches neither; deeper-layer finite-window powers depend on gains, not just layer index.The Pythia layer-2 fitted power is 1.07–1.33; the observed k ≈ l − 1 coincidence in layers 2–4 is not a general law.Source: §4.5 · p. 14
  • All 15 untrained Pythia layer curves are reproduced by the exact solution with R2 ≥0.992, though local-exponent deviations reach 0.08 in layer 15.The layer-1 fit gives P = 28.8 versus a mean prompt length of 25.7 tokens.Source: §4.5 · p. 15
  • The fitted first gains are equal at 0.66, and later gains decrease approximately as l^-1.4 from layers 4–15, reaching 0.02 in layer 15.The first two gains cannot be distinguished because the curves are symmetric in them; the fitted accumulated gain at layer 15 is 2.18.Source: §4.5 · p. 15

5 Forgetting is close to a power law

Across Pythia and Qwen, prompt offsets follow power-law-like decay: training yields deep-layer exponents around 0.52–0.70, while untrained local exponents rise toward 1 over time. The section tests this pattern across prompt types, models, offset measures, and continuation structures, identifying departures in untrained and late-time Qwen.

5.1 Hand-written scenes

  • In hand-written scenes, trained Pythia layers 1–2 have α of 0.9–1.05, while deeper layers settle at 0.57–0.70; untrained matched-layer exponents agree across model widths.Layer 1 is direct dilution without relay; the untrained local exponent rises over time, while continuation noise remains approximately constant for Pythia.Source: §5.1 · pp. 15–16

5 Forgetting is close to a power law

  • After training, deep-layer exponents are 0.52–0.70 in Pythia real-corpus pairs and 0.52–0.70 in Pythia-160m; untrained local exponents rise toward 1 across layers.The real-corpus pattern also appears in Qwen, whose trained deep-layer exponents are 0.54–0.63; untrained Qwen local exponents rise with time as well.Source: §5.1 · p. 17, §5.2 · pp. 18–19

5.2 Real-corpus prompt pairs

  • On real-corpus pairs, trained Pythia layers 1–2 remain at 0.8–1 and deeper layers at 0.52–0.70; paraphrase and language are forgotten most slowly.The trained profiles are similar across Pythia sizes and Qwen, while untrained Pythia sizes agree at matched layers; prompt-difference size affects offset amplitude, not its decay exponent.Source: §5.2 · pp. 18–20

5.3 Signal versus mean squared offset

  • After training, the mean-squared offset decays more slowly than the signal in deep Pythia and Qwen layers, indicating that offset directions decorrelate across continuations.In untrained Pythia, signal and mean-squared-offset exponents instead agree within 0.01 across layers.Source: §5.3 · p. 20

5.4 Where Qwen differs

  • Untrained Qwen departs from Pythia’s linear-relay picture in its first blocks, while retaining rising local exponents; the proposed block-structure explanation remains untested.Its continuation noise also grows slowly at initialization, changing the stated decay rate of prompt decodability.Source: §5.4 · p. 20
  • Trained Qwen shows late-time slowdown: tests leave estimation bias, a prompt-specific attention sink, and long-range continuation structure as excluded explanations.In Qwen2.5-0.5B, the layer-1 slowdown tracks positional oscillations of the direct attention path around 1/t dilution; the rotary-frequency link remains open.Source: §5.4 · pp. 20–21

5.5 Continuation structure

  • Shuffling continuation sentences leaves untrained exponents unchanged and alters trained layers 1–4 by at most 0.04, but trained deep layers generally forget more slowly.The authors suggest coherent text may supply recent content that displaces the prompt, but they have not tested this mechanism.Source: §5.5 · pp. 21–22

6 Mechanism: direct path and relay

The offset’s attention writes divide into direct reads from prompt tokens and relay reads from continuation tokens that already carry prompt information. Across training, relay initially sustains the late offset, then begins to leak as forgetting sets in before first-layer recency bias or attention sinks appear.

6.1 The decomposition

  • At initialisation, relay carries 73–97% of the late offset and barely decays, while the direct path has αdirect ≈0.68–0.70 across layers.The relay exponent falls from 0.18 in layer 2 to 0.01 in layer 11; the long tail in untrained networks is relay.Source: §6.1 · p. 22, Figure 4 · p. 23
  • At step 512, αrelay ≈0.38–0.52 and αdirect ≈0.82–0.96 in layers 2–11, while relay still carries 0.55–0.88 of the late offset.Training makes relay leak and steepens the direct path.Source: §6.1 · p. 22
  • After training, the component exponents do not sum to the measured exponent: layers 6–11 have αdirect ≈0.39–0.41 and αrelay ≈0.19–0.23, but the total has α ≈0.61–0.63.A cross term, with components anti-aligned at late times, means only the sum is reliable.Source: §6.1 · pp. 22–23

6.2 When forgetting appears in training

  • Forgetting is strongest at step 512 and relaxes slightly in deep layers afterward; by the end, deep-layer α is 0.58–0.66 and roughly flat beyond layer 3.At onset, attention entropy falls beyond layer 1 while layer 1 stays near 0.98.Source: §6.2 · pp. 23–24, Figure 5 · p. 25

6.1 The decomposition

  • The direct/relay decomposition reproduces measured α within 0.06 at every layer and checkpoint, while raw or value-weighted prompt attention is misleading outside steps 512–1000.Later, attention sinks dominate the weights despite carrying no prompt-specific information.Source: §6.1 · p. 23, Table 6 · p. 24

6.2 When forgetting appears in training

  • Forgetting begins between steps 128 and 512, when relay starts to leak and deeper-layer exponents rise before first-layer recency bias or attention sinks appear.The first layer remains in the untrained regime until step 1000; token-0 attention mass stays at or below 0.003 through step 1000.Source: §6.2 · pp. 23–24

7 Causal test: compensating dilution

The authors test whether dilution drives forgetting by boosting prompt-token logits to counteract the shrinking prompt share of attention. Compensation removes or reverses measured forgetting, with the interpretation bounded by a large perplexity cost after training.

7.1 The intervention

  • The intervention boosts prompt logits to compensate for the 1/t shrinkage of the prompt’s attention share, uniformly or with per-head calibration.Per-head gains are fitted so each head’s attention to a typical prompt token stops decaying.Source: §7.1 · p. 25

7.2 What the intervention removes

  • After training, uniform compensation leaves α = 0.11–0.29, while per-head compensation reverses decay; at initialisation and step 512, compensation removes forgetting.Per-head compensation brings α to 0.05–0.13 at steps 0 and 512, then overshoots after training so prompt influence grows.Source: §7.2 · p. 26, Figure 6 · p. 26

7.3 What the test does and does not show

  • Because per-head compensation raises perplexity by 140% or more, the test shows dilution is necessary for measured forgetting in these networks, not that the compensated network behaves as the original does.Sufficiency is instead shown by untrained networks and the linear relay model, where uniform attention alone produces decay.Source: §7.3 · p. 26
  • In preliminary hand-written-scene runs, model-generated continuations gave the same α as human-written ones within 0.1, so the measured rate was not inherited from the model’s own text statistics.The reported real-corpus results use human-written continuations.Source: §7.3 · p. 26

8 Exterior: what survives in the generated text

In free generation, later text depends on the prompt through the trajectory it has already caused, rather than through a shared continuation. The results find that language remains decodable from generated text while details, paraphrase, and genre do not.

The probe

  • The observer tests prompt identity using a text-only probe on hashed unigram histograms from generated-token windows, with chance accuracy 0.5.The probe uses logistic regression and five-fold stratified cross-validation on freely generated trajectories.Source: §8 · p. 27

What survives

  • At t = 512, language is decodable from generated text at 0.81 for Qwen2.5-0.5B and 0.84 for Pythia-160m, while details, paraphrase, and genre become undecodable.Language is not always conserved: 4–11 of 32 Qwen trajectories per scene switch language, mostly toward Chinese.Source: §8 · p. 27

Why language survives

  • Language survives through the generated trajectory: the prompt tilts early tokens toward its language, and later tokens follow a history already written in that language until a chance switch.The teacher-forced offset for the language axis still decays, so this persistence is not memory of the prompt in the hidden state.Source: §8 · p. 27

9 Application: a dilution knob, and its costs

The paper tests prompt-logit boosts as a way to counter dilution and increase retention. Mild boosts improve instruction adherence with trade-offs, while matched-mean constant boosts perform comparably to log t boosts in multi-turn tests.

9.1 The trade-off

  • A mild prompt boost of s = 0.5 lowers α by 0.17–0.25 for +2.6% perplexity on the tested real-corpus pairs.At matched retention, prompt boosting costs less than the placebo boost; at matched strength, it costs more on these pairs.Source: §9.1 · pp. 27–28, Table 9 · p. 28

9.2 Instruction adherence and its side effects

  • On Qwen2.5-0.5B-Instruct, the share of samples fully in French after 2048 tokens rises from 31% to 94% at s = 0.5.For Qwen2.5-1.5B-Instruct, the share rises from 84% to 94–100% for all boosts; a constant boost with the same mean bias does as well, with less echo.Source: §9.2 · p. 28
  • In multi-turn conversations, log t boosts trend toward higher French adherence, but show no clear advantage over matched-mean constant boosts.With 32 conversations per condition, these differences are trends rather than significant effects; the benefit is attributed to extra attention to the system prompt.Source: §9.2 · p. 28
  • Under forced-length single-turn generation, instruction echo rises from 19% to 56–69% of outputs for the 1.5B model.Much of the added instruction discussion is filler after the story has ended.Source: §9.2 · p. 28
  • Across the instruction tests, the supported pattern is that extra attention to the instruction helps, not the specific shape of the correction.The multi-turn comparison finds no clear advantage for log t over a matched constant boost.Source: §9.2 · p. 28

10 Discussion and conclusion

The prompt-induced hidden-state offset follows a measurable power-law decay, while geometric dilution and relay explain its asymptotic behavior. Conditional on the generated text, the prompt’s influence vanishes; the findings remain subject to theoretical and empirical scope limits.

10 Discussion and conclusion

  • The prompt-induced hidden-state offset decays approximately as a power law, with a bound attained asymptotically under uniform attention.The exactly solvable linear relay model has one gain per layer and accounts for slow, depth-dependent forgetting in untrained networks.Source: §10 · pp. 28–29
  • Prompt influence is diluted as context grows, while relay through continuation tokens adds at most logarithmic factors.Trained networks suppress relay within the first billion training tokens and come close to pure dilution.Source: §10 · p. 29
  • Compensating dilution removes or reverses forgetting; attention weights alone do not reliably measure prompt influence, and a log t-shaped boost is no better than a constant boost.Extra attention to the instruction, rather than the correction’s shape, helps keep it alive.Source: §10 · p. 29
  • The theorem is informative only for t ≫ Pe2B, its constant can grow exponentially with depth, and larger-scale or instruction-tuned behavior remains open.The core measurements stop at Pythia-1b, and the behavioral tests at Qwen2.5-1.5B.Source: §10 · p. 29
  • Conditional on generated text, the prompt is forgotten, so surviving properties such as language are carried by the text itself.The paper describes this as the conditional no-hair conclusion.Source: §10 · p. 29

A Proofs

The appendix supplies proof details, including softmax bounds along a path and bookkeeping that extends single-head constants to multi-head blocks. It also identifies the role of normalization-layer gains in those constants.

A Proofs

  • The proof appendix expands Section 4 with a softmax perturbation lemma, multi-head constant bookkeeping, position alignment, and a quantitative version of Corollary 2.It also gives a complete proof of Lemma 3.Source: A Proofs · p. 30

A.1 Softmax along a path

  • With logits bounded in magnitude by B, softmax weights along the interpolation path are at most e2B/n.The lemma’s proof differentiates the softmax along the path and integrates over θ from 0 to 1.Source: §A.1 · p. 31

A.2 Constants of a multi-head block

  • The multi-head constants combine per-head bounds without a hidden factor H, with normalization-layer gains absorbed into adjacent weight matrices.The appendix defines combined constants from the per-head quantities and applies Lemma 1 head by head.Source: §A.2 · p. 31

A.3 Positions

Rotary encodings make continuation-token attention logits depend on relative positions, allowing alignment across runs with different prompt lengths. The proof also states which positional and attention setups fall within its scope.

A.3 Positions

  • With rotary encodings, a continuation query-key logit depends on positions through their distance, so it matches across runs even when prompt lengths differ.The relative offset is the same in both runs because continuation positions shift with their respective prompt lengths.Source: §A.3 · p. 31
  • For absolute position embeddings, taking equal prompt lengths restores zero offset at continuation positions.Otherwise the continuation tokens carry different position embeddings across runs.Source: §A.3 · p. 31
  • The proof covers full or partial rotary encodings and several attention and normalization variants, but not unbounded-distance logit biases, growing logit scalings, sliding-window, or non-softmax attention.These exclusions arise because the bounded-logit condition or the stated dilution lemma fails.Source: §A.3 · p. 32

A.4 Proof of Lemma 3

The proof of Lemma 3 separates prompt and continuation contributions to an attention head’s output, then bounds their effects and propagates them through residual and MLP blocks. Its bounds rely on bounded logits and non-degenerate normalization inputs.

A.4 Proof of Lemma 3

  • The head output is decomposed into prompt and continuation averages, weighted by the prompt attention mass π and its complement.The continuation weights are normalized over the i continuation keys.Source: §A.4 · p. 32
  • The prompt contribution is bounded by a term proportional to Pe2B/i, while continuation differences depend on earlier hidden-state offsets.The continuation attention weights are bounded by e2B/i, producing a sum over prior continuation offsets.Source: §A.4 · p. 32
  • Residual and MLP propagation uses a Lipschitz bound on non-degenerate inputs; inserting the attention bound yields constants K1, K2, and K3.For parallel blocks, the stated offset bound is smaller than for sequential blocks.Source: §A.4 · p. 32

A.5 A quantitative form of Corollary 2

The appendix quantifies behavior of the linear relay model's series terms, including their large- and small-parameter regimes. It also specifies when the local-exponent approximation applies and reports numerical checks of the inequalities.

A.5 A quantitative form of Corollary 2

  • Proposition 3 applies the quantitative bounds to the linear relay model using G = Gl, Q = Ql, and u = ln P+i.Corollary 3 states a bound on the weight beyond depth l whenever Gu < (l + 1)(l + 2).Source: §A.5 · p. 33
  • The inequalities were checked against the exact recursion in 1,800 random checks, with no violations.The checks varied depth, prompt length, gains, and i; they do not establish a proof.Source: §A.5 · p. 33
  • For z ≫ 1, the series terms concentrate around k ≈ √z, with m1 ≈ √z and m2 ≈ z.The relative-error expression is not recoverable from the supplied passage.Source: §A.5 · p. 33
  • For z ≲ 1, w1 dominates and corrections are small, but the local-exponent asymptotic expansion is invalid.The fitted Gl values in Section 4.5 are therefore effective parameters; coverage of model local exponents requires δ to approximate φ uniformly over an octave.Source: §A.5 · p. 34

B Non-degeneracy of pre-norm activations

The appendix reports empirical checks of the non-degeneracy assumption, describes the prompt sets, and outlines a toolkit for measuring prompt offsets and related quantities.

B Non-degeneracy of pre-norm activations

  • Across the tested sequences, pre-norm activations stayed away from zero, supporting Assumption 1 empirically but not proving it for other continuations or longer contexts.For Pythia-160m, r/d was 0.112 at initialization and 0.236 when trained; Qwen2.5-0.5B's minimum post-attention norm was 0.012.Source: B Non-degeneracy of pre-norm activations · p. 34

C Prompt sets

  • The prompt set includes twenty hand-written scenes, each followed by the same sixteen WikiText-103 validation passages.Real-corpus pairs are described in Section 3.4, and the exact pairs are released with the toolkit.Source: C Prompt sets · p. 34

D Toolkit

  • The single-laptop toolkit measures prompt offsets under shared continuations, checks non-degeneracy, analyzes attention-write dilution, and tests instruction adherence.It also includes retention–perplexity trade-offs with a placebo control and bootstrap intervals over prompt pairs.Source: D Toolkit · p. 37
Loading 2610.24133v1…