PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data

Marios Papamichalis, Regina Ruane

arXiv:2608.14712v1cs.CLcs.AIcs.LGmath.ST

TL;DR

Attention-row comparisons conflate sink allocation with content allocation under an often-unreported keep-or-drop convention. This paper separates these channels using compositional geometry and finds that 17–47% of closer-head verdicts flip across ten pretrained models, while clustering structure can be convention-driven.

  • Problem

    Attention analyses rarely report whether comparisons keep the dominant sink token or drop and renormalize it, although the choice changes the question being measured.

  • Method

    The paper treats attention rows as compositional data, separating sink and content channels through Aitchison distance, entropy identities, and transformer-symmetry invariances.

  • Results

    17–47% of closer-head verdicts flip across ten pretrained models, and a BERT clustering structure disappears across conventions, with cross-convention ARI as low as 0.01–0.05.

  • Takeaways & Limitations

    Separating sink from content allocation distinguishes convention-dependent comparisons from content-based diagnostics and shows that measured entropy collapse is often mostly sink growth.

  • Takeaways & Limitations

    The reported scale law rests on one model family, and the frozen-regime test produced one confirmation, one abstention, and one failure.

Abstract

from arXiv · show

Each row of a transformer's attention matrix is a probability distribution over tokens, and in trained models most of that probability lands on a single \emph{sink} token, usually the first. Standard tools for comparing attention rows (cosine similarity, Jensen--Shannon divergence, Shannon entropy) therefore hinge on a choice papers rarely report: keep the sink, or drop it and renormalize. This choice can reverse conclusions. On ten pretrained models from five families, 17--47% of verdicts about which of two heads is more similar flip with the convention, and the most prominent structure in a standard BERT head-clustering pipeline is an artifact of it. The reason is that one-number summaries mix two questions: how much attention the sink takes, and how the rest is divided among the content tokens. Treating rows as compositional data separates them exactly: the Aitchison distance splits orthogonally into a sink term and a content term, entropy splits by an exact identity, and the content distance is characterized by invariances the transformer itself possesses. The separation matters in practice: most measured entropy collapse during training is the sink growing, not attention sharpening (30% of the drop at 70M parameters, 95% at 1B, 79% at 1.4B), and pruning heads with the wrong channel can inflate perplexity more than a hundredfold. We map where each convention is safe, test a frozen out-of-sample predictor (one confirmation, one abstention, one failure), and release code regenerating every number.

1 INTRODUCTION

Attention rows are compositions whose dominant sink token makes common similarity and entropy summaries depend on an often-unreported keep-versus-drop convention. The paper separates sink allocation from conditional content allocation and develops exact decompositions and invariance-based tools for analyzing them.

  • 1 INTRODUCTION: Attention rows are probability distributions on the simplex, so they carry relative information and can be analyzed as compositional data.This frames transformer attention within Aitchison’s compositional-data framework.
  • 1 INTRODUCTION: Roughly 80% of attention mass in a typical Llama-3.1-405B prompt sits on the ⟨bos⟩ token, making sink inclusion or exclusion consequential for comparisons.The fraction of sink heads also grows with model scale.
  • 1 INTRODUCTION: Full-row statistics measure total allocation, whereas sink-dropped statistics measure allocation conditional on ignoring the sink; both are legitimate estimands.The failure arises when one-number summaries mix these questions under an unstated convention, affecting taxonomies, pruning, and stability monitoring.
  • 1 INTRODUCTION: The paper proves that keep-versus-drop conventions can reverse cosine, JS, and Euclidean closer-head verdicts, then provides exact Aitchison and entropy decompositions.It also states transformer-nuisance invariances and a five-axiom characterization, including a quotient version for content distance.

2 RELATED WORK

Prior attention analyses use JS divergence, cosine similarity, entropy, and taxonomies, but do not address sink conventions across similarity, diagnostics, taxonomies, and pruning. This work introduces log-ratio geometry to attention analysis and separates sink and content channels, unlike Fisher–Rao geometry.

  • Attention-analysis metrics: Prior work uses summed JS divergence, cosine similarity, entropy, and attention-pattern taxonomies, without studying sink conventions across these analyses.Clark et al. cluster head–head distances with summed JS divergences; cosine similarity is widely used, while other work studies entropy and attention patterns.
  • Measurement gap: Sink-convention diagnostics, taxonomies, and pruning remain unaddressed by prior attention-analysis metrics.The passage identifies these applications as part of the measurement problem addressed here.
  • Information geometry: Fisher–Rao geometry has different invariances, is not perturbation-invariant, and does not decompose sink from content.These properties distinguish it from the geometry required by the sink problem.
  • Novelty: This is the first work to bring log-ratio geometry to attention-analysis measurement and to separate sink and content channels within it.The claimed novelty concerns both the methodological setting and the channel decomposition.

3 AITCHISON GEOMETRY OF ATTENTION ROWS

Attention rows are compositions, so Aitchison geometry provides a principled vector-space representation and separates analyses that remove the sink through subcomposition. Its distance remains coherent under sink removal, unlike cosine, Jensen–Shannon, and Euclidean distance, while masking and structural zeros require common-support comparisons and zero replacement.

  • Aitchison representation: A softmax attention row lies in the simplex, whose perturbation and powering operations form the Aitchison compositional geometry.The simplex is treated as a (D−1)-dimensional real vector space with a uniform identity and compositional inverse.
  • Aitchison representation: The centered log-ratio and isometric log-ratio maps identify compositions with Euclidean coordinates, inducing the Aitchison inner product, norm, and distance.The ILR map uses any orthonormal contrast basis, while Aitchison geometry is Euclidean after clr or ilr transformation.
  • Subcompositions and coherence: Subcompositionally, removing and renormalizing selected parts is an orthogonal projection, so Aitchison distance is dominant and preserves log-ratios among retained parts.This formalizes sink masking: conclusions about retained parts should not depend on whether analysis used the full composition or its subcomposition.
  • Subcompositions and coherence: Cosine, JS, and Euclidean distance are neither subcompositionally coherent nor dominant, and none is stable under sink removal.The instability follows because these measures do not preserve conclusions when the sink column is discarded and the remaining row is renormalized.
  • Sink/content notation: Each row factors as p = (s, (1 −s)π), with sink mass s and content composition π equal to the sink-dropped, renormalized row.The sink coordinate is indexed as 0, while content coordinates run from 1 to D−1; multiple sinks can be grouped identically.
  • Structural zeros and masking: Structural zeros from causal masking are handled on the common unmasked support, while residual numerical zeros use multiplicative replacement with ε = 10−6.The protocol also fixes key support across aggregated query positions and reports sensitivity analyses.

4 THEORY

The theory shows that retaining versus removing the sink can reverse similarity and entropy conclusions because these metrics conflate sink mass with content composition. Aitchison geometry separates the two channels, is characterized by transformer-relevant invariances, and explains why classical metrics lose dynamic range in sink-dominated heads.

  • Sink-driven ranking reversal: All three classical similarity verdicts reverse in the exact witness, while Jensen–Shannon and Euclidean distances also violate subcompositional dominance.The witness uses p = (0.90, 0.08, 0.02), q = (0.70, 0.24, 0.06), and r = (0.90, 0.02, 0.08).
  • Entropy inversion: Entropy comparisons can invert after sink removal because H(p) = H_b(s) + (1−s)H(π), separating sink strength from content spread.In the D = 3 witness, H(P) ≈ 0.233 < 0.697 ≈ H(Q), but H(P′) = log 2 > H(Q′) ≈ 0.0079.
  • Pythagorean sink decomposition: Aitchison geometry decomposes full-row distance orthogonally into sink balance and content distance, with the content term unchanged when sink mass varies at fixed content composition.The content distance is identical in balance-orthogonal full-row coordinates and after dropping the sink and re-closing.
  • Metric characterization: The Aitchison distance is uniquely characterized up to scale by metricity, shared-logit-bias invariance, temperature homogeneity, the parallelogram law, and permutation invariance.The theorem states δ = c d_A for some constant c > 0.
  • Dynamic-range collapse: For heads with sink mass ≥0.8, cosine similarities exceed 0.94, Euclidean distances stay below 0.29, and JS stays below 0.14 nats, compressing classical dynamic range.Llama-3.2-1B’s mean sink mass is 0.84, making within-band orderings and cluster relationships comparable to estimation noise.

5 EXPERIMENTS

Experiments across synthetic scenarios and ten pretrained models show that sink conventions materially alter similarity, entropy, clustering, and pruning conclusions. Separating sink and content channels identifies when each convention is reliable and explains most measured entropy collapse as sink growth.

  • Convention-dependent similarity: 36/39/41% of cosine verdicts, 23/31/40% of JS verdicts, and 27/37/45% of Euclidean verdicts disagree at mean sink masses 0.5/0.7/0.9.Disagreement grows with sink mass toward the 50% chance level.
  • Pretrained-model measurements: 17–23% reversal rates on ViT rise with signature sink mass across ten pretrained models and five model families.Measured row-level sink masses span 0.04 for ViT CLS to 0.86, while signature masses concentrate further around shared sinks.
  • Redundancy pruning: On Llama-3.2-1B, with-sink JS perplexity worsens from 15.3 to 2094±140, while dropped-row JS reaches 70.5±9.6 and d⊥ reaches 63.6±4.1.On Llama-3.2-3B, dropped-row JS wins outright at 25.3 versus 40.2; full dA fails like JS at 2480±211.
  • Redundancy pruning: d⊥ A-L1 is best on every strong-sink model, reaching 45.0, 26.9, 23.4, and 0.581 on Llama-1B, Qwen, Llama-3B, and ViT, respectively.Classical criteria fail severely on at least one strong-sink model, while d⊥ A-L1 fails on GPT-2 with perplexity 449.
  • Collapse diagnostics: 30/53/70/95/79% of entropy-drop magnitude across 70M–1.4B is not attributable to a fall in content entropy, reaching 95% at 1B.In a synthetic sink-growth scenario, Shannon entropy reports 70% collapse while content log-ratio dispersion remains flat; when content collapses and the sink recedes, entropy barely moves.

6 CONCLUSION

The paper concludes that attention rows should be treated as compositions: standard summaries conflate sink and content questions, while exact separation changes empirical conclusions and clarifies functional relevance. It also identifies resampling stability and future extensions as remaining considerations.

  • Conclusion: Attention-row summaries mix sink and content questions under an unreported convention, so separating the channels changes published-style conclusions.The reported effects include near-chance verdict flips, convention-driven clustering structure, and collapse that is mostly sink.
  • Conclusion: The separated channels map when each attention component matters functionally, while Theorems 1–2 make measurement claims rather than broader theoretical claims.
  • Limitations and future work: A-L1 selects unstable sets across resampling, with audits performed at the signature level rather than per row.
  • Limitations and future work: Future work includes fitting d2 and extending the analysis with held-out functional evidence, per-row audits, and value-weighted extensions.

A WORKED EXAMPLE AND THE GEOMETRY OF THE REVERSAL

The worked example shows that Aitchison distance separates sink-balance and content discrepancies exactly, while cosine, JS, and Euclidean distances mix them and can reverse similarity verdicts. For a fixed anchor pair, cosine is convention-dependent across 77.9% of the simplex, whereas total and content Aitchison distances disagree on only 0.1% of triples.

  • Worked example: The exact witness uses p = (0.90, 0.08, 0.02), q = (0.70, 0.24, 0.06), and r = (0.90, 0.02, 0.08) over (sink, a, b).Its parameterization has π = (0.8, 0.2), ˜π = (0.2, 0.8), s = 0.90, and s′ = 0.70.
  • Geometry of the reversal: Aitchison distance decomposes exactly: dA(p, q)2 = 1.214868 is entirely sink balance, while dA(p, r)2 = 3.843624 is entirely content.The sink balances are b(p) = b(r) = 2.5422 and b(q) = 1.4400, with the decomposition verified to machine precision.
  • Geometry of the reversal: Cosine, JS, and Euclidean distance mix sink and content components in a renormalization-dependent way, enabling the reversal.The example’s (p, q) discrepancy is sink-only, whereas (p, r) is content-only.
  • Flip region: 77.9% of the simplex yields a convention-dependent cosine verdict when Figure 6 fixes (p, r) and sweeps q.Near strong sinks, convention-dependence is described as typical rather than exceptional.
  • Estimands: 0.1% of triples produce different closer-head judgments between total Aitchison distance and content distance d⊥, at every sink level.These are distinct declared estimands: sink-inclusive analyses should report dA, while content-focused analyses should report d⊥.

B PROOFS FOR SECTION 4.1

The proofs formalize sink deletion as a compositional transformation: the induced Aitchison distance is a pseudometric whose quotient recovers the re-closed content simplex, but full-row rankings can change. They also establish that entropy comparisons can invert after sink removal, while fixed-content distances remain independent of sink mass.

  • Aitchison geometry: The sink-deletion pullback of Aitchison distance is a pseudometric, becoming a genuine metric on equivalence classes identified by identical re-closed content.Its metric quotient is isometric to the sink-removed simplex with its Aitchison distance.
  • Aitchison geometry: Full Aitchison distance is not generally invariant under sink deletion: dA(ps, q) →∞ as s ↑1, while dA(ps, rs) = B.The comparison dA(ps, q) < B holds only below a threshold sA and reverses above it.
  • Entropy inversion: For every D ≥3, entropy orderings can invert after removing the sink and re-closing the remaining coordinates.The proof selects a sufficiently large sink mass so H(ps0) < H(q), while the sink-removed comparison reverses.
  • Entropy inversion: D = 2 is an exception: every sink-dropped composition equals (1), so H(p′) = H(q′) = 0 for all p, q ∈S2.The entropy inversion result therefore requires D ≥3 and unequal sink masses.
  • Fixed-content invariance: For fixed normalized content π, the sink-deletion Aitchison distance is independent of the mass assigned to the designated sink and equals the CLR norm of the re-closed content.This does not imply invariance when the designated sink coordinate changes.

C PROOFS FOR SECTION 4.2 · FULL STATEMENT OF PROPOSITION 2

The proofs establish an exact orthogonal separation of sink and content variation in Aitchison geometry, with content coordinates independent of sink mass. Proposition 2 characterizes the resulting invariances, equivariances, masking conditions, and subcomposition dominance, while identifying important failures for alternative metrics.

  • C PROOFS FOR SECTION 4.2: The Aitchison distance obeys an exact Pythagorean identity separating sink variation from content variation.The decomposition is orthogonal in CLR space and yields the corresponding distance identity.
  • C PROOFS FOR SECTION 4.2: The remaining ILR coordinates are exactly an ILR system for the sink-dropped composition, so content distance depends only on content distributions.Neither the content term nor its coordinates depend on sink mass; the projection statement concerns CLR space, not Euclidean probability-vector projection.
  • FULL STATEMENT OF PROPOSITION 2: Shared finite logit bias preserves dA but generally changes d⊥A, Euclidean distance, Jensen–Shannon divergence, cosine similarity, and ρA.The content distance is identically zero when D = 2, but is not perturbation-invariant in general for D ≥3.
  • FULL STATEMENT OF PROPOSITION 2: A common temperature scales Aitchison distances and variance rather than preserving their values, while separate positive head temperatures cancel in ρA under common-support, nonuniform signatures.Row-dependent temperatures can change ρA, and no analogous general ranking-preservation claim holds for dA, d⊥A, or Aitchison variance.
  • FULL STATEMENT OF PROPOSITION 2: The content norm is sink-invariant and identical whether computed in full balance-orthogonal coordinates or after dropping and re-closing the sink.Full dA is not sink-invariant because its balance term generally changes when sink masses change.
  • FULL STATEMENT OF PROPOSITION 2: Subcomposition is an orthogonal CLR projection, making dA subcompositionally dominant; masking comparisons require at least two positive parts in the common support.With unequal structural-zero supports, the comparison discards unsupported parts and is not covered by the interior perturbation claim.
  • FULL STATEMENT OF PROPOSITION 2: For every D ≥3, Euclidean distance, Jensen–Shannon divergence, and cosine dissimilarity can increase after subcomposition, so none is subcompositionally dominant.The explicit extension constructs strictly positive counterexamples in every dimension D ≥3.

FULL STATEMENT AND PROOF OF THEOREM 2

Theorem 2 uniquely characterizes, up to a positive dimension-specific constant, the Aitchison distance among metrics satisfying metric, perturbation, scaling, parallelogram, and permutation invariances. Its corollary derives scale, subcompositional, sink-decomposition, temperature, and calibration consequences, while clarifying which require compatibility assumptions.

  • Theorem 2: Theorem 2 states that assumptions (A1)–(A5) force δ_D = κ_Dd_A for a unique κ_D > 0, and every such scaling satisfies the assumptions.The proof reduces the metric to a norm, applies Jordan–von Neumann, and uses permutation irreducibility to force an isotropic inner product.
  • Corollary 2: The corollary establishes scale invariance, subcompositional dominance, shared coordinate-bias and temperature properties, including preservation of pairwise-distance rankings under one common temperature.Cosine similarity is invariant under separate positive powerings, whereas a shared nonconstant coordinate bias generally does not preserve it.
  • Corollary 2: The exact sink decomposition separates full distance into a sink-balance term and an orthogonal content term based on recentered kept CLR coordinates.The content term depends only on content log-ratios and not sink mass; simple deletion of the sink CLR coordinate is not equivalent to the orthogonal projection.
  • Calibration limitations: Cross-dimensional dominance requires a common calibration: separately chosen constants κ_m need not agree, and without compatibility even metrics satisfying (A1)–(A5) can violate dominance.The remark gives δ_3 = d_A and δ_2 = 100d_A as an explicit counterexample and identifies common calibration as a convention, not a fixed-dimensional consequence.
  • Invariance limitations: Shared-bias invariance applies only when both distance endpoints, or all rows and their center, receive the same perturbation; it does not cover arbitrary content bias with a fixed reference.The limitation specifically excludes invariance of one-row concentration or CLR cosine under an arbitrary nonconstant content bias.

D PROOFS FOR SECTION 4.3 … E.1 ARE ATTENTION ROWS COMPOSITIONS? TAIL MASS, RIVAL GEOMETRIES, AND THE CONTENT CHANNEL

The results formalize attention-row analysis around sink mass and conditional content, showing that content-preserving sink changes leave content geometry unchanged while classical distances can disagree with sink removal. Experiments and audits then assess rival geometries, entropy effects, clustering, and pruning consequences.

  • D PROOFS FOR SECTION 4.3: Changing sink masses while preserving conditional content compositions leaves every content-distance matrix and content-concentration value unchanged.This establishes that sink variation can be isolated from content geometry through content-preserving transformations.
  • FULL STATEMENT OF PROPOSITION 3: Sink-dropped content distances equal explicitly computed Aitchison content distances entrywise, so any clustering rule using only that dissimilarity matrix produces identical outputs.The identity applies to deterministic rules with fixed preprocessing, linkage, hyperparameters, tie-breaking, and stopping rules, and also to coupled randomized rules.
  • FULL STATEMENT OF PROPOSITION 3: For a positive-measure open set of triples, cosine, Jensen–Shannon, and Euclidean distances select different unique closest pairs before sink removal but the same pair after removal.Thus sink removal can reverse nearest-neighbor and clustering conclusions simultaneously across all three dissimilarities.
  • FULL STATEMENT OF COROLLARY 1: Entropy decomposes exactly as H(p) = Hb(s) + (1 −s)H(π), so changing sink mass alone can create an apparent entropy collapse while content composition remains fixed.The possible apparent collapse can approach log D, although the supremum is not attained; the content log-ratio dispersion is therefore used as a separate diagnostic.
  • E EXPERIMENTAL DETAILS: ViT-B/16 evaluation uses 2,048 COCO val2017 images and label-free top-1 agreement against the unpruned model.The released script can substitute labeled ImageNet-val data through IMAGENET_DIR.
  • E.1 ARE ATTENTION ROWS COMPOSITIONS? TAIL MASS, RIVAL GEOMETRIES, AND THE CONTENT CHANNEL: Tail coordinates below 10−9 contribute essentially 0%, while Hellinger and Fisher–Rao reversal rankings coincide and content Hellinger selects the same prune sets as dropped-row JS.No coordinate has median signature value below 10−6 in any model, limiting the influence of sub-threshold tails.
  • E.1 ARE ATTENTION ROWS COMPOSITIONS? TAIL MASS, RIVAL GEOMETRIES, AND THE CONTENT CHANNEL: Within the content channel, sink-dropped JS and d⊥A have Spearman ρ ≥0.96 but differ in nearest-neighbor identities for 13 to 27% of heads and can reach cross-ARI 0.33 on Llama-3.2-1B.The paper concludes that conditioning out the sink is the dominant stabilizer, while finer geometry remains metric-dependent.

E.2 CONVENTION STAKES MEASURED: TEMPERATURE AND THE SINK SET

Convention choices materially change attention comparisons and pruning outcomes: temperature and sink-set definitions can alter JS-based verdicts, while Aitchison projections remain invariant and ordered. A frozen dispersion-based regime predictor showed one confirmation, one abstention, and one failure.

  • Temperature: Up to 6.6% of dropped-row JS closer-head verdicts flip across temperatures, while d⊥A rankings, neighbors, and prune sets remain invariant.The JS top-20% prune set retains as little as 62% of its τ = 1 membership; d⊥A invariance was verified on all ten models.
  • The sink set: Between first-token and first-four-token sink definitions, dropped-row JS flips 1–21% of verdicts, whereas d⊥A is uniformly less sensitive.The Aitchison content distances are nested orthogonal projections, with dominance verified without violations across all head pairs in all ten models; JS has no analogous relation.
  • Which regime? A dispersion predictor: Cross-head sink-share dispersion separates regimes: content-regime language models have coefficient of variation ≤0.36, while JS-regime models have ≥0.42.Examples include Llama-3.2-1B at 0.11, Llama-3.2-3B at 0.08, Qwen2.5 at 0.36, GPT-2 at 0.42, and Pythias at 0.43–1.03.
  • Held-out test and outcomes: The frozen predictor confirmed DistilGPT2, abstained on SmolLM2-135M, and failed on OPT-125M.Measured CVs were 0.51, 0.42, and 0.28 respectively; DistilGPT2 matched the predicted JS regime, SmolLM2 fell on the JS side, and OPT showed a third behavior.
  • Exact decomposition of checkpoint collapse: At 70M/160M/410M/1B/1.4B parameters, entropy changes were −0.78/−1.55/−2.11/−1.58/−2.01, while the weighted content term carried the entire fall.The exact decomposition has machine-precision residual, with sink-binary changes of +0.18/+0.26/+0.23/+0.29/+0.25 and weighted-content changes of −0.96/−1.81/−2.34/−1.88/−2.26 nats.

E.3 FULL-SUPPORT CLARK REANALYSIS AND WINDOW ROBUSTNESS

The full-support Clark et al. reanalysis shows that the pipeline’s prominent clustering structure is driven by the sink channel rather than convention-invariant content structure. Window sweeps preserve the phenomenon despite changing the retained support and estimand.

  • Full-support Clark reanalysis: The reanalysis rebuilds Clark et al.’s head-clustering pipeline on 512 exactly 128-token WikiText sequences with shared support and sink set {CLS, SEP}.It uses per-head Aitchison-mean signatures, JS distances, average-linkage clustering, and additional cosine and d⊥ comparisons.
  • Full-support Clark reanalysis: 68 of 144 heads place majority mass on SEP and 12 on CLS, while keep-versus-drop JS clusterings agree at chance with ARI 0.01–0.05.Cosine agreement is likewise only 0.00–0.06 across K ∈{4, 5, 6, 8}.
  • Full-support Clark reanalysis: The with-sink JS clustering recovers the SEP-head block at Jaccard 0.87–0.92, but dropping sinks reduces recovery to 0.46–0.49.Convention-invariant content distances recover the block only at 0.47–0.50, showing the prominent structure is specific to the sink channel.
  • Window robustness: Across W ∈{32, 64, 128}, JS reversal rates shift by at most 2.1 points, while consecutive-window d⊥ distances correlate at Spearman ρ = 0.87–0.98.Retained pre-reclosure mass varies by model: 0.49–0.61 for GPT-2, 0.60–0.71 for Llama, and 0.48–0.67 for Pythia-410M.

E.4 SYNTHETIC EXPERIMENTS (SECTION 5.1) · E.5 PROTOCOL FOR PRETRAINED MODELS (SECTION 5.2) · E.6 PROTOCOL FOR THE PROSPECTIVELY SPECIFIED TESTS (SECTION 5.3)

The protocols test when sink retention changes attention comparisons, using synthetic witnesses, fixed-support pretrained-model measurements, and prospectively specified clustering, scaling, and training-curve analyses. Across these experiments, the sink-separated content distance provides a convention-invariant reference while entropy and other distances can reflect sink variation.

  • E.4 SYNTHETIC EXPERIMENTS (SECTION 5.1): Synthetic rows vary sink mass s and content concentration α, comparing with-sink versus sink-dropped verdicts across cosine, JS, Euclidean, and Aitchison distances.The sampler uses D = 64 content dimensions, with α = 0.1 as the peaked setting and α = 1.0 as the diffuse control.
  • E.4 SYNTHETIC EXPERIMENTS (SECTION 5.1): At mean sink masses 0.5/0.7/0.9, verdict reversals reach 36.4/39.0/41.1% for cosine, 23.4/31.0/39.7% for JS, and 27.3/36.8/44.6% for Euclidean.d⊥ A is 0% by construction, while total-dA-versus-d⊥ A is 0.1% at every grid point.
  • E.4 SYNTHETIC EXPERIMENTS (SECTION 5.1): In collapse simulations, entropy falls 2.71 → 0.82 nats while content dispersion changes 48.13 → 47.98 when sink mass varies, but rises 10.04 → 70.43 when content concentration also changes.The fixed-dispersion scenario isolates sink-driven entropy collapse; the second scenario combines declining sink mass with geometrically decaying α.
  • E.5 PROTOCOL FOR PRETRAINED MODELS (SECTION 5.2): The pretrained-model protocol extracts rows on a common 64-key support, retains the sink column, and aggregates each head using the Aitchison mean.Residual zeros are replaced multiplicatively at ε = 10^-6, with ε ∈ {10^-5, 10^-6, 10^-7} reported in a sensitivity sweep.
  • E.5 PROTOCOL FOR PRETRAINED MODELS (SECTION 5.2): The pretrained measurements sample 50,000 head triples for reversal rates, compare entropy with content dispersion across layers, and prune 10%, 20%, or 30% of heads at matched sparsity.Pruning is evaluated with held-out WikiText perplexity for language models and top-1 agreement with the unpruned model on 2,048 ViT images.
  • E.6 PROTOCOL FOR THE PROSPECTIVELY SPECIFIED TESTS (SECTION 5.3): Prospectively specified taxonomies cluster per-head signatures under both conventions using 1 − cos, JS, Euclidean, d⊥ A, and dA, with stability measured by Adjusted Rand Index.Agglomerative clustering uses average, complete, and single linkage with K ∈ {4, 6, 8}; d⊥ A matrices are asserted equal entrywise at runtime.
  • E.6 PROTOCOL FOR THE PROSPECTIVELY SPECIFIED TESTS (SECTION 5.3): The prospective scale and training tests add Pythia models from 70M to 1.4B parameters and track sink mass, reversal rates, entropy H, content dispersion C⊥ A, and their checkpoint changes.Pythia-160M is evaluated at ten checkpoints using 256 held-out sequences per checkpoint, with transitions shaded when sign(∆H) ≠ sign(−∆C⊥ A).

E.7 MEASURED RESULTS: FULL PRUNING SWEEP AND ε SENSITIVITY

The full pruning sweep shows that pruning outcomes depend sharply on the attention-distance criterion: JS-guided pruning can be catastrophic, while A-guided pruning remains comparatively robust. Reversal rates are bit-identical across ε ∈ {10−5, 10−6, 10−7} under the fixed-support protocol.

  • Pruning sweep: 15.3 →358.0 perplexity makes JS-guided pruning catastrophic on Llama-3.2-1B at 10% sparsity.The result is reported for the complete 10/20/30% pruning sweep.
  • Pruning sweep: 79.8 keeps A-guided pruning within a factor 5.2 of baseline on Llama-3.2-1B even at 30% sparsity.Full-dA is lowest at 10% on Llama at 18.2 but degrades sharply at higher sparsity.
  • Pruning sweep: 18.2 is the full-dA criterion’s Llama perplexity at 10% sparsity, before sharp degradation at higher sparsity.The pattern is consistent with sink-similar heads being redundant in small numbers but not in bulk.
  • ε sensitivity: All Table 2 reversal rates are bit-identical across ε ∈ {10−5, 10−6, 10−7}.Under fixed common support, multiplicative replacement is inactive, so ε affects only numerical clipping.
Loading 2608.14712v1…