PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Emotion Representations Across Scale, Depth, and Instruction Tuning

Adam Hollowell

arXiv:2610.35411v1cs.CLcs.LG

TL;DR

Existing work established emotion geometry in Claude Sonnet 4.5 and Gemma 2 27B base, but whether it holds across Gemma 2 scale and tuning remained open. The study compares six base and instruction-tuned models and finds broadly shared geometry, with changes across scale and depth.

  • Problem

    Whether the emotion geometry previously reported in Claude Sonnet 4.5 and Gemma 2 27B base holds across Gemma 2 scales and tuning states remained an open question.

    Source: §1 · p. 1

  • Method

    The study applies the established extraction pipeline to six Gemma 2 models and compares their emotion geometry across scale, tuning, and depth.

    Source: §9.1 · p. 21

  • Results

    The geometry is largely preserved across scale, while larger models spread variance across more directions and 27B models show a half-network readability gate.

    Source: §9.1 · p. 21

  • Takeaways & Limitations

    Valence structure and cross-model correspondence are already present in cleaned static token-embedding tables, without a forward pass.

    Source: §9.1 · p. 22

    The observational study measures emotion representations, not model behavior, and does not show how Gemma 2 uses them when generating text.

    Source: §9.2 · p. 22

Abstract

from arXiv · show

We measure the emotion geometry of six LLMs spanning Google's Gemma 2 family. Using an extraction pipeline established by Sofroniew et al. (2026) on Claude Sonnet 4.5 and replicated by Hollowell (2026) on Gemma 2 27B base, we compare emotion geometry across base and instruction-tuned variants of Gemma 2 2B, 9B, and 27B. First, all six model cells exhibit the core emotion geometry. A linear map fit between two cells places held-out emotions nearest their own vectors in the target model. Across all fifteen model pairs, fit in one direction each, this failed exactly once, and the median rank was 1 in every pair. Linear maps refit on cleaned embedding-table vectors recover every held-out emotion at rank 1 in all thirty ordered pairs. Second, scale spreads the geometry's variance across more directions without weakening its tie to valence. The leading component's share of variance falls from 31.6 percent (2B) to 29.8 (9B) to 26.7 (27B) in the base models and from 31.9 (2B) to 29.2 (9B) to 27.3 (27B) in the tuned ones, while its correlation with human valence ratings stays between .71 and .78 in all six models. Third, we project each emotion vector onto the output vocabulary and ask at what depth the vector's own emotion word enters the top 10,000 of the 256,000 tokens. In the 2B and 9B models the median emotion word is there from layer 0, while in the 27B models it begins near chance, stays outside the top 10,000 for half the network, and enters at layer 23 in one step. Fourth, tuning leaves every registered analysis-layer measure within its floor. At every size, valence correlation rises and arousal correlation falls by amounts within their floors, the latter against prediction. PC2's variance share also falls, with a pooled decrease significant after correction. Mid-late layer stability exceeds its floor and traces to base 27B's final layer. Code, data, and audit reports are released.

1 Introduction

Building on emotion-geometry findings in Claude Sonnet 4.5 and Gemma 2 27B base, this study asks whether the structure holds across Gemma 2 sizes and tuning states. Across six cells, it finds shared geometry alongside scale- and depth-related differences.

Introduction

  • All six Gemma 2 cells exhibit core emotion geometry, and cross-cell maps generally retrieve held-out emotions at rank 1.The study applies the Sofroniew et al. (2026) pipeline across 2B, 9B, and 27B base and instruction-tuned models.Source: §1 · pp. 1–2
  • The leading component's variance share falls with scale, from 31.6 percent at 2B to 26.7 percent at 27B in base models, while valence correlation stays between .71 and .78.The leading-component variance share also declines from 31.9 percent at 2B to 27.3 percent at 27B in tuned models.Source: §1 · p. 2
  • At layer 23, emotion words in both 27B models enter the top 10,000 output tokens after remaining outside that range for half the network.In 2B and 9B, the median emotion word ranks among the top few thousand tokens from layer 0.Source: §1 · p. 2
  • Instruction tuning leaves every registered analysis-layer measure within its floor, although valence correlation rises and arousal correlation falls at every size.The arousal change is contrary to the prediction; PC2 variance share also falls, with a pooled decrease significant after correction.Source: §1 · p. 2

2 Background and related work

Prior studies examine emotion representations through different models, probes, directions, or feature types. The paper identifies a gap in full-pipeline comparisons across scale and tuning within Gemma 2.

Prior emotion-geometry studies

  • Sofroniew et al. (2026) characterized 171 emotion vectors in Claude Sonnet 4.5, and Hollowell (2026) replicated core structure in Gemma 2 27B base.The reported structure includes valence, arousal, emotion families, and a stable late-middle band.Source: §2 · p. 3

Related Gemma studies

  • Related Gemma studies use classification prompts, supervised probes, sparse-autoencoder features, or a smaller story corpus rather than the full extracted-vector pipeline.Van der Ben et al. (2026) use the closest method, but with a smaller story corpus and neutral set.Source: §2 · p. 3

Related comparisons

  • Other work compares probe accuracy across scale and fine-tuning, or tests one pain direction across models, rather than measuring a set of emotion vectors' geometry.Zhang and Zhong (2025) compare probe accuracy; Tagliabue, Dung, and Berg (2026) assess separation along a pain direction.Source: §2 · p. 3

Research gap

  • No prior study measures emotion geometry across scale and tuning within Gemma 2 using Sofroniew et al.'s (2026) full pipeline.The stated distinction from nearest work includes a base-model arm, whole-vocabulary readout, and human-rating benchmark.Source: §2 · pp. 3–4

3 Methods

The study applies one disclosed extraction and analysis pipeline to six Gemma 2 checkpoints, with measures, thresholds, and resampling procedures specified across the methods.

Model design

  • Six Gemma 2 checkpoints at 2B, 9B, and 27B receive identical raw text, with base and instruction-tuned variants at each size.Scale denotes released model sizes, not parameter count alone, because the 2B and 9B checkpoints were distilled while 27B was trained from scratch.Source: §3 · p. 4

Layer selection

  • The analysis layer is set at two-thirds depth: L17 for 2B, L28 for 9B, and L31 for 27B.L18 is also registered as a robustness check for 2B because two-thirds of its 26 layers falls between layers.Source: §3 · p. 4

Vector extraction

  • Emotion vectors are difference-of-means residual-stream activations, cleaned against each cell's own neutral-dialogue principal components to 50 percent cumulative variance.The story corpus contains 205,200 stories across 171 emotions and 100 shared topics; embedding-table vectors use the same steps without a forward pass.Source: §3 · p. 4

Replication check

  • The base 27B rerun matched Hollowell (2026)'s locked vectors in 24 of 24 checks, to about 1 × 10^-9.This makes its Table 1 row a reproduction of a published measurement within the common pipeline.Source: §3 · p. 4

Registration and uncertainty

  • Registered comparisons use Holm correction within test families, with effect-size floors fixed before grid data existed.Uncertainty estimates include story-level and case bootstraps, while correlation tests assume independent correlations despite shared rated emotions.Source: §3 · pp. 4–5

4 Core emotion geometry

Across Gemma 2 cells, the emotion geometry is closely shared in its axes, similarity structure, and family organization. Cross-cell maps retrieve held-out emotions exceptionally well, though a lexical baseline limits what retrieval alone establishes.

4 Core emotion geometry

  • All six cells show core geometry, with significant valence and arousal axes and arousal weaker than valence.The family cohesion and self-agreement measures describe soft but stable emotion families across cells.Source: §4 · p. 5, §4.1 · p. 6

4.1 Emotion geometry across six Gemma 2 models

  • PC1 explains 26.7 to 31.9 percent of variance and tracks valence, while PC2 tracks arousal more weakly; both axes are significant in all six cells.Valence correlations are .711 to .777 and arousal correlations are .363 to .668; the arousal correlation is weaker in every cell.Source: §4.1 · pp. 5–6, Table 1 · p. 6

Pairwise similarities

  • The nine cell comparisons show pairwise-similarity agreement of .9917 to .9964, above .99 but below the .9976 within-cell reference.The small between-cell differences exceed the variation from resampling stories within a cell.Source: Pairwise similarities · p. 6

PC1 and PC2 planes

  • Eight of nine PC1–PC2 plane comparisons exceed the .0070 within-cell disparity reference, while fitted rotations never exceed 2.2 degrees.The ninth comparison, 9B tuned against 27B tuned, has disparity .0048 within the replicate range; no alignment requires a reflection.Source: PC1 and PC2 planes · p. 7

Emotion families

  • Emotion-family agreement ranges from .53 to .81, and all nine comparisons clear the registered .51 floor for calling families reorganized.The study finds no consistent evidence that tuning pairs match more closely than scale pairs.Source: Emotion families · p. 7

4.3 Cross-cell prediction

  • The cross-cell test fits a rigid Procrustes map on four fifths of the 171 emotions and ranks each held-out emotion among all target vectors.The map is fit without parameters specific to any one emotion, so retrieval tests whether cells arrange emotions similarly.Source: §4.3 · p. 7
  • Across fifteen cell pairs, every transport fit has median rank 1, with one held-out miss in the 2B-tuned to 9B-base fit; the Figure 2 fits sit above the permutation null.The 2B-base to 27B-base fit retrieves all held-out emotions at rank 1, with median cosine .972 and add-one p = 9.999 × 10^-5.Source: Figure 2 · p. 8, §4.3 · p. 8
  • A bag-of-words baseline reaches top-1 .877 and .883 on the cross-story partition, showing lexical information alone is sufficient for strong retrieval on that task.Thus, transport performance does not by itself establish that the shared geometry extends beyond lexical information.Source: §4.3 · p. 8

5 Embedding tables and scale effects

Embedding-table vectors already capture moderate valence structure, while scale spreads emotion variance across more directions without weakening PC1’s valence tie. Arousal trends with scale are not established and depend on the analysis layer used for 2B.

5.1 Embedding tables track valence more than arousal

  • Across all six cells, embedding-table vectors correlate moderately with valence (r = .479 to .507) and weakly with arousal (r = .125 to .179).At 2B and 9B, raw-vector valence correlations are .029 to .074, versus .378 and .380 in the 27B cells; whether the analysis-layer valence direction carries forward the table’s direction is unmeasured.Source: §5.1 · p. 9
  • Refitted on cleaned embedding-table vectors, every cross-cell linear map recovered held-out emotions at median rank 1, with each nearest its own target vector.Median cosines were .943 to .958, below the analysis-layer range of .969 to .982; no uncertainty was computed for that difference.Source: §5.1 · p. 9

5.2 Scale spreads component variance

  • 31.62% to 26.73% (base) and 31.89% to 27.27% (tuned): PC1’s variance share falls from 2B to 27B, opposite the prediction.The drops exceed their registered floor more than fourfold and are Holm-significant at p < 10^-15.Source: §5.2 · p. 9, Figure 3 · p. 10
  • The participation ratio rises with scale in both chains, while TwoNN estimates remain about 4.3 across cells, indicating more linear directions but similar local dimension.TwoNN is an unregistered check, and no uncertainty was computed for either dimension measure.Source: §5.2 · p. 10
  • PC1’s valence correlation remains similar from 2B to 27B, and a held-out-story classifier selects the right emotion about equally well at every scale.The PC1 correlation is .71 at 2B and .72 at 27B in the base chain, and .76 and .77 in the tuned chain; the classifier result is unregistered.Source: §5.2 · p. 10

5.3 Scale may strengthen arousal

  • Arousal tracking and the embedding-to-layer depth gap rise toward 27B, but neither registered endpoint test passes.A bootstrap across the 45 rated emotions supports both rises, but it was unregistered and does not establish generalization beyond those emotions; the contrasts also shift under the 2B L18 check.Source: §5.3 · p. 11

6 Readability across scale

Emotion-word readability begins at layer 0 in Gemma 2 2B and 9B, but at layer 23 in 27B; there, readability onset coincides with geometry consolidation under the top-10,000 criterion.

6.1 The readout instrument

  • Seven to eleven of twelve emotions grade on-theme in each cell; four do so in all six, while nine do so in at least five.The rubric counts an emotion as on-theme when at least two of its top five projected tokens are emotion-related.Source: §6.1 · pp. 11–12, Table 3 · p. 12

6.2 When emotion words become readable

  • Readability onset is the median own-word rank crossing below 10,000 of 256,000 tokens for 140 single-token emotions; normalized vector projections are not layer logits.The 140-emotion test excludes words split into multiple tokens.Source: §6.2 · p. 12
  • At layer 23, both 27B cells’ median emotion word enters the top 10,000, while the 2B and 9B medians are already readable at layer 0.In 27B, median rank falls from near chance to about 6,300 in base and 3,700 in tuned; the figure plots median own-word rank over 140 single-token emotions.Source: Figure 4 · p. 13, §6.2 · p. 13

6.3 The 27B gate at the consolidation seam

  • Layer 23 is both the top-10,000 readability onset and consolidation seam in both 27B cells, but top-1,000 and top-100 crossings lag to layers 29 and 34.No smaller cell shares the coincidence, and the stricter crossings occur six and eleven layers after the seam.Source: §6.3 · p. 14

7 Tuning effects

Instruction tuning leaves the registered geometry measures within their floors, but activation profiles and emotion displacements reveal more specific differences. Across sizes, geometric perturbation grows with depth, while the emotion-displacement ranking is more consistent than activation-shift patterns.

7.1 The tuning battery

  • Tuning shifts pooled valence by +.056 and arousal by −.083 against .15 floors; every other registered analysis-layer measure with a floor also stays within it.PC2’s variance share falls by 0.323 points pooled and remains significant after correction, but stays below its registered floor.Source: §7.1 · pp. 14–15, Table 4 · p. 15

7.2 Activation profiles are largely preserved, but 27B’s shifts diverge

  • Base and tuned activation profiles correlate at .9967, .9993, and .9987 across 2B, 9B, and 27B, but centered agreement is absent at 9B and 27B shifts align weakly with smaller-model shifts.Centered profile correlations are .833 at 2B, .954 at 27B, and −.121 at 9B; 27B activation shifts correlate .23 and .17 with 2B and 9B shifts.Source: §7.2 · p. 16

7.3 Tuning’s perturbation grows with depth

  • Geometric perturbation rises across successive layer thirds at every size, but the 27B late rise depends on its final layer.Perturbation at 27B jumps from .0056 at layer 43 to .0882 at 44 and .1755 at 45; activation-shift magnitude does not rise steadily with depth, and early- and late-layer shifts are largely separate.Source: §7.3 · p. 17, Table 5 · p. 17

7.4 The exception: a stable ranking of displaced emotions

  • Displacement rankings agree across sizes (Spearman .81 between 2B and 27B); regretful, guilty, and remorseful rank highly at every size, but the proposed moral-emotion explanation is inconclusive.Only seven emotions entered the regression for self-conscious moral emotions; its estimate (β = +0.25, p = .81) cannot rule out a large effect in either direction.Source: §7.4 · pp. 17–18

8 Robustness checks

Robustness checks largely preserve registered verdicts across alternative human-rating sets, while showing that arousal’s component rank depends on the emotion set. Layer-stability findings trace to base 27B’s final layer, and the audits reproduce reported values while quantifying precision and hardware noise.

8.1 Other human rating sets leave every verdict but the negative control unchanged

  • No registered verdict changes under four additional human-rating settings except the negative control; the pooled valence delta reaches nominal significance only for the largest overlap, while its registered verdict stays null.The embedding-table asymmetry of moderate valence and weak arousal correlations persists across five rating settings.Source: §8.1 · p. 18, Table 6 · p. 19

8.2 The arousal axis depends on the emotion set

  • With the primary 45-emotion set, arousal falls to the third component in all six cells, but resampling shows its second-versus-third rank is fragile.Across alternative settings, point estimates show no consistent direction, and arousal’s principal-component rank changes from draw to draw in every cell.Source: §8.2 · pp. 19–20

8.3 One divergent layer behind two registered verdicts

  • Two mid-late stability contrasts clear their floors, but both trace to base 27B’s divergent layer 45; dropping that layer sends the tuning contrast to zero and reverses both scale-contrast signs.The divergence replicates across extraction seeds and builds gradually from layers 42 to 45; the authors recommend checking depth-band contrasts with one layer dropped at a time.Source: §8.3 · p. 20

8.4 Audit and the cosine noise floor

  • Seven audits reproduce every reported number from audited files using independent code; bf16-versus-fp32 extraction agrees at mean per-story cosine .999994 or better, while hardware shifts make the fifth decimal noisy on one accelerator and the fourth between accelerators.The hardware bounds are measured on single vectors and are not propagated through the averaged and centered quantities compared in the paper.Source: §8.4 · pp. 20–21

9 Discussion and limitations

The Gemma 2 cells share a stable emotion geometry, while scale and depth change how its variance and emotion words appear. Instruction tuning leaves registered analysis-layer measures within their floors, but the study’s conclusions remain bounded by its assumptions, corpus, model family, and readout.

9.1 Discussion

  • Every cell exhibits the core geometry, and emotion vectors correspond across cells closely enough that one cell’s vector predicts its counterpart in another.The geometry includes a strong valence axis, weaker arousal axis, and soft but stable emotion families.Source: §9.1 · p. 21
  • Across Gemma 2, scale largely preserves the geometry’s shape, while 27B introduces the clearest discontinuities in readability, late-layer behavior, and tuning shifts.The 2B-to-9B change is described as one of degree; new behavior appears from 9B to 27B.Source: §9.1 · p. 21
  • At 2B and 9B, emotion words are readable from layer 0; at 27B, they become readable at mid-depth, while the base model’s final-layer geometry moves.The paper distinguishes the presence of emotion geometry from the readability of emotion words.Source: §9.1 · p. 21
  • Instruction tuning leaves registered analysis-layer measures within their floors, though its perturbation grows with depth and at 27B comes from the last two layers.The emotions most displaced by tuning form a stable ranking across sizes, and the study locates effects in a few emotions rather than the geometry as a whole.Source: §9.1 · pp. 21–22
  • The authors interpret the family’s shared geometry as substantially inherited from the lexicon and corpus, since cleaned embedding-table vectors already carry valence and preserve cross-model transport.Embedding-table transport maps retain median rank 1 in every pair; the six models build nearly the same pairwise similarities, valence-arousal plane, and emotion families.Source: §9.1 · p. 22
  • Valence is the largest component and is already present in the embedding table, whereas arousal gains nearly all its strength from the forward pass.The authors note that these tests cannot adjudicate the possible explanation that valence is lexical and arousal situational.Source: §9.1 · p. 22

9.2 Limitations

  • The study assumes emotion concepts are linear activation-space directions, which may poorly capture blended emotions or directionality toward characters.The authors also cannot be certain the vectors capture all and only the intended concepts or are free of confounds.Source: §9.2 · p. 22
  • The findings concern representations, not use in generated text: the study is observational, has no steering or behavioral measure, and cannot establish how models use these representations.All cells share a story corpus generated by Claude, English-only stories, and validation by one human coder; size comparisons also confound parameter count with training regime.Source: §9.2 · p. 22

10 Open problems

The open problems focus on testing whether the 27B readability gate reflects model computation or the readout, and on extending the results beyond the current family and measurements. The authors outline targeted tests for cross-family generalization, embedding-table contributions, and variance and arousal changes.

What produces the 27B gate

  • An alternative tuned-lens readout could test whether the 27B emotion-word gate depends on the projection or persists under a changed instrument.The proposed test fits a tuned lens on text disjoint from the stories, then reruns the own-word trajectory and seam comparison.Source: §10 · p. 23

The cross-family grid

  • A second model-family grid is needed to test whether Gemma 2’s results generalize, with training regime controlled across sizes and emotion stories from another generator.The proposed grid would test transport within and across families and whether another family’s base model also diverges in its final layers.Source: §10 · p. 23

What the embedding table carries forward

  • It remains open how much later-layer valence strengthens the structure already in the embedding table versus developing a different structure that also tracks human ratings.The authors propose direction-alignment measurements and activation interventions to test dependence rather than alignment alone.Source: §10 · p. 23

The mechanism of variance spreading

  • Whether variance outside leading components contributes emotion-identity or human-rating signal remains untested, motivating held-out prediction across component budgets.If it does, sparse-feature decomposition could help identify that signal.Source: §10 · p. 24

Arousal correlation with scale

  • The rise in arousal correlation at scale is unresolved: it may reflect more arousal information or redistribution across components rather than new information.A proposed held-out, low-dimensional measure across a depth band would distinguish these possibilities.Source: §10 · p. 24

11 Reproducibility statement

The reproducibility statement inventories released artifacts and specifies what users must regenerate or cannot reproduce from the release alone. The appendices also scope comparisons with adjacent studies and map analyses inherited from the replication study to their family extensions.

11 Reproducibility statement

  • The release includes per-cell vectors, embedding-table vectors, activation files, cleanup bases, bootstrap draws, derived tables, analysis scripts, registration materials, and audit records.Vocabulary-readout ingredients are not bundled; scripts export them from the checkpoints and rebuild the vocabulary maps.Source: §11 · pp. 24–25
  • The 205,200-story corpus is not redistributed, and the disjoint-topic bag-of-words baseline cannot be reproduced from the release alone.The release provides story labels and slice mappings but no story text; the Claude comparison also depends on a table not redistributed by either release.Source: §11 · p. 25

A Design-space comparison with adjacent studies

  • Table A.1 scopes overlap in design and method: Zhang and Zhong vary scale and tuning but use supervised probes, while van der Ben et al. use difference-of-means vectors and 171 emotions without systematically varying scale or tuning.The text states that none of the comparison studies combines a base model, unrestricted vocabulary readout, and human-rating benchmark.Source: Table A.1 · p. 28, A Design-space comparison with… · p. 29
  • The 171-emotion nearest-class-mean check remains about equally accurate across scales as PC1’s variance share falls, but its task differs from the seven-class probe studies.Accuracy changes by +1.10 points from 2B to 27B in the base chain and −0.33 in the tuned chain; the authors say this neither matches nor contradicts the earlier result.Source: A Design-space comparison with… · p. 29

B Correspondence with the replication study

C Registration concordance

The registration concordance reports how pre-stated analyses were operationalized and graded, while the transport appendix gives the full battery and a descriptive disjoint-topic extension. The latter’s single topic partition limits what it establishes about robustness.

C Registration concordance

  • The concordance distinguishes registered analyses from operationalized departures, descriptive reporting, and items not reported, and gives the outcomes of directional predictions.It also records that the concordance was graded with measured values in hand, unlike the registration’s pre-stated level of commitment.Source: C Registration concordance · pp. 30–31, Table C.1 · p. 31
  • The concordance reports cross-cell similarity as about stable with a slight rise, neighbor cosines as high everywhere with a slight rise, and a minor pooled +.0024 tuning change as null.The tuning contrast on self-agreement had no registered test; its test was fixed at grading.Source: C Registration concordance · pp. 31, 33
  • Several registered items were not reported or were not graded, including norm variability across depth and the scale-axis test for neighbor cosines.The concordance also describes the embedding-table baseline’s reported correlations while noting that its PC variance, similarity structure, ARI, and neighbor cosines are not reported.Source: C Registration concordance · pp. 31, 33

D The transport battery in full

  • The transport battery comprised eight pre-stated items locked before execution, with one item’s corrected ranges and framing printed after verification found truncated minima.In two of 108 decompositions, a solver was substituted under a recorded amendment and every output remained identical bit for bit.Source: D The transport battery in full · pp. 34–35
  • Under one fixed split of 100 topics, all 32 disjoint-topic cross-cell fits met the median-rank criterion, with top-1 retrieval of .871 to .918; this descriptive result does not test robustness across partitions.Matched random-half and within-cell topic-split comparisons are also reported descriptively.Source: D The transport battery in full · p. 35

E Robustness checks on vector cleaning, norm source, tuning, and readability, full tables

The appendix tests how robust the paper’s vector-cleaning, tuning, and readability findings are to alternative settings. The checks qualify the readability onset and report uncertainty in some rating-dependent comparisons.

Full tables

Rating-dependent contrasts

  • Arousal’s best-correlating component varies across resamples: no cell places it on PC2 in at least 95 percent or at most 5 percent of draws under any tested rating setting.The bootstrap resamples the 45 rated emotions with replacement, using the same sample across cells; its pooled delta is the mean of the three size-specific deltas.Source: E Robustness checks on vector… · pp. 37–38

Scale battery

  • The registered scale prediction’s PC2-versus-PC3 variance-share gap was not operationalized or graded, and PC3’s share was not graded on the scale axis.The two PC1 variance-share p values are reported only as p < 10^-15 because the computed tail underflows double precision.Source: E Robustness checks on vector… · p. 38

Tuning measures outside the Holm correction

  • The two norm correlations and cohesion are outside the Holm correction; the pooled PC1 norm-correlation change is +.095 against a .10 floor, while the PC2 change is +.045 and cohesion moves +.017.The .10 floor relies on an independence assumption that the authors say inflates it by an unmeasured amount.Source: E Robustness checks on vector… · p. 38

Readability checks

  • Across 500 cleanup-basis redraws, the 27B readability onset stays at layer 23, while all four 2B and 9B cells read from layer 0.Five draws needed an alternative solver; where both solvers converged, their outputs agreed to 9.1 × 10^-12 without changing the seam or onset.Source: E Robustness checks on vector… · p. 38
  • Layer-0 readability at 2B and 9B survives removal of normalization in all four cells and removal of cleanup in three; 2B tuned rises to rank 14,800 without cleanup.Random directions score near rank 128,000 at all three widths, near chance.Source: E Robustness checks on vector… · pp. 38–39

The seam threshold

  • Across .95–.98 seam thresholds, smaller-cell seams stay between one-seventh and about one-third of depth, while both 27B seams remain at half depth or move to layer 24.The 27B seam is at layer 23 under .95, .96, and .97.Source: E Robustness checks on vector… · p. 39

Calibration analysis

  • Neither 27B cell reaches rank 1,000 or 100 within layers 18–28; both first cross 1,000 at layer 29 and 100 at layer 34.Under the 1,000 line, the tuned cell’s onset is never at the seam across 500 redraws, unlike 75.6 percent of draws under the 10,000 line.Source: E Robustness checks on vector… · p. 39

F A valence asymmetry in readability

The readability analysis finds an early positive-word advantage in smaller models and at the 27B onset, but negative words become more readable by the final layer. Frequency controls preserve the early pattern in the smaller cells.

Valence asymmetry

  • The analysis compares median vocabulary ranks for 137 single-token emotion words rated by NRC-VAD: 50 positive and 87 negative.The lexicon lacks ratings for energized, insulted, and stimulated.Source: F A valence asymmetry in readability · p. 39
  • At the final layer, negative emotion words are more readable than positive words in all six cells, with medians of 13 to 25 versus 18 to 42.The controlled comparison puts negative words ahead in every cell, significantly in 9B base, 9B tuned, and 27B base.Source: F A valence asymmetry in readability · p. 40
  • At layer 0, positive-word medians are 1,000–1,900 and negative-word medians are 6,700–11,300 in the 2B and 9B cells.In 27B, positive words cross below rank 10,000 at layers 22 and 21, one to two layers before negative words at layer 23.Source: F A valence asymmetry in readability · p. 40

Frequency controls

  • Frequency controls preserve the early positive advantage in all four 2B and 9B cells, while the controlled 27B difference runs the other way and is significant only in tuned 27B.The 27B p values are .026 for tuned and .079 for base; the early positive advantage in smaller cells is also found in base models.Source: F A valence asymmetry in readability · p. 40

G The fixed-probe method block

The fixed-probe analysis compares base and tuned activations using tuned-cell emotion probes, then examines profile stability, layer shifts, and displacement. The reported results include a modest out-of-sample prediction and diagnostics that qualify profile comparisons.

Fixed-probe method

  • The fixed-probe measurement uses each size’s tuned-cell probes on both cells and averages cosine similarity over 500 shared neutral dialogues.The authors call this an analog, not a replication, of Sofroniew et al. (2026), because token position, prompts, and probe extraction differ.Source: G The fixed-probe method block · p. 40

Centered activation profiles

  • The 9B centered base-to-tuned profile correlation rises to .96 after excluding the union of the cells’ 21 highest-variance dimensions.Centering shrinks the mean activation vector’s norm from about .94 to about .02, allowing a few high-variance dimensions to dominate; the remaining profiles correlate .975 and .983 across odd/even dialogue halves.Source: G The fixed-probe method block · p. 41

Layer perturbation

  • At 27B, removing layer 45 alone reduces late-third perturbation to .0103, below the middle third’s .0142; removing both layers 44 and 45 reduces it to .0043.Removing layer 44 alone leaves late-third perturbation at .0165.Source: G The fixed-probe method block · p. 41

Split-search nulls

  • The layer-label shuffle produces one value at least as large as 2B’s observed .620 difference, while none reaches the observed differences at 9B or 27B.The add-one p value for 2B is .0010; the null quantifies the contribution of the split search and grades nothing.Source: G The fixed-probe method block · p. 41
  • Observed same-side versus opposite-side mean correlations are .620 at 2B, .386 at 9B, and .605 at 27B; random-vector split-search nulls never reach those values.The observed differences are at least twelve times each size’s null maximum, but the authors do not interpret where the preferred split falls.Source: G The fixed-probe method block · p. 41

Displacement checks

  • Under 1,000 cleanup-basis redraws, each vindictive emotion ranks among the top ten displacements in at least .999 of 27B redraws.At 2B and 9B, vindictive and vengeful reach the top ten in .42–.66 of redraws, while spiteful does so in .174 and .070.Source: G The fixed-probe method block · p. 41

Regression

  • The regression explains 11 percent of displacement variance across 137 rated single-token emotions, and its moral-emotion indicator has standard error 1.03 versus arousal’s 0.09.The indicator covers seven of eight prespecified self-conscious moral emotions because self-critical lacks an NRC-VAD rating.Source: G The fixed-probe method block · pp. 41–42

The out-of-sample prediction

  • All five out-of-sample probes moved in the predicted direction, but the three primary probes together reached a one-sided p of .056.Only regretful was individually below .05 among the primary probes; reflective, a secondary probe, had p = .004, and the prediction used a threshold of p < .05 adopted at grading.Source: G The fixed-probe method block · p. 42

H The layer-45 replication

The layer-45 replication supports a late-layer divergence in base 27B, while the layer sweep and additional measures locate related changes in the late band. The appendix also documents asset licensing and source-material provenance.

H The layer-45 replication

  • Under a corrected replication rule, base 27B’s late-layer divergence graded genuine on both seeds, despite layer 45’s lower extraction agreement and sensitivity to story sample.Two layer-45 extractions agree at .895, with a square-root-rule ceiling of .946; layer 31’s ceiling is .9856 from subsample draws.Source: H The layer-45 replication · p. 42
  • Across layers 42–45, the divergence is a slope, steepest from layer 43 to 44, and excluding the largest-changing dimension does not restore agreement.Consecutive-layer correlations are .9970 into 43, .9087 into 44, and .9800 into 45; dimension 4140 accounts for 13.8 percent of change.Source: H The layer-45 replication · p. 42
  • Residual-norm drops align with the same late transition in both 27B cells, while emotion-vector rotation concentrates in base 27B’s final layers and at the tuned 27B consolidation seam.The residual-norm curve is a per-dialogue-mean proxy, unlike Hollowell (2026)’s per-token measure.Source: H The layer-45 replication · p. 43
  • Base 27B’s valence correlation falls by .24 from layers 43 to 45, while tuned 27B’s valence correlation never declines.The passage distinguishes this valence-correlation decline from Zhang and Zhong (2025)’s reported decline in emotion representation.Source: H The layer-45 replication · p. 43

I Licenses for existing assets

  • The study inherits the replication study’s asset set unchanged except for replacing one model with six checkpoints; the 171-emotion vocabulary and prompts are not reproduced here.The source materials are in Appendix A of Hollowell (2026), and release decisions are stated in §11.Source: I Licenses for existing assets · pp. 43–44

J Compute

The extraction setup leaves released weights unprocessed and separates accelerator-based extraction from downstream CPU analysis. Twin-path checks were bit-identical, while precision differences were small overall but concentrated in one 27B dimension.

J Compute

  • Extraction used bf16 on 80 GB accelerators, with compatibility mode off and no weight folding or centering; total accelerator use was approximately 20 hours.Analysis ran on a Mac CPU environment, while the permutation floor and seam bootstrap ran on cloud CPUs; package lists are pinned in the release manifest.Source: J Compute · p. 44
  • Downstream analyses recompute from released vectors on a CPU; vocabulary readout additionally requires downloading gated Gemma checkpoints and exporting unembedding ingredients.The bag-of-words baseline also requires the gated Gemma tokenizer and the unreleased story corpus.Source: J Compute · p. 44
  • Every corresponding vector from the two extraction paths was bit-identical before cross-hardware comparison.The twin run extracted 6,840 per-story vectors, 40 for each of 171 emotions, on one accelerator.Source: J Compute · p. 44
  • At 27B, dimension 4140 carries 97.6 percent of the bf16–fp32 difference, while the largest dimensions carry 7 percent at 2B and 3 percent at 9B.The authors describe this scale-associated outlier-feature pattern as documented in quantization literature.Source: J Compute · p. 44
Loading 2610.35411v1…