PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank Measures of Fine-Tuning Updates Need a Noise Baseline

Muhammad Usman Altaf

arXiv:2610.22438v1cs.LG

TL;DR

Fine-tuning updates are often summarized by rank, but the measures can give conflicting readings and need a noise baseline. The paper compares multiple rank measures against noise and other references, finding that updates have a few strong directions over a long tail, while the observed pattern does not single out alignment.

  • Problem

    Rank is commonly used to characterize fine-tuning updates, but counting-based and energy-weighted measures can give different readings of how many directions an update uses.

    Source: §1 · p. 1

  • Method

    The paper compares several update-rank measures with same-shape random matrices, pretrained weights, and matched fine-tunes without alignment content.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Counting measures put updates 2–5% below noise, eRank and 90% energy rank 12–35% below, and stable rank 78–94% below; matched data controls do not single out alignment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Rank measures answer different questions, so update-rank values need noise baselines, and claims about alignment need controls without alignment content.

    Source: §7 · p. 14

    The random-update damage test uses one model, Qwen3-0.6B, and one task, GSM8K.

    Source: §7 · p. 15

Abstract

from arXiv · show

Studies of what fine-tuning does to a language model often summarise the weight update $\Delta W$ by its rank. We compare the rank measures used for this against the same measures on a random matrix of the same shape, and against two harder references: the pretrained weights, and matched fine-tunes without alignment content. Measures that count directions put our fine-tuning updates only a little below noise; stable rank, computed from the same singular values, puts them far below. Both are right: an update has a few strong directions on top of a long tail. The pattern holds for thirteen released pairs up to 8B parameters, and it is not a mark of alignment: with training held fixed, instruction data does not stand out from Wikipedia or code, DPO gives the same update whether its preference labels are real or half swapped at random (one model, modest preference strength), and scrambled text gives the most concentrated update of all. Applied to a recent claim that post-training is superficial because the effective rank of the weights barely changes, the baselines show that a random update of the same size leaves that rank almost unchanged while wrecking the model, and that an edit built on the effective rank of the update has no measurable effect in our tests. Rank measures of weight updates need a noise baseline, and claims about alignment need a control without alignment content.

1 Introduction

Rank measures can make the same fine-tuning update look almost like noise or far more structured, motivating measure-by-measure noise baselines and matched controls for training content. The paper finds a few strong update directions atop a long tail, a pattern not specific to alignment.

1 Introduction

  • Threshold counts put updates only 2–5% below noise, while eRank and 90% energy rank are 12–35% below and stable rank is 78–94% below.Across the Rényi-rank family, the gap grows as measures weight top directions more, then levels off.Source: §1 · p. 2
  • The disagreement reflects updates with a few strong directions and a long tail, and updates are on average more concentrated than the pretrained weights.Source: §1 · p. 2
  • With training held fixed, instruction SFT does not stand out from Wikipedia or code; DPO has the same shape and directions with real or randomly swapped preferences.Scrambled text gives the most concentrated update.Source: §1 · p. 2
  • The pattern holds across thirteen released model pairs up to 8B parameters, though rounding may account for much of the threshold count in three bf16 updates.Source: §1 · p. 2
  • A same-size random update also leaves effective rank of W unchanged while reducing GSM8K accuracy from 60% to 5%; an update-effective-rank edit has no measurable GSM8K effect.The edit also changes the authors’ GPQA-Diamond benchmark by no more than about three points.Source: §1 · p. 2
  • The authors recommend reporting update-rank measures beside noise, checking that low-precision storage did not erase the update, and testing alignment claims against controls without alignment content.Source: §1 · p. 2

2 Measuring the rank of an update

The paper compares five scale-invariant rank measures by how far their update values fall below a shape-matched noise baseline. It also uses pretrained weights and matched non-alignment fine-tunes as harder references, and checks whether rounding obscures released updates.

2 Measuring the rank of an update

  • Each measure is reported as the fraction of maximum rank, enabling averages across matrices of different shapes.For the gap to noise, 0 means the update looks like noise and values near 1 mean it is more concentrated; intervals bootstrap over layers.Source: §2 · pp. 2–3
  • The five measures differ in how strongly the largest singular directions count: threshold count, energy rank, eRank, participation ratio, and stable rank.The threshold count uses singular values at least 1% of the largest; energy rank is the smallest direction count holding 90% of energy.Source: §2 · pp. 2–3
  • For each measure, the reported gap is 1 − mean(update)/mean(noise); the measures have different ceilings, so comparisons across measures are only approximate.Source: §2 · p. 3
  • Noise is a random-init pair for the authors’ runs and a same-shape Gaussian draw for released models; the random-init stable rank matches the Gaussian baseline within 0.01 across Qwen2.5-0.5B matrix shapes.On Qwen2.5-0.5B, stable rank is 0.436 for the random-init pair and 0.434 for Gaussian matrices.Source: §2 · p. 3
  • Beating Gaussian noise is not enough to show an update is special, because pretrained weights and differences between unrelated trained checkpoints can also fall below noise.The paper therefore also compares updates with pretrained weights and fine-tunes trained the same way on data without alignment content.Source: §2 · p. 3
  • The rounding check estimates the share of update singular values above a simulated bf16 rounding floor and also reports the share of exactly unchanged stored entries.The simulated independent rounding errors are a generous model because near-zero weight changes can have dependent errors or store as exactly zero.Source: §2 · pp. 3–4
  • To test measures of W, the paper compares post-update weights with W plus a Gaussian update scaled to match the update’s Frobenius norm.Weyl’s inequality bounds singular-value movement by the update’s spectral norm, but does not by itself settle how much a measure of W changes.Source: §2 · p. 4

3 Setup

The setup combines controlled fine-tunes on two small base models with released checkpoint pairs, measuring attention and MLP projections while checking precision and checkpoint lineage. The training conditions include alignment and non-alignment data, plus public post-training stages and specialist models.

3 Setup

  • The authors fine-tune all parameters of Qwen2.5-0.5B and SmolLM2-360M in four conditions each, using the same optimizer settings and about 1.75M or 1.9M tokens per epoch for three epochs.Source: §3 · p. 4
  • The four conditions are instruction SFT, Wikipedia language-model training, Python-code training, and word-scrambled Wikipedia text.The scrambled condition preserves words and frequencies but removes word order within each document.Source: §3 · p. 4
  • Runs use fp32 weights with bf16 compute; after training, at most 1.01 × 10^-5 of weights are unchanged, avoiding the rounding artifact seen when weights themselves were stored in bf16.Source: §3 · p. 4
  • The released-model analysis covers thirteen official pairs: base/instruct, OLMo 2 post-training stages, and specialist math and coder models.Checkpoint descent is checked using weight cosine; the released OLMo 2 SFT checkpoint at revision 0d85a3d has cosine 0.00 with the main released base revision, so the authors use a mid-training branch checkpoint instead.Source: §3 · pp. 4–5
  • Table 1 reports rank measures as fractions of maximum rank and top-direction energy across attention and MLP matrices, with noise and pretrained W as references.Its DPO updates are measured from the SFT model they start from.Source: Table 1 · p. 5
  • Measurements cover attention and MLP projections, averaging 168 matrices per Qwen2.5-0.5B model and up to 252 for Qwen3-8B; uncertainty intervals resample whole layers.Embeddings and the output head are excluded; intervals describe spread over layers within a run, not variation between training runs.Source: §3 · p. 5

4 What the rank measures say

Across rank measures, fine-tuning updates look close to random noise under direction counts but far from it under stable rank; their spectra combine strong leading directions with a long tail. Matched controls and released-model comparisons show that concentration is not specific to instruction or preference training.

4.1 Counting measures compress the gap to noise

  • For DPO updates, the reported measures are 12–18% and 24–35% below noise, while stable rank is 78–94% below; threshold count differs by a few percent.The gap grows as measures give more weight to top directions.Source: §4.1 · p. 6, Figure 1 · p. 6

4.2 A few strong directions and a long tail

  • The measures differ because updates have a strong spectral head and a slowly decaying tail: the Qwen SFT top direction holds 4.2% of energy versus 0.6% for noise.Counting measures register the many tail directions, while stable rank is dominated by the top; updates are also more concentrated than pretrained weights on average, though not for every matrix type.Source: §4.2 · pp. 6–7, Figure 2 · p. 6

4.3 Instruction and preference training leave no special mark

  • With model, optimizer, and training tokens held fixed, SFT does not stand out from wikitext or code in stable rank, although data type steers which update directions are used.Across different data, top-16 subspace overlap is four to six times random, while same-data runs with different seeds overlap more.Source: §4.3 · p. 7

4.4 Released models

  • Across thirteen released pairs up to 8B parameters, threshold count is 91–99% while stable rank puts every update 1.6–27 times below noise.Dropping the top direction changes little, and every update has tens to hundreds of directions above the same-size Gaussian's largest singular value.Source: Figure 3 · p. 8, §4.4 · p. 8

4.3 Instruction and preference training leave no special mark

  • At the tested preference strength, DPO and a control with half the preference labels swapped have similar update shape and directions, despite the control being at chance on held-out preferences.Their stable ranks are 0.053 and 0.055, respectively, versus 0.436 for noise.Source: §4.3 · p. 8
  • Shuffled-word training produces the most concentrated updates on both models, but the authors do not know why and report this as an observation rather than an explanation.Stable rank is 0.042 on Qwen and 0.009 on SmolLM2; the models also fit this data poorly.Source: §4.3 · p. 8

4.4 Released models

  • For three released updates, rounding may account for much of what threshold count registers; the OLMo 2 DPO and RL updates are tiny and too affected by rounding for the authors to draw preference-training conclusions from them.Only 13% and 3% of directions in those OLMo updates rise above the simulated rounding floor.Source: §4.4 · p. 9
  • Released specialist models show mixed concentration relative to instruct models, but the comparisons are uncontrolled because content and training amount differ.The authors conclude that concentration varies with training amount and type, and does not set instruct updates apart.Source: §4.4 · p. 9

5 Case study: is post-training superficial?

The paper tests whether post-training is superficial by comparing weight-rank measures and edits against noise and other baselines. An unchanged effective rank of the weights does not distinguish useful post-training from destructive random updates, and the tested effective-rank edit adds no measurable benefit.

5 Case study: is post-training superficial?

  • The case study checks Wu et al.’s claim that post-training is superficial because weight effective rank barely changes, and tests their Timber edit against baselines.Timber keeps the top ⌈eRank(∆W)⌉ singular directions and scales the rest by λ < 1; λ = 0.2 is recommended.Source: §5 · p. 9

5.1 An unchanged effective rank is what any update of this size gives

  • Across seven base/instruct pairs, updates sized 1–13% of the weights change effective rank by at most 0.5%, and real updates move it 1.6–3.8 times less than random updates.Qwen3-0.6B and Qwen3-8B base weights have effective ranks of 0.84 and 0.86 of full rank; post-training changes them by 0.13% and 0.06%.Source: §5.1 · p. 10, Figure 4 · p. 10
  • Random updates move Qwen3-0.6B’s effective rank by only 0.5% yet reduce GSM8K accuracy from 59.5% to 4–6%; the real update moves it 0.13% and produces the instruct model.The effective rank of the weights barely changes whether an update wrecks the model or produces an assistant, so that measurement alone cannot decide whether post-training is superficial.Source: §5.1 · pp. 10–11, Figure 4 · p. 10

5.2 The edit does not need the effective-rank cutoff

  • Timber’s effective-rank cutoff keeps 83% of Qwen3-0.6B’s update directions and 98.9% of its energy, leaving an edit that touches only the last 1%.Running the edit in bf16 distorts it by 16% of its size at λ = 0.2 and 25% at λ = 0.5.Source: §5.2 · p. 11
  • On GSM8K, uniform scaling beats both spectral edits: pass@1 rises by +3.4 points and pass@8 by +2.7, while Timber’s edit has no detectable effect.The uniform edit shrinks the update to the stable-rank edit’s size; selecting directions by either eRank or stable rank adds nothing on this benchmark.Source: Table 5 · p. 11, Figure 5 · p. 11, §5.2 · p. 12
  • Across GSM8K and GPQA-Diamond, scaling only the update’s last 1% of energy has no measurable effect; scaling a quarter changes behavior, for better or worse.On GPQA-Diamond, Timber’s edits move accuracy by under a point and intervals include zero; the authors did not reproduce the reported gain under their evaluation setup.Source: §5.2 · p. 12, Table 6 · p. 12
  • Removing Timber’s 1.1%-energy tail changes pass@1 by only +0.5 points, while removing the stable-rank tail, 59% of update energy, costs 4.2 points versus the unedited model.The comparison says little about the weakest directions alone because the stable-rank “tail” comprises most of the update.Source: §5.2 · p. 12
  • On GPQA-Diamond’s 198 questions in non-thinking mode, using eight unedited-model samples and four per edit, authors report wide intervals and say the effective-rank cutoff is not what makes this kind of edit work.Source: Table 6 · p. 12, §5.2 · p. 13

6 Related work

Prior work has studied noise spectra in data and trained weights, fine-tuning update rank, alignment’s weight-level effects, and update editing. This paper distinguishes its approach by comparing each update-rank measure directly with noise and by testing spectral edits against uniform scaling.

Noise baselines for spectra

  • Earlier noise-spectrum work spans parallel analysis, random-matrix studies of trained weights, and comparisons of fine-tuned weights with pretrained weights plus Gaussian noise; this paper compares updates with noise measure by measure.The cited studies use noise to select factors, separate signal from noise, or study spectra of trained weights.Source: §6 · p. 13

Superficial alignment

  • Work on superficial alignment links alignment to output style and examines weight-rank or activation-level effects; this paper’s weight analysis does not address activation patching.The cited literature also questions how far superficial alignment extends, including safety behavior and reasoning improvements with more data.Source: §6 · p. 13

The rank of fine-tuning updates

  • Prior update-rank studies report low-rank subspaces, concentrated spectra, or nearly full-rank updates; this paper agrees with direction-count findings and adds that updates are more concentrated than noise at the spectrum’s top.Reported measures include rank needed to hold 90% of energy and the share of maximum rank reached by preference- and reinforcement-learning updates.Source: §6 · p. 13

Editing updates

  • Update editing includes trimming, merging, and whole-update scaling; this paper asks whether Timber’s spectral cutoff adds anything beyond uniform scaling.Prior work reports that interpolation can improve robustness and extrapolation along an alignment update can improve an aligned model.Source: §6 · p. 13

Low precision

  • Rounding weights to bf16 can cancel small updates and make updates appear sparse; this paper reports the same effect in its runs and in threshold counts for released checkpoints.The paper connects low-precision storage to apparent sparsity in measurements of small updates.Source: §6 · pp. 13–14

7 Discussion

The measures answer different questions: counting measures track how many directions an update touches, while stable rank and participation ratio track energy concentration. The discussion cautions that rank alone cannot establish alignment or validate checkpoint pairing, and identifies boundaries in the experiments and controls.

7 Discussion

  • Counting measures say fine-tuning touches almost all directions, while stable rank and participation ratio say its energy is very unevenly spread; neither should be read without a noise value.Stable rank is also far below noise for pretrained weights and differences between unrelated checkpoints, and an edit based on it did no better than an eRank edit.Source: §7 · p. 14
  • A released OLMo 2 SFT checkpoint is not the parent of the obvious base release; rank measures do not flag the mismatch, but a weight cosine does.The unrelated pair still has a plausible-looking spectrum, with stable rank 4.4 times below noise.Source: §7 · p. 14
  • The DPO runs and swapped-label control use only Qwen2.5-0.5B; reinforcement learning appears only in an OLMo 2 stage too small to measure above rounding.Source: §7 · p. 14
  • The authors’ own fine-tunes are 0.36B and 0.5B parameters, so they do not study training at the scale of released models up to 8B.Source: §7 · p. 14
  • Matching code-run tokens leaves 2.6–2.7 times fewer optimizer steps than SFT, while matching steps uses 2.5–2.7 times more tokens; the two controls yield different code updates.No single code control matches both tokens and steps.Source: §7 · p. 14
  • The evidence is bounded by several controls: one random-update damage test, roughly bounded run-to-run variation from two seeds, and released-model recipes that vary in many ways at once.Released-model comparisons test the measurement, not claims about alignment; SFT also computes loss on answer tokens only, unlike controls trained on every token.Source: §7 · pp. 14–15

8 Conclusion

The paper argues that interpreting fine-tuning through weight-space rank requires noise and no-alignment controls: rank measures can give different views, and alignment-specific claims need matched comparisons. It cautions against using weight-space statistics alone to judge alignment and describes materials planned for release.

8 Conclusion

  • Counting directions makes updates look nearly full rank and only slightly below same-shape random matrices, while top-weighting measures show greater concentration than noise and pretrained weights.The conclusion attributes the disagreement to updates combining a few strong directions with a long tail.Source: §8 · p. 15
  • With training held fixed, instruction tuning does not stand out from Wikipedia or code, DPO is unchanged by half-swapped preferences, and scrambled text gives the most concentrated update.The conclusion frames update shape as a property of fine-tuning, not a mark of alignment.Source: §8 · p. 15
  • A random update of the same size also leaves the weights' effective rank almost unchanged, while an edit based on update effective rank has no measurable effect on two benchmarks.The paper recommends comparing update-rank measures with same-shape random matrices and testing alignment claims against fine-tunes without alignment content.Source: §8 · p. 15

Broader Impact Statement

  • The paper is a measurement-methodology study, not a new capability or training method, and cautions that weight-space statistics alone do not establish whether alignment is shallow or deep.Such claims can affect how much trust is placed in alignment training.Source: Broader Impact Statement · p. 15

Reproducibility

  • The authors say they will release code, configurations, run manifests, update spectra, random-update seeds, edit scripts, and evaluation prompts after review.These materials are withheld while the paper is under review.Source: Reproducibility · p. 15

A Training details

Training settings and threshold choices shape how the update comparisons should be read. Across matrix types, the reported updates remain far below noise, and shuffled words are the most concentrated.

A Training details

  • Language-model runs used the same token count as the same-model SFT run after truncation; DPO shared precision, memory settings, and batch except for listed changes.Table 7 presents settings shared across fine-tunes.Source: Table 7 · p. 19
  • Table 8 lists run steps, mean training loss, and update size; the second SFT seed's logged loss is omitted because it resumed from step 1,600.Its optimizer and scheduler state were resumed, but the logged mean loss covers only the final steps.Source: Table 8 · p. 19
  • DPO runs took 3.6–8.6 hours, with held-out reward accuracy of 65% and 74% for its two seeds versus 51% for the swapped control.Dropping the top direction still leaves each update far below noise; the reported noise stable ranks without that direction are 0.440 for Qwen and 0.357 for SmolLM2.Source: A Training details · p. 20

B Threshold sensitivity and the noise baseline

  • Table 9 reports threshold counts at three cutoffs and notes that the verdict depends on the cutoff: updates look almost like noise at 0.01 σ1 but have a large gap at 0.1 σ1.The supplied table text does not give the individual counts.Source: Table 9 · p. 20
  • Table 10 compares stable rank for the random-init pair with the same-shape Gaussian prediction, using Qwen2.5-0.5B and means over layers.The supplied caption does not report the values.Source: Table 10 · p. 20
  • For Figure 6, the gap below noise grows across the Rényi family as order rises and levels off from about α = 8.The figure plots mean 1 − Rα(update)/Rα(noise) over matrices, with α = ∞ corresponding to stable rank.Source: Figure 6 · p. 21

C Stable rank by matrix type

  • Every matrix type of every update is far below noise in both models; within each model SFT, wikitext, and code are close, while shuffled words are most concentrated.On SmolLM2, every update type is also below the pretrained weights.Source: C Stable rank by matrix type · p. 21, Table 11 · p. 21

D The shuffled-words update

The shuffled-words update is the most concentrated, but late-layer frequency alignment also appears in other fine-tunes and does not explain that result.

D The shuffled-words update

  • Table 12 presents stable rank by quarter of the network.The supplied caption gives no quarter-specific values.Source: Table 12 · p. 22
  • Mean late-layer frequency correlations are 0.33 for shuffled words, 0.25 for SFT, 0.19 for code, and 0.09 for wikitext.In the last third of the network, several runs align with token frequency above the random 95th percentile, about 0.19.Source: D The shuffled-words update · p. 22
  • The authors treat frequency alignment as a pattern common to fine-tuning, not an explanation for why shuffled words are most concentrated.They qualify the analysis by noting 24 layers, four runs, and no multiple-test correction.Source: D The shuffled-words update · p. 22

E Released pairs: more measures

Among released instruct models, Qwen2.5-0.5B is least concentrated on every rank measure except top-1 energy. Direct checks also show substantial variation in exact-zero entries and rounding diagnostics.

E Released pairs: more measures

  • Table 13 reports participation ratio against a same-shape Gaussian, singular values above the Gaussian edge, rounding share, update-to-weight size, and zero-entry share.The caption describes rounding share as a loose upper bound on how much of the threshold count rounding could explain.Source: Table 13 · p. 22
  • Qwen2.5-0.5B is the least concentrated instruct release on every rank measure except top-1 energy, and is less concentrated than the authors' SFT of the same base.The authors say they do not know why.Source: E Released pairs: more measures · p. 23
  • Exact-zero entries account for 14% of Qwen2.5-1.5B updates and 39% and 52% of OLMo DPO and RL updates, while other instruct releases have 1.5–3.1% zeros.Those OLMo updates are under half a percent of the weights; specialists with updates as large as their weights have almost no zero entries.Source: E Released pairs: more measures · p. 23

F Direction overlap between updates

The top-16 update subspaces overlap less across different data than across seeds, even though stable ranks barely differ. Random updates sharply reduce accuracy, while scoring a real update with the base model’s prompt gives a result that is difficult to interpret as damage.

F Direction overlap between updates

  • The overlap measure compares top-16 left singular subspaces; it is 1 for coincident subspaces and 16/m for random 16-dimensional subspaces of R^m.Table 14 reports means over matrices with 95% layer-bootstrap intervals.Source: F Direction overlap between updates · p. 23, Table 14 · p. 23
  • On SmolLM2, cross-data subspace overlap is 0.12–0.13 versus 0.34–0.51 across same-run seeds, while stable ranks differ by at most 0.014.Wikitext and shuffled words also overlap more than wikitext and code.Source: F Direction overlap between updates · p. 23

G Random-update details

  • Random updates leave finished base-model answers only 7–8% correct versus 63% for the base, while 29–59% of updated answers hit the token limit.For the instruct model, 72% hit the limit and the remaining answers are 28% correct versus 66% for the unedited model.Source: G Random-update details · p. 23
  • With the base model’s 8-shot prompt, the real update scores 19.9%, but 59% of answers hit the token limit; finished answers are 42% correct versus 7–8% with random updates.The passage says the fair test of the real update is the instruct model in its own format.Source: G Random-update details · pp. 23–24

H Edit experiment details

The edit experiments compare variants on GSM8K and GPQA-Diamond, including a longer-token-limit check on GPQA-Diamond. At that limit, the edit and unedited model have similar scores, with uncertainty intervals spanning zero difference.

H Edit experiment details

  • 25.6% for the unedited model and 25.4% for the λ = 0.2 edit on GPQA-Diamond at 4,096 tokens; score differences are −0.3 (−3.5, +3.0) and +2.5 (−3.5, +8.6) points.Doubling the limit changes neither score by more than a point.Source: H Edit experiment details · p. 24
  • On GPQA-Diamond, the edit comparison uses four samples per question and pass@4; a check reruns the unedited model and λ = 0.2 edit with 4,096 tokens.The 8,192-token setting did not fit on the 6 GB GPU with four samples at once.Source: H Edit experiment details · p. 24
  • On GSM8K, edits apply to attention and MLP projections in fp32 before bf16 inference, with rounding error ranging from 4% to 88% of the intended change across variants.The 88% error applies to uniform scaling matched to Timber’s edit; the 4% applies to the stable-rank tail edit.Source: H Edit experiment details · p. 24

Related papers

Loading 2610.22438v1…