Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

• Across released VLM–LLM pairs, text-capability gaps reach double digits on multiple format-sensitive tasks; text-only further training degrades substantially less, with matched trajectories isolating modality.These tasks include instruction following and chain-of-thought reasoning whose final answer is parsed under a strict rule.Source: §1 · p. 1
• VL fine-tuning perturbs Wq and Wk, which read sink dimensions; input-magnitude-scaled logit shifts are amplified at the sink, collapsing per-head concentration and potentially destabilizing attention to graded surface tokens.The proposed sink-stabilization role for attention to surface tokens is presented as a hypothesis tested in the paper.Source: §1 · p. 1
• On instruction following, every sink-preserving backbone stays within 5.4 pt of its reference LLM, while every backbone whose sink collapses loses at least 7.9 pt.The introduction says post-VL damage depends on both corruption inherited from pre-training and corruption induced by VL fine-tuning.Source: §1 · p. 2
• Normalization granularity matters: per-head QK-RMSNORM removes raw-magnitude dependence, whereas layer-wide normalization can compress weak heads whose per-head sink is already weak.Theorem 1 formalizes per-head QK-RMSNORM's removal of explicit raw-input-magnitude sensitivity from the sink-gap perturbation bound; the introduction reports margin calibration across five headline pairs and two layerwise controls.Source: §1 · p. 2
• Sink Strength ranks backbones by post-VL damage (Spearman ρ = 0.97) and remains strongly correlated across six VLM–LLM pairs and four format-sensitive tasks (ρ ≥0.88).It is computed on the base LLM using inference-only forward passes, without VL training.Source: §1 · p. 2
• Post-pretraining QK-RMSNORM injection fails to reproduce native protection, and tested post-VL linear, spectral, and sparsified weight-merging methods fail to recover lost capability.These negative controls narrow the intervention space toward head-selective training-time protection.Source: §1 · p. 2

VL fine-tuning and text-side erosion

• Standard decoder-updating VLM adaptation often erodes LLM text capability, especially instruction-following and format-sensitive behavior, despite partial data-, training-, and post-hoc remedies.Prior work reports erosion (Zhang et al., 2024; Zhai et al., 2024; Lee et al., 2025a; Ratzlaff et al., 2025; Srivastava et al., 2026); remedies include text-only re-blending (Lin et al., 2024; Tong et al., 2024; Tu et al., 2025), frozen LLMs (Zhu et al., 2024), low-rank adapters (Hu et al., 2022; Liu et al., 2024), neuron-level re-merging (Yu and Ananiadou, 2025), and weight merging (Choi and Kim, 2026).Source: §2 · p. 2

Attention sinks and mechanistic position

• Prior work characterizes attention sinks at inference, while this study examines fine-tuning shifts and treats corruption of the early-position sink as a source of text-side instability.Sink accounts include massive activations (Sun et al., 2024), over-mixing, embedding-level bookkeeping (Zhang et al., 2025), and emergence conditions (Gu et al., 2025); VL adaptation shifts decoder attention patterns (Zhang et al., 2024).Source: §2 · p. 3
• QK-RMSNorm replaces L2 query–key normalization with RMSNorm, and its presence in Qwen3 but absence in Qwen2.5 and Llama-3 makes it an architectural axis of the analysis.Its motivation is entropy-collapse stability (Zhai et al., 2023); the paper’s bound and saturation lemma follow small-circuit attention and fine-tuning circuit-stability analyses (Olsson et al., 2022; Geva et al., 2021; Wang et al., 2025b).Source: §2 · p. 3

3.1 Problem Setup

• The five headline VLMs show IFEval gaps from −1.8 to −9.6 pt, while a matched text-only Qwen2.5 endpoint avoids the IFEval and GSM8K regressions seen after VL training.The four format-sensitive tasks use strict, parseable surface gates; the matched endpoint comparison also finds larger late-layer query/key displacement for the VL endpoint.Source: §3.1 · pp. 3–4

3.2 Sink Strength

• S is the median per-head sink attention probability over roughly the last 10 layers, measured on the base LLM with 15 inference-only passes in seconds on one GPU.S measures the base-side sink margin: high S means a bounded VL update has more sink lead to overcome; in this sample, S tracks IFEval gaps better than the full margin, ρ = 0.97 vs. 0.89.Source: §3.2 · pp. 4–5, §3.3 · p. 6
• Across six pairs, Sink Strength tracks IFEval gaps with Spearman ρ = 0.97; leave-one-pair-out error is 2.54 pt versus 4.40 pt for a constant-mean baseline.The authors frame this six-point result as a sanity check on the linear relation, not prospective evidence on unseen backbones.Source: Figure 3 · p. 5, §3.2 · p. 5
• Lower S is associated with greater format-sensitive loss, while knowledge and broader capability tasks degrade less; per-head QK-RMSNORM aligns with high S, unlike layerwise-normalized Molmo2-O.Molmo2-O is the boundary case: despite having QK-RMSNORM, it has S = +0.24 and the largest six-pair IFEval drop, −18.7 pt.Source: §3.2 · p. 5

3.3 Generalization and Practical Use

• Across six pairs, S ranks backbones on all four format-sensitive tasks with Spearman ρ ∈ [0.88, 0.97]; on five headline pairs, held-out error is under 2 pt per task.A 17-pair IFEval panel reaches ρ = 0.88, and a nine-pair, four-task subset maintains ρ ≥ 0.82; magnitude prediction requires a task-specific linear fit.Source: Table 4 · p. 6, §3.3 · p. 6

3.2 Sink Strength

• S cannot distinguish backbones with the same score: Qwen3-VL and InternVL3.5 share S = 2.36 yet differ by 3.5 pt in observed IFEval gap.The six-pair evaluation is a sanity check, and VL recipe variation can produce up to 3.5 pt of difference within pairs sharing a reference LLM and S.Source: §3.2 · pp. 5–6

4.1 Setup and Notation

• The bound assumes fine-tuning perturbs only query and key projections while holding hidden states and normalization scales fixed; Qwen3-VL’s scale drift was at most 2.9% across 36 q-norm and k-norm layers.This setup isolates projection drift, and the fixed-scale assumption is described as a close approximation for that pair.Source: §4.1 · p. 7

4.2 Perturbation Bound and Margin Saturation

• Per-head QK-RMSNORM removes explicit raw-input-magnitude dependence from the perturbation bound, whereas layerwise normalization can compress weak heads and attenuate their sink logits.In layerwise normalization, attenuation depends on each head’s share of layer energy; the margin can fail through a smaller Gbase rather than a larger Bgap.Source: §4.2 · p. 7
• The margin Gbase−Bgap controls an upper bound on non-sink attention mass, and preserving sink attention at 1−η requires Gbase−Bgap ≥ log((1−η)/η).Negative margins are consistent with observed sink collapse.Source: §4.2 · p. 8

4.3 Empirical Sink Behavior

• Across seven pairs, median(Gbase)−median(Bgap) was near zero for both per-head QK-RMSNORM pairs and strongly negative for all three pairs without QK-RMSNORM.The calibration used 15 prompts; the reported ordering tracked the behavioral split. Per-head QK-RMSNORM pairs had at least 0.70 attention on position 0, while no-QK-RMSNORM pairs spanned 5×10^-5 to 0.50.Source: Table 5 · p. 8, §4.3 · p. 8
• Molmo2-O’s aggregate position-0 attention was 0.25, similar in order to LLaVA-OV’s 0.50, yet its IFEval damage was worse and its per-head sink was weakest among calibrated pairs.Molmo2-O’s marginal proxy was −3.01, second only to no-QK-RMSNORM Qwen2.5-VL at −3.83; both layerwise pairs had strongly negative margins.Source: Table 5 · p. 8, §4.3 · p. 8

4.4 Channel and Direction Ablations

• Disabling top-10 normalization-scale channels in Qwen3-VL caused a 44-point joint IFEval drop versus 20 points summed across single-scale ablations, a 24-point super-additive interaction.A norm-matched isotropic random perturbation caused an approximately 6× larger IFEval drop than the Qwen2.5-VL gap, indicating that perturbation direction matters beyond weight-space distance alone.Source: §4.4 · p. 9

Within-vendor comparison

• Within-vendor comparisons also associate QK-RMSNORM backbones with smaller text-capability losses, but successive releases differ in more than QK-RMSNORM, so this is not causal evidence.The comparisons are Qwen3-VL vs. Qwen2.5-VL and InternVL3.5 vs. InternVL3; isolating the architectural axis is left to future ablation work.Source: §5 · p. 9

Modality control (VL vs. Tülu trajectory)

• +0.41 to +0.20: in the matched 3B trajectory, VL training weakens S as SEEDBench rises from 60.1 to 64.8, while text-only S stays +0.58–+0.65.The text-only run loses only approximately 5 IFEval points from the shared base. IFEval loses most of its points by step 1k, before S declines, then partially recovers while S continues falling; S predicts the regime a run settles into, not when capability is lost.Source: §5 · p. 9

Post-pretraining QK-RMSNORM injection

• 11 on IFEval at the first evaluated checkpoint, against the 59.9 base: post-pretraining QK-RMSNORM injection lags vanilla throughout matched VL training.Together with the Molmo2-O control, this supports native-pretraining co-adaptation of the projections, rather than the module’s mere presence, as important to the protective regime.Source: §5 · p. 9

Weight-space merging

• Seven off-the-shelf merging settings on Qwen2.5-VL and Qwen3-VL fail to recover IFEval on the low-S side; only the two mildest mixes avoid substantial additional degradation.The settings span linear task-vector mixes, spectral decompositions, and sparsified merges.Source: §5 · p. 9

What the negative results do and do not show

• Text-only optimization preserves the sink whereas VL optimization disrupts it; the data-side remedy is applied uniformly after the backbone is fixed, while S can identify which backbones need it before training.The remedy uses multimodal data budget, and the negative controls narrow the intervention space by showing that post-pretraining injection and post-VL merging do not provide the same protection under the tested settings.Source: §5 · p. 9

6 Conclusion

• Across five headline VLM–LLM pairs from three model families, format-sensitive text-capability gaps track reference-LLM Sink Strength, with per-head QK-RMSNORM the strongest architectural correlate in the sample.The relation also extends from the six-pair diagnostic set to a 17-pair panel spanning MoE and multiple LLM families.Source: §6 · p. 9
• Negative injection and merging controls motivate pre-VL screening and head-selective training-time protection.Source: §6 · p. 9

Limitations

• The study reports this per-head sink erosion and format-sensitive capability cost as uncharacterized in prior multimodal literature, but its sample cannot isolate architecture from vendor, recipe, and pretraining data.Within-vendor comparisons reduce but do not isolate those factors, and whether Sink Strength predicts drift under untested regimes such as continual pretraining remains open.Source: Limitations · p. 10
• The tested merging methods do not recover lost capability, and post-pretraining QK-RMSNORM injection does not reproduce native protection, leaving training-time protection open.Proposed future tests include freezing layers below the first sink layer and projecting sink-head Wq, Wk updates away from the sink direction, as well as disabling per-head QK-RMSNORM during Qwen3 VL training to test IFEval effects.Source: Limitations · p. 10

A Theory and Proofs

• The theory defines Bgap as the largest sink-anchored logit-gap perturbation and G as the aggregate sink gap; orthogonal RoPE preserves rotated-vector norms.The sum-of-norms term follows from bounding differences across positions by the triangle inequality.Source: A Theory and Proofs · p. 15

A.1 Single-Head Perturbation Bound (Theorem 1)

• The single-head perturbation proof treats projection dependence as linear without QK-RMSNORM and nonlinear through the RMS denominator with it.The QK-RMSNORM argument uses a local Taylor expansion through that denominator.Source: §A.1 · pp. 15–16
• With QK-RMSNORM, the RMS denominator cancels input-magnitude dependence, but near-null calibration inputs weaken the bound through small projection-RMS lower bounds.The argument assumes nondegeneracy; near-null cases correspond to small mq or mk.Source: §A.1 · p. 16

A.2 Aggregate-Gap Saturation Transfer (Lemma 1)

• The perturbed attention mass outside the sink is at most σ(Bgap − G), with the weaker bound exp(Bgap − G).Here Bgap is the maximum perturbation in the sink-anchored logit gap, and G is the aggregate sink gap.Source: A Theory and Proofs · p. 15, §A.2 · p. 17
• Using the top-2 gap instead of G would require an extra log N term, making the bound vacuous unless T > log N + Bgap.The aggregate-gap form avoids this condition and matches the measured quantity directly.Source: §A.2 · p. 17

A.3 Frobenius / Stable-Rank Restatement of Theorem 1

• Replacing each operator norm by its Frobenius norm divided by the square root of its projection’s stable rank yields a restatement, not a tighter bound.The expression uses separate query and key terms with inputs rq, rk, ∥∆Wq∥F, ∥∆Wk∥F, mq, mk, ∥γq∥∞, and ∥γk∥∞; a zero perturbation contributes zero.Source: §A.3 · p. 17
• On the same 15-prompt calibration set, Bgap is 2.06 nats for Qwen3 versus 5.01 nats for Qwen2.5, a roughly 2.4× smaller value.The comparison is consistent with the theorem’s prediction that the QK-RMSNorm side is less sensitive to raw input magnitude, but does not isolate QK-RMSNorm causally because the models also differ in pre-training data, RL stage, and tokenizer.Source: §A.3 · p. 17

A.4 Multi-Layer Composition Remark

• Theorem 1 and Lemma 1 are single-layer results; multi-layer chaining is only an informal intuition requiring additional control and Lipschitz assumptions.The remark motivates the per-layer diagnostic but does not formalize composition; a marginal proxy is negative in Qwen2.5’s top late layers, and a tight multi-layer theorem is left for future work.Source: §A.4 · pp. 17–18

A.5 Why Layerwise QK-RMSNorm Does Not Provide the Same Per-Head Guarantee

• Per-head QK-RMSNorm gives a head scale independent of its energy, whereas layerwise normalization tracks each head’s share of layer energy and can compress low-energy heads.Compression can reduce a head’s sink-logit lead so its margin fails on the base side; aggregate sink mass may remain high despite weak per-head margins. Molmo2-O shows low per-head Gbase with unremarkable Bgap, while no-QK-RMSNorm Qwen2.5-VL shows the reverse profile.Source: §A.5 · p. 18

A.6 Numerical Sanity Checks

• In a synthetic no-QK-RMSNorm case, the measured sink-anchored quantity is 0.169 against a specialized bound of 0.197, a ratio of 0.86.The case uses d = 16, dh = 4, N = 8, and a size-10 sink channel; the RoPE-free specialization uses the sink/non-sink input difference before the triangle-inequality relaxation.Source: §A.6 · p. 18
• For G = 3.0, Bgap = 0.4, and N = 64, direct softmax computation gives non-sink mass 0.069, matching the logistic bound with ratio 1.00.The worst-case gap perturbation sets non-sink δp = +Bgap/2 and sink δpsink = −Bgap/2, attaining the bound.Source: §A.6 · p. 19

B Evaluation Setup

• The evaluation appendix collects harness protocols, backbone extraction, and hardware/software details for reproducibility.Source: B Evaluation Setup · p. 20

B.1 Evaluation Protocols

• Two lm-evaluation-harness protocols use either chat-template, zero-shot instruct settings or per-task community-default few-shot settings with the chat template disabled.The instruct protocol uses batch size 8 and bf16; the community-default protocol provides a leaderboard-style comparison.Source: §B.1 · p. 20
• The nine-task suite covers MMLU, MMLU-Pro, BoolQ, MBPP, RULER, GSM8K-CoT, IFEval, GPQA-Diamond-CoT, and EQ-Bench, with IFEval as the headline sink-corruption metric.IFEval uses prompt_level_strict_acc and directly grades surface format; MBPP uses pass@1 and EQ-Bench reports raw points.Source: §B.1 · p. 20
• Matched-protocol deltas compare each reference LLM with its VLM-LM, while OpenCompass checks reproduce Qwen2.5-7B-Instruct values within 0.93 pt on GSM8K-CoT (90.67 vs. 91.6) and 0.46 pt on MMLU-Pro (55.84 vs. 56.30).The checks used matched OpenCompass configurations; same-protocol comparisons reduce sensitivity to framework-specific evaluation differences.Source: §B.1 · p. 20

B.2 Backbone Extraction

• Extracted backbones preserve the VLM’s input embeddings and language-model head, retain Qwen3-VL q_norm/k_norm scales, and reproduce original language-module logits on a held-out probe.Parameters are remapped to the corresponding ForCausalLM namespace; the injected QK-RMSNorm variant uses a self-contained trust_remote_code module to match its training pathway.Source: §B.2 · p. 20

B.3 Implementation Details and Reproducibility

• Training ran on four NVIDIA H100 GPUs, while text evaluation and calibration measurements ran on a single NVIDIA A6000 using bf16.Sink Strength calibration took about 8 s per 7B backbone, and the full Gbase, Bgap pipeline took about 52 s per pair; software versions included PyTorch 2.10+cu128 and transformers 4.57.6.Source: §B.3 · p. 20

B.4 Models and Datasets

• The 15 calibration prompts are short instruction-following prompts written in an IFEval style, with no IFEval instances used, keeping calibration data disjoint from evaluation.The prompt set supports Sink Strength and Gbase, Bgap measurements.Source: §B.4 · p. 22

C Per-Task and Behavioral Evidence

• Appendix C expands the behavioral split with capability-versus-format-sensitive coverage, IFEval failure categories, an outcome-bucketed sink probe, and an extended capability panel.The extended panel covers coding, reasoning, and long-context retrieval.Source: C Per-Task and Behavioral Evidence · p. 22

C.1 Capability Preservation vs. Format-Sensitive Degradation

• MMLU and BoolQ drops never exceed 2.3 pt, while IFEval gaps reach −9.6 pt for no-QK-RMSNORM and −18.7 pt for layerwise.Across six pairs, MMLU and BoolQ even improve on three; Table 6 compares these benchmarks with format-sensitive tasks under the same instruct protocol.Source: §C.1 · p. 22, Table 6 · p. 22
• GPQA-Diamond-CoT drops −7.1 to −9.8 pt on three no-QK-RMSNORM pairs, while GSM8K-CoT stays within ±2.1 pt on two per-head QK-RMSNORM pairs.These contrasting MCQA and open-generation results indicate that preservation tracks the probed capability and reference-LLM QK protection, rather than task format alone.Source: §C.1 · p. 23

C.2 Failure-Category Breakdown

• IFEval regressions are prompts the LLM passes but the VLM-LM fails; Table 7 counts their violated instruction categories per pair.Categories include punctuation, combination, length constraints, language, start/end, detectable format, change case, and keywords; prompts can count in multiple categories, so columns need not sum to the total.Source: §C.2 · p. 23, Table 7 · p. 23

C.3 Outcome-Bucketed Sink Probe

• The outcome-bucketed sink probe partitions IFEval prompts by strict accuracy at 0.5 and bootstraps position-0 attention over 100 prompts per outcome bucket.It aggregates late-layer, head, and last-eight-query-position attention probabilities; the 95% percentile confidence interval is based on 100 bootstrap samples per outcome bucket.Source: §C.3 · p. 23

C.4 Extended Capability Panel

• Coding rises by mean +7.3, long-context retrieval is nearly flat at −1.1, and advanced reasoning declines by −2.7 mean, worst −7.4 on Qwen2.5-VL.Across the same five pairs, format-sensitive means range from −5.6 to −8.5 pt; unlike the format-sensitive tasks, these benchmarks do not separate by per-head QK-RMSNORM.Source: §C.4 · p. 23, Table 8 · p. 24

D Mechanism Details

• Appendix D expands the mechanism analyses with calibration-set-size choice, two layerwise controls, a channel-level ablation, and a random-direction perturbation control.These are the mechanism analyses covered in this appendix section.Source: D Mechanism Details · p. 24

D.1 Calibration Sample-Size Robustness

• With N = 15, calibration curves reach the N = 100 asymptote within seed scatter, and pair ordering remains unchanged across all 108 cells.At N = 5, Sink Strength shifts by at most 0.11 (mean 0.09), yet its correlation with IFEval degradation remains ρ = 0.971 with no pair reordering; N = 15 remains the default for broader-panel variance protection.Source: §D.1 · p. 24

D.2 Layerwise Backbones: Aggregate vs. Per-Head Sink

• Molmo2-O’s layerwise QK-RMSNorm preserves model-mean position-0 attention of 0.250, yet only 8.5% of late-layer observations are strong-sink heads, alongside an 18.7-point IFEval drop.The authors interpret this as evidence that aggregate sink mass is insufficient and per-head sink strength is relevant; MolmoE-1B likewise has S = +1.34, margin −2.84, and a 9.4-point IFEval drop, but the two layerwise pairs are convergent controls, not a controlled comparison.Source: §D.2 · pp. 25–26

D.3 Channel Ablation Sweep

• Replacing the top K = 10 amplifier channels jointly across three Qwen3-VL-8B normalization scales costs 44.36 IFEval points, 24.40 points more than summed individual costs.The individual costs sum to 19.96 points; the ablation replaces the largest-magnitude channels with each layer’s channel mean while leaving residual streams and projections untouched.Source: §D.3 · p. 26

D.4 Random-∆W Control

• Random perturbations matched to Qwen2.5-VL’s per-submodule relative Frobenius distances drop Qwen2.5-7B-Instruct IFEval to 10.72, a −61.37 delta from 72.09.The observed Qwen2.5-VL gap is −9.6 points, while the random-∆W control leaves all RMSNorm γ parameters untouched.Source: §D.4 · p. 26

E.1 Per-Pair Sink Strength Predictions

• Per-pair Sink Strength predictions report reference-LLM S, observed VLM–reference IFEval gaps, and leave-one-pair-out linear predictions.This is the numerical detail behind Figure 3 and the Section 3 analysis.Source: §E.1 · p. 26

E.2 IFEval-Specific Predictor (17 pairs)

• Across 17 pairs, Sink Strength correlates +0.88 with IFEval gaps, but orders backbone risk rather than predicting the degradation magnitude of a particular VL recipe.The panel has 12 distinct Sink Strength values because pairs sharing a reference LLM share S; 13 of 17 pairs are Qwen-family, and two examples expose the resolution limit.Source: §E.2 · p. 26

E.3 Multi-Task Generalization (9 pairs)

• Across nine pairs, Sink Strength correlates +0.945 with IFEval deltas versus +0.971 on six pairs, and +0.82 to +0.88 on the other three tasks.The panel adds three sub-10B Qwen-family pairs, extending coverage to 1.5B reference LLMs while retaining all four format-sensitive tasks on a unified protocol.Source: §E.3 · p. 26

E.4 Sink Strength Design Ablations

• The late decoder third alone reproduces the full predictor at ρ = 0.97, while the early third carries almost no signal.Sink concentration and its correlation with the outcome rise together with depth; the late third overlaps the default last-∼10-layer window for every headline pair.Source: §E.4 · p. 27
• For K ∈{3, 5, 10, 15, 20}, the rank correlation remains ρ = 0.971, with no pairwise ordering reversals.The magnitudes of S shift with K, making the last-∼10-layer window a robust rather than tuned choice.Source: §E.4 · pp. 26–27
• The median measures whether a typical head forms a sink, while the mean is inflated by a few strong heads; substituting the mean preserves sign and pair ordering.The median’s per-head statistic exposes cases where a strong model-level aggregate coexists with a weak per-head distribution.Source: §E.4 · p. 27

F Controls

• The controls cover a text-only Qwen2.5-family endpoint, a shared-base VL versus text-only trajectory, post-pretraining QK-RMSNorm injection, and weight merging.These are detailed as controls for the modality comparison and Section 5.Source: F Controls · p. 27

F.1 Modality Comparison: Text-Only Endpoint

• The text-only endpoint is Qwen2.5-7B-Instruct-1M, the Qwen team’s official long-context text-only release, used for Qwen2.5-family further training.The passage gives an Appendix B.4 Hugging Face URL, but no URL is reproduced here.Source: §F.1 · p. 27
• Both text-only Qwen2.5-7B-Instruct-1M and Qwen2.5-VL-7B-Instruct are evaluated against the same Qwen2.5-7B-Instruct text-only reference.The text-only variant uses the paper’s 0-shot instruct protocol, batch size 8, on a single A6000.Source: §F.1 · p. 27

F.2 Modality Control: VL vs. Text-Only Trajectory

• The matched modality comparison starts from Qwen2.5-3B-Instruct and uses the same effective batch size, learning rate, and number of steps.The vision run uses image-text instruction data; the text-only run uses Tülu general instruction SFT.Source: §F.2 · p. 28
• In the matched text-only run, Sink Strength stays between +0.58 and +0.65 while IFEval gives up about 5 points.The run uses Tülu general instruction SFT, in contrast to vision fine-tuning on image-text instruction data.Source: §F.2 · p. 28
• IFEval loses most of its score by vision-training step 1k before Sink Strength falls, then recovers slightly as Sink Strength continues declining.Sink Strength tracks the regime at the run’s end rather than step-level fluctuations; step-0 IFEval is 59.9 versus the published 58.2 (Yang et al., 2024b), within 1.7 points.Source: §F.2 · p. 28
• On Qwen2.5-3B-Instruct, the vision run raises SEEDBench-IMG accuracy from 60.1 to 64.8 as Sink Strength falls from +0.41 at the shared base to +0.20 at step 10k.Both runs use the same base, batch size, learning rate, and step count, while differing in training modality and data.Source: §F.2 · p. 28, Table 14 · p. 29

F.3 Injection Experiment Training Recipe

• The injection experiment compares vanilla and unit-scale-initialized QK-RMSNorm variants trained from the same Qwen2.5-3B-Instruct base under the same LLaVA Stage-1/Stage-2 recipe.Only the QK-RMSNorm module differs; each run takes about 47 hours on a dedicated 2× NVIDIA H100 node with DeepSpeed ZeRO-2.Source: Table 15 · p. 29, §F.3 · p. 29

F.4 Weight-Merging Sweep on Format-Sensitive Benches

• Mild merges avoid substantial additional degradation, while full-strength operators and especially sparsified merges are more damaging across the tested pairs and benches.The reported pattern covers IFEval, EQ-Bench, GSM8K-CoT flexible-extract, and GPQA-Diamond-CoT flexible-extract; full-strength TA, TSV, and STAR reduce the non-protected pair by tens of points.Source: §F.4 · p. 29