Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

Minsik Choi; Geewook Kim; Young Geun Kim

Source-linked AI summary

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

Minsik Choi, Geewook Kim, Young Geun Kim

arXiv:2609.00746v1Sep 1, 2026cs.LG

TL;DR

VLM fine-tuning can erode text capability on format-sensitive evaluations, but the mechanism and predictors of this loss remain unclear. The paper traces degradation to attention-sink corruption, introduces base-model Sink Strength as a pre-training diagnostic, and finds that tested injection and merging remedies do not restore the lost capability. These results support screening backbones before VL training and motivate head-selective training-time protection.

Problem
VLM adaptation can erode text capability, particularly on evaluations requiring exact output formats, while the source of this degradation remains insufficiently explained.
Method
The paper relates post-VL degradation to attention-sink corruption and computes Sink Strength from inference-only base-LLM forward passes before VL training.
Results
Across six VLM–LLM pairs and multiple format-sensitive tasks, Sink Strength tracks relative degradation; post-pretraining QK-RMSNorm injection and tested post-VL merging fail to restore the lost capability.
Takeaways & Limitations
Sink Strength enables pre-VL backbone screening, while the negative controls narrow protection toward head-selective training-time intervention.
Takeaways & Limitations
Whether Sink Strength predicts instruction-following drift under adaptation regimes not sampled, such as continual pretraining, remains open.

Abstract

from arXiv · show

Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.

1 Introduction

VL fine-tuning can erode text capability, especially on format-sensitive tasks, and the paper attributes this loss to corruption of early attention sinks. Sink Strength measures base-model sink concentration before training and predicts post-VL degradation, while tested post-training remedies do not restore the lost capability.

Motivation: VL adaptation damages text capability most on tasks requiring exact output rules, including instruction following and strictly parsed chain-of-thought answers.Across released pairs, the gap reaches double digits on multiple format-sensitive tasks.
Mechanism: VL fine-tuning perturbs query and key projections, amplifying input-magnitude effects at the early sink and collapsing per-head attention concentration.The sink normally absorbs diffuse attention mass and helps stabilize attention to surface tokens graded by format-sensitive evaluators.
Evidence: Preserved-sink backbones remain within 5.4 pt of their reference LLM on instruction following, whereas collapsed-sink backbones lose at least 7.9 pt.This separates inherited backbone weakness from corruption induced during VL adaptation.
Diagnostic: Sink Strength is a scalar computed on the base LLM with inference-only forward passes that ranks backbones by post-VL damage without VL training.Across six VLM–LLM pairs and four tasks, rank correlations remain strong, with Spearman ρ = 0.97 reported for the backbone ranking.
Interventions: Post-pretraining QK-RMSNorm injection fails to reproduce native protection, and tested post-VL merging methods fail to recover lost capability.These negative controls narrow the intervention space toward head-selective protection during training.

2 Background & Related Work

Prior work documents text-side erosion in VLM adaptation and studies attention sinks in fixed networks, while this paper connects sink behavior to format-sensitive text-capability gaps during adaptation.

VL adaptation: Standard VLM recipes update LLM decoder weights across alignment and instruction-tuning stages, with reported losses in instruction following and other format-sensitive behaviors.The paper positions this erosion as a recurring observation across the VLM literature.
Attention sinks: Attention-sink research has linked the phenomenon to massive activations, over-mixing, embedding-level bookkeeping, and its emergence conditions.These studies primarily characterize sinks in fixed networks at inference time.

3 Sink Strength: A Pre-VL Diagnostic

Sink Strength is a base-LLM diagnostic that tracks text-capability loss after vision-language adaptation, especially on format-sensitive tasks. Across model pairs and tasks, it provides pre-training architectural insight and useful rank prediction, while its magnitude predictions have clear resolution limits.

Observed text-capability gaps: The affected tasks use strict surface gates, so small attention-pattern changes can flip outcomes through formatting failures such as truncation, exact-phrase violations, language mismatches, and missing markers.The four format-sensitive tasks are IFEval, EQ-Bench, GSM8K-CoT, and GPQA-Diamond-CoT.
Observed text-capability gaps: Format-sensitive evaluations expose broad degradation: IFEval gaps reach at least 7.9 points for three headline pairs, with the largest headline gap at −9.6 points.Regressions span 43–126 distinct prompts per pair, indicating broad instruction-following degradation rather than a few outliers.
Observed text-capability gaps: A text-only further-training endpoint degrades substantially less than its matched VL endpoint, supporting modality-specific attribution of the observed loss.For the Qwen2.5 comparison, the text-only variant does not regress on IFEval or GSM8K and loses at most 0.42× the VL gap on EQ-Bench and GPQA.
Sink Strength: Sink Strength S measures per-head sink concentration on the reference LLM using inference-only forward passes, with no VLM forward pass or VL training.The metric uses 15 inference-only calibration-prompt passes and aggregates attention probabilities over the last approximately 10 layers.
Architectural correlate: S aligns more precisely with QK-RMSNORM structure than a binary module-presence test, identifying layerwise normalization as a boundary case and Molmo2-O as highest risk.Molmo2-O has QK-RMSNORM applied layerwise, S = +0.24, and the largest measured IFEval drop at −18.7 points.
Sink Strength: Across six pairs, S strongly tracks the IFEval gap with Spearman ρ = 0.97 and predicts held-out gaps with 2.54-point average error versus 4.40 points for a constant-mean baseline.The outcome range is 17 points, from −1.8 to −18.7, and leave-one-pair-out fitting is used for the prediction errors.
Architectural correlate: The predictor cannot resolve all recipe variation: Qwen3-VL and InternVL3.5 share S = 2.36 but differ by 3.5 points in observed gap.The paper describes this as the resolution floor of a single-feature predictor and attributes the within-S variation to VL recipe differences.
Sink Strength: The same S ranks backbones across all four format-sensitive tasks, achieving Spearman ρ ∈[0.88, 0.97] across six pairs and under-2-point average held-out error on five headline pairs.Magnitude prediction requires a per-task linear fit, whereas rank prediction reuses the single reference-LLM measurement.

4 Mechanism

The mechanism attributes format-sensitive capability loss to attention-sink corruption: projection drift can collapse per-head sink margins, while pre-existing weak sinks worsen vulnerability. Empirical controls show that per-head, rather than aggregate, sink strength tracks the behavioral split.

Perturbation bound and margin: Without QK-RMSNORM, projection perturbations retain raw query- and key-input magnitude factors, amplifying logit shifts at massive-activation sinks.Per-head QK-RMSNORM removes this explicit raw-magnitude dependence by normalizing query and key projections separately.
Per-head versus layerwise normalization: Layerwise QK-RMSNORM absorbs input magnitude but scales each head by its share of layer energy, compressing weak heads instead of normalizing them.A weak head can therefore enter VL training with an already weak per-head sink, reducing Gbase before post-VL perturbation is considered.
Perturbation bound and margin: Attention-sink survival depends on the margin Gbase − Bgap, which bounds non-sink attention mass through σ(Bgap − Gbase).Preserving sink level 1 −η requires Gbase − Bgap ≥ log((1 −η)/η); negative margins are consistent with sink collapse.
Empirical sink behavior: ≥0.70 of attention concentrates on position 0 for the two per-head QK-RMSNORM pairs, versus 5×10−5 to 0.50 for no-QK-RMSNORM pairs.InternVL3 and InternVL3.5 span 5×10−5 to 0.74, while Qwen2.5-VL and Qwen3-VL span 0.03 to 0.70.
Per-head versus aggregate sink: Molmo2-O shows that aggregate position-0 attention can be misleading: its 0.25 aggregate sink resembles LLaVA-OV’s 0.50, despite much worse IFEval damage.Its weakest calibrated per-head sink and strongly negative G − B indicate that head-level protection is not delivered by layerwise normalization.
Empirical sink behavior: A joint kill of three Qwen3-VL normalization-scale channel groups costs 44 IFEval points, versus 20 points summed across singles.The 24-point super-additive interaction supports a redundant multi-norm sink amplifier contributing to format-sensitive capability.

5 Controls and Discussion

Controls distinguish modality-driven sink disruption from text-only training and test whether proposed architectural or post-hoc remedies restore capability. Injection fails to reproduce native protection, while weight merging fails under the tested settings.

Discussion: Within-vendor comparisons show less loss for QK-RMSNORM models, but the releases differ in more than that architectural feature.The authors therefore treat the association as an alternative explanation addressed, not controlled causal evidence.
Modality control: A matched 3B trajectory weakens the sink under vision fine-tuning but preserves it under text-only Tülu SFT.The vision run changes S from +0.41 to +0.20, whereas the text-only run holds S at +0.58–+0.65 and loses only ∼5 IFEval points.
Negative controls: Post-pretraining QK-RMSNORM injection fails to reproduce native protection during matched VL training.The injected model scores 11 on IFEval at the first evaluated checkpoint, versus 59.9 for the base, supporting projection co-adaptation during pretraining.
Negative controls: Seven off-the-shelf weight-merging settings fail to recover IFEval on low-S Qwen2.5-VL and Qwen3-VL backbones.Only the two mildest mixes avoid substantial additional degradation under the tested settings.
Discussion: Text-only re-blending preserves the sink but consumes multimodal data budget and is chosen only after the backbone is fixed.Sink Strength instead identifies vulnerable backbones before VL training begins.

6 Conclusion

Across model families, format-sensitive text-capability gaps track the Sink Strength of reference LLMs, with per-head QK-RMSNORM the strongest architectural correlate in the sample. Negative controls motivate pre-VL screening and head-selective training-time protection.

Conclusion: Across five headline pairs and a 17-pair panel, format-sensitive text-capability gaps track reference-LLM Sink Strength across multiple model families.Per-head QK-RMSNORM is the strongest architectural correlate in the reported sample.
Conclusion: Negative injection and merging controls further motivate pre-VL screening and head-selective training-time protection.These controls narrow the intervention space beyond simpler post-pretraining and post-VL shortcuts.

Limitations

The paper’s limitations concern scope, theoretical coverage, and intervention evidence. The diagnostic is validated for sampled VL adaptation settings, while multi-layer guarantees and causal training-time protections remain unformalized or untested.

Scope: S is validated on VL adaptation outcomes, so its predictive reach under unsampled regimes such as continual pretraining remains open.Within-vendor comparisons reduce but do not isolate the architectural axis.
Interventions: Post-VL merging and post-pretraining QK-RMSNORM injection do not recover the lost capability or reproduce native protection, leaving head-selective training-time intervention untested.The paper proposes freezing lower layers and projecting sink-head Wq, Wk updates away from the sink direction, but does not test this intervention.
Theory: Near-null query or key projections reduce mq or mk and loosen the QK-RMSNORM bound, although exact degeneracy is excluded by Assumption 2.The measured calibration values mq, mk > 0 avoid exact degeneracy but do not remove this sensitivity.

A.3 Frobenius / Stable-Rank Restatement of Theorem 1

The restatement rewrites the perturbation bound using Frobenius norms and stable ranks while preserving its structural message: QK-RMSNORM removes explicit raw-input-magnitude dependence. The analysis also explains why per-head normalization protects weak heads better than a shared layerwise denominator.

Stable-rank restatement: The operator-norm perturbation bound can be rewritten using Frobenius norms divided by per-projection stable ranks.The rewriting uses rq, rk, ∥∆Wq∥F, ∥∆Wk∥F, mq, mk, and the γ scale constants, without tightening the bound.
Structural implication: The Frobenius form contains no explicit raw-input-magnitude factor on the QK-RMSNORM side; only stable ranks, γ values, and projection-RMS lower bounds remain.The directly measured Bgap is the quantity used in the main text.
Empirical comparison: Qwen3’s measured Bgap is 2.06 nats versus 5.01 nats for Qwen2.5 on the same 15-prompt calibration set, approximately 2.4× smaller.The comparison is consistent with, but does not causally isolate, QK-RMSNORM because the models also differ in data, RL stage, and tokenizer.
Per-head versus layerwise: Per-head QK-RMSNORM cancels each head’s input magnitude, whereas layerwise normalization additionally scales each head by its share of layer energy.Both remove input magnitude, but only per-head normalization provides a scale independent of head energy.
Per-head versus layerwise: A weak layerwise-normalized head can be compressed, reducing its sink-logit lead and causing the margin Gbase − Bgap to fail from the base side.Aggregate sink mass may remain high even when individual compressed heads carry little lead.
Synthetic checks: A synthetic Lemma 1 case with G = 3.0, Bgap = 0.4, and N = 64 yields non-sink mass 0.069, exactly matching the logistic bound.The reported ratio is 1.00 under the worst-case perturbation.

B.1 Evaluation Protocols

The evaluation uses lm-evaluation-harness under instruct and community-default protocols, extracting each VLM’s language backbone for matched text-only comparison. The suite combines format-sensitive, general capability, coding, reasoning, and retrieval tasks with reproducibility checks and disjoint calibration prompts.

Protocols: All text-side evaluations use lm-evaluation-harness v0.4.12 under an instruct protocol and a community-default leaderboard-style protocol.The instruct setting uses zero-shot chat-template evaluation, while the community-default setting uses task-specific few-shot counts without the chat template.
Task suite: The nine-task suite includes MMLU, MMLU-Pro, BoolQ, MBPP, RULER, GSM8K-CoT, IFEval, GPQA-Diamond-CoT, and EQ-Bench.IFEval is the headline sink-corruption metric because its checker directly grades surface format.
Reproducibility: Pipeline sanity reproduces reference Qwen blog values within 0.93 point on GSM8K-CoT and 0.46 point on MMLU-Pro under matched OpenCompass configurations.Reported deltas compare each reference LLM and VLM-LM under the same protocol.
Backbone extraction: Each VLM language backbone is extracted as a standalone causal LM while preserving its original embeddings and output head for matched text evaluation.Qwen3-VL extraction also preserves q_norm and k_norm scale parameters, and round-trip logits reproduce the original language module on a held-out probe.
Models: The study evaluates six headline VLM–LLM pairs, with additional controls and extended panels covering multiple model families and adaptation variants.The supplied model lists include Qwen, InternVL, LLaVA, Molmo, Ovis, MiniCPM, Idefics, and related references.
Calibration: Sink Strength and Gbase, Bgap use 15 short instruction-following calibration prompts written in IFEval style but disjoint from IFEval instances.This keeps calibration prompts separate from the evaluation data.

C Per-Task and Behavioral Evidence

The behavioral evidence shows that VL adaptation disproportionately harms format-sensitive capabilities while preserving or mildly changing several knowledge, coding, retrieval, and reasoning tasks. Per-prompt and task-level analyses connect the loss pattern to instruction-following failures rather than a simple output-format confound.

Knowledge and comprehension: MMLU and BoolQ drops never exceed 2.3 points across six pairs, while IFEval gaps reach −9.6 and −18.7 points.The contrast is measured under the same instruct protocol.
Format confound: GPQA-Diamond-CoT drops by −7.1 to −9.8 points on three no-QK-RMSNORM pairs, whereas GSM8K-CoT stays within ±2.1 points on two per-head pairs.These results weaken the explanation that only multiple-choice versus open-generation format drives the contrast.
Behavioral breakdown: IFEval prompts are partitioned into both_pass, regression, recovered, and both_fail buckets to localize instruction-following changes.The regression bucket contains prompts passed by the LLM but failed by the VLM-LM.
Extended capabilities: Coding rises by a mean +7.3 points, long-context retrieval is roughly flat at mean −1.1, and advanced reasoning declines mildly at mean −2.7.The four format-sensitive tasks instead average −5.6 to −8.5 points on the same five pairs.

D.1 Calibration Sample-Size Robustness

The calibration analysis finds that the Gbase − Bgap margin stabilizes by roughly 15 prompts, while Sink Strength remains predictive even with only 5 prompts. The pair ordering is robust across calibration sizes, supporting N = 15 as the default.

Calibration margin: By N = 15, Gbase, Bgap, and G − B are indistinguishable from the N = 100 asymptote within seed scatter for every pair.Between N = 10 and N = 100, seed-mean shifts are at most 0.05 for Gbase and Bgap and 0.06 for G − B.
Calibration margin: σ(G − B) falls from approximately 0.15 at N = 5 to at most 0.07 by N = 15 and at most 0.02 by N = 50.The largest N = 5 scatter occurs for InternVL3.5.
Calibration margin: Across all 108 pair, calibration-size, and seed cells, the sign of G − B and the magnitude ordering remain unchanged.The sweep uses six pairs, six calibration sizes, and three random seeds per cell.
Sink Strength: ρ(S, ∆IFEval) remains 0.971 at both N = 15 and N = 5, with no reordering of the six pairs.N = 5 shifts S by at most 0.11 and runs approximately three times faster, about 3 seconds versus 8 seconds per 7B backbone.
Default choice: The authors retain N = 15 because the extra prompts reduce seed-variance risk on broader panels, despite the smaller set preserving the six-pair predictor ordering.The figure plots Gbase, Bgap, and their margin against N, with seed means, ±σ bands, and the G − B = 0 boundary.

E.4 Sink Strength Design Ablations

Sink Strength is designed as a late-layer, per-head statistic: its predictive signal is concentrated in deep decoder layers, remains stable across window sizes, and is not dependent on using a median rather than a mean.

Evaluation scope: Sink Strength is measured on reference LLMs and evaluated against VLM–reference IFEval deltas across panels spanning dense and MoE architectures, multiple families, and 1.5B–32B backbones.The 17-pair panel includes per-head, layerwise, and no-QK-norm classes.
Layer selection: The late decoder third alone reproduces the full predictor at ρ = 0.97, whereas the early third carries almost no signal.The late third overlaps the default last-approximately-10-layer window for every headline pair.
Window robustness: Across last-K-layer windows with K ∈ {3, 5, 10, 15, 20}, the rank correlation remains ρ = 0.971 and no pairwise ordering reverses.The magnitudes of S shift with K, but the ranking does not.
Statistic definition: The median over (ℓ, h, q) represents the typical head, while a mean can be inflated by a few unusually strong heads.Replacing the median with the mean preserves both the sign and pair ordering, so the design choice changes none of the conclusions.

F.2 Modality Control: VL vs. Text-Only Trajectory

A matched trajectory from the same 3B base shows divergent modality effects: vision fine-tuning improves image understanding while weakening the sink, whereas text-only training strengthens the sink but slightly reduces IFEval.

Vision trajectory: The vision run raises SEEDBench-IMG from 60.1 to 64.8 while Sink Strength falls from +0.41 to +0.20 by step 10k.IFEval remains in the 34–44 band throughout the vision trajectory.
Text-only trajectory: The matched text-only run holds Sink Strength between +0.58 and +0.65 but gives up about 5 points of IFEval.Both runs start from the same Qwen2.5-3B-Instruct base and use matched batch size, learning rate, and step count, while differing in modality and data.
Temporal alignment: IFEval loses most of its vision-run performance by step 1k, before Sink Strength falls, then recovers slightly while Sink Strength continues declining.Thus, Sink Strength tracks the ending regime rather than step-level fluctuations.
Baseline: The step-0 IFEval score is 59.9, within 1.7 points of the published Qwen2.5-3B-Instruct score of 58.2.This establishes the reported trajectory from a closely reproduced baseline.
Control design: The injection experiment compares vanilla and QK-RMSNorm-injected 3B VLMs trained from the same base under the same LLaVA Stage-1 and Stage-2 recipe.The variants differ only in the injected QK-RMSNorm module.