PaperlayerView exact revision on arXiv

Source-linked AI summary

LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar, Alham Fikri Aji, Yova Kementchedjhieva

arXiv:2604.00829v3cs.CVcs.CL

TL;DR

Multimodal adaptation can damage a pretrained LM’s native linguistic capability, while standard fine-tuning may not fully recover it. LinguDistill uses the frozen LM as a teacher, shares KV caches to provide multimodal conditioning, and selectively distills on language-intensive data. It recovers language and knowledge benchmark performance while maintaining comparable performance on vision-heavy tasks and adds no parameters to the final VLM.

  • Problem

    Multimodal adaptation causes representation shift and cross-modal interference that degrade native language capability and are difficult to reverse with standard fine-tuning.

  • Method

    LinguDistill uses the frozen original LM as a teacher, layer-wise KV-cache sharing for vision-conditioned supervision, and selective distillation on language-intensive data.

  • Results

    LinguDistill recovers language and knowledge benchmark performance while maintaining comparable performance on vision-heavy tasks.

  • Takeaways & Limitations

    Targeted cross-modal distillation can preserve backbone capabilities in multimodal systems without additional modules or final-model inference overhead.

  • Takeaways & Limitations

    The teacher’s language signal makes the model worse at exact copying in document extraction tasks, including correcting strings, selecting nearby fields, and paraphrasing.

Abstract

from arXiv · show

Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representation shift and cross-modal interference introduced during multimodal adaptation. Such loss is difficult to recover, even with targeted task-specific fine-tuning using standard objectives. Prior recovery approaches typically introduce additional modules that act as intermediate alignment layers to maintain or isolate modality-specific subspaces, which increases architectural complexity, adds parameters at inference time, and limits flexibility across models and settings. We propose LinguDistill, an adapter-free distillation method that restores linguistic capability by utilizing the original frozen LM as a teacher. We overcome the key challenge of enabling vision-conditioned teacher supervision by introducing layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without modifying the architecture of either model. We then selectively distill the teacher's strong linguistic signal on language-intensive data to recover language capability, while preserving the student's visual grounding on multimodal tasks. As a result, LinguDistill recovers $\sim$10% of the performance lost on language and knowledge benchmarks, while maintaining comparable performance on vision-heavy tasks. Our findings demonstrate that linguistic capability can be recovered without additional modules, providing an efficient and practical solution to modality-specific degradation in multimodal models.

1 Introduction

Multimodal adaptation can degrade pretrained language models’ native linguistic ability, and standard fine-tuning may not fully reverse this loss. LinguDistill addresses the problem with adapter-free, teacher-guided cross-modal distillation using shared KV caches.

  • Multimodal adaptation degrades pretrained language models’ native language capabilities, including on purely textual inputs.
  • Representation shift and cross-modal interference make language degradation difficult to recover through lightweight language-focused fine-tuning.
  • LINGUDISTILL uses a frozen pretrained LM as teacher and a VLM as student optimized with a standard distillation objective.
  • Layer-wise KV-cache sharing exposes the teacher to the student’s multimodal context without modifying either model’s architecture.
  • The adapter-free method recovers language and knowledge benchmark performance by ∼10% while showing comparable degradation on vision-heavy tasks.

2 Related Work

Prior approaches preserve language ability by freezing the LM or use cross-modal distillation and KV-cache sharing to bridge modalities. These methods motivate LinguDistill’s parameter-free training-time reuse of the original LM teacher.

  • Frozen-backbone approaches preserve language capability by keeping the LM unchanged while adding modules that process visual inputs.
  • These approaches add parameters and therefore increase inference cost, whereas LinguDistill removes the teacher after training and adds no final-model parameters.
  • Cross-modal knowledge distillation trains a student to learn information from a teacher operating on a different modality.
  • KV-cache sharing lets one model attend to another component’s context without re-encoding it.
  • LinguDistill extends the student VLM’s layerwise KV cache to a frozen LM so the teacher can attend to visual context without parameter modification.

3 LINGUDISTILL

LinguDistill uses a VLM student and its frozen pretrained LM backbone as a teacher. Layer-wise KV-cache sharing provides multimodal teacher conditioning, while data-dependent distillation weighting emphasizes language-intensive examples and preserves visual supervision elsewhere.

  • LINGUDISTILL is a training-time framework that recovers a VLM’s native linguistic capability using its frozen instruction-tuned LM backbone as teacher.
  • The teacher is removed after training, leaving a standard VLM with no additional parameters or inference overhead.
  • KV Sharing Architecture: The student processes the full multimodal sequence and produces layerwise KV caches, while the teacher reprocesses the text prompt to form query states.
  • KV Sharing Architecture: The teacher directly reuses the student KV cache at every layer and attends with its queries over that transported memory.
  • Selective Distillation Objective: The KD objective combines the frozen teacher’s soft distillation with the student’s hard next-token supervision.
  • Selective Distillation Objective: Data-dependent weighting applies more teacher supervision to language-intensive data and more hard supervision to visual-heavy sources.

4 Setup

Experiments use a balanced subset of multimodal instruction-tuning data and compare baseline, uniform-KD, and selective-KD variants across distinct capability axes. Selective routing is designed to recover language-heavy performance while limiting harm to visual tasks.

  • Dataset Design: The experiments retain 17 core sources from The Cauldron, covering VQA, OCR/document, knowledge, and domain-specific tasks, with fewer than 600k examples.
  • Dataset Design: Language-heavy sources receive α > 0 KD, whereas OCR/document sources receive CE only with α = 0 in selective variants.
  • Experiment Runs: The study compares 10 run variants sharing the same nanoVLM-460M-8k base model and common data pool.
  • Experiment Runs: Baselines measure multimodal-adaptation degradation and data restriction, while uniform KD tests teacher-based recovery across all data.
  • Experiment Runs: Additional temperature and distillation-weight ablations test whether gains arise from selective routing rather than hyperparameter choice.
  • Evaluation: Benchmarks are grouped into language and knowledge-heavy, document/OCR/vision-specific, and general multimodal capability axes.

5 Results

Standard fine-tuning degrades language capability, while selective distillation recovers the strongest language gains with substantially less vision-task degradation than uniform distillation.

  • Baseline degradation: Standard fine-tuning consistently reduces language-intensive benchmark performance, including COCO captioning, MME cognition, HellaSwag, ARC Easy, and ARC Challenge.The supplied results also report minimal degradation on DocVQA and MMStar, indicating that the regression concentrates on linguistic capability.
  • Data filtering: Language-heavy data filtering partially improves some benchmarks but does not fully correct representation drift introduced by multimodal fine-tuning.Improvements are inconsistent across reasoning and knowledge benchmarks, with slight regressions on AI2D and ScienceQA.
  • Uniform distillation: Uniform distillation improves several language benchmarks but severely damages OCR and document tasks by applying the language-teacher signal across all sources.OCRBench drops by 37.7% and DocVQA by 16.6% relative to nanoVLM-full.
  • Selective distillation: Selective distillation applies KD only to language-heavy examples and achieves the strongest improvements across language and text-only reasoning benchmarks.Reported gains include COCO +28.7%, MME cognition +31.0%, HellaSwag +20.9%, ARC Easy +15.0%, and ARC Challenge +14.0%.
  • Modality balance: Selective distillation substantially reduces vision-task degradation relative to uniform distillation while improving InfographicVQA.DocVQA drops 3.5% versus 16.6% under uniform distillation, and OCRBench drops 17.4% versus 37.7%.
  • Overall result: Overall, selective distillation maximizes language gains while preserving visual capability significantly better than uniform distillation.The comparison identifies a stronger trade-off between linguistic recovery and visual performance.

6 Discussion

The discussion analyzes what LinguDistill recovers, where visual-text failures arise, and why selective distillation and balanced optimization are important.

  • What Does Distillation Recover?: On ScienceQA, LinguDistill fixes 854 examples lost by nanoVLM-full while losing 498, yielding a net gain of +8.39 percentage points.The recovered examples span natural science, language science, and social science.
  • What Does Distillation Recover?: On AI2D, LinguDistill wins 645 examples and loses 362, for a +9.16 percentage-point gain concentrated in food-web reasoning and biology diagrams.InfoVQA similarly improves because its questions require interpreting infographic meaning rather than only reading text.
  • Effect on Visual Text Tasks: OCRBench degradation is concentrated in Key Information Extraction and Doc-oriented VQA, which together account for 71% of the total loss.Basic text recognition changes by only −4/150 points, while the larger losses involve selecting fields in structured documents.
  • Effect on Visual Text Tasks: LinguDistill’s document-extraction failures involve correcting nonsense strings, selecting nearby fields, and paraphrasing instead of copying exactly.These failures arise in document extraction tasks and do not spread to other benchmarks.
  • Selective Distillation: Data-subset choice alone does not significantly affect the gains, indicating that recovery requires architectural or optimization-level intervention.The comparison finds no substantial difference between full and language-focused subsets for baseline and uniform KD settings.
  • Selective Distillation: Selective distillation preserves OCR capability and improves text-only benchmarks by applying teacher supervision where it is reliable and suppressing it on vision-heavy tasks.This design is presented as preventing misleading supervision while preserving visual grounding.
  • Distillation Hyperparams: The main LinguDistill setting achieves the lowest CE curve among the selective variants, whereas both stronger and weaker KD configurations are worse.The analysis indicates that excessive teacher pressure harms direct supervision, while too little KD is also suboptimal.

7 Conclusion

LINGUDISTILL recovers linguistic capability in VLMs through adapter-free distillation from a frozen LM teacher and layer-wise KV-cache sharing. It improves language and knowledge benchmarks while maintaining comparable vision-heavy performance without additional parameters.

  • LINGUDISTILL uses a frozen LM teacher, layer-wise KV-cache sharing, and selective distillation to restore linguistic capability after multimodal adaptation.The framework is adapter-free and surpasses the standard fine-tuning objective with zero additional parameters.
  • LINGUDISTILL recovers performance on language and knowledge benchmarks while maintaining comparable performance on vision-heavy tasks.
  • Targeted cross-modal distillation provides a simple approach to preserving backbone capabilities in multimodal systems without additional modules.

A Training Configuration

All variants share a common training configuration built around nanoVLM-full, a frozen SigLIP2 vision encoder, and a SmolLM2-360M-Instruct language decoder. Training uses The Cauldron with BF16 precision, AdamW, cosine decay, and a 1024-token sequence limit.

  • Table 4 lists hyperparameters shared across all variants.
  • The shared configuration uses BF16 precision, AdamW, a 1×10−4 projector learning rate, cosine decay, 3% warmup, and norm-1.0 gradient clipping.The vision encoder remains frozen with learning rate 0.
  • All experiments use nanoVLM-full, a 460M-parameter model pairing a frozen SigLIP2-B/16 vision encoder with a SmolLM2-360M-Instruct decoder.Runs initialize from the lusxvr/nanoVLM-460M-8k checkpoint and train on a filtered subset of The Cauldron.
  • Distillation variants use a frozen copy of the original SmolLM2-360M-Instruct backbone as teacher, with zero teacher learning rate and evaluation at step 4000.

B Evaluation Subsets

Evaluation uses lmms-eval under a standardized zero-shot protocol and groups tasks by language and knowledge, document and OCR, and general multimodal capabilities. The optimization pseudocode describes shared-KV continuation from a multimodal left decoder to a frozen right decoder.

  • Evaluation protocol: All models are evaluated with lmms-eval using a standardized zero-shot protocol, fixed batch size 1, and single-GPU inference.Predictions are logged for further analysis.
  • Evaluation subsets: Language and knowledge-heavy tasks include ARC Easy, ARC Challenge, HellaSwag, ScienceQA, and COCO2017 captioning.
  • Evaluation subsets: Document and OCR tasks include OCRBench, DocVQA, InfoVQA, and RealWorldQA, while general multimodal tasks include AI2D, MMStar, MME, and MMMU validation.
  • Optimization pseudocode: The optimization algorithm updates left-side VLM parameters using detached right-decoder generation built by continuation over shared KV memory.The objective returns a scalar loss and updated left-side parameters.
  • Optimization pseudocode: Figure 3 shows the image mapped into the language subspace before the left decoder materializes prompt KV memory for the frozen right decoder.The objective combines source-selective distillation with hard next-token supervision and updates only left-side parameters.

D Sequence Length Analysis

The 1024-token training limit covers nearly all examples while prioritizing throughput under compute constraints. Long-context document benchmarks may still regress, and MMMU’s multi-image reasoning exceeds the single-image training setting.

  • Token-length statistics: The 1024-token training limit leaves ample headroom because most examples fit within the budget and the overall mean length is under 120 tokens.Only 2.4% of Robut SQA and 2.1% of ScienceQA examples exceed the threshold; OCR and document sources average under 130 tokens.
  • Evaluation impact: Evaluation includes short-sequence tasks such as MME and ARC alongside longer-sequence tasks including DocVQA, InfoVQA, OCRBench, and MMMU.Generation uses a separate maximum output length, so the training limit does not directly constrain evaluation.
  • Limitations: Residual regressions of −3.8% on DocVQA and −7.6% on InfoVQA are consistent with indirect effects from insufficient long-sequence training exposure.The passage presents this as an explanation for long-context document-understanding regressions.
  • Limitations: MMMU requires multi-image reasoning, which the single-image training constraint does not support.
  • Compute trade-off: Fewer than 1% of training examples exceed 1024 tokens, so the setup prioritizes training throughput over marginal coverage.Future work can explore longer sequence lengths for document-heavy benchmarks.

E.1 Sub-task Breakdown

LINGUDISTILL largely preserves decoding-only text recognition, while regressions concentrate in OCR tasks requiring spatial grounding, especially locating the correct fields in structured documents.

  • Text recognition: Text recognition remains largely preserved, scoring 133/150 for LINGUDISTILL versus 137/150 for nanoVLM-full.Regular text drops by one point, while artistic text improves by one point.
  • Handwriting & non-semantic text: Handwriting and non-semantic tasks drop by 23 points, reaching 153/250 versus 176/250 for nanoVLM-full.The largest decline occurs on non-semantic text, where ground-truth strings are intentionally not real words.
  • Layout-dependent tasks: Layout-dependent tasks account for most of the OCRBench regression, with Key Information Extraction and Doc-oriented VQA contributing 89 of the 126-point total drop.These tasks require selecting the correct field among nearby text regions in structured documents.
  • Failure patterns: In failure analysis, the model often converts ambiguous or non-semantic strings into plausible words, including PEAEC becoming peace.LINGUDISTILL fails on 11 of 50 non-semantic text items versus 4 for nanoVLM-full; 9 of 20 handwriting failures are attributable to distillation.
  • Failure patterns: The model also selects fluent text from the wrong document location, such as returning store hours instead of a receipt date or a subtotal instead of the total.These answers come from the document but from nearby incorrect fields.
  • Failure patterns: A further failure mode is paraphrasing rather than copying, causing numerical precision loss, value substitutions, and shortened descriptions or names.The model may identify the correct region but rewrite its contents instead of reproducing them exactly.
Loading 2604.00829v3…