PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Teach Large Multimodal Models New Skills

Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem

arXiv:2510.08564v2cs.AIcs.CVcs.LG

TL;DR

The paper asks how to teach LMMs new skills without degrading existing capabilities, since fine-tuning can degrade broad vision–language ability. The study sequentially fine-tunes five target skills while monitoring eight held-out benchmarks and probes output-token distribution shift with counting bias. Across a five-target task sequence, eight held-out benchmarks, and three model families, tuning MLP up&gate layers achieved +30.5 learning / -2.1 forgetting.

  • Problem

    The paper asks how to teach LMMs new skills without degrading existing capabilities, since fine-tuning can degrade broad vision–language ability.

    Source: §1 · p. 2

  • Method

    The study sequentially fine-tunes five target skills while monitoring eight held-out benchmarks and probes output-token distribution shift with counting bias.

    Source: §1 · p. 3

  • Results

    Across a five-target task sequence, eight held-out benchmarks, and three model families, tuning MLP up&gate layers achieved +30.5 learning / -2.1 forgetting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Selective tuning of SA Proj. or MLP Gate&Up, and distillation, can limit output-distribution shift while supporting sequential skill learning across three model families.

    Source: §6 · p. 14

    The evaluation does not cover massive or commercial multimodal models or training data scaled to billions of samples.

    Source: §6 · p. 15

Abstract

from arXiv · show

How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. Surprisingly, we find that performance lost on held-out tasks after fine-tuning on one skill can partly recover when the model is subsequently tuned on a different skill. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that shows the shift co-varies with forgetting. Guided by this insight, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers (SA Proj., $\Delta$ learning +24.9 / $\Delta$ held-out forgetting -0.6), and (ii) updating only the MLP Gate&Up while freezing the Down projection (+30.5 / -2.1). Both substantially outperform full-LLM tuning (+31.8 / -23.3) in the learning-forgetting trade-off. We also compare against common forgetting mitigation methods: Learning without Forgetting (LwF), LoRA, Mixture-of-Experts, and weight-space interpolation (WiSE-FT), and find that our selective tuning recipes match or exceed their learning-stability balance while remaining simpler, requiring no replay, auxiliary parameters, or per-stage tuning. These results hold across LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL, confirming that the key to teaching LMMs new skills without forgetting lies in controlling output distribution shift by choosing which components to tune. Code will be made available.

1 Introduction

The paper studies how to teach LMMs new skills while preserving existing abilities, and finds that forgetting varies with the parameters tuned and can partly recover across stages. It links forgetting to output-token distribution shift and identifies selective tuning recipes that retain learning with limited drift.

1 Introduction

  • The study frames rehearsal-free skill adaptation as a way to avoid costly retraining, while noting that naïve task-by-task fine-tuning can degrade broad vision–language ability.The authors report that forgetting depends strongly on which parameters are updated.Source: §1 · p. 2
  • The evaluation sequentially fine-tunes five practical skills while measuring average performance drops across eight held-out vision–language benchmarks.The skills span bird classification, counting, medical VQA, OCR, and time reading.Source: §1 · p. 2
  • Performance lost on held-out tasks after one stage can partly recover when the model is tuned on another specialized task.The paper reports this recovery alongside its finding that SA Proj. tuning has little or no measurable forgetting across a five-task sequence.Source: §1 · pp. 2–3
  • A counting-bias probe links increased numeric-token likelihood to held-out performance drops; SA Proj. tuning learns counting with little numeric-token bias.The shift in numeric-token likelihood grows with MLP tuning and remains near baseline for self-attention tuning.Source: §1 · p. 3
  • The authors present their work as a systematic study of rehearsal-free sequential skill adaptation with broad held-out evaluation and partially recoverable forgetting.Their evaluation covers five target skills, eight held-out benchmarks, and three model families.Source: §1 · p. 3
  • +30.5 learning / -2.1 held-out forgetting for MLP Gate&Up, and +24.9 / -0.6 for SA Proj., across five target skills, eight held-out benchmarks, and three model families.Both selective recipes limit forgetting while learning new skills.Source: §1 · p. 3

2 Related work

Prior continual-learning research spans regularization, replay, and network expansion, with additional work on prompt tuning and mitigation strategies for LLMs and LMMs. This study focuses instead on generative LMM skill adaptation and decoder-component tuning, relating its findings to output-distribution shift.

2 Related work

  • Continual-learning methods include regularizing representations, replaying stored examples, expanding networks, and adding learnable prompts while freezing model weights.The cited foundational work largely concerns closed-vocabulary image classification.Source: §2 · p. 4
  • LLM forgetting studies compare full tuning, adapters, LoRA, prompts, and interpolation; prior findings include both reduced learning and forgetting with LoRA and retention benefits from WiSE-FT.Related approaches also include replay, pseudo-data, regularization, and orthogonal LoRA updates.Source: §2 · p. 4
  • The study distinguishes its setting from vision-text contrastive-model approaches by addressing generative LMMs and investigating which decoder components to update.Its stated evaluation covers rehearsal-free sequential adaptation across five skills, eight held-out benchmarks, and three model families.Source: §2 · p. 5
  • Prior LMM continual-learning work includes replay, distillation, and architectural expansion, often for VQA or captioning; this study targets generative LMM decoder components.The paper also situates its view of attention as routing and MLPs as key–value memories within mechanistic transformer research.Source: §2 · pp. 4–5
  • The authors identify SA Proj. and MLP Gate&Up with Down frozen as tuning recipes that learn with limited forgetting and require neither added modules nor rehearsal data.Their mechanistic analysis ties much observed forgetting to output-token distribution drift.Source: §2 · p. 6

3 Method

The method evaluates rehearsal-free adaptation of pretrained LMMs by measuring target-task learning and held-out performance, then compares which model components to tune. Its design is motivated by the roles of attention and MLPs in shaping decoder representations and output distributions.

3 Method

  • The evaluation adapts a pretrained LMM to single target tasks or a sequential task stream, without rehearsal unless noted, and assesses target and held-out performance after each stage.Learning and forgetting are measured as target-task improvement and average held-out-suite drop.Source: §3 · p. 6

3.1 Model

  • An LMM combines a vision encoder and projector with a decoder-only language model; visual tokens are projected to language width and contribute to the decoder input.The decoder updates residual states through self-attention and MLP sublayers, and its final state contributes to output logits.Source: §3.1 · pp. 6–7

3.2 Which parts to tune?

  • The component-selection rationale treats attention as routing and MLPs as key–value memories that influence next-token distributions, motivating updates to SA Proj. or MLP Gate&Up with Wdown frozen.The compared variants include SA Proj., SA Proj. (QKV), MLP, and MLP (Gate & Up).Source: §3.2 · p. 8

3.3 Training objective

  • Training uses teacher-forced next-token cross-entropy on the current target dataset; an optional Learning-without-Forgetting term distills outputs from the previous-stage frozen model.The distillation term uses a random subset of token positions capped at 1,000, with a coefficient balancing new-task fitting and preservation of earlier behavior.Source: §3.3 · p. 8

4 Experiment Design

The evaluation is organized around four questions: which components to tune, why forgetting occurs, whether selective tuning generalizes across model families, and how it compares with other mitigation approaches. It uses sequence-level metrics to track learning on current and previously trained target tasks and performance on held-out benchmarks.

4 Experiment Design

  • The study asks where to tune, whether forgetting tracks output-distribution shifts, whether selective tuning generalizes, and how it compares with other mitigation approaches.Additional task and implementation details, per-task tables, and extended analyses are in the Appendix.Source: §4 · pp. 8–9

4.2 Sequence-level metrics

  • Four sequence-level metrics distinguish gains on the current task, losses on earlier target tasks, end-of-sequence target performance, and changes on held-out benchmarks.Negative held-out values indicate forgetting; positive values indicate positive transfer.Source: §4.2 · pp. 9–10

4.1 Tasks and evaluation suites

  • The metrics are applied to a sequential curriculum and held-out evaluation suite, with Table 1 summarizing component-tuning results averaged over three five-task sequences.The table uses a blue-orange color scale for performance changes and underlining to mark results not significantly different from the best under paired tests.Source: Table 1 · p. 10, §4.2 · p. 10

5 Results

Selective tuning of self-attention projections or MLP Gate&Up offers a strong learning–stability balance, and the paper relates forgetting to output-distribution shifts. The results also report similar tuning patterns across additional backbones and comparisons with common mitigation methods.

5.1 Component tuning on LLaVA-OneVision

  • On LLaVA-OneVision, MLP (Gate&Up) delivers +30.5 Target Learning and +27.1 Target Overall with Held-out Forgetting -2.1; SA Proj. reaches +23.1 Target Overall with Held-out Forgetting -0.6.Source: §5.1 · p. 11

5.2 Output-distribution probe (counting bias)

  • Numeric-token bias is strongly associated with held-out forgetting across training steps and methods (Spearman ρ = −0.84), and subsequent fine-tuning can partly reverse forgetting.The counting-tuned MLP also reframes captions as counting statements, while SA Proj. retains detailed captioning ability.Source: Figure 3 · p. 11, §5.2 · p. 12

5.3 Beyond LLaVA-OneVision: generalization to other backbones

  • Across LLaVA-NeXT and Qwen2.5-VL, SA Proj. remains a stable choice, while MLP (Gate&Up) preserves much of full-MLP learning with limited forgetting.The preferred recipe depends on the objective: SA Proj. for held-out stability, or MLP (Gate&Up) for near-maximal target learning with limited forgetting.Source: §5.3 · p. 13, Table 2 · p. 13

5.4 Mitigating forgetting

  • SA Proj. and MLP (Gate&Up) match or exceed the compared mitigation methods' learning–stability balance without extra modules, replay, or per-stage weight blending.WiSE-FT requires task-dependent blending coefficients, while LwF can reduce target-task gains.Source: §5.4 · p. 14, Figure 5 · p. 14

6 Conclusion

The study concludes that selective tuning or LwF-like distillation can teach LMMs new skills while limiting forgetting by constraining output-distribution shift. The evaluation does not cover massive or commercial models or training data scaled to billions of samples.

6 Conclusion

  • Restricting updates to SA Proj. or MLP Gate&Up, or using LwF-like distillation, can teach new skills while limiting forgetting.The authors link these approaches to constraining output-distribution shift.Source: §6 · p. 14
  • A counting-bias probe and layerwise logit analysis show a strong relationship between measured forgetting and output shift, alongside recovery across task stages.The reported trends hold across three model families, five target skills, and eight broad benchmarks.Source: §6 · p. 14
  • The evaluation does not extend to massive or commercial multimodal models or test selective tuning with billions of training samples.These are identified as primary limitations of the work.Source: §6 · p. 15

A.1 Tasks and evaluation suites

The appendix specifies a five-skill sequential curriculum, eight held-out benchmarks, and task-specific evaluation procedures. It also documents the curriculum orders, implementation settings, and parameter-group accounting across the evaluated model families.

A.1 Tasks and evaluation suites

  • Five target skills—bird classification, counting, medical VQA, OCR reading, and time reading—form a 107,910-sample curriculum, evaluated alongside eight held-out vision–language benchmarks.The held-out suite includes AI2D, ChartQA, DocVQA, InfoVQA, RealWorldQA, SeedBench, ScienceQA, and MMStar; ANLS scores are averaged with accuracies.Source: §A.1 · p. 22

A.2 On building target tasks

  • Task construction uses multiple-choice bird labels and task-specific prompts and scoring, including exact match for counting and medical VQA and parsed hour-and-minute accuracy for time reading.The bird task mixes its correct label with 31 randomly chosen labels; counting validation numbers are reported because some download links were invalid.Source: §A.2 · pp. 22–23

A.3 Task curriculum

  • Three task orders are provided, with the CUB200 → PixmoCount → PathVQA → TextVQA → TimeClock sequence used by default unless stated otherwise.The other sequences begin with PathVQA and TimeClock, respectively.Source: §A.3 · p. 23

B.1 Implementation details for LLaVA-OneVision

  • LLaVA-OneVision experiments use a 7B Qwen2 checkpoint, primarily on four H100 GPUs, with single-epoch training and a 5×10^-6 learning rate for the language model.The vision encoder and projector use a smaller 2×10^-6 learning rate; LwF uses λ = 1 and τ = 2.Source: §B.1 · pp. 23–24

B.2 Parameter count for LLaVA-OneVision

  • The parameter accounting identifies the language model as the model’s largest capacity component and its MLP as the largest parameter consumer within that model.Table 3 lists parameter groups and counts for LLaVA-OneVision Qwen2-7B.Source: §B.2 · p. 24, Table 3 · p. 24

B.3 Implementation details for Qwen2.5-VL and LLaVA-NeXT (LLaMA 3)

  • Qwen2.5-VL uses a 7B Qwen2.5 checkpoint, while LLaVA-NeXT uses an 8B LLaMA-3 checkpoint; both use single-epoch training.Qwen2.5-VL uses an effective batch size of 64 and a 2e-5 learning rate; LLaVA-NeXT follows LLaVA-OneVision’s learning rate, warm-up, batch-size, and accumulation settings.Source: §B.3 · p. 24

B.4 Implementation details for Sec. 5.2: Output-distribution probe (counting bias)

The counting-bias probe estimates a checkpoint’s tendency to assign probability to numeric tokens during generated captions. The paper relates this measure to held-out performance and specifies the token set used.

B.4 Implementation details for Sec. 5.2: Output-distribution probe (counting bias)

  • The probe measures the maximum next-token probability among a fixed subset of numeric tokens at each caption position, averages across positions and examples, and aggregates the result by checkpoint.It uses a held-out batch of 100 image–caption pairs and deterministic greedy decoding with identical settings across checkpoints and methods.Source: §B.4 · pp. 24–25
  • Spearman’s ρ = −0.84 relates average held-out score to numeric-token bias across checkpoints and methods; higher bias accompanies worse held-out performance.The correlation is reported for Fig. 3.Source: §B.4 · p. 25

B.5 Numerical token list used for counting bias probe

  • For the counting bias probe, the set comprised 38 numeric tokens, including numeric digits and words such as “one” and “ONE”; Table 4 lists the tokens and their tokenizer indices.Source: §B.5 · p. 25, Table 4 · p. 25

C Discovery Process

The discovery process moved from documenting forgetting and unexpected cross-task recovery to testing selective tuning and an output-distribution explanation. The authors report robust trends across model families and task orders, while noting resource limits on further experiments.

C Discovery Process

  • SA Proj. tuning learned across the five-task sequence with no measurable held-out drop, while forgetting after one stage could recover after another specialized task.The initial recovery observation followed counting-task training and subsequent PathVQA tuning.Source: C Discovery Process · p. 26
  • Across LLaVA-NeXT and Qwen2.5-VL, major trends held; LLaVA-NeXT SA Proj. tuning had a 7.7-point held-out drop, less than half the forgetting from MLP tuning and less than one-quarter of full-LLM tuning.The authors also report recovery when reversing task order and LwF-like distillation as effective for mitigating forgetting when tuning the MLP.Source: C Discovery Process · p. 26
  • The authors hypothesized that full-LLM tuning can bias the output distribution, with SA Proj. avoiding direct MLP updates and varied subsequent tasks potentially reversing narrow shifts.They describe MLP up layers as activating concepts and down layers as applying them to modify the output token distribution.Source: C Discovery Process · p. 27
  • The counting-bias tests supported the hypothesis: numeric-token bias increased during counting and correlated strongly with forgetting, while tuning MLP up layers without down layers achieved the best overall target performance with little more forgetting than SA Proj.The proposed second experiment tested whether leaving down layers untuned would mitigate output bias and forgetting.Source: C Discovery Process · p. 27
  • For Table 1, the 7B model was fine-tuned on five tasks 21 times and evaluated on eight broad benchmarks and five target tasks 105 times, yielding 105 trainings and 1,365 evaluations.Each target-task evaluation followed training on that target task; the authors said further experiments were constrained by resources.Source: C Discovery Process · p. 27

D.1 Single task fine-tuning

Across single-task and sequential tuning, updating selected language-model components can improve target skills while limiting held-out forgetting. The analysis links greater output-distribution drift to forgetting and shows that tuning the first 23 layers can retain strong learning with less forgetting than full-LLM tuning.

D.1 Single task fine-tuning

  • +31.6 learning with -1.4 forgetting for MLP tuning, while SA Proj. achieves +24.2 learning with no measurable forgetting in single-task tests.Full-network and full-LLM tuning achieve +33.3 and +31.8 learning, respectively, alongside -3.2 and -2.8 forgetting; task-specific results also vary, with CUB200 and PathVQA benefiting almost exclusively from language-model updates.Source: §D.1 · p. 28, Table 5 · p. 28

D.2 Sequential fine-tuning

  • MLP (Gate&Up) gives the best overall target-task performance in the five-task sequence, while SA Proj., SA Proj. (QKV), and Gate&Up keep held-out results stable.Held-out performance can also rebound after a later task: full-model performance fell from 0.76 to 0.63 after PixmoCount, then rose to 0.74 after PathVQA.Source: §D.2 · p. 29, Figure 6 · p. 29

E.1 Composing stable tuning strategies: SA Proj. + MLP Gate&Up

  • Combining SA Proj. with MLP (Gate&Up) matches or slightly underperforms the standalone settings in target learning, and neither composition consistently dominates Gate&Up alone.The QKV-only composition has better held-out performance than the other composition.Source: §E.1 · p. 30, Figure 9 · p. 31

E.2 Layer-wise residual–to–logit contribution analysis

  • MLP pathways contribute more logit-space change than self-attention, with the last 4–5 layers—especially the final two—accounting for most of the shift.Drift increases with training in forgetting-prone settings, while Gate&Up and LwF shrink late-layer MLP magnitudes relative to full MLP.Source: §E.2 · pp. 31, 33, Figure 10 · p. 32

E.3 Effect of tuning different LLM layers

  • Tuning the first 23 layers yields target-task performance comparable to or above full-LLM tuning while largely alleviating forgetting, including the PixmoCount drop.Tuning only the last five layers yields minimal learning gains and little forgetting.Source: §E.3 · p. 33, Figure 11 · p. 33

E.4 Qualitative results of MLP and SA Proj. tuning

  • After PixmoCount, MLP tuning can reframe a captioning request as a counting task, whereas SA Proj. retains detailed captioning while answering counting examples correctly.The MLP example still recognizes image contents, illustrating a changed response behavior rather than loss of object concepts.Source: §E.4 · pp. 33–34

E.5 Training efficiency comparison of different forgetting mitigation methods

The efficiency comparison reports throughput and trainable-parameter costs for the evaluated tuning variants. A three-stage probe finds that numeric-token likelihood rises alongside held-out forgetting after counting training and recedes as performance recovers after PathVQA, without fully returning to baseline.

E.5 Training efficiency comparison of different forgetting mitigation methods

  • SA Proj. achieves highest throughput at 1.46 samples / sec. using only 0.82 B trainable parameters; MLP (Gate&Up) follows closely at 1.45 samples / sec.LwF maintains a teacher network, resulting in the largest total parameter count of 16.06 B and a reduced throughput of 0.81 samples / sec. because of the extra forward pass and distillation loss.Source: §E.5 · p. 34
  • Among the compared forgetting-mitigation methods, LoRA uses the fewest trainable parameters, at 0.5 B; the study compares model size, trainable parameters, and training speed.Experiments used four NVIDIA H100 96 GB GPUs with DeepSpeed in bfloat16 and no competing processes; each run processed exactly 384 training samples.Source: §E.5 · p. 34

E.6 Output Distribution Dynamics Across Forgetting and Recovery

  • After PixmoCount, numeric-token likelihood rises as held-out performance drops; after PathVQA, likelihood falls and held-out performance recovers, but not fully to baseline.The authors hypothesize that PathVQA introduces other task-specific shifts, and note that quantifying overlapping shifts remains future work.Source: §E.6 · pp. 35–36, Figure 12 · p. 35

E.7 Qualitative Analysis of Forgetting and Recovery

Qualitative examples show captioning behavior shifting toward counting after PixmoCount and recovering after PathVQA. The accompanying method descriptions specify how LoRA, WiSE-FT, and MoE implement alternative forgetting-mitigation strategies.

E.7 Qualitative Analysis of Forgetting and Recovery

  • After PixmoCount, LLM and MLP models can answer a general caption prompt with a counting-style response, then regain descriptive captioning after PathVQA.The example retains object recognition while changing the response framing.Source: §E.7 · pp. 36–37, Figure 13 · p. 36

F Forgetting mitigation approach descriptions

  • LoRA uses trainable low-rank matrices A and B, merging each task's update into the backbone before discarding the adapters.This keeps the backbone frozen during task optimization and avoids accumulating task-specific modules.Source: F Forgetting mitigation approach… · p. 37
  • WiSE-FT interpolates base and fine-tuned weights, with β trading off current-task specialization against retention of general capabilities.The reported comparison uses β=0.3 after a sweep of candidate coefficients.Source: F Forgetting mitigation approach… · pp. 37–38
  • MoE freezes a pretrained expert and updates a copied expert plus its gate, reusing the pair for each new task.The pretrained expert remains available as a safeguard when the tuned expert fails.Source: F Forgetting mitigation approach… · p. 38

G Detailed task performance

The appendix provides detailed sequential-tuning performance tables for forgetting-mitigation baselines and component choices. The tables cover LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL.

G.1 Forgetting mitigation methods sequential tuning tables

G.2 Sequential tuning detailed performance tables on LLaVA-OneVision

  • LLaVA-OneVision tables report sequential tuning results for the full model, vision encoder, projector, LLM, SA projections, MLP, and MLP (Gate&Up).The appendix also includes the MoE performance table for this model.Source: §G.2 · p. 38, Table 7 · p. 39, Table 11 · p. 41

G.3 Sequential tuning detailed performance tables on LLaVA-NeXT (LLaMA 3)

  • LLaVA-NeXT (LLaMA 3) tables cover the full model, vision encoder plus projector, LLM, SA projections, MLP, and MLP (Gate&Up).The individual table captions specify these tuning configurations.Source: §G.3 · p. 39, Table 19 · p. 45, Table 20 · p. 46

G.4 Sequential tuning detailed performance tables on Qwen2.5-VL

  • Qwen2.5-VL tables cover the full model, vision encoder plus projector, LLM, SA projections, MLP, and MLP (Gate&Up).The individual table captions specify these tuning configurations.Source: §G.4 · p. 39, Table 25 · p. 48, Table 26 · p. 49
Loading 2510.08564v2…