MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models

• Smaller LVLMs are more susceptible than larger models to spurious language cues and prompt formatting in multimodal ICL.Prompt configuration has sought to improve ICL, but its effectiveness remains constrained by intrinsic model capabilities.Source: §1 · p. 1
• Output-distribution matching leaves unresolved whether a student uses relevant contextual evidence or superficial correlations, while attention-pattern distillation adds cost and training instability.Source: §1 · p. 1
• MCD uses structure-preserving, attribute-matched token replacements and teacher gradients to identify evidence, then verifies it by retaining or removing selected tokens.It transfers both evidence sufficiency and causal dependence without mechanism annotations or attention alignment.Source: §1 · p. 2
• 7.23 on average improvement and 4.68 over vanilla distillation are reported for MCD across three LVLM families and seven benchmarks.Source: §1 · p. 2

• Multimodal ICL lets models infer visual grounding, output formats, and input-output mappings from a few examples without parameter updates (Sun et al., 2024).Source: §2 · p. 2
• Prompt-configuration approaches provide limited improvements to underlying reasoning, while smaller models may rely on textual cues or superficial structural patterns (Li et al., 2026a; Chen et al., 2025a; Xu et al., 2025a; Zheng et al., 2025; Yang et al., 2024; Fu et al., 2025).Source: §2 · p. 2
• LVLM distillation methods model visual-textual interactions through attention or relational alignment, but attention-map reliance may not faithfully capture causal dependencies (Zhou et al., 2026a; Feng et al., 2025; Cai et al., 2025b; Kim et al., 2025; Chen et al., 2026).Source: §2 · p. 2
• LeaF uses interventions to reveal token interactions in long-context text tasks, but its pruning-based interventions limit applicability to multimodal ICL (Guo et al., 2026).Source: §2 · p. 2

3.1 Overview

• MCD discovers and verifies teacher-used evidence through structure-preserving interventions, then transfers evidence sufficiency and causal dependence alongside standard distillation objectives.It separates prompt structure from content-bearing tokens and uses attribute-matched replacements that preserve functional or spatial position while changing semantics.Source: §3.1 · p. 2

3.2.1 Vanilla Vision-language Distillation

• Vanilla output distillation imitates the teacher but does not sufficiently supervise how to exploit cross-modal semantics, limiting generalization.In multimodal ICL, the same answer can arise from relevant evidence or superficial correlations such as textual biases and demonstration order.Source: §3.2.1 · p. 3

3.2.2 A Causal View of Multimodal ICL

• Query mechanisms and demonstration evidence jointly shape the model state, while prompt structure and modality bias can provide competing paths to its prediction.Interventions replace selected query or demonstration content upstream while preserving prompt structure.Source: §3.2.2 · pp. 3–4

3.3 Causal Evidence Discovery

• Gradient scores rank content-bearing text and visual tokens, and MCD selects a fixed fraction r as candidate evidence while leaving structural tokens unchanged.The candidate set is determined by the prompt; token replacements preserve specified structural and positional properties.Source: §3.3 · p. 4

3.4 Causal Effect Verification

• Only examples where retaining selected evidence reproduces the teacher response more faithfully than removing it receive causal supervision.The method retains the top fraction ρ of examples with cT > 0; other examples use only standard supervised terms.Source: §3.4 · pp. 4–5

3.5 Causal Behavior Distillation

• Students are trained to recover the teacher’s full-prompt prediction when selected evidence is retained and to match the teacher’s prediction change when it is removed.Both behaviors are evaluated on fixed teacher-generated prefixes.Source: §3.5 · p. 5

3.6 Training Objective and Efficiency

• The training objective combines supervised learning, vanilla distillation, and causal distillation; after distillation, students reason more effectively over token-level evidence in multimodal ICL.Teacher generations, verification scores, and output distributions are cached before student training; MCD uses input embeddings and output distributions without aligning attention matrices.Source: §3.6 · pp. 5–6

4.1 Setup

• Training uses 60K teacher-correct prompts; evaluation covers three LVLM families and seven benchmarks, with main experiments using four-shot prompts.The training prompts are drawn from multimodal ICL benchmarks and additional prompts constructed with TACO.Source: §4.1 · p. 6

4.2 Main results

• MCD achieves the best student performance on every benchmark, averaging 47.95, 58.22, and 66.38, and beating Vanilla KD by 3.46 to 6.06 points.It also outperforms Align-TI by 1.47 to 2.16 points across the three families.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Ablation Study

• MCD maintains substantial gains over the original student and Vanilla KD across ICD counts and retrieval strategies, improving further as prompt quality increases.The comparisons vary both the quantity of demonstrations and their retrieval quality.Source: Figure 3 · p. 7, §4.3 · p. 7
• Average score falls from 66.34 to 64.89 without ℓkeep, 64.36 without ℓeffect, and 63.47 without verification; removing verification hurts most.The objectives capture distinct aspects of causal behavior, and verification checks whether selected evidence induces a reliable teacher response.Source: §4.3 · pp. 7–8, Table 2 · p. 8
• Replacing gradient ranking with random selection, attention selection, or unrestricted replacement lowers 66.34 to 62.26, 64.59, and 63.85, respectively.The ablations support using teacher-response gradients and replacements that preserve tokens’ functional and spatial roles.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Analysis

• On held-out prompts, MCD reduces evidence-sufficiency and causal-response errors by 33.3% and 30.0% versus Vanilla KD, while raising correlation by 0.23.The analysis measures how closely each student matches the teacher’s responses to evidence interventions.Source: Figure 4 · p. 8, §4.4 · p. 8
• Verified evidence shifts by task: query tokens are 55% and 54% on VQAv2 and MMStar, ICD tokens 58% on MathVision, and LogicVista is 52% ICD versus 48% query.Evidence composition also varies by modality across the four benchmarks.Source: §4.4 · p. 8

5 Conclusion

• MCD identifies and verifies mechanism-bearing evidence through token-level interventions, then transfers its sufficiency and causal influence to the student.The framework transfers multimodal ICL capabilities from larger LVLMs to smaller models within the same family.Source: §5 · p. 8
• Experiments across three LVLM families and seven benchmarks show consistent improvements over original students and existing distillation methods.The paper presents this as progress toward compact LVLMs with stronger multimodal reasoning capabilities.Source: §5 · p. 9

Limitations

• MCD currently requires teacher and student models from one LVLM family that share a tokenizer and image processor.The paper identifies cross-family distillation with differing tokenization schemes and visual encoders as a future direction.Source: Limitations · p. 9
• Experiments cover standard few-shot ICL prompts with a moderate number of demonstrations and a single question-answering query.Long-context prompts with many interleaved images, tool-augmented agents, and interleaved image-text chain-of-thought are identified as future extensions.Source: Limitations · p. 9
• The explored setting does not include emerging think-with-image scenarios requiring iterative inspection and integration of multiple visual evidence pieces.The paper presents these scenarios as a natural next step for extending MCD’s causal supervision.Source: Limitations · p. 9

A.1 Training Details

• Models train for three epochs with AdamW, a 2×10−5 learning rate, batch size 32, cosine decay, and linear warmup.The maximum sequence length is 4,096, and experiments use eight NVIDIA H200 GPUs.Source: §A.1 · p. 11
• VL-ICL requires extracting task information from demonstrations, while TrueMICL limits shortcuts by language priors, formatting, and demonstration order.SMMILE adds expert-curated medical ICL cases across imaging modalities and diagnostic tasks, emphasizing fine-grained visual features and multi-step clinical reasoning.Source: §A.1 · pp. 11–12
• GQA targets compositional visual question answering, and HatefulMemes tests whether predictions use image-caption interactions rather than unimodal shortcuts.MME-RealWorld covers high-resolution images and expert questions across practical scenarios; BlindTest diagnoses precise visual-perception errors, while VisuLogic tests diagram-based logical reasoning.Source: §A.1 · p. 12

A.2 Benchmarks

• The benchmarks probe visual question answering, logical and mathematical reasoning, abstract patterns, and noisy user images.Examples include VQAv2, LogicVista, MathVision, MMIQ, and VizWiz, respectively.Source: §A.2 · pp. 12–13
• Additional benchmarks target vision-indispensable questions, curriculum-based multimodal reasoning, and abstract or diagram-based visual tasks.MMStar filters text-answerable samples, while MDK12 draws on K–12 curricula and VisuLogic evaluates structured diagram problems.Source: §A.2 · pp. 12–13

A.3 Baselines

• The baseline comparison includes output and relational alignment, attention-map distillation, and cross-modal alignment with attention matching.For multi-stage baselines, the experiments use distillation fine-tuning and omit subsequent supervised fine-tuning for fairness.Source: §A.3 · pp. 13–14

B Attribute-matched Replacement

• Text replacements match prompt role, field, and normalized position; visual replacements match image role and normalized spatial location.Replacements change input embeddings while preserving sequence length, structural tokens, image boundaries, positional indices, and attention masks.Source: B Attribute-matched Replacement · p. 14

C.1 Sensitivity to Hyperparameters

• The default r = 0.25, ρ = 0.50, and λ = 1.0 provides a balanced, robust setting across Qwen3-VL and LLaVA-OneVision.For both models, increasing r from 0.10 to 0.25 improves scores, while increasing it further to 0.50 reduces them; the verification sweep also peaks at ρ = 0.50.Source: §C.1 · p. 14, Figure 5 · p. 15

C.2 Scope of Causal Supervision

• 66.34 average for full MCD exceeds Text-only (64.95), Vision-only (64.24), ICD-only (64.50), and Query-only (64.87).The combined result indicates that effective multimodal ICL depends on jointly transferring visual and textual evidence from the demonstrations and query.Source: Table 4 · p. 15, §C.2 · p. 15

C.3 Decomposition of Evidence Verification

• 66.34 for Joint verification exceeds Remove-only (65.16) and Retain-only (64.73), supporting complementary roles for retained and removed evidence.The joint criterion requires evidence to preserve the answer when retained and influence it when removed.Source: Table 5 · p. 15, §C.3 · p. 15

C.4 Robustness to Replacement Sampling

• Fixed-1 offers the best balance between performance and preprocessing cost, since teacher preprocessing grows approximately linearly with each additional replacement.The authors conclude that MCD is insensitive to the fixed replacement assignment.Source: §C.4 · pp. 15–16
• 0.19 is the average-score change from Fixed-1 to Fixed-4, while epoch-wise resampling reaches 65.42, 0.92 below the fixed default.Fixed-4 improves over Fixed-1 by at most 0.53 points on individual benchmarks; epoch-wise resampling has larger drops on MathVision and LogicVista.Source: Table 6 · p. 16, §C.4 · p. 16

D.1 Metrics for Causal Behavior Transfer

• Each held-out prompt uses the teacher-discovered evidence set to create full, retain-evidence, and remove-evidence variants, evaluated against the same teacher answer and prefixes.The answer length and sparse-support distributions are used in the evaluation.Source: §D.1 · p. 16
• The reported errors are averaged over the held-out prompt set.Source: §D.1 · p. 16
• Removal-effect measures and Pearson correlation assess whether students preserve variation in the strength of the teacher’s intervention response.Source: §D.1 · p. 16

D.2 Efficiency Analysis

• For accepted MCD examples, caching the full- and remove-prompt distributions adds at most 16K + 4 bytes per answer token.The comparison uses a cache of K = Kcache; the efficiency table reports cache values using Kcache = 128, mean answer length 8, and acceptance rate 0.5.Source: §D.2 · pp. 16–17, Table 7 · p. 17
• 5.25 student evaluations per example over three epochs for MCD compare with 3 for Vanilla KD, while prompt variants are evaluated sequentially and do not multiply peak activation memory.The efficiency comparison covers the fixed 60K-example training set; MCD uses 1.75 student evaluations per epoch at a realized acceptance rate of 0.5.Source: §D.2 · p. 16, Table 7 · p. 17
• MCD incurs a larger one-time teacher cost than output KD, but removes the teacher from training and uses fewer student evaluations than the two-evaluation token-interaction objective.All methods retain the unmodified student architecture and therefore have identical deployment cost.Source: §D.2 · pp. 16–17

D.3 Failure Case Study

• Position-matched replacements can still break local image-text consistency, so output changes may reflect off-manifold prompts rather than removal of mechanism evidence.Verification filters many such cases; context-conditioned replacement is identified as a possible way to make interventions more precise.Source: §D.3 · p. 17
• On low-quality VizWiz images, rare mechanisms, and open-ended questions with multiple valid answers, selected evidence can vary with the teacher’s response.Correctness and causal-effect screening reduce this noise; multi-response or ensemble verification is proposed for preserving supervision across reasonable teacher predictions.Source: §D.3 · p. 17