Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

1 Introduction

• Earlier approaches rebalance attention, extract task-specific steering vectors, or fit retrieval policies, but are tailored to a task or model and do not yield general vICL ability.VLMs are routinely prompted with demonstrations but struggle when the task must be inferred from images rather than text.Source: §1 · p. 1
• The authors attribute vICL difficulty to textual shortcuts: Qwen3-VL-8B accuracy drops from 78.3% to 44.8% when a task-specific instruction becomes generic.The paper hypothesizes that dedicated vICL training without textual shortcuts can teach models to learn from demonstration images.Source: §1 · p. 1
• MIME contains diverse, task-agnostic demonstrations whose hidden rule must be inferred from images; DTAD randomly withholds demonstration labels to encourage image use.Training on MIME with the rule stated sharply reduces the gain and demonstration-image attention.Source: §1 · p. 2
• 8.1–18.0 points: MIME improves average vICL accuracy across 11 tasks and three VLMs; DTAD matches MIME accuracy with stronger, more consistent attention to demonstration images.MIME-trained Qwen3-VL-8B matches frontier models on TrueMICL, while DTAD surpasses them.Source: §1 · p. 2
• MIME also preserves single-image performance and improves multi-image and text-only in-context learning, while making vICL more robust to prompt-format changes.The reported evidence suggests the learned ability extends beyond an image-specific patch.Source: §1 · p. 2
• Adding MIME to LLaVA-1.5-13B’s instruction-tuning mix more than doubles TrueMICL accuracy, from 18.9 to 39.7, without single-image regression.A same-size multi-image set reaches 21.5 and lowers POPE by 30 points.Source: §1 · p. 2

2 Preliminary Study: The Text Shortcut in vICL

• Near-perfect accuracy in text-ICL and transcribe-and-answer settings indicates that neither visual perception nor textual ICL alone explains the vICL performance gap.The study uses two TrueMICL tasks and progressively removes textual cues across four settings.Source: §2 · p. 3
• Accuracy falls from 78.3% with a task-specific instruction to 44.8% with a generic instruction, suggesting reliance on textual cues rather than learning from visual demonstrations.The generic instruction asks the model to inspect demonstrations and answer the final query, without a task-specific hint.Source: §2 · p. 3
• The study motivates the hypothesis that VLMs are insufficiently trained to perform ICL over visual demonstrations and may underuse visual tokens in contextual tasks.The authors propose that vICL training data designed to eliminate textual shortcuts could encourage attention to visual contextual signals.Source: §2 · p. 3

3.1 MIME: A Task-Agnostic vICL Training Set

• MIME has 23,578 instances with 1–5 image-label demonstrations and a query image whose label is held out; each rule must be inferred across shots from the images.Queries do not repeat demonstration images or their underlying instances, preventing direct matching.Source: §3.1 · p. 3
• MIME requires the image-label pairs to carry the rule, never states the rule in its generic instruction, and leaves text-only Qwen3-8B near random chance without images.The schema spans five sources with different rule types and visual domains; examples include polygon rules and CLEVR counting.Source: §3.1 · p. 3

3.2 DTAD: Demonstration Text Attention Dropout

• DTAD independently withholds each demonstration label with probability p while retaining its image, so a label-free demonstration can inform the query only through its image.Each instance can mix fully visible and image-only demonstrations; the query remains fully visible and is the sole loss target.Source: §3.2 · pp. 3–4
• DTAD leaves inference unchanged and uses either placeholder replacement or attention blocking to withhold labels while preserving conversation structure, sequence length, and image content.The two withholding implementations are compared against fine-tuning on the same data.Source: §3.2 · p. 4

4 Experimental Results

• MIME's gains come from demonstration structure, while existing attention-rebalancing methods leave the baseline unchanged.Source: §4 · p. 4

4.1 Experimental Setup

• Evaluation averages accuracy over 1/2/4 shots and three seeds across 11 tasks, using generic instructions that require inferring rules from demonstrations.Source: §4.1 · p. 4

4.2 MIME Drives the Accuracy Gain

• MIME fine-tuning adds +15.1, +8.1, and +18.0 average vICL accuracy points across the three backbones under generic instructions.Source: §4.2 · p. 4, Table 1 · p. 5
• On TrueMICL, Qwen3-VL-8B fine-tuned on MIME matches Claude Opus 5, while DTAD outperforms all compared models.Source: §4.2 · p. 4, Table 2 · p. 5

4.3 The Data Belongs in the Training Mix

• In LLaVA-1.5-13B instruction tuning, MIME raises TrueMICL accuracy from 18.9 to 39.7, versus 21.5 with same-size multi-image data, without single-image regression.Source: §4.3 · p. 5, Table 3 · p. 5

4.4 DTAD: Improved Visual Pathway at No Accuracy Cost

• DTAD performs on par with MIME fine-tuning across three models and attends to demonstration images more strongly and consistently across tasks.Source: §4.4 · p. 5

4.5 Beyond vICL Benchmarks

• MIME preserves single-image performance, improves multi-image accuracy by 2.6 points and MARVEL from 27.0 to 29.1, and transfers to text-only ICL.Source: §4.5 · p. 5, Table 4 · p. 5

5.1 Trained models rely on the demonstrations

• Shuffling demonstration images or answers reduces trained-model accuracy by 33–52 points, and removing demonstrations reduces it by 42–63 points.Foreign-rule demonstrations cause DTAD’s largest drop, showing it follows the demonstrated rule even when it is wrong for the query.Source: §5.1 · p. 6

5.2 Hiding the rule is necessary

• With the rule stated, gains fall to +12.1 / +12.3 points rather than +17.0 / +22.0 when hidden, and visual share falls below the base model in mid-to-late layers.A single-image control scores 61.5 on average, versus 76.2 with MIME and 59.0 for the untrained model; stating the rule without demonstrations nearly eliminates the gain.Source: §5.2 · p. 6, Table 6 · p. 6

5.3 Training data analysis

• Evaluation benchmarks are 70–170% calibrated distance from their nearest training source, while CLEVR-based tasks are excluded because 93% of images exceed 0.95 cosine similarity to training images.Text-only Qwen3-8B is near chance on average with images removed, and there are zero query–demonstration instance overlaps.Source: §5.3 · pp. 6–7

5.4 DTAD builds a stable, usable visual pathway

• DTAD raises visual share above the base model from middle layers onward, with a wider late-layer gap and nearly constant values across shot counts.Fine-tuning raises visual share mainly in middle layers and has wider task-to-task spread in late layers; rule-stated training lowers it below the base model.Source: §5.4 · p. 7, Figure 2 · p. 8
• With demonstration text blocked, DTAD redirects 31.3% of freed attention to images at 4 shots, versus 24.6% for the untrained model, and exceeds it in 14 of 15 task–shot pairs.Fine-tuning redirects 29.7% at 4 shots and exceeds the untrained model in 12 of 15 pairs; rule-stated training stays near the untrained model.Source: §5.4 · p. 7, Table 7 · p. 7

5.5 Scaling with shot count

• 69.0% ERR for DTAD and 67.5% for MIME fine-tuning exceed the untrained model’s 53.9% as shots increase from 4 to 16.Trained models begin at 88.6 and 90.0 4-shot accuracy, versus 66.6 for the untrained model, leaving less room to improve.Source: §5.5 · p. 7, Table 8 · p. 8

Visual ICL models and benchmarks

• Interleaved VLM training has treated few-shot ability as an intended or assumed capability, while few benchmarks directly test visual ICL.VL-ICL Bench covers perception, induction, reasoning, and long-context tasks; TrueMICL focuses on tasks requiring both modalities, and SEED-Bench-2 includes in-context captioning.Source: §6 · p. 8

Addressing the visual ICL gap

• Prior accounts locate the vICL gap in attention allocation or in how task mappings are built and propagated across layers.DARA and inference-time attention reweighting target attention, but provide only marginal gains over the untrained model.Source: §6 · p. 8
• Other approaches inject task or in-context vectors from many-shot activations or fit per-task demonstration-selection policies, while Jia et al. (2025) penalize text shortcutting.These methods are typically task- or domain-specific and do not yield a general capability.Source: §6 · p. 8

7 Conclusions

• MIME improves vICL by 8.1–18.0 points across architectures, preserves single-image ability, and improves multi-image and text-only ICL; DTAD performs on par while strengthening late-layer visual attention.In a controlled LLaVA-1.5-13B retrain, MIME outperforms same-size multi-image data.Source: §7 · p. 8

Limitations

• The controlled instruction-tuning experiment covers only LLaVA-1.5-13B because current state-of-the-art VLM training data and pipelines are not public.The full effect of adding MIME to modern models’ pretraining and instruction tuning therefore remains unmeasured.Source: Limitations · p. 9
• Fine-tuning tests use 4B–8B models with LoRA adapters, leaving full-parameter training and larger backbones unexplored.The authors note that an 8B model matches frontier models on TrueMICL in this lightweight setup.Source: Limitations · p. 9
• MIME and DTAD are evaluated in isolation, not combined with other vICL approaches.Their combination with inference-time methods such as retrieval or task vectors remains untested.Source: Limitations · p. 9

• The appendix begins with the motivating experiment designed to isolate the visual in-context learning bottleneck.Source: A Appendix · p. 13
• It examines prompt-format ablations, training-set composition, and the contribution of individual data sources.Source: A Appendix · p. 13
• It also reports full-model retraining on LLaVA-1.5-13B, including regression checks and a multi-image QA control.Source: A Appendix · p. 13

• The study evaluates Qwen3-VL-8B on the TrueMICL Clock and Operator Induction tasks while progressively removing textual descriptions of demonstrations.Source: §A.1 · p. 13, Table 9 · p. 13
• Near-perfect pure text-ICL and transcribe-and-answer accuracy indicates that perception and textual ICL are not the bottlenecks in these tasks.Source: Table 9 · p. 13
• Accuracy falls with a generic instruction, indicating reliance on textual cues rather than direct inference from visual demonstrations.Source: Table 9 · p. 13, §A.1 · p. 13

A.2.1 Official protocol results

• Under the official protocol, MIME fine-tuning gains are +7.8, +9.7, and +31.6 points across the three backbones.The protocol retains each benchmark’s task-specific question; Table 10 reports per-task accuracy averaged over 1/2/4 shots and three demonstration seeds.Source: §A.2.1 · p. 14, Table 10 · p. 14

A.2.2 Prompt Format Ablation

• Trained models score higher with multi-turn prompts; removing instructions costs the baseline and fine-tuning models 6.8–7.2 pp, while DTAD models lose 0.5 and 2.1 pp.Source: Table 11 · p. 15, §A.2.2 · p. 15

A.2.3 Beyond vICL tasks

• MIME preserves single-image performance (76.1 vs. 76.4 on average) while improving both multi-image benchmarks, MARVEL, and all three text-only ICL benchmarks.Source: §A.2.3 · p. 16, Table 12 · p. 16

A.3.1 Statistics; A.3.3 Dataset design choices

• The five-source set contributes up to 5,000 episodes per source with 1–5 demonstrations; audits test demonstration-only and hidden-rule properties and report text predictability and instance violations.Source: §A.3.1 · p. 16, Table 13 · p. 16, §A.3.3 · p. 16

A.3.2 Single-Image Control

• 61.5 vs. 59.0 overall, the text-rule control remains below visual-ICL training at 76.2; on Clock it scores 11.0 vs. 12.8, while DTAD reaches 70.1.The control uses the same query images but states each rule in text and omits demonstrations.Source: §A.3.2 · p. 16, Table 14 · p. 16

A.3.4 Train-Test Separation

• Evaluation benchmarks lie 70–170% calibrated distance from their nearest training source, but CLEVR-based benchmarks lie 8% away and are excluded.Figure 3 is a t-SNE projection of SigLIP image embeddings; 0% denotes a held-out split and 100% denotes COCO photographs.Source: Figure 3 · p. 17, §A.3.4 · p. 17

A.3.5 Source Contribution

• Removing any source costs 3.0–16.7 points under either DTAD variant, while naïve fine-tuning has a clear source-removal effect only for Triangle (−10.2).Removing CLEVR costs 16.7 points under DTAD (AM) but 1.0 under naïve fine-tuning.Source: §A.3.5 · p. 17, Table 16 · p. 17

A.4 Full-Model Retraining on LLaVA-1.5-13B

• +20.8 pp average TrueMICL gain follows adding MIME to LLaVA-1.5-13B instruction tuning, with no POPE or MME regression; the Mantis control gains +2.6 pp and incurs a −30 pp POPE regression.Source: Table 17 · p. 18, §A.4 · p. 18