PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction Distillation: Text Instructions as Visual Examples

Hardik Jindal, Soumyabrata Pal, Sayak Ray Chowdhury

arXiv:2608.28696v1cs.CV

TL;DR

Visual ICL is effective for fine-grained classification but expensive because retrieved images consume substantial context. Instruction Distillation converts each training image into a structured textual rule and evaluates image, instruction, and hybrid contexts, finding that instruction pipelines match or exceed image ICL while reducing token use and latency, with hybrids showing complementary benefits.

  • Problem

    Retrieved image examples make large-K visual ICL expensive because each image consumes roughly 550 context tokens, limiting affordable inference-time context.

  • Method

    Instruction Distillation uses an MLLM to generate per-image instructions containing appearance cues, discriminative features, and confusion points, then retrieves them through a shared CLIP index for instruction-only or hybrid ICL.

  • Results

    Instruction ICL matches image ICL at K = 1 across seven benchmarks, exceeds it on five, and at K = 5 reduces per-query tokens by 2.9× and mean latency by 3.27×.

  • Takeaways & Limitations

    Visual and textual ICL signals are complementary: hybrid contexts consistently outperform unimodal contexts where both modalities are individually useful.

  • Takeaways & Limitations

    Instruction quality depends on the generator MLLM’s visual understanding, and offline generation incurs a one-time cost proportional to training-set size.

Abstract

from arXiv · show

Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved image example consumes several hundred context tokens, making large-$K$ settings prohibitively expensive at inference scale. We propose Instruction Distillation: an offline procedure in which the MLLM itself generates, for each individual training image, a structured identification instruction encoding general appearance cues, features that differentiate the class from visually similar ones, and a common confusion point. Unlike prior work that produces a single description per class, our instructions are generated per training image, preserving the intra-class visual diversity that per-class descriptions collapse. At inference time, we study five configurations sharing a single CLIP retrieval index: zero-shot, image ICL, instruction-only ICL, and two hybrid variants in which retrieved neighbors are split between images and instructions. Across seven fine-grained benchmarks and two MLLM backbones, instruction based pipelines match, or exceeds image ICL at $K{=}1$ and reduces per-query tokens by $2.9\times$ and inference latency by $3.3\times$ at $K{=}5$. Hybrid configurations further show that visual and textual ICL signals are complementary, images give visual patterns to learn and see, while instructions give explicit rules and logic. When both of these are provided, the quality of context improves, which is noticeable in the performance.

1 Introduction

Instruction Distillation replaces retrieved image examples with per-image textual instructions for multimodal in-context classification, addressing the token and latency costs of visual ICL while preserving instance-level diversity. The approach is evaluated alongside image, instruction, and hybrid configurations using a shared retrieval basis.

  • Motivation: Instruction Distillation asks whether text distilled from each individual training image can replace the image as the in-context signal inside an MLLM.The question differs from class-level textual-description methods because the instructions remain tied to retrieved training instances.
  • Instruction Distillation: Each structured instruction captures general appearance cues, class-discriminative features, and a common confusion point with a concrete visual test.Instructions are generated offline by the MLLM and retrieved using the same CLIP similarity process as image ICL.
  • Instruction Distillation: Per-image instructions preserve intra-class visual diversity that class-level descriptions collapse, while providing query-relevant textual guidance during retrieval.The instructions compress visual signal into a compact textual form without pixel-level detail.
  • Results: At K = 1, instruction ICL matches image ICL across all seven benchmarks and exceeds it on five, including a +57.9% gain on EuroSAT.At K = 5, instruction ICL reduces per-query tokens by 2.9× and mean latency by 3.27×, from 0.549s to 0.168s.
  • Results: Hybrid configurations combine complementary visual and textual signals, with images supplying visual patterns and instructions supplying explicit rules and logic.The evaluated configurations share one CLIP retrieval index and include image-primary and instruction-primary hybrids.

2 Methodology

The paper replaces retrieved image demonstrations with per-image distilled instructions while preserving a shared CLIP retrieval process. It evaluates image, instruction, and hybrid pipelines that vary how retrieved neighbors provide visual evidence and textual guidance.

  • Instruction Distillation: Instruction distillation generates structured rules from individual training images for use as textual in-context demonstrations.The rules capture general appearance, differentiating features, and a common confusion point with a visual test.
  • Inference Pipelines: At K=5, image demonstrations cost roughly 550 tokens each, while an identification rule costs approximately 95 tokens.The token-cost comparison motivates instruction-based context at larger retrieval sizes.
  • Instruction Distillation: The method preserves per-image visual diversity by storing each training image with its label and distilled instruction.This contrasts with a single class-level description and keeps the retrieved image and instruction aligned.
  • Inference Pipelines: Five configurations vary whether retrieved evidence is presented as images, instructions, or both, while sharing one CLIP retrieval backbone and frozen MLLM.The suite includes zero-shot, image ICL, instruction ICL, image-primary hybrid ICL, and instruction-primary hybrid ICL.
  • Inference Pipelines: P2 replaces retrieved images with their distilled instructions, retaining the query image while reducing context requirements.Because retrieval is unchanged from P1, accuracy differences between the two are attributable to modality rather than retrieval quality.
  • Inference Pipelines: Hybrid P3 presents top-ranked neighbors as images and lower-ranked neighbors as instructions, whereas P4 reverses those modality roles.The designs treat images as visual grounding and instructions as discriminative guidance, with ordering used to manage misleading or noisy neighbors.

3 Experiments

Experiments evaluate seven fine-grained benchmarks using macro-averaged top-1 accuracy across token budgets and targeted instruction-format and granularity ablations. Image-primary hybrids perform best overall, while instruction-using pipelines substantially outperform pure image ICL at low budgets.

  • Experimental Setup: Experiments cover seven fine-grained visual classification benchmarks and report mean per-class accuracy across low, mid, and high token budgets.The metric is macro-averaged top-1 accuracy, and each pipeline is evaluated at the configuration closest to each budget.
  • Ablation Study: The three-rule instruction format outperforms captions, descriptive paragraphs, single discriminative rules, and two-rule formats across K.The ablation attributes the result to structured decomposition plus general and error-avoidance cues.
  • Main Results: P3 is the best aggregate pipeline at every budget on both backbones, averaging 74.8–81.0% on Gemma and 78.0–83.3% on Qwen.These ranges span the Low, Mid, and High budgets reported for the two backbones.
  • Main Results: At the Low budget, instruction-using pipelines exceed pure image ICL by approximately +20pp on Gemma and +7pp on Qwen.Gemma compares 56.3% for P1 with 74.8–75.9% for instruction-using pipelines; Qwen compares 71.6% with 78.0–79.4%.
  • Ablation Study: Per-class instructions consistently underperform per-image instructions across K values.The reported explanation is that class-level instructions lose representativeness and can mismatch the retrieved visual instance.

4 Analysis

Across token budgets and backbones, instruction-based pipelines outperform pure image ICL most strongly at low budgets, while the image-primary hybrid remains the most consistent overall winner. Results also show substantial gains on EuroSAT and complementary modality benefits, alongside limitations when cues are predominantly perceptual.

  • Budget-Bucketed Comparison: At Low budget on Gemma, P1(a) averages 56.3%, versus 75.9% for P2 and P4 and 74.8% for P3 across seven benchmarks.The roughly +20 percentage-point gap is the largest single effect reported; on Qwen, the corresponding gap is +7 points.
  • Budget-Bucketed Comparison: P3 leads P1(a) at every evaluated budget, reaching 81.0% versus 79.8% at High budget on Gemma and 83.3% versus 82.9% on Qwen.Neither pure image pipeline is ever the highest-scoring configuration on either backbone.
  • EuroSAT as a Limiting Case: On EuroSAT at Low budget, Gemma accuracy rises from 27.8% with P1(a) to 89.2% with P2, a 61.4-point gap.The authors associate this with high intra-class variation in satellite imagery, where retrieved images may share low-level statistics instead of class-discriminative content.
  • Hybrid Configurations Dominate: P3 improves over both unimodal baselines on FGVC and FLWR, reaching 46.5% and 93.8% where the individual image and instruction pipelines are weaker.The results indicate complementary modality signals, with the hybrid correcting failures not corrected by either modality alone.
  • Instruction Quality: Generated instructions score 4.57 for groundedness, 3.20 for genericness, and 3.87 for hallucination on 1–5-scale judge evaluations.ESAT is an outlier with lower groundedness and hallucination scores, consistent with satellite imagery’s abstract, low-texture content.
  • Boundary Conditions and Failure Modes: Pure instruction ICL is weaker when discriminative cues are predominantly perceptual, such as flower geometry or aircraft configuration, and redundant instructions can dilute gains as budgets increase.On Qwen at Mid budget, P2 trails P1(a) by 7.2 points on FLWR and 2.9 points on FGVC; the image-primary hybrid recovers performance.

Limitations

The findings are bounded by instruction-generation quality and the upfront cost of producing instructions for the training set.

  • Instruction quality depends on the visual understanding of the generator MLLM, so weaker generators may produce generic or hallucinated rules.Such rules may mislead the inference model.
  • Offline instruction distillation requires a one-time generation budget proportional to training-set size.This cost is amortized across later test queries but may be substantial for very large corpora.
  • The supplementary material organizes additional implementation details and extended analyses across dataset, retrieval, results, prompting, and token-calculation appendices.

A Datasets

The evaluation uses standard train/test splits across seven fine-grained visual classification benchmarks, with dataset details summarized in Table 4.

  • All datasets use their standard train/test splits without modification.
  • Table 4 summarizes the evaluation datasets and their details.

B Implementation Details

The implementation retrieves visually similar training examples with CLIP and uses visual similarity to select either images or image-grounded instructions for inference.

  • Experiments run on NVIDIA A100 GPUs, with models up to 7B parameters served on a single GPU.Instruction generation takes from a few minutes to at most an hour, while inference time varies by dataset size.
  • CLIP retrieves the K training examples most visually similar to each query image rather than using random demonstrations or a fixed support set.The design follows evidence that visually similar demonstrations are more informative for few-shot visual classification.
  • Training and query images are embedded with a normalized CLIP ViT-B/32 encoder, making cosine similarity equivalent to a dot product.
  • Visual CLIP similarity retrieves instructions because each instruction is anchored to a training image and can preserve intra-class visual diversity.Class-level retrieval could miss appearance variation within a class.

D Full Results

The supplementary results report mean per-class accuracy across five inference pipelines, token budgets, datasets, models, shot counts, and prompt configurations.

  • Full Results: The full-result tables report mean per-class accuracy for individual datasets and models across different shot counts K.Several tables distinguish full-resolution from compressed image ICL, while ESAT reports coinciding image variants because its images are already low-resolution.
  • Full Results: Figure 5 compares five pipelines across seven benchmarks and Low, Mid, and High per-query token budgets.The pipelines include full-resolution image ICL, compressed-image ICL, instruction ICL, and two hybrid variants; compressed-image ICL is reported only at Low and Mid budgets.
  • Full Results: Additional tables provide shot-count results for FLWR, S397, FOOD, DTD, PETS, FGVC, and ESAT under Gemma or Qwen.
  • Full Results: The appendix includes prompt templates for zero-shot, image ICL, instruction ICL, and hybrid configurations.The templates specify visual cues, differentiating characteristics, common confusion points, and target-classification instructions.

F Instruction Quality

Instruction quality is evaluated with automated scores and human validation, while ablations test instruction format and granularity. Structured, per-image instructions provide the strongest performance and preserve image-specific diversity.

  • F Instruction Quality: Mean groundedness is 4.57 and mean hallucination is 3.87 across seven datasets, indicating generally visible and non-fabricated features.
  • F Instruction Quality: Hallucination scores are highest for DTD, FGVC, and FLWR at 4.42–4.66, whereas ESAT scores 2.63.
  • F Instruction Quality: Fifty judge scores per dataset were manually validated against human ratings, revealing consistent agreement patterns in instruction quality.
  • G.1 Ablation A: Instruction Format: The three-part format combines general appearance, inter-class discrimination, and confusion-point resolution, whose complementary signals exceed a single discriminative rule.
  • G.1 Ablation A: Instruction Format: Structured three-part instructions outperform simple captions, single rules, and unstructured paragraphs across every tested K value.
  • G.2 Ablation (b): Instruction Granularity (Per-Image vs. Per-Class): Per-class instructions use one centroid representative and deduplicate retrieved examples by class, reducing generation cost but losing localized guidance.
  • G.2 Ablation (b): Instruction Granularity (Per-Image vs. Per-Class): Per-image instructions consistently outperform per-class instructions because they preserve visual diversity and align each instruction with its retrieved neighbor.

G.3 Ablation (c): Potential label-prior confound

The label-prior control tests whether per-image instructions gain from image-specific evidence rather than merely class-conditioned generation. Across accuracy and diversity analyses, per-image conditioning consistently produces stronger and less collapsed instructions than label-only conditioning.

  • Accuracy: Per-image instructions outperform label-only instructions at every tested K on both Oxford-IIIT Pets and FGVC.The comparison covers K ∈ {1, 2, 3, 5, 10, 25} with no exceptions.
  • Accuracy: Per-image instructions improve accuracy by 1.3–1.8pp across FGVC and by 0.5pp at K=1 to 3.7pp at K=25 on Oxford-IIIT Pets.The gains remain directionally consistent across the full tested K range.
  • Diversity metrics: Within-class cosine similarity falls from 0.1873 to 0.1424 with per-image instructions, indicating less redundancy among same-class instructions.Lower similarity is interpreted as greater within-class variation.
  • Diversity metrics: Per-image instructions have higher vocabulary breadth and unique-trigram ratios than label-only instructions, including 0.0497 versus 0.0236 for unique trigrams.The TTR also rises from 0.0029 to 0.0038.
  • Diversity metrics: Removing the image produces the same instruction for a class in roughly two-thirds of cases, whereas image conditioning reduces this collapse to under 2%.The combined metrics support image-specific variation rather than label-only repetition.

H Efficiency Analysis

The efficiency analysis benchmarks wall-clock latency and throughput across inference pipelines to test whether token-count reductions translate into practical serving gains.

  • Benchmark setup: The study measures wall-clock latency and throughput using 100 queries per configuration with matched query samples on a single A100 GPU.The benchmark covers all inference pipelines.
  • Benchmark setup: Latency and throughput are evaluated across configurations and shot counts K to characterize efficiency under different context sizes.The corresponding comparison is reported in Table 21.
  • Purpose: The benchmark is designed to validate whether the token-count reductions reported elsewhere produce real inference speedups.Both latency and throughput are evaluated rather than token counts alone.

K Pipeline Latency (s) QPS Speedup

Instruction-based inference becomes substantially more efficient as shot count grows, while retaining a one-time offline generation cost. The analysis also separates token-count savings from modality-specific processing advantages and projects deployment-scale effects.

  • Latency and throughput: Latency speedup widens from 1.74× at K=1 to 5.44× at K=10 because image ICL adds a full ≈550-token image for each shot.Instruction ICL latency grows much more slowly with K.
  • Latency and throughput: At K=5, instruction ICL reduces mean latency from 0.5486s to 0.1678s per query, a 3.27x speedup over image ICL.Throughput simultaneously rises from 2.57 QPS to 24.40 QPS, a 9.49x improvement.
  • Modality effect: At a token-matched Mid budget, instruction ICL is 39.4% lower in latency than image ICL despite near-identical token counts.The comparison uses P1 K=5 with 3,400 tokens and P2 K=25 with 3,305 tokens.
  • Deployment-scale projection: For 100,000 queries at K=5, projected wall-clock reductions are 69.4% in serial serving and 89.5% in throughput serving.The token-matched Mid-budget comparison still yields a 39.4% serial reduction.
  • Deployment caveat: Throughput projections assume benchmarked QPS remains valid at scale, so production savings vary with GPU memory and batching configuration.The reported direction and approximate magnitude are expected under similar serving conditions.
  • Amortization: Instruction ICL requires one-time offline generation, with the worst-case cost amortized after approximately 9,500 queries in the K=5 serial comparison.The crossover occurs when cumulative instruction-based serving becomes cheaper than image-based serving.

I Token Cost Derivation

The token-cost derivation sums prompt components and compares image-based, compressed-image, and instruction-based pipelines. At K=5, instruction-based context is substantially cheaper than image-based context, while generated instructions encode three visual reasoning rules per image.

  • Measurement procedure: The reported token counts are measured from tokenized prompts produced by the native HuggingFace processors of the vLLM-served models.With the CLS token, each image yields 577 vision tokens before accounting for model-specific image special tokens.
  • Per-component costs: A single uncompressed image example consumes approximately 550 patch tokens plus approximately 12 label tokens.Compressed retrieved images reduce the per-image patch cost to approximately 258 tokens at 224×152 resolution and 75% JPEG quality.
  • Per-component costs: System prompts contribute between 15 and 40 tokens, while class-name text contributes between 2 and 5 tokens.Instruction token counts are measured as the mean across generated instructions.
  • Per-pipeline token costs: At K=5, CP1(a) costs approximately 3,390 tokens, CP1(b) approximately 1,940, and CP2 approximately 1,165.The corresponding P1(a)/P2 ratio is 2.91×, while P1(b)/P2 is 1.67×.
  • Instruction structure: Each generated identification instruction consists of three visual reasoning rules based on an image and its class label.The examples include general identification cues, distinctions from similar classes, and checks intended to avoid misclassification across food, remote-sensing, and flower categories.
Loading 2608.28696v1…