PaperlayerView exact revision on arXiv

Source-linked AI summary

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors

Shu Zhao, Tan Yu

arXiv:2610.16977v1cs.CVcs.AI

TL;DR

Visual ICL depends on which demonstrations are selected, and counterfactual selection methods can require multi-stage attribute and image-matching pipelines. SMILE mines a target VLM’s past errors and uses its prediction to select semantically aligned hard negatives; across four benchmarks and four VLMs, it gains +3.09 average task-score points over the SOTA baseline at 5× lower latency.

  • Problem

    Visual ICL performance is sensitive to demonstration choice, while counterfactual selection such as CIRCLES requires attribute extraction, caption generation, and composed image matching.

    Source: §1 · p. 1

  • Method

    SMILE mines a target VLM’s incorrectly predicted labeled examples offline and uses the query’s zero-shot prediction to select semantically aligned hard negatives.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    Across four benchmarks and four VLMs, SMILE gains +3.09 average task-score points over the SOTA baseline at 5× lower latency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SMILE’s strongest gains are on open-ended VQA, where model failure modes are most diverse.

    Source: §6 · p. 9

    Prediction-conditioned selection depends on a meaningful zero-shot error direction; when the prediction is entirely off-manifold, it cannot identify corrective hard negatives.

    Source: H Limitations · p. 19

Abstract

from arXiv · show

Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/

1 Introduction

Visual ICL depends strongly on demonstration quality, while attribute-based counterfactual selection requires a multi-stage pipeline. SMILE instead uses a VLM’s own errors as counterfactuals and selects them using the model’s prediction as a cue.

1 Introduction

  • Counterfactual selection can reveal fine-grained visual distinctions, but CIRCLES requires attribute extraction, faithful caption generation, and composed image matching.The paper identifies three potential failure points in that multi-stage pipeline.Source: §1 · p. 1
  • A VLM’s past errors can expose the same failure mode as a query, making plausible but misleading evidence explicit for the model to reject.The illustrated case contrasts SMILE with direct answering, positive-only retrieval, and attribute-based retrieval.Source: §1 · p. 1, Figure 1 · p. 2
  • +3.09 average task-score points over the SOTA baseline across four benchmarks and four VLMs, at 5× lower latency.SMILE’s prediction-conditioned selection is presented as a unified mechanism for closed-set classification and open-ended VQA.Source: §1 · p. 2
  • SMILE mines model-specific hard negatives offline, then uses the zero-shot prediction to find past mistakes semantically aligned with the query.Positive demonstrations are identified by image-embedding similarity; hard negatives are selected in text-embedding space.Source: §1 · p. 2

2 Related Work

Prior visual ICL methods largely select examples to match the query, while CIRCLES constructs counterfactual demonstrations through attribute extraction and composed image matching. SMILE instead uses the target VLM’s own mistakes as model-specific counterfactuals, without fine-tuning.

2 Related Work

  • Visual ICL selectors range from nearest-neighbor retrieval and multimodal heuristics to learned policies, coreset optimization, and sequence configuration.These approaches share a focus on selecting examples that match the query.Source: §2 · p. 2
  • CIRCLES constructs counterfactual demonstrations through attribute extraction and composed image matching, whereas SMILE uses the target VLM’s own errors as natural counterfactuals.SMILE’s approach is described as having minimal pipeline overhead.Source: §2 · p. 2
  • Prior error-driven approaches use failures for parameter updates; SMILE instead places model-specific hard negatives directly in the prompt without fine-tuning or external verifiers.Figure 2’s caption distinguishes concentrated exact confusion-pair reuse in classification from sparse reuse in VQA.Source: §2 · p. 3, Figure 2 · p. 3

3 Model Errors Reveal Reusable Failure Modes

The paper tests whether VLM errors recur as exact ground-truth/prediction pairs or remain useful through semantic proximity. Classification errors often repeat exactly, while VQA errors rarely do but tend to remain near the correct answer in semantic space.

3 Model Errors Reveal Reusable Failure Modes

  • A wrong answer can still reveal a plausible answer region, and recurring similar mistakes can serve as warnings at test time.This is the paper’s central assumption about why a labeled example the model previously answered incorrectly can be useful.Source: §3 · p. 3
  • Top-100 source error pairs cover 45.5% of validation mistakes, and 94.2% of wrong validation predictions repeat a source pair in fine-grained classification.The paper characterizes these recurring confusions as decision-boundary traps.Source: §3 · p. 3
  • Only 3.0% of wrong VQA validation predictions repeat an exact source pair, as synonyms, paraphrases, and partial answers fragment exact strings.The paper therefore examines semantic proximity rather than relying on exact pair identity.Source: §3 · p. 3
  • Across all VLMs and both task regimes, wrong predictions are closer to ground truth than random answers, with positive gaps in every model.For the reference model, the mean gap is ∆=+0.117 on classification and ∆=+0.144 on VQA.Source: §3 · p. 4, Figure 3 · p. 4
  • Matching the query’s zero-shot prediction to stored mistakes supports exact confusion matching for classification and semantic matching for VQA.This turns the prediction into a cue for selecting examples from the relevant failure mode.Source: §3 · p. 4

4 Method

SMILE constructs visual in-context prompts with image-similar positive examples and model-specific hard negatives mined from prior errors. It selects negatives by matching the query’s zero-shot prediction to stored mistaken predictions in text-embedding space.

4.1 Problem Setup

  • SMILE starts from a labeled image-question-answer corpus and uses a frozen target VLM and embedding model to select demonstrations for each test query.The embedding model supplies image and text representations for similarity-based selection.Source: §4.1 · p. 4

4.2 Framework Overview

  • The framework combines image-similar positive demonstrations with hard negatives whose stored mistaken predictions match the query’s zero-shot prediction.The hard-negative block is appended before the query and labeled as examples previously confused with the predicted answer; duplicates in the positive set are excluded.Source: §4.2 · pp. 4–5, Figure 4 · p. 5

4.3 Offline Hard-Negative Mining

  • SMILE builds a separate hard-negative pool for each target VLM by storing labeled examples whose self-consistency-voted predictions are wrong.Each stored example includes both its ground-truth answer and mistaken prediction, so its label contradicts the model’s likely prediction.Source: §4.3 · p. 5

4.4 Prediction-Conditioned Selection

  • At test time, SMILE retrieves positive examples by image similarity and ranks mined negatives by cosine similarity between the query’s zero-shot prediction and stored mistaken predictions.Text-embedding matching accommodates lexical variation in open-ended VQA predictions, such as “park,” “public garden,” and “green space” for one error mode.Source: §4.4 · p. 5

5 Experiments

The experiments test whether SMILE improves visual ICL across tasks and models, why mined mistakes help, and which selection choices matter. SMILE leads the baselines, with especially large gains on open-ended VQA, and the analyses examine error-pool coverage, transfer, matching, and latency.

5.2 Main Results

  • +3.09 average task-score points over CIRCLES, with SMILE also exceeding the strongest similarity-only baseline by +3.3 to +9.7 points.SMILE has the highest average task score and F1 for every backbone despite using 12–20 examples, compared with 32 for each baseline.Source: §5.2 · pp. 6–7
  • Gains over CIRCLES are largest on open-ended VQA, where plausible wrong answers are diverse; classification gains are smaller.The authors attribute the smaller classification gains to strong visual-neighbor label signals and accuracy being closer to its ceiling.Source: §5.2 · p. 7

5.3 Why Mined Errors Help

  • On CUB, hard-only retrieval costs 1.81 points versus clean-only, but beats it by +68.34 EM on queries from classes with no clean examples.Those classes account for 39% of queries; where both pools cover the true class, clean-only instead leads by +9.15 EM. The pool comparison is organized around uncovered and fair slices.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 Cross-Model Hard-Negative Pool Transfer

  • Across architecture families, another model’s hard-negative pool can be a viable substitute, though a self-mined pool remains safest for maximum accuracy.The transfer comparison reports changes in CUB accuracy or OK-VQA EM relative to the same inference backbone’s self-mined pool.Source: §5.4 · p. 8, Table 3 · p. 8

5.5 Ablations

  • On OK-VQA, prediction-conditioned negatives outperform random negatives (47.80 vs. 42.86 EM), while random negatives remain competitive on CUB.Positive demonstrations alone provide the largest single jump, especially on CUB, where accuracy rises from 14.83 to 81.18.Source: §5.5 · p. 8
  • On OK-VQA, semantic matching reaches 47.80 EM, while exact-match selection falls to 40.46 EM; the three matchers perform within 0.25 Acc on CUB.The comparison covers exact string match, token-F1 overlap, and CLIP text similarity.Source: §5.5 · p. 8

5.6 Hyperparameter Sensitivity

  • CUB peaks at K=8, N=4, while OK-VQA prefers K=12, N≥8 and remains within 0.3 EM of its best cell for K≥8, N≥8.Performance saturates and eventually degrades when K+N exceeds the model’s effective context.Source: §5.6 · pp. 8–9, Figure 6 · p. 9

5.7 Inference-Time Cost and Latency

  • SMILE runs 3.6–6.0× faster than CIRCLES on H100 and 4.2–6.0× faster on H200, with under 1 s per query on every tested H200 backbone.The reported H200 per-query cost is 0.40–0.90 s for SMILE versus 1.7–5.3 s for CIRCLES; inference overhead is under 5%.Source: §5.7 · p. 9, Figure 7 · p. 9

6 Conclusion

SMILE replaces attribute-engineered counterfactual selection with model-specific mining of a VLM’s own errors. Across four benchmarks and four VLMs, it outperforms the compared baselines, with the strongest gains on open-ended VQA.

6 Conclusion

  • SMILE collects labeled examples the target VLM predicts incorrectly and selects those whose mistaken predictions align semantically with the query’s likely error.This surfaces targeted contradictions without attribute extraction, caption generation, or composed image matching.Source: §6 · p. 9
  • Across four benchmarks and four VLMs, SMILE consistently outperforms the compared baselines, including CIRCLES.The strongest gains occur on open-ended VQA, where model failure modes are most diverse.Source: §6 · p. 9
  • The conclusion presents mined mistakes as targeted contradictions that challenge the model’s failure modes.Source: §6 · p. 9

A Implementation Details

The implementation evaluates SMILE and six baselines on four visual ICL benchmarks using shared backbone and similarity-encoder settings. Hard-negative pools are mined per target VLM, and the one-time mining cost is modest even for the largest dataset.

A Implementation Details

  • The evaluation covers CUB-200-2011, Flowers-102, OK-VQA, and VizWiz, with accuracy/weighted F1 for classification and exact match/token F1 for VQA.The four datasets span fine-grained classification, knowledge-based VQA, and real-world VQA; about 32% of VizWiz questions are unanswerable.Source: A Implementation Details · p. 15
  • For each target VLM, self-consistency voting identifies labeled examples whose voted zero-shot prediction differs from the ground truth for its hard-negative pool.Pool sizes range from about 20% of the labeled corpus for strong models on easy tasks to about 80% for smaller models on difficult tasks. Baselines and SMILE share CLIP ViT-G/14 as the similarity encoder and use a unified prompting template.Source: A Implementation Details · p. 15

B Offline Mining Time Breakdown

  • Mining VizWiz’s 20,523 examples takes 6.7–13.3 minutes per backbone on 8 H100 GPUs, and Flowers-102 takes under one minute.Mining is one-time per backbone and dataset; a previously mined pool can often be reused for another VLM.Source: B Offline Mining Time Breakdown · p. 16, Table 6 · p. 16

C Additional Error-Structure Analysis

The error-structure analysis finds recurring exact confusion pairs in classification, but more fragmented exact answers in VQA. Across both regimes, wrong predictions remain semantically closer to the truth than random answers.

C Additional Error-Structure Analysis

  • On CUB, the top 100 source confusion pairs cover 45.5% of validation mistakes, and 94.2% of wrong validation predictions repeat a source pair.For OK-VQA, only 3.0% of wrong validation predictions repeat an exact source pair; its top 100 source pairs cover 3.4% of source error patterns.Source: C Additional Error-Structure Analysis · p. 16
  • Wrong predictions are closer to ground-truth answers than random answers across all four VLMs and both task regimes; reference-model mean gaps are +0.117 for classification and +0.144 for VQA.Across all eight model-and-task cells, the gaps range from +0.085 to +0.168.Source: C Additional Error-Structure Analysis · p. 17
  • Classification errors often stay within visually related species, while VQA errors often preserve meaning through synonyms or category-level alternatives.Examples include song sparrow for fox sparrow, children for kid, and round for circle.Source: C Additional Error-Structure Analysis · p. 17

D Effect of the Similarity Encoder

SMILE remains effective across a range of similarity encoders, with performance rising as encoder capacity increases within a family. The paper’s CLIP ViT-G/14 baseline is near the top of the tested set, and even the smallest CLIP encoder substantially outperforms the reported baselines.

D Effect of the Similarity Encoder

  • 83.88 EM with CLIP ViT-H/14 is the best tested encoder result, while the CLIP ViT-G/14 baseline reaches 82.55 EM.The baseline is within 1.3 points of the best encoder and is second-best overall.Source: D Effect of the Similarity Encoder · p. 17
  • CLIP EM rises monotonically from 70.85 with ViT-B/32 to 83.88 with ViT-H/14, while SigLIP 2 rises from 76.68 at base to 78.68 at large.These comparisons hold within each encoder family.Source: D Effect of the Similarity Encoder · p. 17
  • With CLIP ViT-B/32, SMILE reaches 70.85 EM, above zero-shot at 14.83 and the Random baseline at 26.27 on Qwen2.5-VL-7B.The paper interprets this as evidence that the benefit persists with a significantly downsized retriever.Source: D Effect of the Similarity Encoder · p. 17

E Additional Backbone Results

This block reports additional Qwen3-VL backbone results, analyzes how pool composition and class coverage affect retrieval, and describes SMILE’s unified prompt structure. The pool analysis finds that hard examples matter especially when clean examples do not cover the query’s class.

E Additional Backbone Results

  • Table 8 reports SMILE results on two additional Qwen3-VL backbones using the same CLIP encoder and task-specific K, N settings as the main experiments.The table caption identifies accuracy and weighted F1 for CUB and Flowers, and exact match and token-level F1 for OK-VQA and VizWiz.Source: E Additional Backbone Results · p. 17, Table 8 · p. 18

F Pool Decomposition: Phase Transition with k

  • On fair subsets, natural wins at small k, clean-only dominates at k≥4, and hard-only never wins; the hard pool matters because it covers classes the model gets wrong.CLIP image similarity can surface hard examples implicitly, without explicit pool-routing logic at inference time.Source: Table 9 · p. 18, F Pool Decomposition: Phase Transition… · p. 18
  • The pool contains 1,113 clean and 4,881 hard CUB images; 77 of 200 species have no clean examples, defining an uncovered slice of 2,255 queries (39% of test).The fair ≥k slice restricts queries to classes with at least k examples in both pools; at k=4 it contains 1,584 queries (27%).Source: F Pool Decomposition: Phase Transition… · p. 18

G Prompt Template

  • SMILE presents the query, then K image-similar positive demonstrations and N prediction-conditioned hard negatives, and repeats the query before answering.Hard negatives show the labeled example’s ground-truth answer, not the model’s stored mistaken prediction; classification is closed-set, while OK-VQA and VizWiz are open-ended.Source: G Prompt Template · pp. 18–19

H Limitations

The paper identifies boundaries in the informativeness of the zero-shot prediction, fixed prompt structure, and benchmark scope. These constrain when and where the current inference-time approach has been evaluated or may help.

H Limitations

  • Hard-negative selection depends on a meaningful zero-shot prediction; when it is entirely off-manifold, better demonstration selection may not address the model’s knowledge or reasoning limitation.The paper notes that parameter updates such as supervised fine-tuning or reinforcement learning may be needed in such cases.Source: H Limitations · p. 19
  • The implementation uses a fixed prompt structure and capped hard-negative count rather than adapting organization or negative budget to query difficulty.Adaptive prompt choices are outside the paper’s scope.Source: H Limitations · p. 19
  • Evaluation covers four benchmarks with available labeled corpora; extension to web-scale example retrieval or settings without a labeled pool remains open.Source: H Limitations · p. 19

Related papers

Loading 2610.16977v1…