PaperlayerView exact revision on arXiv

Source-linked AI summary

SAVER: Selective Auditing of Verbal Evidence for Error Recovery in VLM Change Reasoning

Youdi Li

arXiv:2608.22857v1cs.CL

TL;DR

VLMs often fail to express visual evidence even when relevant information is available, creating a gap in change reasoning. SAVER audits responses for task-critical evidence and selectively triggers structured reprompting. Across benchmarks and models, it improves expression-dominated tasks, with gains up to +25.8% on CLEVR-Change, while its limitations remain measurable before deployment.

  • Problem

    VLMs can contain relevant visual information yet produce change descriptions with incorrect objects, change types, or insufficient evidence.

  • Method

    SAVER uses a rule-based evidence gate to inspect response evidence and triggers a structured, systematic-comparison reprompt when evidence is missing or conflicting.

  • Results

    Across three benchmarks and four VLMs, SAVER improves accuracy on expression-dominated tasks by up to +25.8% on CLEVR-Change, with no significant effect on Spot-the-Diff.

  • Takeaways & Limitations

    The evidence gate, rather than reprompting alone, drives improvement, and evidence patterns can be hand-written or LLM-generated and checked.

  • Takeaways & Limitations

    SAVER depends on domain-specific evidence patterns and cannot check confident perceptual errors that satisfy the expected keywords.

Abstract

from arXiv · show

Vision-language models (VLMs) frequently fail at visual change reasoning, even when their vision encoders contain sufficient information. We observe that correct VLM outputs tend to contain explicit verbal evidence (object names, colors, spatial locations) that supports the claimed change, while incorrect outputs often lack such evidence. We propose SAVER (Selective Auditing of Verbal Evidence for Error Recovery), a lightweight, rule-based method that parses VLM responses for this evidence and triggers structured reprompting only when evidence is missing or inconsistent. Across three change detection benchmarks and four VLMs, SAVER significantly improves accuracy on tasks where errors stem from the model failing to articulate what it saw (expression failures), with gains up to +25.8% on CLEVR-Change. The evidence patterns can also be generated by an LLM in a single call, matching the hand-tuned gate on CLEVR-Change. Ablation experiments confirm that the evidence gate, not reprompting alone, drives the improvement.

1 Introduction

VLMs can detect visual changes yet fail to express the correct object, change type, or supporting evidence. SAVER audits verbal evidence and selectively reprompts evidence-poor responses, improving expression-dominated change reasoning.

  • VLMs may miss differences, hallucinate nonexistent changes, or describe changes without specific detail despite visually available evidence.
  • Expression failures arise when models process relevant visual information but fail to express the evidence needed to support their answers.
  • SAVER parses responses for object names, colors, spatial locations, and change-type evidence before accepting or reprompting the answer.The evidence gate accepts present and consistent evidence; otherwise, structured reprompting guides systematic comparison.
  • The gate adds no cost unless it triggers a reprompt, and its keyword-and-rule decisions remain inspectable and transparent.
  • +25.8% accuracy is achieved for Qwen on CLEVR-Change, while GPT-4o gains +13.4% on CLEVR-Change and +5.3% on MagicBrush over baseline.SAVER has no significant effect on Spot-the-Diff, consistent with weaker benefits when errors are primarily perceptual.
  • Across three benchmarks and four VLMs, selective auditing helps expression-dominant errors but not perception-dominant errors.The method fixes more samples than it degrades, according to the stated contribution summary.

2 Related Work

Prior work improves change reasoning through model training, selective verification, or additional reasoning resources. SAVER instead operates at inference time on frozen models by inspecting the response already produced.

  • Image difference captioning: Image difference captioning has motivated specialized architectures and multi-image benchmarks, while these methods improve the model through training.
  • Perception–expression gap in VLMs: Research on the perception–expression gap indicates that VLMs can contain or process correct visual information without decoding it into their outputs.
  • Selective verification: Selective verification methods include dynamic early exit, adaptive reasoning escalation, and reinforcement-learning-based reasoning triggers.
  • Selective verification: Unlike methods requiring logits, attention maps, extra models, or multiple API calls, SAVER parses the response already produced and adds no cost unless reprompting occurs.

3 Method: SAVER

SAVER audits baseline VLM responses for task-specific verbal evidence and selectively reprompts only when evidence is unclear, missing, or conflicting. The structured reprompt enumerates objects in both images before systematic comparison.

  • Evidence extraction: SAVER sends a baseline prompt, extracts task-specific verbal evidence, and accepts the response when that evidence supports the claimed change type.Evidence categories can also be generated by an LLM and then applied deterministically at inference time.
  • Evidence extraction: Evidence extraction converts a free-text response into a binary vector using case-insensitive keyword pattern matching across visual-evidence categories.CLEVR-Change uses absence, presence, color_change, and material_change categories; other datasets adapt the taxonomy.
  • Gate decision: The gate triggers when the predicted change type is unclear, required evidence is absent, or conflicting evidence is detected; otherwise it accepts the baseline response.The gate uses required and conflicting evidence mappings for each change type.
  • Case study: In the illustrated drop case, missing explicit change evidence triggers the structured reprompt, which identifies the removed green cylinder after two API calls.By contrast, an explicit red-to-yellow transition satisfies the color-change gate and is accepted after one call.
  • Selective reprompting: When triggered, SAVER reprompts the model to enumerate objects in Image 1, enumerate objects in Image 2, compare them systematically, and state the answer.The structured reprompt costs one additional API call and is contrasted with a generic retry in the method description.

4 Experiments

Experiments across three datasets and four VLMs show that SAVER helps most when errors reflect omitted verbal evidence, while perception-dominated errors limit its gains. Selective gating also reduces damage to correct baseline answers, and ablations show that both gate selection and structured reprompt content matter.

  • Evaluation setup: SAVER evaluates four VLMs across CLEVR-Change, MagicBrush, and Spot-the-Diff, spanning synthetic, edited, and natural-image change detection.Spot-the-Diff uses recall for multiple simultaneous changes, unlike the fixed taxonomies in CLEVR-Change and MagicBrush.
  • Robustness: SAVER breaks 3–9.5× fewer correct answers than always-structured prompting while fixing more than it breaks across evaluated datasets.For GPT-4o on CLEVR-Change, B1 broke 76 correct samples (12%) versus 8 (1.3%) for SAVER.
  • Per-type analysis: Structured prompting helps GPT-4o on color (+17%) and texture (+49%) but hurts drop (−22%) and add (−13%), so SAVER selectively applies it by evidence need.The gate preserves baseline answers for drop/add while capturing most gains on color/texture; correct drop answers often contain absence evidence, whereas color/texture answers require explicit property transitions.
  • Ablations: SAVER outperforms all four ablations, while a generic retry with the same gate is significantly worse in 6 of 12 model–dataset combinations.On CLEVR, Random and Inverted gates fall 11.5% and 17.8% below SAVER, and generic retry can fall below baseline, showing that structured enumeration—not a second chance alone—drives improvement.
  • Generated gates: Single-shot LLM-generated evidence patterns match the manual gate on CLEVR-Change but fail on MagicBrush and change nothing on Spot-the-Diff.One critique pass restores manual-gate accuracy for Gemini and LLaMA on MagicBrush, partly narrows GPT-4o’s gap, but fails for Qwen.

5 Discussion

SAVER distinguishes expression failures, where models omit evidence they captured, from perception failures, where they miss the visual signal but produce plausible evidence-rich answers. Its benefit depends on this boundary and must be weighed against the cost of overwriting correct answers.

  • Failure modes: Expression failures are detectable when outputs omit key evidence, whereas perception failures produce confident, evidence-rich but incorrect responses that pass the gate.Expression failures dominate CLEVR-Change, while perception failures dominate Spot-the-Diff.
  • Boundary: The gate is above random on CLEVR-Change, at or below random on Spot-the-Diff, and intermediate on MagicBrush.Comparing gate precision with a same-rate random gate measures where evidence auditing identifies baseline errors beyond baseline error prevalence.
  • Limitations: SAVER can accept factually wrong evidence-rich responses, and structured reprompting can occasionally corrupt correct answers, especially for drop/add changes.Forced enumeration may cause confusion about which object changed after listing objects present in both images.

6 Conclusion

SAVER selectively audits task-critical verbal evidence and triggers structured reprompting when evidence is missing or conflicting. The method improves expression-dominated change reasoning while preserving transparency and revealing boundaries where perceptual errors dominate.

  • Method: SAVER checks task-critical verbal evidence before deciding whether to trigger structured reprompting.The gate maps extracted evidence vectors to a binary accept-or-reprompt decision per dataset.
  • Findings: SAVER significantly improves accuracy on expression-dominated tasks, reaching gains up to +25.8% on CLEVR-Change, but does not help perception-dominated Spot-the-Diff errors.The approach is intended for settings where missing verbal evidence is detectable and structured prompting helps selected inputs.
  • Findings: Ablations show that the evidence gate’s selectivity, rather than reprompting alone, drives improvement and preserves more correct answers than always-structured prompting.The gate avoids structured prompting when it would hurt, reducing unnecessary answer overwrites.
  • Scope and transparency: Adapting SAVER to a new domain requires defining evidence patterns, which can be hand-written or LLM-generated and then checked.The gate uses predefined vocabularies and a fixed proximity window that are not tuned per dataset.
  • Method: The structured reprompt enumerates objects in both images, compares them systematically, and then states the changed object, change type, and location.This procedure is used in the B1/SAVER structured prompt.

E Gate Precision Analysis

Gate precision measures whether triggered baseline responses are disproportionately wrong relative to a random gate at the same trigger rate. SAVER helps in cells where this precision exceeds the random baseline, especially on CLEVR-Change, but not consistently on Spot-the-Diff.

  • Precision definition: Gate precision is the fraction of triggered samples whose baseline response is wrong, compared against the random-gate precision Prand.Prand equals the baseline error rate at the same trigger rate.
  • Results: On CLEVR-Change, precision exceeds Prand by 0.08 to 0.26 absolute for all four models.These margins indicate that the gate selectively triggers on incorrect baseline responses.
  • Results: On Spot-the-Diff, precision is at or below Prand for all four models, while MagicBrush ranges from −0.08 to +0.18.The cells where the gate beats the random baseline are the cells where SAVER helps end-to-end.
  • Generated gates: The generated-gate experiments report the same precision measure for single-shot and self-critiqued patterns across all 12 model–dataset cells.Prand is shared with the manual-gate analysis because it depends only on baseline error rate.

F Keyword Quality Analysis

Keyword quality determines whether SAVER selectively identifies responses needing reprompting. Over-broad patterns fail differently depending on conflict checks, while missing branches over-trigger but remain a comparatively graceful failure mode.

  • Breadth: The breadth ablation broadens CLEVR-Change patterns cumulatively from wider proximity windows to generic and near-universal words.The experiment measures accuracy, trigger rate, and gate precision for GPT-4o using cached responses.
  • Breadth: With conflict checks intact, over-broad patterns over-trigger and degrade accuracy gracefully toward B1, reaching 80.2 at L3 versus B1 at 80.7.The principal cost is wasted reprompt calls rather than silent failure.
  • Breadth: Without conflict checks, broadening collapses the trigger rate from 0.34 to zero and returns accuracy to the 72.8 baseline without signaling the failure.Required evidence appears present in nearly every response, making missing evidence undetectable.
  • Coverage: Removing 25%, 50%, or 75% of pattern branches causes over-triggering and lowers precision toward Prand, while accuracy remains between 81.6 and 86.6.Missing keywords are therefore the comparatively graceful failure mode.
  • Cost and comparison: On CLEVR-Change, SAVER outperforms B1 for GPT-4o by +5.5% and LLaMA by +2.7% while using 1.37–1.39 calls, whereas SC-5 reaches 77.4% at 5.0 calls.For Gemini and Qwen, B1 is more accurate; on MagicBrush, B1 outperforms SAVER for all models by 7–13%.

H Ablation Details

The ablations show that SAVER’s evidence-matched gate outperforms keyword-only and overly loose or strict alternatives, while improving accuracy with less collateral damage and lower cost than self-consistency.

  • Gate sensitivity: 88–100%: C6 trigger rates on CLEVR-Change make it approximate always-structured prompting.C6 scores below SAVER for GPT-4o and LLaMA, whose drop/add performance is harmed by structured prompting.
  • Gate sensitivity: 5–33%: C7’s conservative trigger rates cause it to miss evidence-poor errors.On CLEVR-Change, C7 scores 4–16% below SAVER; Qwen falls from 82.3% to 66.0%.
  • Self-consistency comparison: 88 samples: SAVER corrects cases that SC-5 misses, while SC-5 corrects only 32 cases SAVER misses.SC-5 also uses 3.6× more API calls, and majority voting barely helps texture errors.

J Exploratory: Claude 3.5 Sonnet

An exploratory 200-sample CLEVR-Change comparison with Claude 3.5 Sonnet finds that SAVER improves accuracy over always-structured prompting without breaking correct baseline answers.

  • Exploratory comparison: 77.5% vs. 76.5%: SAVER outperforms B1 on Claude 3.5 Sonnet’s 200-sample CLEVR-Change subset.SAVER breaks zero correct baselines, whereas B1 breaks 12 samples.
  • Exploratory comparison: 17 versus 12 net corrections: SAVER fixes 17 samples and breaks none, while B1 fixes 27 and breaks 12.SAVER therefore achieves a net gain of +17 compared with B1’s net gain of +15.
  • Domain applicability: SAVER is intended for tasks with a structured output taxonomy and known evidence requirements for each output type.Change detection supplies change types such as drop, add, color, and texture, with corresponding verbal evidence requirements.
  • Domain applicability: Medical image comparison may offer less benefit because subtle changes and registration artifacts can produce perception errors.The paper suggests domain-specific preprocessing when perceptual noise dominates.
  • Domain applicability: Adapting SAVER requires defining the taxonomy, specifying evidence and conflicts, and validating the gate on a small labeled sample.The first three steps can be automated by prompting an LLM with a task description and example outputs.

L Spot-the-Diff Case Study: Perception Failures

Spot-the-Diff illustrates SAVER’s boundary: many responses contain relevant evidence language yet describe the wrong objects or changes, indicating perception rather than expression failures.

  • Perception failures: 27–46%: gate-accepted Spot-the-Diff responses are wrong despite containing evidence language.The gate accepts these responses because it detects change-related keywords, although the described changes do not match ground truth.
  • Perception failures: The gate cannot detect confident, evidence-rich descriptions when the underlying objects or changes are wrong.Three GPT-4o examples show accepted responses describing vehicles or people that do not correspond to the actual changes.
  • Interpretation: Spot-the-Diff errors are perceptual rather than expressive when valid evidence keywords accompany incorrect change descriptions.In such cases, auditing verbal evidence alone cannot identify the mismatch.

M Hyperparameters and Reproducibility

The experiments use fixed generation, token, image, retry, seed, bootstrap, and significance-testing settings, with documented model endpoints and experiment dates for reproducibility.

  • Generation settings: Temperature is 0 for all conditions except self-consistency, which uses k=5 and temperature 0.7.
  • Generation settings: Maximum output lengths are 300 tokens for baseline, 600 for structured/SAVER reprompts, and 400 for standard CoT.
  • Input settings: Image detail is low for GPT-4o and auto for OpenRouter models, with images supplied as base64-encoded PNGs.
  • Reliability settings: Retries use exponential backoff from 30–120 seconds, with up to three retries for rate-limit or server errors.
  • Statistical settings: Random seed 42 governs sample selection, bootstrap procedures, and ablation randomization.
  • Statistical settings: 1,000 bootstrap resamples produce 95% confidence intervals, while McNemar’s test uses continuity correction and Bonferroni adjustment.The original tests cover 12 model–dataset pairs; revision experiments use a separate 24-test correction family.
  • Experiment chronology: Main experiments ran in January–February 2026, while LLM-generated-pattern and self-critique experiments ran in July 2026.The revision experiments reused cached responses where gate decisions matched prior conditions.
Loading 2608.22857v1…