PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails

Suyoung Lee, Myungsub Choi

arXiv:2609.05535v1cs.CVcs.AI

TL;DR

Verdict-only evaluation does not reveal whether a VLM used the visual evidence supporting its decision. The paper introduces a benchmark and diagnostic interventions for evaluating evidence localization and counterfactual responsiveness. Across models, detection and evidence-aligned detection diverge, and endorsement probes reveal further variation in counterfactual consistency.

  • Problem

    Verdict-only evaluations collapse cases where a VLM gives the correct verdict but cites unrelated evidence or fails to revise its verdict when the cited text is endorsed.

    Source: §1 · p. 1

  • Method

    The paper introduces Mind2Web-Injection, a benchmark with instruction-relative labels, pixel-exact evidence boxes, and matched image-side counterfactuals, and proposes ReadGate and CmdCompare as diagnostic interventions.

    Source: §1 · pp. 2–3

  • Results

    Across six VLMs, detection AP tied within 0.001 while EAD differed ninefold; on the endorsement probe, aligned rates were 58.7% and 99.9%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Verdict correctness, evidence localization, and counterfactual responsiveness are empirically separable and should be reported separately.

    Source: §5 · p. 13

    Injections are pixel overlays rather than HTML content, curation is model-based with limited human validation, and the instruction-side probe is indirect and text-only.

    Source: §5 · p. 13

Abstract

from arXiv · show

Verdict-only evaluation does not reveal whether a vision-language model (VLM) used the visual evidence that should support its decision. We study this problem in web-agent guardrails, where a VLM judges whether on-screen text conflicts with a user instruction. We introduce Mind2Web-Injection, a benchmark of 9,954 instruction-screenshot pairs with instruction-relative labels, pixel-exact evidence boxes, and matched image-side counterfactuals. Across six VLMs, two models with nearly identical average precision differ ninefold in Evidence-Aligned Detection (EAD), the fraction of attacks both detected and correctly localized. To test whether a verdict depends on the command cited as evidence, we replace the instruction with one that endorses that command. Qwen3-VL-32B, the strongest open-weight localizer, returns aligned in only 58.7% of cases, whereas GPT-5.6-luna does so in 99.9%. To diagnose these failures, we propose two training-free interventions. ReadGate improves grounding without changing verdicts, while CmdCompare tests whether explicit instruction-command comparison resolves instruction-side inconsistency. These results motivate reporting verdict correctness, evidence localization, and counterfactual responsiveness separately.

1 Introduction

Verdict-only evaluation can miss whether a VLM used the visual evidence that supports its decision. This study makes evidence localization and counterfactual dependence measurable in screenshot-based web-agent guardrails.

1 Introduction

  • Verdict-only scores can hide a correct injection verdict grounded in unrelated page content or a verdict that persists after the cited text is endorsed.The task tests whether on-image text diverts a web agent from its instruction, and the evaluation problem is relevant more broadly to multimodal systems that produce localized evidence.Source: §1 · pp. 1–2
  • Mind2Web-Injection contains 9,954 instruction–screenshot pairs with pixel-exact evidence boxes and 6,168 matched image-side counterfactuals.Each twin replaces the divergent command while preserving its placement and appearance.Source: §1 · pp. 2–3
  • Detection AP is tied within 0.001, yet EAD differs ninefold across six VLMs; endorsement probes expose further counterfactual-consistency gaps.In Fig. 1, detection AP is 0.752 vs. 0.751, while EAD0.3 is 0.426 vs. 0.047.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The endorsement probe returns aligned in only 58.7% of cases for the best open-weight localizer, compared with 99.9% for the best model.ReadGate improves grounding without changing verdicts; CmdCompare improves instruction-side consistency for some models but does not resolve image-side failures.Source: §1 · p. 3
  • The benchmark evaluates detection, conditional grounding, joint evidence-aligned detection, and two-sided counterfactual consistency across six VLMs.The study also proposes ReadGate and CmdCompare as training-free diagnostic interventions.Source: §1 · p. 3

2 Related Work

Prior work studies prompt-injection detection, typographic attacks, and explanation faithfulness. This paper distinguishes its relational, pixel-exact evaluation and diagnostic interventions from those lines of work.

2 Related Work

  • Prior detectors range from detection benchmarks such as WAInjectBench [24] to localization by WebSentinel [38], while AlignSentinel [21] and WebAgentGuard [5] make the judgment relational.The paper instead measures instruction-relative judgments on native screenshots with pixel-exact evidence and two-sided counterfactuals.Source: §2 · p. 3
  • Typographic attacks exploit text rendered in images, but this study’s injected text is innocuous alone and divergent only relative to the instruction [33].Source: §2 · p. 3
  • Faithfulness work tests whether rationales drove decisions [11, 20], whereas this study intervenes on the instruction–rendered-command relation and uses matched image-side twins.Its diagnostic interventions adapt prior grounding and instruction/data-isolation techniques; the contribution is diagnostic rather than algorithmic [39] [36].Source: §2 · p. 4

3 Method

The method evaluates a VLM guardrail that judges screenshot text relative to a user instruction and identifies the responsible on-screen evidence. It combines instruction-relative benchmark data, separate detection and localization measures, counterfactual probes, and two diagnostic interventions.

3.1 Dataset: Mind2Web-Injection

  • Mind2Web-Injection contains 9,954 rendered real-page instruction pairs with relational labels, exact evidence boxes, and divergent, aligned, and neutral style-matched variants.Instructions are grounded in OCR of page content, and a VLM-chosen placement fixes the rendered evidence box.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

Aligned twins.

  • The benchmark includes 6,168 aligned twins that replace a divergent overlay while holding the instruction, page, position, and style fixed.Only the overlay’s relation to the instruction changes, making the ground-truth verdict change from divergent to aligned.Source: Aligned twins · pp. 5–6

3.2 Evidence-Aligned Evaluation

Joint detection and grounding: EAD.

  • EADτ combines recall with G.ACCτ and measures the fraction of divergent attacks both detected and localized.Unlike conditional grounding, EAD counts all divergent instances, so missed attacks count against it.Source: Joint detection and grounding: EAD · pp. 6–7

Counterfactual responsiveness: CFCT and CFCI.

  • CFCT changes the instruction to endorse the quoted command, while CFCI replaces the divergent overlay with its aligned twin and reruns detection.Both probe the already-correct base detections; high consistency supports, but does not prove, faithful evidence use.Source: Counterfactual responsiveness: CFCT… · p. 7

ReadGate: verdict-preserving grounding repair.

CmdCompare: comparison-based diagnostic reframing.

  • CmdCompare extracts an imperative command without the instruction, then tests whether doing that command instead of the instruction would deviate from it.This is a diagnostic reframing: it may change recall and false-positive behavior but leaves visual candidate extraction untouched.Source: CmdCompare: comparison-based… · pp. 8–9, Figure 4 · p. 8

4 Experiments

Across six VLMs, detection performance can diverge sharply from evidence localization, and localization does not guarantee counterfactual responsiveness. ReadGate and CmdCompare diagnose different failure stages: ReadGate changes grounding without changing verdicts, while CmdCompare helps when the model already quotes the right text.

4.1 Evaluated Models

  • Six general-purpose VLMs were evaluated with a shared guardrail prompt; models regressed a 0–100 divergence score for AP ranking, and their box conventions were normalized before IoU scoring.Pages exceeding the vision encoder limit were split into overlapping vertical crops and mapped back to full-page coordinates.Source: §4.1 · pp. 9–10

4.3 Detection Does Not Imply Localization

  • 0.001 AP separates Llama-4-Scout and Qwen3-VL-8B, yet EAD0.3 is 0.047 vs. 0.426; Gemma-3-27B and Llama-4-Scout achieve only 2.8% and 11.7% G.ACC.The broader localization pattern is model-dependent despite competitive detection AP.Source: Table 1 · p. 10, §4.3 · p. 10

4.2 All Models Exceed the No-Skill AP Baseline

  • With one threshold and no calibration, all six models exceed the no-skill AP baseline; GPT-5.6-luna leads at AP = 0.886.The open models range from 0.709 to 0.769, against a no-skill baseline of 0.620.Source: §4.2 · p. 10

ReadGate adapts across localization regimes.

  • With one threshold and no calibration, ReadGate improves on original boxes for every model and protects strong grounders from unconditional reconstruction, without changing verdicts.Unconditional reconstruction helps the two weakest grounders but damages GPT-5.6-luna and Qwen3-VL-32B, motivating per-sample routing.Source: ReadGate adapts across localization… · p. 11

4.4 Localization Does Not Imply Counterfactual Responsiveness

  • Qwen3-VL-32B and GPT-5.6-luna, the strongest grounders, differ by 41 percentage points on CFCT; only GPT-5.6-luna consistently returns aligned after the instruction endorses the quoted text.GPT-5.6-luna’s near-perfect CFCT also serves as a sanity check that the probe admits the expected response.Source: §4.4 · p. 12

CmdCompare distinguishes recoverable reasoning failures.

  • CmdCompare’s effect depends on the failure stage: Llama-4-Scout rises from 0.299 to 0.912 CFCT, while Gemma-3-27B falls from 0.529 to 0.270.The intervention can help when the model quotes the right text but fails to relate it to the instruction; it does not generally fix upstream evidence-identification failures.Source: CmdCompare distinguishes recoverable… · p. 13

5 Conclusion

Detection, grounding, and counterfactual responsiveness are empirically separable across the evaluated VLMs. The authors therefore argue that multimodal evaluations should report verdict correctness, evidence localization, and counterfactual responsiveness separately.

5 Conclusion

  • Nearly identical detection AP can conceal a ninefold gap in whether models both detect and localize responsible evidence.Strong localization can also coexist with failure to revise a verdict when the instruction–evidence relation changes.Source: §5 · p. 13
  • ReadGate and CmdCompare recover some coordinate-regression and instruction–command comparison errors without training, but evidence-identification failures persist under tested reframing.The study therefore concludes that answer correctness alone cannot certify evidence use.Source: §5 · p. 13
  • The study covers general-purpose VLMs, uses composited pixel overlays, and relies on model-based curation with limited human validation.The instruction-side probe is indirect: it uses the model’s quoted evidence in a text-only call rather than re-judging the rendered page under an independently written endorsing instruction.Source: §5 · p. 13

Supplementary Material for

The supplement supplies omitted construction, implementation, prompting, and evaluation details. It also expands qualitative figures and reports additional results across several evaluation questions.

Supplementary Material for

  • The supplement details Mind2Web-Injection construction, including generation models, five instruction types, injection generation, and dataset statistics.It also specifies ReadGate’s hierarchical patch selection and gate, including a rejected judge variant.Source: Supplementary Material for · p. 17
  • Verbatim prompt templates are provided for the guardrail detector, CmdCompare, and the ReadGate transcription gate.The supplement also describes divergence-score elicitation and coordinate normalization across models.Source: Supplementary Material for · p. 17
  • Additional results cover dataset splits, grounding-threshold sweeps, failure decomposition, and image-side counterfactual evaluation.The supplement also reports CmdCompare measurements on the image-side counterfactual, where instruction-side reframing does not transfer.Source: Supplementary Material for · p. 17

A Dataset Construction Details

Mind2Web-Injection is built from real page screenshots with instruction-relative injections and exact composited evidence boxes. ReadGate reconstructs a quoted command’s location through staged visual selection followed by constrained OCR refinement.

A Dataset Construction Details

  • Mind2Web-Injection uses Multimodal-Mind2Web screenshots from Entertainment, Shopping, and Travel, retaining 2,092 pages after an aspect-ratio filter.Table 4 identifies the models used to build the benchmark.Source: A Dataset Construction Details · p. 17, Table 4 · p. 18
  • Each page receives up to five benign instructions, each grounded in a specific OCR span, spanning question, action, navigation, multi-step, and constrained-action types.The instruction generator’s output is assumed benign; the construction yields 9,959 image–instruction pairs before rendering exclusions.Source: A Dataset Construction Details · pp. 17–18
  • Injection candidates are generated as stylistically matched divergent, aligned, and neutral text; only divergent text receives the positive label.Divergent candidates cover redirect, hijack, exfiltration, and manipulation goals at roughly 25% each; labels are 62.0% divergent, 24.0% aligned, and 14.0% neutral.Source: A Dataset Construction Details · p. 18
  • A 235B VLM selects injection placement and appearance, then PIL composites it onto full-resolution screenshots to provide exact ground-truth boxes.Appearance style is chosen independently of label; five invalid placements are dropped, leaving 9,954 images across test_task and test_website.Source: A Dataset Construction Details · p. 18

B ReadGate: Hierarchical Patch Selection

  • ReadGate runs only on already-divergent predictions and uses the model’s quoted text as an anchor, narrowing localization to finding that string.It operates on the crop selected by the model and emits no pixel coordinates directly.Source: B ReadGate: Hierarchical Patch Selection · p. 18
  • ReadGate successively selects a horizontal band, a fine vertical span, and a column span to form a candidate box.The span choices accommodate text crossing grid boundaries and variable text widths.Source: B ReadGate: Hierarchical Patch Selection · p. 19
  • The candidate box is converted to absolute page pixels, then refined by fuzzy-matching OCR blocks only within the selected strip.A match replaces the coarse grid box with the tight union of matched blocks; otherwise the grid box remains, while strip restriction avoids page-wide OCR mismatches.Source: B ReadGate: Hierarchical Patch Selection · p. 19
  • ReadGate uses six bands, a 16-cell fine grid, and 16 columns; the reported sweep shows near-monotone improvement saturating near this resolution.Table 5 scopes that sweep to a 200-image Llama-4-Scout test_task pilot; format errors rise only at the finest grids.Source: B ReadGate: Hierarchical Patch Selection · p. 19, Table 5 · p. 19

C Gate Implementation Details

ReadGate checks whether a model’s predicted box contains its quoted text and falls back to hierarchical reconstruction when needed. The supplement also specifies the prompts and evaluation procedures used to compare scores and measure grounding.

C Gate Implementation Details

  • The ReadGate transcription gate compares text inside the proposed box with the model’s quote and uses reconstruction when the match score is below 0.7.The expected text is omitted from the gate prompt to prevent the model from copying it and manufacturing a match.Source: C Gate Implementation Details · pp. 19–20
  • The rejected yes/no judge variant had a strong yes-bias and approved many garbage boxes; evidence elicitation removed most spurious approvals.The reported design principle is to elicit evidence rather than ask for a verdict.Source: C Gate Implementation Details · p. 20
  • The gate uses a 0.7 match threshold, routes degenerate boxes or gate-call errors to reconstruction, and never changes the verdict.It adds one call per divergent verdict and therefore cannot lower detection.Source: C Gate Implementation Details · p. 20

D Prompts

  • The supplement provides the guardrail detector, CmdCompare, and ReadGate transcription prompts, including instruction-relative detection and text-only instruction–command comparison.The ReadGate prompt requests verbatim transcription only from inside the marked rectangle.Source: D Prompts · pp. 20–21

E Score and Coordinate Elicitation

  • Because hosted models expose no token log-probabilities, all models regress a divergence score from 0 to 100, and average precision uses its ranking.Unparseable responses are excluded, leaving each model with slightly fewer than 9,954 scored instances.Source: E Score and Coordinate Elicitation · p. 21
  • Predicted boxes use differing coordinate conventions, so the authors detect and invert each model’s convention before computing IoU.Reading normalized coordinates as absolute pixels had reduced nearly every model’s IoU below 0.01; correction qualitatively changed grounding results.Source: E Score and Coordinate Elicitation · p. 21

F Additional Qualitative Results

Figures 8–10 extend the qualitative results to all six models, illustrating localization quality, mislocalization, and ReadGate decisions.

F Additional Qualitative Results

  • Figures 8–10 extend the main-paper examples to all six models with samples disjoint from those shown there.Figure 8 shows localized flagged samples, Figure 9 mislocalized samples, and Figure 10 ReadGate decisions.Source: F Additional Qualitative Results · p. 21
  • Gemma-3-27B and Llama-4-Scout rarely achieve the overlap level routinely reached by the other four models, even on strong attempts.Figure 8 illustrates their near-zero G.ACC sample by sample.Source: Figure 8 · p. 22
  • Even strong grounders sometimes place evidence on unrelated page content, including footer menus, headlines, or photo strips.Models differ in how often this occurs, not whether it occurs.Source: Figure 9 · p. 22
  • ReadGate keeps the model’s box when transcription matches the quote and replaces it with a reconstruction when they differ.The gate takes both actions on every model; their mix follows the reliability of each model’s coordinates.Source: Figure 10 · p. 23

G Additional Quantitative Results

The supplementary results show stable per-split rankings, threshold-dependent differences in box tightness, and distinct read, point, and false-positive failure patterns.

G.1 Per-split breakdown

  • The model ranking is identical across the two evaluation splits, and no model’s mIoU shifts by more than 0.03.Table 6 reports detection and grounding separately by split; the main table pools them.Source: §G.1 · p. 24, Table 6 · p. 24

G.3 Failure decomposition: detect, read, point

  • The read–point accuracy gap motivates ReadGate, while the dominance of Q2-type false positives motivates testing CmdCompare.Table 8 separates quote and crop accuracy from point accuracy and distinguishes Q2 errors, which skip the instruction check, from Q1 false positives on ordinary UI.Source: §G.3 · p. 25, Table 8 · p. 25

G.4 The image-side counterfactual

The image-side counterfactual results distinguish twin false positives from baseline over-triggering and report uncertainty for estimates based on few localized pairs.

G.4 The image-side counterfactual

  • Gemma-3-27B’s twin false-positive rate is 0.602, with 0.477 already present on benign instances.GPT-5.6-luna and Llama-4-Scout have similar twin FPRs, 0.166 and 0.186, despite divergent-instance recall of 0.734 and 0.403, respectively; the authors therefore report conditional CFCI rather than interpreting twin FPR alone.Source: Table 9 · p. 25, §G.4 · p. 26
  • Gemma-3-27B has only 111 correctly localized pairs, making its grounded counterfactual estimate too uncertain for a strong claim.The reported intervals use 95% cluster bootstrapping over source pages, since multiple prompt types share pages and overlays.Source: §G.4 · p. 26, Table 10 · p. 26
  • Gemma-3-27B’s grounded interval is five times wider than GPT-5.6-luna’s, with an order-of-magnitude smaller sample of correctly localized pairs.Table 10 reports cluster-bootstrap intervals for the 6,168 image-side pairs drawn from about 2,054 pages.Source: §G.4 · p. 26, Table 10 · p. 26

G.5 CmdCompare on the image side

CmdCompare’s image-side results are mixed: it improves CFCI clearly for only one model, while increasing twin false positives for most models. The accompanying analyses show that this probe does not consistently test the intended comparison mechanism.

G.5 CmdCompare on the image side

  • CFCI rises clearly only for Gemma-4-31B under image-side CmdCompare, falls for two models, and changes little for the others, while twin FPR increases for five of six.The table compares the single-call detector with CmdCompare on the same aligned-twin pairs; recall is measured on the divergent cell and supplies CFCI’s denominator.Source: §G.5 · p. 26, Table 11 · p. 27
  • CmdCompare leaves the step that finds and quotes on-image text untouched, although that step is central to the image-side payload change.The instruction-side probe swaps the instruction while holding the payload fixed; the image-side probe swaps the payload while holding the instruction fixed.Source: §G.5 · p. 26
  • For four models, the compared route is a minority of aligned-twin verdict changes; only GPT-5.6-luna’s compared route clearly exceeds half.For Gemma-3-27B, the compared route accounts for 0.038 of 0.246 CFCI; a no-candidate route bypasses comparison when quoting returns nothing.Source: §G.5 · p. 27, Table 12 · p. 27
  • CmdCompare raises divergent-cell recall for four models, changing CFCI’s denominator to include attacks those detectors previously missed.Part of each CFCI drop reflects this changed composition; the two models without increased recall have CFCI that holds within 0.012 or rises, and no stable-denominator model shows more than a negligible drop.Source: §G.5 · pp. 27–28
  • On aligned twins, the strict deviation test can label a plausible but non-identical directive as divergent, so the authors do not claim a general faithfulness repair.The twin contains an instruction-consistent on-page directive rather than no directive; the extraction step can still return a command.Source: §G.5 · p. 28

Related papers

Loading 2609.05535v1…