When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

• Unlike chart QA, Web Agents use charts as evidence for downstream actions, so misinterpretation can propagate into an observable action and unsafe recommendation.The wildfire example describes non-comparable historical and modern records visually suggesting low current risk.Source: §1 · pp. 1–2
• MISVISAGENTBENCH contains 280 executable Web-agent task instances in matched pairs that differ only in whether chart evidence is clean or misleading.Source: §1 · p. 2
• The diagnostic framework identifies mechanism-aligned wrong action branches and uses browser traces to analyze extra actions, repeated selections, and action switching.Source: §1 · p. 2
• Misleading visual evidence reduces task success, induces mechanism-aligned wrong actions, and increases decision instability.Source: §1 · p. 2

2.1 Web-Agent Evaluation Benchmarks

• Web-agent and safety benchmarks assess task completion, persistent browsing, or policy compliance, but generally do not control misleading visual evidence.Source: §2.1 · p. 2

2.2 Deceptive Interface Design and Dark Patterns

• Unlike studies of manipulative interface affordances or embedded instructions (Cuvin et al., 2026; Ersoy et al., 2025; Chen et al., 2025a), this benchmark changes chart evidence while holding workflow, actions, and scoring fixed.Source: §2.2 · p. 3

2.3 Misleading Visualization

• Misleading-chart studies assess distorted judgments, chart detection, or correction in isolation (Chen et al., 2025b; Mahbub et al., 2025; Tonglet et al., 2025a,b); this work follows distorted judgments into submitted browser actions.Source: §2.3 · p. 3

3 MISVISAGENTBENCH Construction

• The benchmark models an execution chain in which an agent observes web-page evidence, forms an intermediate judgment, and submits a browser action that can be scored.Source: §3 · p. 3

3.1 Misleading Visual Evidence Pool

• The misleading-evidence taxonomy covers magnitude, trend, comparison, and generalization errors in chart interpretation.Source: §3.1 · p. 3
• Feasibility screening reduced 508 candidate visualizations to 243 evidence units with recoverable judgments, identifiable mechanisms, and plausible paths to browser actions.The candidate pool covers nine major misleading mechanisms.Source: §3.1 · pp. 3–4

3.2 Workflow Archetypes

• Four workflow archetypes turn chart judgments into resource prioritization, risk escalation, trend response or capacity planning, and evidence governance or decision approval.Source: §3.2 · p. 4

3.3 Task Specifications and Hidden Scoring

• Each task specification connects visual evidence, page content, action choices, and hidden scoring through expected, misleading, and irrelevant action branches.Source: §3.3 · pp. 4–5

3.4 Executable Web Environments

• Tasks use a generated executable browser shell with a dashboard chart and context, an action-selection form, and a confirmation page; hidden scoring is not agent-visible.Source: §3.4 · p. 5

3.5 Human Review and Quality Control

• Human reviewers checked chart mechanisms and recoverable judgments, pair consistency, and browser execution; the final benchmark contains 140 clean–misleading task pairs.Source: §3.5 · p. 5

4.1 Benchmark Environment Overview

• 140 paired tasks each use clean and misleading chart evidence in a fixed workflow from task overview and dashboard to form submission.The dashboard supplies chart evidence for an executable downstream decision.Source: §4.1 · p. 5

4.2 Evaluation Metrics

• Paired Robust Accuracy measures the proportion of tasks an agent solves under both clean and misleading evidence.Source: §4.2 · p. 5

4.2.1 Task Outcome and Failure Attribution Metrics

• Clean and misleading accuracy report success in each condition; AFRc isolates failures among clean-success cases, while MSRc tracks redirection to the mechanism-aligned wrong branch.Source: §4.2.1 · pp. 5–6

4.2.2 Execution Behavior Metrics

• Trace metrics complement outcomes: SI measures additional browser steps, while DI captures primary-decision hesitation through repeated selections, switches, and decision-related max-step failures.Source: §4.2.2 · pp. 6–7, Table 1 · p. 7

5.1 Experiment Setup

• The 13 agents used browser screenshots and page-state summaries to choose actions and submit a final action across executable web tasks.The evaluation covered three open and ten proprietary models.Source: §5.1 · p. 7

5.2 Main Result

• 77.42% average task success under normal charts fell to 43.24% when only the dashboard chart was replaced with its misleading counterpart, a 34.18 percentage-point decline.The headline robustness score was 39.56% for model–task pairs solved under both chart conditions.Source: §5.2 · p. 7

5.3 Failure Analysis

• 48.9% AFRc means nearly half of model–task pairs solved with clean evidence failed when only the chart evidence became misleading.Of 689 attributable failures, 616 (89.40%) followed the mechanism-aligned wrong action branch; 25 were irrelevant actions and 48 were timeouts.Source: Figure 4 · p. 7, §5.3 · p. 8
• Misleading evidence increased browser-action steps by SI=+0.20 and primary-decision instability by ∆DI=+0.78 on average across thirteen agents.Attributable failures had larger step increases than both-success cases, while cases failing under both chart versions had negative average deltas; audited traces also show a wrong selection followed by correction.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Prompt and Policy-System Defenses

• A generic prompt warning raised misleading-chart accuracy from 40.8% to 45.6%, while clean-chart accuracy remained 76.4%, leaving the robustness gap open.The effect varied across models, with some changing little or degrading.Source: §5.4 · pp. 8–9, Figure 6 · p. 9
• With GPT-5.4 and CUGA 0.3.0, generic policy guidance changed misleading-task success from 40.7% to 42.9% and clean-task success from 80.0% to 82.9%, so the gap did not narrow.Across misleading pairs, the Playbook produced 20 recoveries and 17 regressions (exact McNemar p = 0.7428).Source: §5.4 · p. 9

• MISVISAGENTBENCH isolates misleading visual evidence as a controlled variable in Web Agent evaluation.Source: §6 · p. 9
• More than 34 percentage points: replacing dashboard charts with misleading counterparts reduced average task success across thirteen agents.The failures predominantly followed the targeted misleading action branch rather than causing random breakdowns.Source: §6 · p. 9
• Misleading evidence also induced extra browsing steps and decision hesitation, including in tasks agents ultimately completed.Source: §6 · p. 9
• The findings support treating visualization robustness as a first-class dimension of Web Agent safety, alongside task competence and policy compliance.Source: §6 · p. 9
• Strong clean-chart performance did not predict robustness, and neither a generic warning nor a CUGA policy Playbook closed the gap.Source: §6 · p. 9

• The evaluation focuses on executable web tasks grounded in dashboard-style visual evidence.Source: Limitations · p. 9
• Although the benchmark covers multiple misleading mechanisms, it does not exhaust web visualizations, interaction patterns, or organizational workflows.Source: Limitations · p. 9
• Results depend on the prompting and browser-control protocol used by the agent runners, and other frameworks may exhibit different failure modes.Source: Limitations · p. 9

Ethics Statement

• Deceptive charts can redirect agents toward incorrect downstream actions, presenting an attack vector that could be exploited in financial or political contexts.Source: Ethics Statement · p. 9
• The benchmark is designed for diagnostic and defensive evaluation, and exposing vulnerabilities is presented as a first step toward robust, verifiable Web Agents.Source: Ethics Statement · pp. 9–10
• Benchmark visual evidence comes from public, anonymized datasets without personally identifiable information or offensive content; its synthetic web shells do not interact with live systems.Source: Ethics Statement · p. 10
• Authors and academic researchers conducted human review as part of their standard academic duties, with domain expertise and without undercompensated crowdworkers.Source: Ethics Statement · p. 10

A Appendix Overview

• Appendix B covers benchmark composition and mechanism definitions; Appendices C–H cover robustness, policy results, construction, agent processes, traces, and outcome counts.Source: A Appendix Overview · p. 13

B Benchmark Composition

• The clean-chart and official misleading-chart benchmarks contain the same 140 paired task identities and differ only in rendered chart evidence.Mechanism labels in the clean condition identify the misleading mechanism of the paired official instance, not a misleading property of the clean chart.Source: B Benchmark Composition · p. 13
• Table 2 reports the chart-type distribution across the 140 paired tasks.Source: Table 2 · p. 13
• Other mechanisms include unsupported annotations, cherry-picked subsets, unconventional axis directions, inappropriate scale functions, and marks disproportionate to their data.Source: B Benchmark Composition · p. 13
• Represented mechanisms include dual-axis comparisons, exaggerated or suppressed variation from axis ranges, invalid cumulative or categorical comparisons, and charts too small for reliable inspection.Source: Table 3 · p. 13, B Benchmark Composition · p. 13

C.1 Ability–Robustness Relationship

• r = −0.07 shows clean-task accuracy is not a reliable predictor of misleading-chart retention, and even the highest-retention agents lose about 40% of clean-solved tasks.Models with similar clean accuracy retain substantially different fractions of their clean-success cases.Source: §C.1 · p. 13

D CUGA Policy-System Comparison

• The Playbook yields a three-case net increase on misleading tasks, but the clean–misleading gap shifts from 39.3 to 40.0 percentage points with no statistically reliable change.The controlled comparison concludes that generic procedural guidance alone does not resolve misleading-visualization failures.Source: Table 4 · p. 14, D CUGA Policy-System Comparison · p. 15, Table 5 · p. 15

E.1 Visual Evidence Unit

• Each visual evidence unit records chart, mechanism, judgment, data, and trap information before conversion into a web task.The listed fields include chart type, misleading mechanism, affected judgment, clean and misleading assets, provenance, recoverable ground truth, and expected visual trap.Source: §E.1 · p. 15

E.2 Workflow Archetype Mapping

• The construction maps evidence units to four workflow archetypes, then specifies an intermediate judgment and downstream action goal.The archetypes are resource allocation and priority routing; risk triage and exception escalation; monitoring, trend response, and capacity planning; and evidence governance and decision approval.Source: §E.2 · p. 15

E.3 Task Specification Fields

• Task specifications separate agent-visible page content from hidden scoring fields to avoid evaluator-information leakage while retaining deterministic scoring.The fields distinguish visible page and action options from ground truth, expected actions, misleading targets, and reasoning operations.Source: §E.3 · p. 15, Figure 8 · p. 15

E.4 Executable Shell Generation Prompt

• LLM shell generation is constrained not to change hidden scoring or action identifiers, while benchmark and defense prompts are specified separately.The benchmark prompt is used for both chart conditions; the generic warning prompt is added only for misleading-chart runs and reveals neither mechanism type nor hidden scoring.Source: §E.4 · pp. 15–16, §E.5 · p. 16, §E.6 · p. 16

F.1 Agent Decision Process

• The browser runner supplies page screenshots and visible controls, records browser and decision-submission actions, and assigns outcomes after a terminal state.Stored traces support auditing final decisions and intermediate instability.Source: §F.1 · p. 16, §F.3 · p. 16, §F.4 · p. 16

G Step-Level Failure and Recovery Cases

• The four worked traces were selected post hoc from successful official-condition runs, so they illustrate patterns but do not estimate prevalence or causal effectiveness.Executed actions and selected options, rather than model-emitted text as evidence of latent reasoning, are the primary basis for verification.Source: G Step-Level Failure and Recovery Cases · p. 17

G.1 Worked Example: Recovering from Visual Disproportion

• Groundwater is visually tallest but reports 29.8%, while Surface water reports 41.2%; the agent first selects Groundwater, then corrects and submits Surface water.The task asks for the largest reported urban water-supply source; the corrected selection yields success.Source: §G.1 · pp. 17–18, Table 6 · p. 18, Figure 12 · p. 19

G.2 Complementary Evidence-Verification Patterns

• Three additional successful traces cover title–trend, represented-population, and separate-axis conflicts, illustrating revisions without establishing a causally sufficient defense.The table’s parentheses report how many of the thirteen main-model runs on each task ended in its pre-specified directed misleading branch.Source: §G.2 · pp. 18–19, Table 7 · p. 19

H Detailed Experimental Tables

• The appendix reports exact counts for the failure-attribution and trajectory-behavior figures in Section 4.The tables distinguish paired outcome counts from trajectory-level behavior changes.Source: H Detailed Experimental Tables · p. 19
• Table 8 gives exact paired outcome and attributable-failure counts across 140 paired tasks per model, including directed, irrelevant, and timeout outcomes.AFR denotes clean-success runs that fail under misleading evidence.Source: Table 8 · p. 19
• Table 9 reports trajectory-level changes from clean to misleading tasks, including average step and decision-instability deltas and counts with deltas of at least two.Step Δ and DI Δ are misleading-minus-clean averages.Source: Table 9 · p. 19