Source-linked AI summary
InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
Maeve Hutchinson, Syed Mahbubul Huq, Mohammad Albinhassan, Radu Jianu, Aidan Slingsby, Pranava Madhyastha
TL;DR
Existing visual-language benchmarks largely test static, one-shot interpretation, whereas analytical evidence can be hidden, distributed across views, or revealed conditionally. InSight introduces a benchmark for agentic claim verification in interactive visualizations, requiring navigation and recording interaction traces; evaluation shows that the task remains difficult, with GPT-5.5 reaching 57.2% accuracy.
Problem
Static, one-shot benchmarks do not capture the epistemic demands of interactive visual analysis, where evidence may be occluded, distributed across views, or conditionally revealed.
Method
InSight evaluates agents that navigate fully interactive web visualizations to verify claims and treats their action traces as proxies for reasoning behavior.
Results
57.2% accuracy: GPT-5.5 is the best-performing model, yet interactive claim verification remains challenging.
Takeaways & Limitations
InSight enables controlled analysis of whether models acquire visual evidence and whether correct answers reflect evidence-based reasoning.
Takeaways & Limitations
Interaction traces are complementary rather than exhaustive evidence of reasoning, because correct actions may be taken for the wrong reasons and plausible reasoning may still fail.
Abstract
from arXiv · showhide
Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interactive environments. Existing benchmarks are predominantly constrained to static imagery and one-shot question answering and fail to capture the epistemic demands of this domain, where evidence is frequently occluded, distributed across linked views, or conditionally revealed through user agency. In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations. The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments. Agents must navigate these environments to determine whether a natural language claim is supported, refuted or not verifiable given the available evidence. Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence. We evaluate state-of-the-art models, revealing that interactive verification remains a non-trivial challenge. We release InSight at https://github.com/maevehutch/insight.
1 Introduction
InSight addresses the gap between VLMs’ fluent chart responses and reliable visual grounding by evaluating claim verification in interactive, rather than static, analytical environments. It introduces a benchmark that requires agents to navigate visualizations and makes their interaction traces explicit evidence of reasoning strategy.
- VLMs often lack fundamental visualization literacy, grounding answers in textual context or prior training data rather than visual features.These failures are especially evident on modified or decontextualised charts.
- Static chart benchmarks assess models in closed settings but overlook that meaningful visualizations are commonly interactive environments.Dashboards, faceted browsers, and exploratory tools can reveal information conditionally through interaction.
- InSight reframes visual reasoning as sequential decision-making for agentic claim verification over interactive visualizations.Agents navigate web-based environments using actions such as clicking, hovering, and scrolling to verify data-related claims.
- The benchmark derives claims from human-authored analytical notebooks and preserves custom Vega-Lite visualizations to represent authentic analytical workflows.This contrasts with synthetic templates, contrived task setups, and crowd-sourced descriptions of static images.
- InSight treats interaction traces as first-class evidence, exposing an agent’s attentional focus and reasoning strategy beyond whether its final answer is correct.Mouse interactions and navigation paths enable analysis of how models seek and synthesize visual evidence.
- The benchmark contains over 21k claims and evaluates how interaction strategies correlate with reasoning fidelity.It formulates Interactive Visual Claim Verification for agentic, multimodal environments and explicitly penalises passive perception.
2 Background
Prior benchmarks separately study interactive task completion, static chart reasoning, and textual or multimodal fact-checking. InSight combines claim-verification supervision with interactive visual environments, where evidence must be actively acquired across partially observable views.
- Existing agentic benchmarks frame perception and action as sequential decision-making but primarily measure whether a goal state is reached.They highlight challenges in long-horizon planning, grounding, and robustness in realistic environments.
- InSight instead focuses on visual claim verification, using interaction explicitly to acquire evidence and analyze how agents seek, evaluate, and integrate multimodal information.The task is tightly scoped to controlled analysis of evidence-seeking behavior.
- Chart reasoning datasets progressed from synthetic or semi-real charts toward greater realism but generally retained one-shot question answering over static images.This limits their ecological validity for interactive analytical work.
- ChartCheck introduced claim verification for charts, yet its static inputs and benchmark-specific claims do not capture analysts’ interactive discovery of evidence.Such datasets also miss compositional and multi-view aspects of visualization use.
- InSight preserves interactive visualization environments and grounds claims in human-authored analytical narratives rather than synthetic schemas.This allows the benchmark to capture realistic tasks absent from existing chart-understanding datasets.
- Text-only and static multimodal fact-checking provide evidence in fixed textual or visual inputs, without sequential evidence acquisition.InSight adds interactive evidence discovery to the epistemic framing of claim verification, including NEI supervision.
3 InSight: Task and Dataset
InSight defines interactive claim verification as deciding whether a claim is true, false, or not sufficiently evidenced while navigating partially observable web visualizations. Its dataset is built from analyst-authored interactive notebooks, with claims extracted, decomposed, mutated, and human-validated.
- 3.1 Task Definition: Each instance pairs a natural-language claim with a web-based visualization environment containing multiple views and interactive UI elements.The task labels claims True, False, or Not Enough Information (NEI).
- 3.1 Task Definition: Relevant evidence may be occluded, distributed across views, or conditionally revealed, requiring a sequence of actions such as clicking, hovering, and scrolling.The environment is partially observable and updated through a fixed action space.
- 3.1 Task Definition: Interaction traces serve as intrinsic proxies for reasoning by recording which information the model seeks, how it navigates views, and when it stops.The formal task is sequential evidence acquisition.
- 3.2 Dataset Construction: The corpus combines interactive Vega-Lite visualizations with narratives written by trained analysts who selected datasets, questions, and custom designs.This construction reflects authentic analytical provenance.
- 3.2 Dataset Construction: Candidate spans are extracted at statistical or visual/perceptual semantic levels, then stabilized across three LLM runs using agreement-based consolidation.Single-run spans are discarded to filter hallucinated or irrelevant extractions.
- 3.2 Dataset Construction: Retained spans are decomposed into atomic, decontextualized propositions that can be independently verified against visualizations.Self-verification and majority voting support claim grounding and semantic labeling.
- 3.2 Dataset Construction: Controlled mutations generate False claims through antonym or in-lexicon substitutions and NEI claims through out-of-lexicon substitutions.An NLI model validates contradiction and entailment thresholds before invalid claims are discarded.
- 3.2 Dataset Construction: Human validation produced 475 annotations across 294 sampled claims, with 81.3% raw agreement with dataset labels.Annotators performed the interactive verification task while blind to original labels and construction.
4 Evaluation and Behavioural Analysis
InSight evaluates interactive claim verification by requiring agents to acquire visual evidence through bounded navigation and interaction. Results show that interaction quality varies substantially across models, with stronger performance associated with more effective exploration, while interactive verification remains unsaturated.
- Experimental Setup: The evaluation renders interactive Vega-Lite notebooks in a fixed-viewport Chromium browser and gives agents a bounded action space for navigation and mouse interaction.The action space includes scrolling, page-level navigation, dragging, clicking, and related mouse actions.
- Evaluation Metrics: IES measures whether agents both answer correctly and produce observable state changes through interaction, complementing accuracy rather than replacing it.The metric is designed to distinguish productive evidence acquisition from passive answering.
- Evaluation Metrics: Correct answers submitted without interaction receive IES = 0 because no interaction indicates parametric knowledge or chance rather than visual grounding.The benchmark assumes claims cannot be verified from the initial viewport alone.
- Evaluation Metrics: Scrolling is usually counted as effective because it reveals views that do not fit in the viewport and helps agents locate relevant evidence.The metric intentionally treats exploratory navigation as useful in a partially observable environment.
- Model Results: At Tmax = 10, GPT-5.5 achieves the highest overall accuracy at 57.2%, while all models exceed random performance but remain below human-level reliability.Random performance on the balanced three-class task is 33.3%; accuracy generally declines with smaller model scale within Qwen and Gemma families.
- Model Results: Gemini’s accuracy is non-monotonic across budgets, reaching 50.0% at Tmax = 25 versus 44.2% at 1 and 41.6% at 10, while its IES rises from 24.93% to 31.0%.Longer interaction allows Gemini to complete more exploration and inspect previously discovered views, although its effective action ratio falls slightly.
- Behavioural Analysis: GPT-5.5 averages 3.13 actions with the highest accuracy, whereas Gemini averages 6.27 actions and Qwen 3.5 0.8B averages zero.The traces indicate a contrast between economical targeted exploration and immediate answer submission.
- Behavioural Analysis: GPT-5.5 leads IES at 26.98%, followed by Gemini 3.5 Flash at 24.93%, while Gemma 4 31B’s comparable accuracy is reduced by a lower effective action ratio.The accuracy–IES gap indicates that some Gemma actions fail to advance the observable environment state.
5 Conclusion
InSight frames claim verification as interactive evidence acquisition over web-based visualizations, using 21,349 claims grounded in human-authored analytical notebooks. Evaluation finds interactive verification remains difficult, with limited accuracy, ungrounded correct answers, and disproportionate difficulty falsifying claims.
- InSight comprises 21,349 claims grounded in human-authored analytical notebooks and requires active navigation of web-based environments.
- The benchmark moves beyond static visual question answering by capturing the sequential, exploratory nature of real analytical workflows.
- 57.2% accuracy is achieved by GPT-5.5, the best-performing model evaluated, showing that interactive claim verification remains challenging.
- IES exposes correct answers produced without environmental interaction as ungrounded prediction rather than evidence-based reasoning.
- Falsification is disproportionately difficult because it requires targeted evidence-seeking interaction that most models struggle to produce.
6 Limitations
InSight’s evaluation is constrained by its fixed, high-level action space and by the limited evidentiary scope of interaction traces. These choices support controlled comparison but do not fully represent human interaction modalities or internal deliberation.
- The fixed, high-level action space abstracts common mouse and navigation interactions and may omit gesture-based interactions or interface-specific semantic shortcuts.
- Different action-space designs may produce different interaction strategies and performance profiles.
- Interaction traces do not fully capture internal model deliberation because correct actions may be taken for the wrong reasons, or plausible reasoning may fail.
7 Ethical Considerations
The corpus was governed by institutional ethical oversight and drawn from graduate capstone projects. Participation in the associated study was voluntary and unpaid.
- The corpus construction was formally ratified by the university’s Research Ethics Committee.
- The corpus comprises artefacts from capstone projects in a graduate-level Advanced Data Visualisation module.
- Data science master’s students participated voluntarily and were not paid.
A Dataset Construction Details
The appendix documents the implementation of the paper’s four-stage dataset construction pipeline, complementing the summary of that pipeline in Section 3.2.
- The appendix provides full implementation details for the four-stage dataset construction pipeline.
- The documented pipeline concerns dataset construction rather than the benchmark’s evaluation procedure.
- The appendix expands the pipeline summary presented in Section 3.2.
A.1 Data Sources and Filtering
InSight is built from analyst-authored interactive notebooks, extracting and stabilizing verifiable claims before generating controlled FALSE and NEI variants. Grounding checks and annotator analysis support dataset quality.
- Data sources: InSight retains 297 notebooks combining human-authored analytical narratives with corresponding interactive visualizations.
- Claim extraction: Candidate spans target statistical and visual insights that can be verified or refuted by visual evidence.
- Claim extraction: Cross-run agreement retains extractive, stable spans and filters out isolated or semantically irrelevant extractions.
- Claim construction: Retained spans are decomposed into decontextualized atomic claims with resolved references, then self-verified and relabeled across three passes.
- Claim mutation: True claims anchor controlled FALSE and NEI mutations, while NLI validation filters mutations that fail to establish the intended semantic relationship.
B Dataset Construction Method Example
The worked example shows how narrative statements become atomic claims and controlled mutations. Antonym and argument substitutions produce FALSE candidates, while out-of-lexicon substitutions target NEI candidates subject to NLI validation.
- Span and proposition processing: A traffic-accident narrative is converted into labeled spans and atomic propositions with implicit references resolved.
- Antonym substitution: Antonym substitution reverses directional language, producing false claims such as changing “decreased” to “increased.”
- In-lexicon argument substitution: In-lexicon argument substitution changes dates, categorical values, or numerical quantities using alternatives from the same dataset.
- Contradiction validation: A numerical mutation with contradiction score 0.068 is rejected, illustrating that low-confidence contradictions do not pass validation.
- NEI generation: Out-of-lexicon substitutions target NEI claims, but examples are rejected when entailment or contradiction scores indicate another relationship.
C Dataset Statistics
InSight contains diverse visualization and interaction environments rather than single-chart snapshots. Its notebooks commonly combine multiple specifications and interaction mechanisms, creating multi-view evidence-search tasks.
- Mark types: Bars are most common, but notebooks also include lines, geoshapes, circles, text marks, and layered combinations.
- Specification and interaction density: Most notebooks contain multiple visualization specifications and multiple distinct interaction mechanisms, with long right tails indicating highly complex environments.
- Dataset composition: Figure 3 summarizes interaction primitives, Vega-Lite mark types, visualization specifications per notebook, and distinct interaction types per notebook.
D Model Evaluation Examples
The evaluation examples contrast correct and incorrect interaction traces across models. The traces include successful verification of true claims and errors that label true or NEI claims incorrectly.
- Correct verification: GPT 5.5 is shown correctly verifying a true claim in an interaction trace.
- Incorrect verification: Gemma 4 31B Flash incorrectly labels a true claim as NEI while verifying it.
- Incorrect verification: Gemini 3.5 Flash likewise incorrectly labels a true claim as NEI in its interaction trace.
- Correct verification: GPT 5.5 is also shown correctly verifying a true claim in a second interaction trace.
- NEI errors: Additional traces show Gemma 4 31B and Gemini 3.5 Flash incorrectly labeling an NEI claim as False.