Source-linked AI summary
DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis
Chuhan Zhang, Qi Xie, Ziyue Wang, Jianing Yin, Yunfan Zhou, Dazhen Deng, Yingcai Wu
TL;DR
Interactive dashboard analysis requires GUI agents to reveal and connect evidence through stateful interactions, while existing benchmarks provide limited diagnostic detail about failures. DashAct evaluates shared annotated interactions across execution, action prediction, and grounding under varied support; experiments find persistent challenges and model-dependent responses, while Qwen-based adaptation improves localization and online Milestone Coverage.
Problem
Existing dashboard benchmarks primarily report answers or task success and provide limited insight into whether failures concern analytical-process continuity, action selection, or target grounding.
Source: §1 · pp. 1–2Method
DashAct evaluates the same human-verified dashboard interactions across end-to-end execution, verified-context action prediction, and target-conditioned grounding with varied spatial support.
Source: §1 · p. 2Results
Takeaways & Limitations
DashAct provides diagnostic evidence for interpreting online failures, and its Qwen-based adaptation study illustrates the practical value of diagnosis-guided agent improvement.
Source: §1 · p. 2, §5 · p. 8Fine-grained trajectory annotation limits the collection to a bounded range of analytical workflows.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, a progressive diagnostic benchmark that evaluates the same annotated dashboard interactions under systematically varied contextual and spatial support. DashAct contains 348 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its three stages evaluate end-to-end execution, verified-context action prediction, and target-conditioned grounding under four visual conditions. Aligned action and spatial annotations connect outcomes across these settings, providing diagnostic evidence for interpreting online failures. Experiments reveal persistent difficulties in action prediction and fine-grained grounding, alongside model-dependent responses to contextual and spatial support. A Qwen-based adaptation study further improves target localization and online Milestone Coverage, illustrating how DashAct can guide practical agent improvements.
1 Introduction
Interactive dashboard analysis depends on stateful interactions that reveal and connect evidence, creating sequential and spatial demands for GUI agents. DashAct links outcomes on the same annotated interactions across execution, action prediction, and grounding to help diagnose failures.
1 Introduction
- Dashboard analysis requires agents to obtain and connect evidence through interactions with charts, tables, filters, and coordinated views.Configuring a view, inspecting marks, and interpreting the resulting state can be necessary to answer a question.Source: §1 · p. 1
- Stateful, interdependent actions require agents to track interface changes and retain intermediate evidence as filters or selections alter views and subsequent targets.Errors can propagate through the process and prevent successful task completion, while dense controls and visualization marks require precise screen localization.Source: §1 · p. 1
- Existing benchmarks assess dashboard question answering, online progress, action prediction, or grounding, but their separate measures do not connect missed online interactions to action or localization failures.Overall grounding accuracy also does not distinguish locating the relevant view from identifying a fine-grained target within it.Source: §1 · pp. 1–2
- DashAct links end-to-end execution, verified-context action prediction, and target-conditioned grounding on the same annotated interactions under varied contextual and spatial support.The benchmark contains 348 analytical questions and human-verified trajectories from 142 real-world dashboards.Source: §1 · p. 2
- Experiments find that action prediction and fine-grained grounding remain challenging even with additional support, while models respond differently to contextual and spatial guidance.Visualization marks pose greater grounding challenges than interface controls; a Qwen-based adaptation improves target localization and online Milestone Coverage.Source: §1 · p. 2
2 Related Work
Prior work spans static chart understanding, GUI agents, and interactive dashboard benchmarks. DashAct combines online evaluation with aligned milestone, action-prediction, and grounding assessments on the same interactions.
Visual Data Understanding and Interactive Agents
- Chart-understanding benchmarks focus on static visualizations, while general-purpose GUI agents and specialized systems address rendered interfaces, planning, analytical assistance, or agent-oriented interaction.Examples include LightVA, ProactiveVA, and VACP.Source: §2 · pp. 2–3
Benchmarks for Interactive Agents
- Interactive-agent benchmarks separately assess online execution, next-action prediction, intermediate progress, planning, and visual grounding.DashboardQA evaluates interactive question answering on real dashboards, and VizAgentBench assesses coordinated multi-view analysis through declarative commands.Source: §2 · p. 3
- DashAct combines online evaluation, reference-action milestone matching, verified-context action prediction, and target-conditioned grounding, linking outcomes for the same annotated interactions.The linked results provide diagnostic evidence for interpreting online failures and examining responses to contextual and spatial support.Source: §2 · p. 3
3 Benchmark
DashAct represents dashboard questions as screenshot-and-action trajectories and uses verified milestones and targets to connect its construction, dataset, and evaluation. Its four visual grounding conditions and three diagnostic stages support paired comparisons across the same interactions.
3.1 Problem Setup
- A dashboard task is a dashboard, natural-language question, and reference answer, followed by screenshot observations, GUI actions, and a submitted answer.Actions can update filters, selections, or coordinated views, requiring evidence to be revealed and connected across successive states.Source: §3.1 · p. 3
- Human-verified milestones record reference actions, target descriptions, and item and containing-view boxes, while a dependency DAG encodes required precedence.Independent milestones may be completed in any order, and the annotations provide a shared evaluation unit across diagnostic stages.Source: §3.1 · p. 3
3.2 Benchmark Construction
- DashAct uses real-world multi-view dashboards, primarily from Tableau Public, with experienced annotators recording executable trajectories across analytical domains.Annotators replay trajectories to verify reproducibility, questions, answers, actions, targets, and precedence relations; a second annotator reviews each completed annotation.Source: §3.2 · p. 3
- The adapted DashboardQA set excludes questions requiring information from multiple dashboards and uses the same trajectory and milestone verification procedure.Tableau metadata is used only for offline dataset construction and verification.Source: §3.2 · p. 3
3.3 Dataset Statistics
- 348 DashAct questions span 142 dashboards and average 2.15 milestones per trajectory; the adapted DashboardQA set has 242 questions across 83 dashboards and averages 3.99 milestones.Mean initial-state view counts are 9.53 for DashAct and 2.94 for adapted DashboardQA.Source: §3.3 · p. 4
3.4 Evaluation
- End-to-end execution measures both milestone progress and final-answer correctness, with Milestone Coverage counting matched annotated milestones and Answer Accuracy measuring semantic agreement with the reference.Milestones are matched against reference action types and target boxes, with a dependency DAG enforcing required ordering while allowing independent milestones in any order.Source: §3.4 · p. 4
- Verified-context action prediction gives each coordinate-requiring milestone its verified preceding history and pre-action screenshot, then scores the predicted action type and target.Action Type Accuracy, Target Hit Rate, and Action Success measure type match, target-box hit, and both together; other action arguments are not evaluated.Source: §3.4 · p. 4
- Target-conditioned grounding asks models to localize described targets under Full Image, Red Box, Gray Mask, and Local Crop conditions.The conditions use the same target descriptions, annotated instances, and reference screenshots; Target Hit Rate uses a deterministic point-in-box test.Source: §3.4 · pp. 4–5
4 Experiments
DashAct evaluates nine core models across three stages, from autonomous dashboard execution to action prediction and visual grounding. Results show persistent gaps across stages, while diagnostic profiles and qualitative trajectories distinguish several sources of failure.
4.1 Experimental Setup
- The evaluation covers nine core models, uses screenshots without DOM or accessibility-tree information, and runs each task once for at most 20 turns.Stage 1 uses live dashboards; Stages 1–2 use full-dashboard screenshots, while Stage 3 evaluates four visual conditions.Source: §4.1 · p. 5
4.2 Main Results on DashAct
- Gemini 3 Flash Preview leads Stage 1 with 70.9% Milestone Coverage and 51.7% Answer Accuracy, while Claude Sonnet 5 leads Stage 2 Action Success at 49.8%.Even the strongest end-to-end model answers only just over half of tasks correctly, and the strongest action predictor succeeds on fewer than half of actions in one attempt.Source: §4.2 · p. 5, Table 1 · p. 6
- Gray masking improves Stage 3 grounding for eight of nine core models, while local cropping helps some models but reduces Claude Sonnet 5 from 75.2% to 62.0%.Cropping narrows the search area but removes global layout cues and changes image dimensions and target positions.Source: §4.2 · p. 5
4.3.1 Grounding across Spatial Levels and Target Types
- Grounding profiles separate view selection from within-view localization, while all nine core models ground interface controls better than visualization marks.Gemini 3 Flash Preview reaches 84.5% View Hit, 61.3% Target Hit, and 72.5% Target Hit|View; on full screenshots, its control and mark accuracies are 90.3% and 64.7%.Source: §4.3.1 · p. 6, Table 2 · p. 6, Table 3 · p. 6
4.3.2 Diagnostic Profiles of Online Failures
- Verified-context action prediction recovers 30.3% of Claude Sonnet 5’s Stage 1-unmatched milestones, while explicitly specifying targets recovers 38.3% for Gemini 3 Flash Preview.Target Full is the largest additional success category for eight of nine models, indicating that some online failures persist even when the target is specified for localization.Source: §4.3.2 · p. 7
- View support localizes an additional 36.3% of GPT-5’s online failures but 13.5% of Claude Sonnet 5’s, while 55.4% of GPT-5’s failures remain unresolved.View Support is an oracle union across Gray Mask, Red Box, and Local Crop, not sequential online recovery.Source: §4.3.2 · p. 7, Figure 2 · p. 7
4.4 Diagnosis-Guided Agent Adaptation
- Qwen3-VL-8B-Instruct’s Grounder fine-tuning raises Target Hit Rate from 36.0% to 49.4%, and the SFT pipeline raises online Milestone Coverage from 14.0% to 51.2%.Online Answer Accuracy rises from 5.7% to 8.6% on the same 35 tasks and 86 reference milestones, showing limited gains in final-answer accuracy.Source: §4.4 · pp. 7–8, Table 4 · p. 8
4.4.1 Qualitative Case Analysis
- Qualitative trajectories show that agents may repeat actions after failing to recognize state changes, or answer incorrectly despite matching all annotated milestones.Errors also arise when agents overlook prerequisite actions; interaction feedback can help some models refine inaccurate grounding.Source: §4.4.1 · p. 8
5 Conclusion
DashAct links human-verified dashboard interactions across three diagnostic stages to help interpret online failures. Experiments find persistent action-prediction and fine-grained localization difficulties, while a Qwen-based adaptation improves localization and online Milestone Coverage.
5 Conclusion
- DashAct links the same interactions across end-to-end execution, verified-context action prediction, and target-conditioned grounding to provide diagnostic evidence about online failures.The benchmark contains 348 human-verified trajectories from 142 real-world dashboards.Source: §5 · p. 8
- Experiments reveal persistent difficulties in action prediction and fine-grained localization.Source: §5 · p. 8
- A Qwen-based adaptation improves target localization and online Milestone Coverage, demonstrating the practical value of diagnosis-guided agent improvement.Source: §5 · p. 8
Limitations
The benchmark’s coverage is bounded by its platform sources, annotation demands, and single-dashboard task formulation. Broader platform, trajectory, and workflow coverage remains a possible extension.
Limitations
- DashAct primarily draws dashboards from Tableau Public, and other platforms may have different visual conventions and interaction patterns.Evaluation uses rendered screenshots; construction-time metadata is used only offline.Source: Limitations · p. 8
- Human replay and verification of milestones, dependencies, and spatial targets constrain the collection to a bounded range of analytical workflows.Longer trajectories with more varied dependency structures are proposed as future extensions.Source: Limitations · p. 9
- The task formulation covers analytical questions within a single dashboard, not workflows combining evidence across multiple dashboards.Cross-dashboard navigation and evidence integration are proposed as extensions.Source: Limitations · p. 9