PaperlayerView exact revision on arXiv

Source-linked AI summary

DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios

Jinxiang Meng, Shaoping Huang, Fangyu Lei, Jingyu Guo, Haoxiang Liu, Jiahao Su, Sihan Wang, Yao Wang, Enrui Wang, Ye Yang, Hongze Chai, Jinming Lv, Anbang Yu, Huangjing Zhang, Yitong Zhang, Yiming Huang, Zeyao Ma, Shizhu He, Jun Zhao, Kang Liu

arXiv:2604.25914v1cs.CL

TL;DR

Existing data-visualization benchmarks underrepresent native spreadsheet workflows, visualization evolution, and ambiguous user intent. DV-World evaluates these lifecycle capabilities across 260 tasks, finding that state-of-the-art agents remain below 50% overall performance.

  • Problem

    Existing benchmarks underrepresent professional visualization workflows by emphasizing sandboxed, creation-only tasks and assuming fully specified user intent.

  • Method

    DV-World benchmarks visualization agents across native spreadsheets, cross-paradigm evolution, and ambiguous multi-turn interaction using numerical checks and rubric-based visual evaluation.

  • Results

    Agents struggled across all three benchmark domains; DV-Sheet performance peaked at 40.48%.

  • Takeaways & Limitations

    DV-World provides a standardized testbed for diagnosing and tracking progress in end-to-end data-visualization workflows.

  • Takeaways & Limitations

    The benchmark preserves range-level provenance only; aggregation correctness is not guaranteed, and post-hoc cell edits may break bindings.

Abstract

from arXiv · show

Real-world data visualization (DV) requires native environmental grounding, cross-platform evolution, and proactive intent alignment. Yet, existing benchmarks often suffer from code-sandbox confinement, single-language creation-only tasks, and assumption of perfect intent. To bridge these gaps, we introduce DV-World, a benchmark of 260 tasks designed to evaluate DV agents across real-world professional lifecycles. DV-World spans three domains: DV-Sheet for native spreadsheet manipulation including chart and dashboard creation as well as diagnostic repair; DV-Evolution for adapting and restructuring reference visual artifacts to fit new data across diverse programming paradigms and DV-Interact for proactive intent alignment with a user simulator that mimics real-world ambiguous requirements. Our hybrid evaluation framework integrates Table-value Alignment for numerical precision and MLLM-as-a-Judge with rubrics for semantic-visual assessment. Experiments reveal that state-of-the-art models achieve less than 50% overall performance, exposing critical deficits in handling the complex challenges of real-world data visualization. DV-World provides a realistic testbed to steer development toward the versatile expertise required in enterprise workflows. Our data and code are available at \href{https://github.com/DA-Open/DV-World}{this project page}.

1 Introduction

DV-World benchmarks data visualization agents across the full real-world lifecycle, addressing environmental decoupling, creation-only tasks, and perfect-intent assumptions. Its hybrid evaluation combines numerical fidelity checks with rubric-guided visual-semantic assessment, while experiments show substantial performance limits across native spreadsheet, evolution, and interaction scenarios.

  • Motivation: Existing benchmarks leave agents ill-equipped for real-world visualization through environmental decoupling, creation-only myopia, and perfect-intent assumptions.The environmental gap includes bypassing spreadsheet-native chart object models, data-to-chart bindings, and GUI constraints.
  • Benchmark scope: DV-World evaluates agents across native spreadsheet manipulation, cross-modal logic evolution, and proactive iterative interaction in real software environments.These scenarios are organized as DV-Sheet, DV-Evolution, and DV-Interact.
  • DV-Sheet: DV-Sheet establishes native grounding by evaluating manipulation of native spreadsheet object models across chart creation, dashboard creation, and diagnostic repair.The supplied passage explicitly identifies DVSheet-Create and describes DV-Sheet as spanning these native spreadsheet workflows.
  • Evaluation: The hybrid evaluation framework combines Table-value Alignment for numerical fidelity with hierarchical MLLM-as-a-Judge assessment guided by expert-annotated, fine-grained rubrics.The framework is designed to go beyond conventional code verification and capture visual-semantic quality.
  • Results: 40.48, 51.44, and 40.43 are the peak scores for DV-Sheet, DV-Evol, and DV-Inter, respectively, revealing difficulty with native object models, dynamic bindings, and semantic logic migration.The results describe state-of-the-art agents as struggling with multi-tool visualization across all three benchmark settings.

2 Benchmark Construction

DV-World constructs a real-world data visualization benchmark around native spreadsheet editing, visualization evolution, and interactive intent clarification. Its task-specific hybrid evaluation combines expert rubrics, quantitative fidelity measures, logic-based success criteria, and gold standards.

  • Task Coverage: DV-World covers three real-world challenges: native spreadsheet charting, visualization evolution, and interactive visualization.These correspond to DV-Sheet, DV-Evol, and DV-Inter task families.
  • DV-Sheet: DV-Sheet evaluates native spreadsheet editing through chart creation with dynamic bindings, defective-chart repair, and professional dashboard composition.The creation task requires dynamic range bindings rather than hard-coded values, while Fix diagnoses and repairs defective charts.
  • DV-Evol: DV-Evol requires reverse-engineering a reference visualization and producing functional plotting code plus a resultant data table in a target language.The formulation takes a reference image, new dataset, modification requirements, and target language as inputs.
  • DV-Inter: DV-Inter uses an ask_user tool and dual-stage user simulator to clarify ambiguities in visualization requests within a stateful data-and-language environment.The simulator includes an interaction gatekeeper that detects and refuses cheating attempts.
  • Evaluation: The hybrid evaluation combines expert qualitative rubrics with quantitative metrics for visual quality and data fidelity, while operational tasks use logic-based appraisal and human-established gold standards.Rubrics assess dimensions including Reliability, Appropriateness, Aesthetics, Integrity, and Consistency; DVSheet-Fix uses a strict Success Rate with τ ≥0.95.
  • Construction Pipeline: 18 visualization specialists build DV-World, adapting over 800 authentic problem threads through structure retention, value perturbation, and privacy-preserving processing.The construction pipeline targets functional realism and semantic precision.

3 Experiments and Analysis

Experiments show that current agents struggle across native spreadsheet visualization, cross-framework chart evolution, and iterative intent alignment, with top scores below or near 51.44%. Error analyses attribute failures to data grounding, framework complexity, visual feedback dependence, and gaps between clarified intent and execution.

  • DV-Sheet results: 40.48% is Gemini-3-Pro’s peak DV-Sheet score, while GPT-5.2 and DeepSeek-V3.2 remain below 38.00%.The results indicate a persistent gap from human performance and a ceiling in native environment execution.
  • DV-Sheet analysis: Fix and Dash are the main DV-Sheet bottlenecks, exposing weaknesses in data-to-chart binding, diagnostic repair, and multi-view workbook planning.These tasks combine reliable chart binding, error correction, and professional spatial organization within complex workbooks.
  • DV-Evol results: 51.44% is Gemini-3-Pro’s highest DV-Evol overall score, followed by Gemini-3-Flash at 49.46% and GPT-4.1 at 44.67%.Top models perform better in Python and Vega-Lite than in D3.js and Plotly.js.
  • DV-Inter results: 40.43% is Grok-4’s peak DV-Inter score, while most top-tier models fail to reach 38.00% because they miss ambiguities or provide inadequate clarifications.The interactive process fails to bridge underspecified user intent and the logic required for accurate visualization.
  • DV-Evol analysis: Removing load_image universally lowers DV-Evol success rates, while performance consistently decays as target Lines of Code increase.The analysis links longer code to greater syntactic density and reasoning burden, especially for frameworks such as D3.js.
  • DV-Inter analysis: 60.07% is GPT-5.2’s Cognitive Execution Gap rate, while GLM-4.7’s Interactive Avoidance reaches 59.47%.These error profiles reflect failures to turn aligned intent into grounded results and overconfidence-driven task downgrades.

4 Meta-Evaluation of the Framework

The meta-evaluation finds that simulator intelligence drives agent performance and interaction cost, with GPT-5-mini offering the most efficient and stable user-simulation environment. Human audits and agreement analyses further support the framework’s alignment and evaluation reliability.

  • Simulator intelligence and cost: GPT-5-mini is identified as the optimal user simulator, combining high instructional fidelity with reduced operational costs and stable measurement.The GPT-5 series leads overall, while other simulators remain useful for stress tests or average-complexity settings.
  • Human user alignment: 150 interaction trajectories were manually audited to assess simulator alignment with human behavior, response reliability, and sensitive-information prevention.The audit specifically examined whether simulated interactions reliably reflect human behavior while preventing sensitive information exposure.
  • Simulator intelligence and cost: 88.67% Faithfulness and 0.86 Pearson correlation (p < 0.04) indicate strong alignment between the GPT-5-mini simulator and human judgment.Removing Reaction Rules or Stage 1 Filtering significantly reduces alignment, producing less reliable and potentially insecure interactions.
  • Human-model alignment: ICC(A,1)=0.932 and weighted κ = 0.903 show strong agreement among human annotators across 210 tasks, supporting consistent rubric-based judgment.The agreement provides a practical upper bound for automated MLLM judges.
  • Cross-judge consistency: Relative rankings are largely preserved across proprietary judge families on the same 210 tasks, despite score shifts; Gemini-2.5-Flash is selected as primary judge.The selection is based on its strongest item- and case-level alignment with human experts.

5 Related Work

Prior agentic benchmarks emphasize execution-grounded interaction, long-horizon tool use, feedback adaptation, and intent clarification, but remain largely domain-general. Data-visualization benchmarks focus mainly on single-shot specification or code generation, alongside chart and spreadsheet manipulation tasks.

  • Agentic benchmarks: Agentic benchmarks evaluate LLM agents through execution-grounded and interactive settings, emphasizing long-horizon tool use and adaptation to feedback.Representative suites address repository-level software engineering and requirement-driven code generation.
  • Agentic benchmarks: Other agentic suites model realistic SQL/data workflows and interactive protocols that stress intent clarification and recovery.However, these benchmarks remain largely domain-general and do not explicitly model professional data-visualization workflows.
  • Benchmarks for data visualization: Data-visualization benchmarks mainly study single-shot translation from natural language into visualization specifications or plotting code, complemented by chart understanding, editing, and chart-to-code generation.Spreadsheet benchmarks additionally evaluate long-horizon table manipulation and control.

6 Conclusion … A.3 Detailed Data Statistics

DV-World constructs a 260-task benchmark spanning native spreadsheet automation, visualization evolution, and multi-turn interaction under ambiguous intent. Its task construction uses real-world data sources, executable ground truths, targeted perturbations, and quantitative or property-based evaluation.

  • 6 Conclusion: DV-World evaluates DV-Sheet, DV-Evol, and DV-Inter across chart creation, debugging, dashboards, cross-paradigm refinement, and ambiguous multi-turn interaction.The benchmark targets real-world workflows beyond traditional benchmarks, including spreadsheet-centric tasks and visualization evolution.
  • A.1 Data Sources: The data sources combine community Q&A from forums and blogs with open datasets to capture realistic spreadsheet questions, answers, and tables.Sources include Kaggle, Chandoo.org, Excelguru, ExcelForum, and MrExcel, covering diverse tabular data and naturally occurring spreadsheet issues.
  • A.2.1 Data Annotation: DVSheet-Crea: DVSheet-Crea contains 80 test cases built through categorized data selection, instruction and solution generation, and quantitative rubric design.Ground truths are native Excel chart objects with correct bindings, while binary checks assess data fidelity, visualization logic, and related constraints.
  • Appendix Contents; A Task Construction Details; A.3 Detailed Data Statistics: Across the construction process, executable scripts define ground truth alongside rendered artifacts or inspected object states, supporting reproducibility, auditability, and robust evaluation.The supplied passages describe native Excel objects, refined scripts and images, code-based intent enforcement, and binary or property-based checks.
  • A.2.2 Data Annotation: DVSheet-Fix: DV-Sheet Fix includes 50 defective-spreadsheet cases requiring diagnosis and repair through error taxonomies, controlled inverse injection, and state-based property verification.Evaluation inspects internal chart properties rather than relying only on visual similarity.
  • A.2.3 Data Annotation: DVSheet-Dash: DVSheet-Dash contains 30 tasks requiring cohesive interactive dashboards, using hybrid reference construction, user archetypes, and rubrics for insight, consistency, visual structure, and professionalism.The workflow combines data selection, coded baseline prototyping, and expert refinement, including interactive controls such as Slicers.
  • A.2.4 Data Annotation: DV-Evol: DV-Evol contains 80 refinement tasks that isolate executable subplots, perturb schemas or distributions, sample edits across four operation categories, and enforce no-hardcode verification.Ground truth includes the refined script, rendered image, and metadata, requiring visual elements to derive dynamically from input data.
  • A.2.5 Data Annotation:DV-Inter: DV-Inter models vague requests through ambiguity injection, hidden intent facts, reaction rules, executable ground truths, and rubrics covering process, correctness, and presentation.Ambiguities include metric definitions, temporal scope, and aggregation, while the simulator responds according to predefined clarification rules.

A.3.1 Taxonomy of fix types for the DVSheet-Fix task

The DVSheet-Fix taxonomy defines four primary repair categories spanning data preparation through visual specification, making visualization defects actionable and evaluable. It emphasizes that many failures arise from upstream data inconsistencies or mis-specified encodings and scales rather than cosmetic issues.

  • Taxonomy scope: The taxonomy organizes DVSheet-Fix repairs into four primary categories spanning data preparation to visual specification.It maps recurring semantic and presentation defects to actionable edits that restore faithful communication.
  • Primary fix categories: Visual Encoding & Mapping covers geometry reselection, occlusion resolution, and multi-view composition for dense or heterogeneous data.These operations aim to better express intended relationships.
  • Primary fix categories: Coordinate System & Scaling addresses range and limit control, transformation or orientation correction, and aspect-ratio or alignment enforcement.The category supports consistent comparability across views.
  • Primary fix categories: Data Transformation & Integrity repairs data-source cleaning, aggregation logic, and continuity or missing-value handling.These fixes prevent spurious patterns caused by erroneous data or mismatched analytic intent.
  • Evaluation utility: The taxonomy supports fine-grained error analysis and category-level reporting of repair performance.Its distribution indicates that failures often stem from upstream data inconsistencies or mis-specified encodings and scales that distort comparisons and trends.

A.3.2 The ambiguity-point taxonomy for the DV-Inter · B User Simulator Design Details · B.1 Problem Setting

DV-Inter models visualization as an intent-recovery problem: agents must clarify underspecified natural-language queries before producing charts. Its ambiguity taxonomy covers semantic boundaries, computational logic, and visual encoding uncertainty, while the interact environment pairs agents with a constrained user simulator.

  • A.3.2 The ambiguity-point taxonomy for the DV-Inter: DV-Inter requires agents to ask clarification questions because a single-shot chart specification may be incorrect for an underspecified query.The agent interacts with a user simulator to recover intent before generating the visualization.
  • A.3.2 The ambiguity-point taxonomy for the DV-Inter: Semantic Boundary ambiguities include missing Top-K values, relative or absolute thresholds, temporal windows, era boundaries, and domain-concept mappings.Examples include “top performers,” “high,” “recent,” undefined year ranges, and business jargon without direct column matches.
  • A.3.2 The ambiguity-point taxonomy for the DV-Inter: Rank-Based Top-K is reported for 14% of ambiguity points, where the rank cutoff is implied but K is unspecified.The accompanying example is “Top, Best, ‘top performers’ (no number).”
  • A.3.2 The ambiguity-point taxonomy for the DV-Inter: Relative Temporal Windows account for 12% of ambiguity points because the requested time span is underspecified.Examples include “Recent,” “Latest,” and “past few years/months.”
  • A.3.2 The ambiguity-point taxonomy for the DV-Inter: Volume & Counting represents 13% of ambiguity points, reflecting unspecified aggregation such as sum, count, or distinct count.The ambiguity concerns which aggregation operator should be applied to the data.
  • A.3.2 The ambiguity-point taxonomy for the DV-Inter: Logic ambiguities cover unclear aggregation operators, magnitude interpretations, numerical bins, hierarchical granularity, derived-metric formulas, and data-cleaning assumptions.Examples distinguish sum, count, and distinct count; total versus mean/median/rate; undefined bin edges; grouping levels; formulas; and missing-value handling.
  • B.1 Problem Setting: The interact environment pairs a plotting agent with a constrained user simulator to recover hidden intent z from ambiguous mappings or filters before chart generation.This setup treats strategic dialogue as part of the visualization task rather than assuming perfect intent specification.

B.2 Two-Stage Simulator Architecture … B.3 Evaluation Signals

DV-World’s interaction simulator separates security control from response generation to prevent agents from bypassing clarification, while grounding simulated users in hidden intent and available facts. Evaluation combines chart adherence with interaction efficiency and safety compliance signals.

  • B.2 Two-Stage Simulator Architecture: The simulator uses a two-stage pipeline that decouples safety control from behavior generation to prevent interaction-logic bypass.This design prevents the agent from bypassing the interaction logic, or “cheating.”
  • B.2 Two-Stage Simulator Architecture: The episode provides the initial instruction, hidden intent, table schema, and interaction trajectory as simulator inputs.These inputs define the context for routing and response generation.
  • B.2.1 Gatekeeper (Router): Security-Oriented Filtering: The Router is a strict binary gate that requires agents to earn information through clarification instead of requesting privileged data.It mitigates shortcut extraction similarly to prompt-injection defense.
  • B.2.1 Gatekeeper (Router): Security-Oriented Filtering: The Router outputs REFUSE for code, SQL, exact schema, direct answers, sensitive information, or indirect implementation extraction, and ANSWER otherwise.ANSWER permits clarification, verification, submissions, errors, and trap-triggered feedback; the output is restricted to one word.
  • B.2.2 User Generator: Grounded Interaction: The user generator models a goal-driven agent whose responses remain non-technical, grounded in fact_source, natural, and consistent with the trajectory.Responses contain 1–3 sentences and must not hallucinate or expose code, SQL, or exact technical column names.
  • B.2.2 User Generator: Grounded Interaction: Generator behavior prioritizes refusal, clarification, correction, and confirmation according to the agent’s request and the true intent.Clarifications reveal only minimal missing information, while corrections preserve the fact source’s complaint without inventing new errors.
  • B.3 Evaluation Signals: Evaluation treats final-chart adherence to hidden constraints as primary success and additionally measures average turns, clarification frequency, and REFUSE-trigger rate.The REFUSE-trigger rate operationalizes Safety Compliance and is interpreted as a measure of operational honesty.

C Evaluation Methods Details … C.5 DV-Inter

The evaluation framework combines native spreadsheet checks, chart-repair similarity, dashboard rubrics, reference-preserving evolution criteria, and interaction-aware assessment. It emphasizes operational correctness, data fidelity, visual quality, and alignment with user intent across DV-World tasks.

  • C.1 DVSheet-Crea: DVSheet-Crea first applies Existence, Spatial, and Dynamic Reference Gates before semantic scoring to ensure charts exist, avoid data obstruction, and bind series to cell ranges.Table Coverage uses Name-First, Type-Second alignment, while values are compared with zero tolerance and missing values excluded.
  • C.1 DVSheet-Crea: DVSheet-Crea semantic scoring covers Reliability, Appropriateness, and Aesthetics, assessing data grounding, analytical fit, readability, layout, and enterprise design compliance.Rubric criteria receive direct binary scores, and the final score sums scores across requirements.
  • C.2 DVSheet-Fix: DVSheet-Fix parses native Excel ChartSpec objects through xlwings and scores repairs with Sim(C, G) = 0.2Stype + 0.6Sseries + 0.15Saxis + 0.05Stitle.Sseries includes name, category, and numerical-value alignment.
  • C.2 DVSheet-Fix: DVSheet-Fix reports strict Success Rate, marking a case successful only when every must-fix attribute reaches similarity threshold τ ≥0.95.The broken-gated protocol identifies required repairs by comparing the broken file with the gold standard.
  • C.3 DVSheet-Dash: DVSheet-Dash evaluates Insightfulness, Accuracy, Professionalism, and Aesthetics across analytical synthesis, data bindings, spatial organization, and visual quality.Dashboard rubric scores are computed by summing direct binary criterion scores.
  • C.4 DV-Evol: DV-Evol scores Integrity, Consistency, and Aesthetics, requiring complete data migration, preserved visualization semantics, and professional output across programming languages.Its rubric compares candidate and ground-truth charts using 1-point criteria for data integrity, style imitation, layout, and aesthetics.
  • C.4 DV-Evol: DV-Evol’s rubric awards 1 point per satisfied criterion and defines a Total Max Score of 16.0 across trend, completeness, labels, style, layout, and balance.Style checks include exact or highly similar colors, chart elements, markers, line styles, and gridlines.
  • C.5 DV-Inter: DV-Inter introduces Interaction Success Rate using clarification requests, inappropriate refusals, and successful turns to reward proactive, efficient, non-leaking dialogue.Its rubric separately evaluates interaction, final-artifact accuracy, and aesthetics through trajectory, code, and chart inspection with binary criterion scores.

D Experimental Details … D.4 Human Evaluation

The experimental protocol evaluates visualization agents in native, sandboxed environments across spreadsheet creation, repair, dashboards, visualization evolution, and interactive clarification. It also defines a structured airport-elevation rubric and establishes a human reference through independently cross-scored augmented workflows.

  • D Experimental Details: The airport-elevation task awards 22 points for p90-based airport identification, a dual-view Top-10-country dashboard, and user-led visual refinement dialogue.The rubric requires identifying high-elevation airports using a p90 threshold, generating two coordinated views, and maintaining dialogue discipline.
  • D.1 Experimental Setup: All models use top-p = 1.0, a 120-turn interaction horizon, containerized Python 3.11, native visualization frameworks, and four trials per task.Supported frameworks include D3.js, Apache ECharts, and Vega-Lite.
  • D.2 Agent Details: SheetCopilot serves as the DV-Sheet baseline by translating natural-language instructions into structured spreadsheet actions through an iterative Observe–Propose–Revise–Act loop.Its actions cover sheet manipulation, cell editing, formatting, and chart construction.
  • D.2 Agent Details: OpenHands provides the DV-Evol baseline through CodeAct in sandboxed environments supporting 120 interaction rounds, Chinese and English instructions, and three tool suites.Runs terminate after three consecutive repetitions of the same action or a 120-second single-action timeout.
  • D.2 Agent Details: DV-World-Agent is a unified ReAct-based baseline running in sandboxed Python 3.11, orchestrating specialized tools for data manipulation, multimodal perception, and proactive interaction.It supports up to 120 tool-interaction rounds per task across benchmark domains.
  • D.3 Tasks System Prompt: The task prompts enforce domain-specific execution: native editable Excel objects for creation and dashboards, in-place repair, reference-style adaptation with final-table saving, and clarification before ambiguous plotting.The prompts distinguish new-sheet construction for creation and dashboards from existing-sheet modification for repair, while DV-Evol requires reference-image inspection and DV-Inter permits ask_user clarification.
  • D.4 Human Evaluation: 10 evaluators completed 50 sampled tasks using external resources, establishing an augmented human performance ceiling across all benchmark domains.Participants could use search engines and AI assistants to simulate professional workflows.
  • D.4 Human Evaluation: Human artifacts were anonymized and independently scored by two peers, with discrepancies exceeding 20% adjudicated by a third reviewer.Scoring followed the benchmark’s standard multidimensional rubrics.

D.5 Additional Experimental Results · E Additional Analysis · E.1 Evaluation Framework Analysis

Sensitivity analyses show that DV-World’s hybrid evaluation remains stable across alternative structural-visual weights and ISR mixing coefficients. Model rankings remain consistent, while adjacent ISR settings achieve Spearman correlations of at least 0.96.

  • D.5 Additional Experimental Results: The scoring mechanism was tested with correlation coefficients r ∈{0.4, 0.5, 0.6} on DV-Evol and DVSheet-Crea.The coefficient controls the relative weighting between structural alignment and visual fidelity.
  • D.5 Additional Experimental Results: DV-Evol model rankings remain strictly consistent across alternative structural-versus-visual weighting settings.The analysis covers cross-framework chart migration and reports results under w = 0.5, 0.4, 0.6.
  • D.5 Additional Experimental Results: DVSheet-Crea scores fluctuate minimally for top-tier agents including Gemini-3-Pro and GPT-5.2 under varied correlation coefficients.DVSheet-Crea evaluates atomic chart generation from spreadsheets.
  • D.5 Additional Experimental Results: The ISR sensitivity analysis varies λ across 0.6, 0.5, and 0.4 while reporting total DV-Inter Scores and adjacent-setting Spearman correlations.This tests whether multiplicative aggregation is overly sensitive to minor interaction-completion variations.
  • D.5 Additional Experimental Results: ≥0.96 Spearman rank correlations across adjacent λ settings indicate robust model rankings under reasonable ISR mixing-weight variations.The result supports stable evaluations using the multiplicative scoring scheme Sfinal = Srubric · ISR.
  • E.1 Evaluation Framework Analysis: The multiplicative scoring scheme Sfinal = Srubric · ISR yields consistent evaluations across reasonable choices of the mixing coefficient.This conclusion follows from the consistently high adjacent-setting rank correlations.

E.1.1 User Simulator Analysis … H Error Analysis

The merged sections define DV-World’s simulator and judge validation, spreadsheet-native and cross-framework visualization workflows, and representative creation, repair, dashboard, evolution, and interaction tasks. They also document provenance limitations and the benchmark’s diagnostic scope across heterogeneous real-world scenarios.

  • E.1.1 User Simulator Analysis: Nine LLM user simulators form three behavioral profiles, with GPT-5.2 and GPT-5-mini as Ideal Mentors whose Score Lifts exceed 21%.The profiles are distinguished using Final Score, Score Lift, Ask Rate, Clarity, and Correlation.
  • E.1.1 User Simulator Analysis: GPT-5-mini achieves 0.86 correlation and 88.67% Faithfulness, while removing Reaction Rules or Stage 1 Filtering significantly degrades both measures.The ablation effects are reported as significant at p < 0.05.
  • E.1.2 Human–LLM Judge Analysis: DV-World validates LLM judges at item, case, and model levels using ordinal agreement, ICC(A,1), and Kendall’s τb to assess rubric fidelity, score reliability, and leaderboard consistency.These granularities correspond to rubric items, aggregated task scores, and final agent rankings.
  • F Visualization Techniques Overview; F.1 DV-Sheet: Traceable In-Sheet Visualization; Core Concept: Auditable Provenance: Spreadsheet-native charts preserve auditable, live cell-range provenance through deterministic table materialization, native chart creation, reference binding, and in-sheet embedding.The chart remains editable and updates when referenced worksheet cells change.
  • System Properties & Limitations: The spreadsheet approach provides inspectable provenance and deterministic reproducibility but preserves only range-level lineage, does not guarantee aggregation correctness, and may break after cell edits.These limitations distinguish spreadsheet auditability from guaranteed semantic correctness.
  • F.2 DV-Evol: Visualization Framework Diversity: DV-Evol evaluates agents across five frameworks—Python, Apache ECharts, Vega-Lite, D3.js, and Plotly.js—whose evolution trajectories require framework-specific idioms.The examples span imperative plotting, structured options, declarative grammar, direct DOM manipulation, and interactive chart APIs.
  • F.3 DV-Inter: Python-Based Visualization: The Python-based DV-Inter pipeline deterministically ingests and normalizes CSV/XLSX data, renders primarily with matplotlib’s OO API, and serializes outputs as PDF/SVG and PNG.Centralized rcParams and fixed random seeds support reproducibility, while programmatic auditability replaces spreadsheet cell-based lineage.
  • G Examples; G.1 DVSheet-Crea; G.2 DVSheet-Fix; G.3 DVSheet-Dash; G.4 DV-Evol; G.5 DV-Inter; H Error Analysis: The benchmark examples cover Excel chart creation, diagnostic repair, interactive dashboards, percentage-based chart evolution, and clarification-driven multi-view visualization.Tasks include age-binned bubble charts, profit-trend repair, NYC housing analysis, preserved dual-axis encodings, and airport elevation ambiguity resolution.

H.1 DV-Sheet Error Analysis

DV-Sheet agents primarily fail on quantitative fidelity, while also violating visualization semantics, damaging source integrity during repairs, and providing insufficient analytical depth in dashboards. Fix tasks show higher average Data Accuracy errors than Create tasks, indicating that debugging remains a major challenge.

  • DVSheet-Crea: 50.74% average Data Accuracy errors dominate Create tasks, while chart structure errors remain comparatively low at 10.77%.The analysis identifies quantitative fidelity as the main bottleneck, with instruction adherence secondary and high-level insight generation less problematic.
  • DVSheet-Dash: 45.71% average Visual Design errors dominate Dashboard tasks, driven partly by wrong chart-type selections, while Insight & Resolution averages 6.92%.Cross-view Data Consistency also remains substantial in multi-plot dashboards.
  • Data grounding: Agents frequently omit reference values or mishandle derived quantities, producing numerically invalid visualizations through missing baselines and incorrect aggregation logic.Examples include omitting an initial population anchor and summing absolute increases and decreases instead of computing net changes.
  • Visualization semantics: Agents produce technically rendered but semantically unusable charts through text overlap, redundant labels, and misuse of categorical and other visual encodings.These convention violations reduce readability and distort the intended meaning of visual elements.
  • DVSheet-Dash: Dashboard failures often present raw volumes without benchmarks, omit actionable KPIs or analytical dimensions, and use unsuitable aggregations for business questions.The central limitation shifts from syntactic correctness toward analytical adequacy and business-oriented reasoning.
  • DVSheet-Fix: 69.31% average Data Accuracy errors in Fix tasks exceed 50.74% in Create tasks, while repairs can truncate datasets, duplicate elements, or corrupt axis labels.The comparison suggests existing erroneous frameworks may increase cognitive load and expose the need for stronger debugging and auditing.

H.2 DV-Evol Error Analysis

DV-Evol errors are dominated by layout/readability and data-consistency failures across visualization languages, with recurring breakdowns in visual mapping, transformations, scaling, and backend rendering. Model tendencies also differ: GPT-5.2 often preserves structure while omitting layers or references, whereas Gemini-3-Pro more frequently exhibits other realization weaknesses.

  • Aggregate error distribution: Layout & Readability errors average 42.43%, followed by Data Consistency at 31.98% and Visual Style at 25.59%.This imbalance is consistent across all evaluated models.
  • Language-specific patterns: Layout & Readability errors remain high across all five languages, exceeding 48% for GPT-5.2 and 51% for Gemini-3-Flash in Vega-Lite.Data Consistency peaks in ECharts at 55.15% for Gemini-3-Flash and 47.63% for GPT-5.1.
  • Cross-backend visual mapping: Agents often preserve high-level chart intent but fail target-backend constraints, producing overlapped ridges and misaligned axes in ridgeline visualizations.Failures include incorrect stacking order, spacing, proportionality, and axis alignment.
  • Data-consistency bottlenecks: Incorrect transformations and scale choices cause Data Consistency violations despite nominally correct chart types.Examples include plotting normalized 0–100% scores instead of raw median scales and altering the country set or order.
  • Rendering omissions: Backend-specific omissions can make visualizations degenerate, such as correlation heatmaps collapsing into diagonal line plots with missing matrix cells, colormaps, colorbars, and annotations.These failures indicate that agents did not instantiate the required mark or trace structure.
  • Model-dependent tendencies: GPT-5.2 often approximates the target structure but omits crucial layers or truncates global references across ECharts and Vega-Lite outputs.Examples include missing violin layers, truncated percentile or trend lines, truncated trendlines, and insufficient points.

H.3 DV-Inter Error Analysis

DV-Inter failures are dominated by cognitive–execution and intent-retention problems, while model families exhibit distinct secondary error patterns. Detailed cases show that ambiguous requirements, brittle data parsing, semantic visual mismatches, and tool instability can each derail interactive visualization workflows.

  • Aggregate error patterns: 38.44% average CEG makes cognitive–execution failure the primary DV-Inter error, with Intent Drift reaching 45.38% for Gemini-2.5-Pro and 41.74% for GPT-4.1.These results identify multi-turn constraint retention as a key bottleneck despite strong code generation.
  • Failure mechanisms: Inquiry Deficit causes agents to make unilateral design choices under ambiguity, producing visual and logical misalignment when clarification is absent.One example substitutes a dual-axis bar–line chart for a comparative decomposition of volume versus value across tiers.
  • Failure mechanisms: Brittle intermediate reasoning can convert correct intent into empty outputs, as an incorrect year-extraction heuristic assigns one timestamp to every row and removes all data during filtering.This is a representative Cognitive–Execution Gap failure.
  • Failure mechanisms: Visual Design Mismatch reduces complex statistical distribution requirements to semantically inadequate charts, while Technical Collapse prevents visualization when runtime errors and environment mismanagement persist.Examples omit medians, IQRs, and categorical encoding, or produce no output after failed filtering and insufficient debugging.
Loading 2604.25914v1…