DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios

• Existing benchmarks miss real-world complexity through environmental decoupling, creation-only evaluation, and assumptions that user intent is fully specified.These gaps leave native spreadsheet workflows, adaptation to new data and requirements, and clarification dialogue under-tested.Source: §1 · p. 1
• DV-World evaluates the full visualization lifecycle through native spreadsheet manipulation, cross-modal logic evolution, and proactive iterative interaction.These scenarios are named DV-Sheet, DV-Evolution, and DV-Interact.Source: §1 · p. 2, Figure 1 · p. 2
• DV-Sheet tests native chart creation, repair of broken visualization bindings, and dashboard synthesis using spreadsheet object models.Its tasks assess direct cell references, diagnostic reasoning, and spatial planning for professional analytical views.Source: §1 · p. 2
• DV-Evol tests whether agents can convert reference images and datasets into executable code, then update and migrate visualization logic across five paradigms.It goes beyond static reproduction to test generalization across languages and frameworks.Source: §1 · p. 2
• DV-Inter evaluates whether agents clarify underspecified requirements or data ambiguities and maintain state consistency across turns to match users’ analytical goals.Source: §1 · p. 2
• The hybrid evaluation combines Table-value Alignment for data fidelity with expert-rubric-guided hierarchical MLLM judging of visual semantics and aesthetics.The paper reports that this multidimensional system aligns closely with human judgment.Source: §1 · p. 2
• DV-Sheet scores peak at 40.48, while DV-Evol and DV-Inter are capped at 51.44 and 40.43, respectively, exposing challenges in native manipulation, logic migration, and iterative alignment.The paper argues that progress requires lifecycle management integrating environmental mastery, semantic portability, and proactive intent alignment.Source: §1 · pp. 2–3

2.1 Task Definition

• The benchmark evaluates three distinct workflows: native spreadsheet editing, adapting reference visuals into code and tables, and clarifying ambiguous visualization requests.DV-Sheet covers chart creation with dynamic ranges, chart repair, and dashboard composition; DV-Evol requires reverse-engineering visual semantics across frameworks; DV-Inter uses a simulator to align outputs with latent user objectives without leaking implementation details.Source: §2.1 · p. 3

2.2 Evaluation Metrics

• Evaluation combines expert rubrics with quantitative checks, using task-specific measures for visual quality, data fidelity, operational correctness, and interaction success.Creation and evolution scores combine rubric ratings with Table Coverage; Fix uses a strict success criterion on must-fix chart attributes, while Dashboard uses rubric scores and Inter multiplies rubric scores by Interaction Success Rate.Source: §2.2 · p. 4

2.3 Annotation Pipeline

• Eighteen visualization specialists built the benchmark from authentic scenarios, adapting source data and constructing task-specific prompts, cases, interaction trajectories, and evaluation standards.They curated over 800 community problem threads, retained workbook structure, perturbed values while preserving distributions, and anonymized metadata. They also created gold tables, gold plots, healthy-file ground truth, and consensus-checked rubrics tested against outputs from five models.Source: §2.3 · pp. 4–5

2.4 Dataset Statistics

• Dataset statistics characterize DV-Sheet’s native grounding, DV-Evol’s cross-framework updates, and DV-Inter’s ambiguity resolution.DV-Sheet workbooks average 36.53 columns and 11,583.36 rows, with 74.5% noisy data in Fix tasks and 40% Reliability weighting. DV-Evol spans 31 chart types and 52,584.58 rows, with Integrity and Consistency weights of 25% and 37%; DV-Inter averages 3.17 ambiguities and 734.54 user tokens per task, with Interaction and Accuracy weights of 24% and 41%.Source: §2.4 · p. 5

3.1 Experimental Setup

• The experiments evaluate models and domain-specific baselines using a unified ReAct agent, human evaluations, and metrics averaged across four independent runs.The setup uses GPT-5-Mini as the DV-Inter user simulator and Gemini-2.5-Flash as the MLLM judge; DVSheet-Crea and DV-Evol combine rubric scores with table-coverage signals at equal weight, and sensitivity analysis found model rankings stable across weight configurations.Source: §3.1 · p. 5

3.2 Main Results

• 40.48% is Gemini-3-Pro’s peak DV-Sheet score; GPT-5.2 and DeepSeek-V3.2 remain below 38.00%, with Fix and Dash especially weak.These results indicate a performance gap from human benchmarks and persistent difficulties with data-to-chart binding, diagnostic repair, and multi-view planning.Source: §3.2 · pp. 5–6, Table 3 · p. 6
• 51.44% is Gemini-3-Pro’s highest DV-Evol overall score, ahead of Gemini-3-Flash (49.46%) and GPT-4.1 (44.67%), yet still far below human performance.Top models perform better on Python and Vega-Lite than on D3.js and Plotly.js, and face challenges preserving semantics and design across frameworks.Source: Table 4 · p. 6, §3.2 · p. 6
• 40.43% is Grok-4’s peak DV-Inter score, while most top-tier models remain below 38.00%; interactive alignment remains the primary bottleneck.Models often fail to identify critical ambiguities or ask precise clarifying questions, leaving a gap between underspecified intent and visualization requirements.Source: Table 5 · p. 6, §3.2 · pp. 6–7

3.3 Analysis of Native Grounding in Spreadsheet

• Spreadsheet agents struggle with precise fixes and large dashboards, while better table coverage correlates with stronger visual aesthetics in creation tasks.Data accuracy accounts for over 50% of creation errors, 69% of fix errors, and nearly 46% of dashboard errors; filtering logic is easier to fix than axis scaling or encoding errors.Source: §3.3 · p. 7, Figure 5 · p. 7

3.4 Analysis of cross-paradigm evolutionary adaptation

• Performance declines as visualization code length grows, and removing visual feedback universally lowers success, with a 7.69% peak drop for Gemini-3-Pro on D3.js.Error profiles vary by framework: Gemini-3-Flash has a 40.96% visual style error rate; Gemini 3 Pro has 45.20% data consistency issues in ECharts; and Gemini-3-Flash has 51.80% layout errors in Vega-Lite.Source: §3.4 · pp. 7–8, Figure 6 · p. 8, Table 6 · p. 8

3.5 Analysis of interactive agency and alignment

• DV-Inter’s primary error bottlenecks are the Cognitive Execution Gap and Interactive Avoidance, reflected in high error rates for GPT-5.2 and GLM-4.7.The Execution Gap reaches 60.07% for GPT-5.2, while Interactive Avoidance reaches 59.47% for GLM-4.7; the latter is linked to overconfidence and simplified task downgrades.Source: §3.5 · p. 8

4.1 User Simulator Analysis

• GPT-5-mini provides the best balance of interactive-agent capability and interaction cost, with high instructional fidelity and stable, consistent evaluation.The GPT-5 series leads, and GPT-5-mini is described as the efficiency frontier and the most stable environment for measuring the agent’s capability ceiling.Source: §4.1 · p. 8, Figure 8 · p. 9
• 88.67% Faithfulness and a 0.86 Pearson correlation (p < 0.04) show that GPT-5-mini simulator ratings align strongly with human judgment.Removing Reaction Rules or Stage 1 Filtering significantly reduces alignment, making interactions less reliable and potentially insecure.Source: Table 7 · p. 9, §4.1 · p. 9

4.2 Validation of MLLM-Judge Method

• Human annotators achieve ICC(A,1)=0.932 and weighted κ = 0.903, while Gemini-2.5-Flash has the strongest judge alignment (κ = 0.821, ICC(A,1)=0.850).The pooled evaluation spans 210 tasks across four MLLM-judged settings; human agreement indicates the rubric supports consistent expert judgment and provides a practical upper bound for automated judges.Source: §4.2 · p. 9
• Across 210 shared tasks, judge-family rankings are largely preserved despite score shifts, and Gemini-2.5-Flash is standardized as the primary judge.The choice is based on its strongest item- and case-level alignment with human experts.Source: §4.2 · p. 9

Agentic benchmarks

• Agentic benchmarks increasingly evaluate execution-grounded interaction, long-horizon tool use, and adaptation to feedback.Representative suites address software engineering, requirement-driven code generation, realistic SQL/data workflows, and interactive intent clarification and recovery.Source: §5 · pp. 9–10
• These benchmarks remain largely domain-general and do not explicitly model professional data-visualization workflows, where underspecified intent and design choices are central.Source: §5 · p. 10

Benchmarks for data visualization

• Data-visualization benchmarks mainly assess single-shot specification or plotting-code translation, chart understanding/editing, chart-to-code generation, or spreadsheet manipulation.Few evaluate spreadsheet-native chart/dashboard creation together with interactive intent alignment and diagnostic repair, motivating DV-World.Source: §5 · p. 10

6 Conclusion

• Strong models including Gemini-3-Pro and GPT-5.2 still struggle with error correction, faithful data binding, and consistent evolution.DV-World combines rubric-based judgments, quantitative checks, and interaction success metrics to support diagnosis and progress tracking across end-to-end visualization workflows.Source: §6 · p. 10

A.1 Data Sources

• The benchmark’s data sources combine community Q&A and open datasets to capture real-world spreadsheet questions, answers, and tables.Sources include Kaggle and Excel-focused communities and learning sites, including Chandoo.org, Excelguru, ExcelForum, and MrExcel.Source: §A.1 · p. 17

A.2.1 Data Annotation: DVSheet-Crea

• DVSheet-Crea’s 80 cases cover direct, extraction-required, and multi-sheet chart tasks, using native Excel ground truths and binary fidelity, logic, and aesthetics checks.Instructions span explicit constraints and abstract intent; generated chart objects preserve data bindings and series definitions for auditability and reproducibility.Source: §A.2.1 · pp. 17–18

A.2.2 Data Annotation: DVSheet-Fix

• DV-Sheet Fix’s 50 cases test agents’ ability to diagnose and repair defective spreadsheets through deterministic repairs verified by chart-state and side-effect checks.Defects cover data binding, axis and scaling, chart-type mismatches, and visual clutter; controlled inverse injection creates each repair task from a verified chart.Source: §A.2.2 · p. 18

A.2.3 Data Annotation: DVSheet-Dash

• DVSheet-Dash’s 30 tasks assess planning and implementation of interactive dashboards, using expert-refined references and rubrics for insight, consistency, visual quality, and professionalism.Instructions span executive directives, technical specifications, and ad-hoc queries, while dashboards include KPI cards and interactive elements such as Slicers.Source: §A.2.3 · pp. 18–19

A.2.4 Data Annotation: DV-Evol

• DV-Evol’s 80 tasks test modifying complex visualizations for new data or requirements through isolated subplots, data and schema perturbations, taxonomy-based edits, and executable no-hardcode verification.Perturbations include distribution shifts and semantic column renaming; edit operations cover data, visual encodings, layers, and styles.Source: §A.2.4 · p. 19

A.2.5 Data Annotation:DV-Inter

• DV-Inter tests multi-turn clarification of vague requests using schema-grounded ambiguities, a simulator with hidden intent and reaction rules, executable ground truths, and process, correctness, and presentation rubrics.Ambiguities include metric definitions, temporal scope, and aggregation; reverse polishing aligns the simulator’s responses with practical adjustments in the expert solution.Source: §A.2.5 · p. 20

A.3.1 Taxonomy of fix types for the DVSheet-Fix task

• The repair taxonomy specifies operations such as geometry reselection, occlusion resolution, range control, orientation correction, data cleaning, aggregation repair, context annotation, and legend or layout improvement.These subtypes address intended relationships, comparability, erroneous data patterns, and interpretation cost.Source: §A.3.1 · p. 20
• DVSheet-Fix groups repairs into Data Transformation & Integrity (30%), Visual Encoding & Mapping (28%), Coordinate System & Scaling (28%), and Presentation & Readability (14%).The taxonomy treats defects as actionable edits across the visualization pipeline, including upstream data inconsistencies and mis-specified encodings or scales.Source: §A.3.1 · p. 20, Table 10 · p. 21
• The DVSheet-Fix taxonomy supports fine-grained error analysis and category-level reporting of repair performance.Source: §A.3.1 · p. 22
• 44.69% of DV-Evol tasks concern Data Processing and Transformation, 29.41% Data Encoding and Mapping, and 20.30% Graphics and Visual Optimization.Data Cleaning and Aggregation accounts for 27.5%, Normalization and Standardization 8.97%, and Sorting and Mathematical Transformation 8.22%.Source: Figure 9 · p. 22, §A.3.1 · p. 22
• DV-Evol also covers Layout and Coding (16.28%), Geometric Attributes (13.13%), Style and Color Matching (12.27%), Layer Superposition and Fitting (8.03%), and Chart Structure and Layout Adjustment (5.60%).The task distribution spans data integrity, visual aesthetics, and professional presentation, including chart conversion and subplot tuning.Source: Figure 9 · p. 22, §A.3.1 · p. 22

A.3.2 The ambiguity-point taxonomy for the DV-Inter

• DV-Inter evaluates whether agents identify missing intent, ask minimal informative clarification questions, and iteratively refine a chart specification with a simulator that knows the intended visualization.Its ambiguity taxonomy includes Semantic Boundary cases such as unspecified Top-K values, cutoffs, temporal windows, and concept-to-field mappings.Source: §A.3.2 · p. 22, Table 11 · p. 23
• Computation Logic ambiguities include unclear operators, aggregation meaning, binning, grouping, derived-metric formulas, or data-cleaning assumptions; Visual Encoding ambiguities include chart type, normalization, stacking, and denoising choices.The categories structure analysis of question-asking and whether dialogue efficiently converges on the intended visualization.Source: §A.3.2 · p. 24

B.1 Problem Setting

• The interact environment pairs a plotting agent with a constrained user simulator, requiring strategic dialogue to recover the user’s hidden intent z before generating the target chart.Queries may be underspecified, for example through ambiguous mappings or filters.Source: §B.1 · p. 24

B.2 Two-Stage Simulator Architecture

• The simulator decouples safety control from behavior generation in a two-stage pipeline that uses the instruction, hidden intent, table schema, and dialogue trajectory.Source: §B.2 · p. 24

B.2.1 Gatekeeper (Router): Security-Oriented Filtering

• The binary router refuses requests for code, internal schema details, or direct answers to prevent shortcut extraction and privileged-data access.Source: §B.2.1 · pp. 24–25
• The router answers legitimate clarification and verification questions, and lets the simulator provide feedback when submissions are wrong or trigger a trap.Source: §B.2.1 · p. 25

B.2.2 User Generator: Grounded Interaction

• The user generator models a goal-driven user and is constrained to respond non-technically, naturally, and without hallucinating beyond its fact source.Source: §B.2.2 · pp. 25–26
• Its response logic prioritizes refusal for technical-detail requests, then clarifies intent, corrects deviations, or confirms progress using grounded information.Source: §B.2.2 · pp. 26–27

B.3 Evaluation Signals

• An episode succeeds when the final chart adheres to hidden-intent constraints; evaluation also tracks turns per episode, clarification frequency, and the REFUSE-trigger rate.The REFUSE-trigger rate is used to assess operational honesty.Source: §B.3 · p. 27

C.1 DVSheet-Crea

• DVSheet-Crea uses three native-chart gates and Table Coverage (ST C) to check operational validity and exact alignment with gold-standard data.The Existence, Spatial, and Dynamic Reference Gates check chart presence, obstruction, and live cell-range bindings; alignment uses normalized names, inferred types, and multiset matching with zero-tolerance string comparisons.Source: §C.1 · p. 28
• DVSheet-Crea rubric scoring covers reliability, appropriateness, and aesthetics, assessing data grounding, fit to user intent, and professional visual quality.The rubric decomposes tasks into requirements and sub-standards, with binary criterion scores.Source: §C.1 · pp. 28–29

C.2 DVSheet-Fix

• DVSheet-Fix parses native chart specifications and scores repair success only when every required attribute reaches at least 0.95 similarity to the gold chart.Excel COM via xlwings extracts chart types, axis scales, and SERIES formulas; similarity weights type 0.2, series 0.6, axes 0.15, and title 0.05.Source: §C.2 · pp. 30–31

C.3 DVSheet-Dash

• DVSheet-Dash rubrics assess insightfulness, accuracy, professionalism, and aesthetics across analytics, artifact correctness, layout, and visual quality.Criteria receive binary 0/1 scores that are summed into requirement and total scores.Source: §C.3 · pp. 31–32

C.4 DV-Evol

• DV-Evol rubrics evaluate integrity, consistency, and aesthetics, checking data and logic migration, preservation of reference design semantics, and professional readability.Candidate charts are compared with ground-truth charts, with each criterion scored 1 if fully satisfied and 0 if failed.Source: §C.4 · pp. 33–34
• DV-Evolution evaluates data integrity, style imitation, and layout aesthetics, including trend and label fidelity, visual styling, readability, alignment, and chart proportions.Its rubric sets a maximum score of 16, calculated by summing individual criteria.Source: §C.4 · pp. 34–35

C.5 DV-Inter

• DV-Inter’s Interaction Success Rate uses inappropriate refusals, successful turns, and user inquiries to reward proactive, quality dialogue while discouraging redundancy and information leakage.The simulator records refusals for inappropriate requests, such as seeking implementation code; valid inquiries proceed to response generation, where successful turns are counted.Source: §C.5 · pp. 35–36
• DV-Inter rubrics assess whether agents clarify underspecified prompts, produce visualizations aligned with the user’s analytical objectives, and meet professional standards for visual quality and clarity.Evaluation checks dialogue records, the assistant’s final Python code, and the resulting chart.Source: §C.5 · p. 36
• DV-Inter assigns binary 0/1 scores to individual rubric criteria, sums them into requirement scores, and totals those scores for the final result.The rubric decomposes tasks into requirements and sub-standards with explicit checkpoints and point allocations.Source: §C.5 · pp. 36–37

D Experimental Details

• The Airport Elevation rubric (22 points) assesses p90-based airport identification, a dual-view Top-10-country dashboard, and dialogue discipline during user-led refinements.The task also requires ranking countries by high-elevation counts with maximum elevation as a tie-breaker.Source: Table 14 · p. 38, D Experimental Details · p. 38

D.1 Experimental Setup

• All models use top-p = 1.0, a 120-turn interaction limit, containerized Python 3.11 with visualization frameworks, and four trials per task.The stated limit balances support for complex reasoning and multi-stage evolution with computational efficiency.Source: §D.1 · p. 38

D.2 Agent Details

• SheetCopilot provides a spreadsheet-agent baseline, OpenHands CodeAct (Wang et al., 2024) a DV-Evol baseline, and DV-World-Agent a unified ReAct baseline across all domains.The setups enable spreadsheet action refinement, sandboxed code interaction, and native visualization execution with proactive requirement alignment, respectively.Source: §D.2 · pp. 38–39

D.3 Tasks System Prompt

• Spreadsheet prompts distinguish native-chart creation, in-place chart repair, and dashboard construction on a new result sheet while imposing headless, tool-only execution.Creation requires editable Excel chart objects; repair preserves the existing sheet and chart placement where possible; dashboards combine KPI cards, tables, and charts without modifying raw data.Source: §D.3 · pp. 39–42
• DV-Evol prompts require reference-style inspection, adaptation to new data and requirements, and saved specs and outputs; DV-Inter prompts require Python charts and user clarification rather than guessed intent.DV-Evol saves the filtered plotting data as result.csv and renders result.png; DV-Inter uses ask_user for ambiguities and saves charts as image files without displaying them.Source: §D.3 · pp. 43–45
• Before a tool call, agents must give a one-sentence English plan and must not mix narrative text with tool calls.Source: §D.3 · p. 45

D.4 Human Evaluation

• 10 evaluators completed 50 sampled tasks across all domains and could use external resources, including search engines and AI assistants.The setup was intended to reflect maximum achievable proficiency in solving benchmark challenges.Source: §D.4 · p. 45
• Each anonymized artifact was independently scored by two participants, with discrepancies exceeding 20% resolved by a third reviewer.Reviewers used the standard multi-dimensional rubrics to provide a reference for comparison with agent baselines.Source: §D.4 · p. 45

D.5 Additional Experimental Results

• The ISR sensitivity analysis tests whether minor variations in interaction completion affect multiplicative score aggregation.It varies λ in Sfinal = Srubric · ISR and compares model rankings across settings.Source: §D.5 · p. 45
• The stable rankings support the reliability of Sfinal = Srubric · ISR across reasonable choices of λ.The paper concludes that the multiplicative scoring scheme yields consistent evaluations under these variations.Source: §D.5 · p. 45
• Score fluctuations for top-tier DVSheet-Crea agents Gemini-3-Pro and GPT-5.2 remain within ±1.6%.The analysis varies the correlation coefficient for structural and visual alignment.Source: §D.5 · p. 45, Table 17 · p. 46
• Across DV-Evol weight settings, the relative rankings of all evaluated models remain consistent.The analysis varies the correlation coefficient across 0.4, 0.5, and 0.6 for structural alignment versus visual fidelity.Source: §D.5 · p. 45, Table 18 · p. 46
• Spearman rank correlations remain ≥0.96 across adjacent ISR mixing-weight settings.Table 19 reports total DV-Inter Scores at λ = 0.6, 0.5, and 0.4.Source: §D.5 · p. 45, Table 19 · p. 46

E.1.1 User Simulator Analysis

• Across nine user simulators, three profiles emerge: mentors, standard users, and hard clients, with mentor Score Lifts exceeding 21%.The profiles vary in cooperativeness, clarity, and difficulty; the text identifies GPT-5.2 and GPT-5-mini as mentors, o4-mini and Gemini-3-Pro as standard users, and Gemini-2.5-Pro and GPT-4.1 as hard clients. Figures 11 and 12 depict the capability matrix and profile landscape.Source: §E.1.1 · p. 47, Figure 11 · p. 47, Figure 12 · p. 47
• GPT-5-mini achieved 0.86 correlation and 88.67% Faithfulness; removing Reaction Rules or Stage 1 Filtering significantly reduced both measures (p < 0.05).The audit covered 150 interaction trajectories across reliability and safety dimensions; Faithfulness is the alignment rate between simulator outputs and human-grounded intent.Source: §E.1.1 · p. 47

E.1.2 Human–LLM Judge Analysis

• Judge validation spans ordinal item agreement, ICC(A,1) for absolute agreement on aggregated scores, and Kendall’s τb for human–judge leaderboard consistency.The three granularities align with rubric items, per-example scores, and final agent rankings, respectively.Source: §E.1.2 · p. 48

Core Concept: Auditable Provenance

• Native spreadsheet charts retain auditable, live cell-range bindings, while rasterized figures sever data links; provenance remains range-level and edits can break bindings.Explicit bindings are inspectable through the spreadsheet UI’s “Select Data” interface, but this does not guarantee aggregation correctness.Source: Core Concept: Auditable Provenance · p. 48, System Properties & Limitations · p. 49

F.2 DV-Evol: Visualization Framework Diversity

• DV-Evol covers five frameworks—Python, ECharts, Vega-Lite, D3.js, and Plotly.js—and illustrates how agents adapt visualizations using framework-specific idioms.The case study evolves a shared trend-and-uncertainty task through porting and refinement, with examples ranging from Python smoothing to Plotly range sliders and hover-coordinated legends.Source: §F.2 · pp. 49, 51, 52, Figure 15 · p. 51

F.3 DV-Inter: Python-Based Visualization

• The Python visualization pipeline uses canonical pandas data, primarily matplotlib’s object-oriented API, and reproducible style and random-seed controls.It exports PDF/SVG and PNG; unlike spreadsheet-native charts, it sacrifices interactive cell-based lineage for programmatic auditability.Source: §F.3 · p. 52

G.1 DVSheet-Crea; G.2 DVSheet-Fix; G.3 DVSheet-Dash

• Spreadsheet tasks cover chart creation from varied data layouts, repair of broken or misleading charts, and interactive dashboards integrating multiple views and controls.Examples include a binned bubble chart, diagnosing a flat Profit series, and a housing dashboard with borough and property-type comparisons, log-scale price distributions, transaction volumes, and data cleaning.Source: §G.1 · p. 52, §G.2 · p. 52, §G.3 · p. 53

G.4 DV-Evol

• The DV-Evol example converts a count-based dual-axis stacked bar chart to within-category percentages while preserving its structure, binning, and legend positions.The task also specifies score-range and line colors and a revised chart title.Source: §G.4 · pp. 53–54

G.5 DV-Inter

• The task tests agents’ ability to resolve ambiguities such as elevation thresholds or aesthetic preferences through multi-turn interaction and refine visual outputs.An illustrated scenario requires resolving elevation-threshold ambiguity before generating coordinated geographical visualizations.Source: §G.5 · p. 54
• The task asks agents to identify high-elevation airports and compare countries in a complex multi-view visualization, resolving threshold ambiguity through dialogue.Source: §G.5 · p. 54
• Required outputs include a latitude–longitude scatter plot sized by elevation, country-level elevation box plots, and IATA codes ranked by elevation.Source: §G.5 · p. 54

Error patterns distribution across DV-Sheet

• Data Accuracy errors average 69.31% in Fix versus 50.74% in Create, while Dashboards are dominated by Visual Design errors (45.71%) and have 6.92% Insight & Resolution errors.Fix tasks maintain structural integrity better than Create tasks, but the higher Data Accuracy error rate may reflect cognitive load or misdirected attention from erroneous baselines.Source: §H.1 · pp. 54, 55, 57, Table 21 · p. 56, Table 22 · p. 57

Systematic breakdowns in data grounding and numerical reasoning

• Create-task grounding failures include omitted baselines and incorrect aggregation, while ranking and dual-axis charts can plot wrong fields or distort values.Examples include omitting an initial population anchor or summing absolute increases and decreases instead of calculating net changes.Source: §H.1 · p. 55

Readability degradation and semantic misuse of visual conventions

• Create-task charts can be unreadable from overlapping text or redundant labels, and semantic misuse includes encoding decreases as upward bars.Agents may place categorical information in data labels rather than axes or use negative signs for demographic quantities.Source: §H.1 · p. 55

Destructive edits and loss of source integrity in fix tasks

• Fix-task agents sometimes truncate data, duplicate visuals, corrupt axis labels, or fail to normalize heterogeneous units, making targeted repairs destructive.The errors indicate a lack of locality and consequence awareness, with downstream effects left unvalidated.Source: §H.1 · p. 55

Insufficient analytical depth and business-oriented reasoning in dashboards

• Dashboard failures include raw volumes without benchmarks, missing actionable KPIs, inappropriate aggregation, unfiltered noise, and absent annotations or reference lines.These shortcomings can prevent dashboards from answering business questions or showing relevant correlations and trade-offs.Source: §H.1 · p. 55

Comparison of fix and create tasks

• Across tasks, agents can produce structurally plausible charts but often fail to validate fidelity to the data, task objective, and professional visualization conventions.The comparison suggests existing frameworks help preserve structural integrity in Fix, while erroneous baselines may contribute to its higher Data Accuracy error rate than Create.Source: §H.1 · pp. 55, 57

Error distribution

• 42.43% of DV-Evol errors are Layout & Readability issues, compared with 31.98% Data Consistency and 25.59% Visual Style; GPT-5.2 has the lowest rates in the first two categories.The reported GPT-5.2 rates are 39.31% for Layout & Readability and 29.12% for Data Consistency.Source: §H.2 · p. 57
• Layout & Readability errors are high across all five languages, especially Vega-Lite, while Data Consistency peaks in ECharts and Visual Style varies more by language and model.Examples include Vega-Lite Layout & Readability exceeding 48% for GPT-5.2 and 51% for Gemini-3-Flash, and ECharts Data Consistency reaching 55.15% for Gemini-3-Flash and 47.63% for GPT-5.1.Source: §H.2 · p. 68

Cross-backend visual mapping failures

• Agents often preserve chart intent but violate target-backend marks, layers, or structural constraints, producing overlapping ridges and misaligned legends or titles.Ridgeline outputs can also violate stacking order, spacing, proportionality, and axis alignment, making otherwise plausible plots difficult to interpret.Source: §H.2 · p. 68

Transformation and scale inconsistencies

• Wrong transformations and scale choices cause Data Consistency failures even when the chart type is correct, including incompatible normalization, log transforms, and truncated reference lines.Examples include using normalized scores instead of the required raw median scale, changing country order, and reference lines that fail to span the required domain.Source: §H.2 · p. 68

Backend-specific rendering omissions and chart-type degeneration

• Backend-specific omissions can remove essential components or degrade chart types, such as a Plotly correlation heatmap collapsing into a line plot or an ECharts violin layer disappearing.Other failures include missing matrix cells, colormap, colorbar, or annotations; legend mismatches; and axis-range drift that adds whitespace.Source: §H.2 · p. 68

Model-dependent error tendencies across five backends

• GPT-5.2 often approximates the target structure but omits layers or truncates references, whereas Gemini-3-Pro more often preserves distribution trends while producing unstable ridgeline layouts.GPT-5.2 examples include missing violin layers and truncated percentile or trend lines; Gemini-3-Pro examples show drift in labels, spacing, and proportionality.Source: §H.2 · p. 68

H.3 DV-Inter Error Analysis

• Interactive Avoidance occurs when agents execute despite ambiguity, as in choosing an arbitrary top-k filter for individual players instead of clarifying aggregation needs.The resulting chart has low semantic value and fails the rubric’s expectation of logical aggregation.Source: §H.3 · p. 79, Figure 40 · p. 81
• Inquiry Deficit leads agents to make unilateral design choices, exemplified by a dual-axis bar–line chart that misrepresents a requested volume-versus-value comparison across tiers.The example also sums quantities instead of counting identifiers, compounding visual and logical misalignment.Source: §H.3 · p. 79, Figure 41 · p. 81
• A brittle year-extraction heuristic assigns every row a constant timestamp, causing temporal filtering to discard the dataset and produce an empty visualization.This illustrates how a parsing mistake can cascade when agents lack robust checks between reasoning and execution.Source: §H.3 · p. 79, Figure 42 · p. 82
• Visual Design Mismatch reduces a requested distribution chart with statistical summaries to a bubble scatter plot, omitting medians, IQRs, and categorical color encoding.Legend occlusion and continuous color for categorical variables further impair readability.Source: §H.3 · p. 80, Figure 43 · p. 82
• Technical Collapse occurs when repeated runtime errors and environment mismanagement prevent completion, leaving no visualization after data-filtering exceptions.Insufficient debugging and state inspection can force premature task abandonment.Source: §H.3 · p. 80, Figure 44 · p. 83
• CEG was the leading DV-Inter error (38.44% average), while frontier models showed severe Intent Drift (Gemini-2.5-Pro 45.38%; GPT-4.1 41.74%).GLM-4.7 and Qwen3-235B-A22B showed high Speculative Logic (33.32% and 30.11%); GPT-5.2 concentrates on Heuristic Logic Approximation (24.74%). DeepSeek-V3.2 peaked in Visual Design Mismatch (13.61%) and Layout Conflict (16.00%), while Qwen3-8B showed Debugging Deadlock (33.40%).Source: §H.3 · pp. 68, 79, Table 28 · p. 80, Table 29 · p. 80