PaperlayerView exact revision on arXiv

Source-linked AI summary

Where LLMs Fail with Visualization DSLs

Chang Han, Andrew McNutt, Katherine Isaacs

arXiv:2610.01873v1cs.HCcs.CL

TL;DR

Visualization DSLs were designed around human authoring constraints, but what is easy for people may be difficult for models increasingly asked to author chart specifications. They compare 10 JSON-style visualization DSLs across 41 tasks and identify recurring failures across 1,230 trials (1,223 generated specifications) from three LLMs. Across 1,230 trials, the authors identify four recurring failures: encoding mismatch, layout arithmetic failure, missed renderer defaults, and failed workaround.

  • Problem

    Visualization DSLs were designed around human authoring constraints, but what is easy for people may be difficult for models increasingly asked to author chart specifications.

    Source: §1 · p. 1

  • Method

    They compare 10 JSON-style visualization DSLs across 41 tasks and identify recurring failures across 1,230 trials (1,223 generated specifications) from three LLMs.

    Source: §1 · p. 1

  • Results

    Across 1,230 trials, the authors identify four recurring failures: encoding mismatch, layout arithmetic failure, missed renderer defaults, and failed workaround.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors recommend explicit encoding fields, composition primitives that own layout budgeting, and visible renderer defaults and hidden contracts.

    Source: §1 · p. 1

    The study uses a single prompting style without repeated trials and does not claim to isolate DSL design as the sole cause; its proposals await broader validation.

    Source: §3 · p. 3

Abstract

from arXiv · show

As LLMs take up the role of authoring charts using visualization domain-specific languages (DSLs), the human constraints that shaped those languages may no longer apply, as what is easy for a person is not necessarily easy for a model. To understand how LLMs might work better with DSLs, we explore where and how they fail with current DSL designs. We evaluate 10 JSON-style visualization DSLs with 41 tasks across 3 LLMs, then assess the generated specifications with JSON and rendering checks, and qualitative coding of failed cases. Analyzing how this specification generation process fails, we identify four recurring failure patterns, link each to specific DSL features, and discuss design considerations for future DSL designs.

1 INTRODUCTION

As LLMs increasingly author chart specifications, human-oriented DSL design assumptions may no longer fit model authors. This study examines failures across JSON-style visualization DSLs and derives design considerations from recurring patterns.

1 INTRODUCTION

  • The study compares 10 JSON-style DSLs across 41 visualization tasks, including single charts, layered views, facets, and composite charts.JSON-style DSLs are used in declarative visualization grammars and share enough surface structure for cross-DSL failure comparison.Source: §1 · p. 1
  • Across 1,230 trials, three LLMs exhibited four recurring failures: encoding mismatch, layout arithmetic failure, missed renderer defaults, and failed workaround.Failed workarounds approximate unsupported chart types with primitives or custom compositions that do not preserve requested semantics.Source: §1 · p. 1
  • The authors propose exposing common encodings as fields or channels, letting composition primitives own layout budgeting, and surfacing renderer defaults and hidden contracts.They argue that AI-native design is not simply shorter syntax or broader permissiveness, but structures and semantics LLMs can reliably interpret and compose.Source: §1 · p. 1

2 RELATED WORK

Prior work spans visualization grammars, LLM-driven chart generation, and evaluations that often hold the DSL fixed. This paper varies visualization DSLs to study how their designs relate to LLM generation failures.

2 RELATED WORK

  • The Grammar of Graphics tradition inspired visualization grammars ranging from Polaris and its successor VizQL, which compiles one declarative description into database queries and graphics, to JSON-based specifications.Source: §2 · p. 1
  • Existing visualization-DSL evaluations examine human authoring trade-offs [34] or design-pattern differences [17], while this study probes LLM generation failures across languages.A survey of JSON-centered DSLs highlights variation in specification structure, layout primitives, and chart-type vocabularies [23].Source: §2 · p. 2
  • LLM visualization research includes end-to-end systems [7] [25] [40] [41] and chart-generation benchmarks [5], while some systems generate SVGs directly.Existing authoring benchmarks fix one DSL, such as Vega-Lite or matplotlib [42] [5]; declarative JSON is also used as an intermediate representation in agentic and code-generation pipelines [41] [40].Source: §2 · p. 2

3 STUDY DESIGN

The study builds a 1,230-trial test suite by crossing selected JSON-style DSLs, constrained chart tasks, and three LLMs. It uses staged checks and audited rubric scores diagnostically, while noting scope and design limitations.

3 STUDY DESIGN

  • The DSL set comprises nine surveyed or general-purpose grammars plus a distinct, actively maintained ChartML project used as a model-familiarity baseline.Candidates were filtered for publicly installable runtimes, active maintenance, and headless rendering without proprietary tooling.Source: §3 · p. 2
  • Tasks draw on a chart taxonomy, exclude cases centered on custom drawing or bespoke layout, and add composite charts and constraints to test preservation of data and presentation semantics.Prompts used explicit, verifiable constraints because pilot runs found that models handled simple chart requests well.Source: §3 · p. 2
  • Each task uses a distinct synthetic dataset with fewer than 10 columns and 5–365 rows, and the three models span providers and capability tiers.Datasets include semantically plausible field names and irrelevant noise fields to test selection of intended fields over decoys.Source: §3 · p. 2
  • The test suite crosses 41 tasks, 10 DSLs, and three models for 1,230 trials, serving to investigate possible issues rather than rank models or DSLs.The authors infer that errors correlate with DSL design choices when patterns differ across DSLs while other factors remain constant.Source: §3 · p. 2
  • The evaluation first checks JSON parsing and image rendering, then scores rendered cases with an 11-item pass/fail rubric across structure, encoding, fidelity, and style/readability.Scores are diagnostic for locating and comparing failures, not for ranking models or DSLs.Source: §3 · p. 2
  • A vision-language model evaluated rubric items, after which a human reviewer audited scores and failure notes; recurring patterns were grouped across DSLs, chart types, and models.Reported rubric numbers use post-audit results, and multiply reviewed cases use their most recent review.Source: §3 · p. 2
  • The study focuses on DSL design and error forms, uses a single zero-shot prompting style, and does not claim to isolate DSL design as the sole cause of failure.It did not run repeated trials and presents its design considerations as directions awaiting validation with broader models, prompting strategies, and task distributions.Source: §3 · pp. 2–3

4 RESULTS

Across the tested DSLs, models repeatedly misplaced intended encodings, failed to coordinate layout, missed renderer behavior, or improvised workarounds that changed chart semantics. Qualitative coding linked these failures to DSL design choices and motivated design considerations for future DSLs.

Overall results

  • 416 of 964 rendered cases (43.2%; 33.8% of all trials) passed all rubric items; 1222/1230 passed JSON validation and 964/1230 rendered.The authors present these counts as descriptive data for qualitative analysis, not as fair comparisons among DSLs or models.Source: §4 · p. 3

Encoding Mismatch

  • Encoding mismatches arose when DSLs entangled visual roles with chart-specific data structures, so models placed the right fields in locations with different semantics.Chart.js polarArea, for example, uses data.labels for wedge categories and dataset-level labels and colors rather than per-wedge color encodings.Source: §4 · p. 3, Figure 2 · p. 3

Design considerations

  • Field-level encoding references may reduce errors by making common visual roles explicit rather than burying them in series structures.The authors describe series-based APIs as usable, but at greater risk of errors.Source: §4 · p. 3

Layout Arithmetic Failure

  • Layout arithmetic failures occurred when models had to coordinate competing space demands across titles, legends, axes, and panels.Examples include Plotly title–legend collisions, dense Vega-Lite facets, clipped Vega panels, and separately coordinated Plotly subplot settings.Source: §4 · p. 3, Figure 3 · p. 4

Design considerations

  • Declarative layout budgeting and visible renderer behavior may make DSLs more legible without reducing expressiveness.The authors also recommend predictable composition tools for assembling less common chart types when a named primitive is absent.Source: §4 · p. 4

Missed Renderer Defaults

  • Missed renderer defaults and hidden contracts changed charts despite valid-looking specifications and correct field selection.Examples include alphabetized weekday labels without an explicit sort and unrequested ApexCharts donut labels enabled by default.Source: §4 · p. 4

Failed Workaround

  • Failed workarounds substituted nearby primitives or custom compositions that did not preserve requested chart semantics.Vega-Lite radar marks produced almost no geometry, while Plotly stacked areas lacked the centered baseline that defines a streamgraph; a Chart.js flattened faceting workaround was usable.Source: §4 · p. 4

ChartML unfamiliarity

  • All ChartML cases failed parsing because the models lacked its unfamiliar syntax, while the four failure patterns also appeared in familiar DSLs.The authors note that adoption and public examples are prerequisites for authoring without documentation augmentation.Source: §4 · p. 4

5 CONCLUSION

This paper offers an initial exploration of how LLMs fail when specifying charts through visualization DSLs. It distinguishes challenges shared with human authors from potentially LLM-specific patterns and points to questions for future study.

5 CONCLUSION

  • The study is a first exploration of LLM failures in chart specification through visualization DSLs.Source: §5 · p. 5
  • Layout arithmetic and missed renderer defaults reflect challenges that human authors also face, while failed workarounds appear more LLM-specific.The authors contrast LLMs’ improvised compositions with human authors who would typically stop or switch tools when a chart primitive is unavailable.Source: §5 · p. 5
  • The authors propose investigating notational properties for LLMs, including Training-set Exposure and Tokenization Alignment.They suggest viscosity may be less relevant for LLMs because tedious transforms are relatively straightforward for them.Source: §5 · p. 5
Loading 2610.01873v1…