PaperlayerView exact revision on arXiv

Source-linked AI summary

DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?

Jiahui tang, Kuicai Dong, Dexun Li, Hongchao Gu, Haocheng Yu, Wei Han, Chen Zhang, Yong Liu, Hao Wang, Enhong Chen

arXiv:2608.26757v1cs.AI

TL;DR

Faithful chart generation requires recovering scattered evidence, deriving chart-ready quantities, and rendering them correctly, but plausible charts can hide upstream data errors. DEEPCHART operationalizes this challenge with an expert-annotated 1,482-instance benchmark and stage-wise Extract–Reason–Visualize evaluation, finding persistent extraction and reasoning failures in realistic long and multimodal settings.

  • Problem

    Existing chart-generation benchmarks often use clean inputs and endpoint-only scoring, leaving hidden data hallucinations and intermediate extraction or reasoning failures difficult to inspect.

  • Method

    DEEPCHART evaluates task-conditioned chart generation as Extract–Reason–Visualize over long real-world documents, with auditable intermediate references and expert verification.

  • Results

    Current models often produce visually plausible charts while failing in extraction and reasoning, and full-document inputs further degrade end-to-end faithfulness.

  • Takeaways & Limitations

    Faithful chart generation requires reliable evidence extraction and quantitative reasoning before rendering; larger context windows alone are insufficient.

  • Takeaways & Limitations

    The benchmark covers Academic, Finance, and Ecosystem but does not exhaust all data-science chart-generation scenarios, and its value-level F1 scores are approximate indicators of numerical faithfulness.

Abstract

from arXiv · show

Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately. Modern LLMs can produce visually plausible, instruction-compliant charts, yet data-level hallucinations remain difficult to detect in long, noisy, and multimodal contexts. To measure this gap, we introduce DEEPCHART, an expert-annotated benchmark of 1,482 task-conditioned chart-generation instances drawn from real-world scientific papers, financial filings, and ecosystem reports. DEEPCHART formulates chart generation as an Extract--Reason--Visualize pipeline and evaluates source-data extraction, derived-data reasoning, and chart rendering stage by stage. Experiments with state-of-the-art models show that visually plausible charts often conceal data-level hallucinations, with extraction and reasoning errors common in realistic long and multimodal settings. These findings suggest that larger context windows alone are insufficient; faithful chart generation also requires reliable evidence extraction and quantitative reasoning before rendering. Our benchmark and associated resources are available at https://github.com/tangdouer1005/DeepChart.

1 Introduction

DEEPCHART addresses the difficulty of generating faithful charts from long, heterogeneous, multimodal evidence, where extraction and reasoning errors can remain hidden behind plausible rendering. It introduces an Extract–Reason–Visualize benchmark and finds persistent stage-level failures in current models.

  • Real-world faithful chart generation requires extracting scattered data, applying transformations, and rendering the resulting visualization.
  • Visually plausible charts can conceal hidden hallucinations caused by mis-extracted or miscomputed numbers that evade appearance-based evaluation.
  • Existing benchmarks often use predigested inputs, omit intermediate data references, and evaluate only final charts, limiting failure verification and localization.
  • DEEPCHART models generation as Extract, Reason, and Visualize stages, enabling failure isolation across evidence retrieval, quantitative derivation, and rendering.
  • The benchmark contains task-conditioned instances from real-world documents spanning long-text and multimodal settings, with source, derived-data, and chart references for stage-wise evaluation.
  • Zero-shot experiments show visually plausible charts can conceal data-level hallucinations, while extraction and reasoning remain difficult and full-document inputs reduce faithfulness.

2 Task Definition

DEEPCHART defines faithful chart generation from a context and chart intent as an auditable Extract–Reason–Visualize process. Its intermediate data path supports checking source values and derived quantities rather than judging only the final chart.

  • Given a long heterogeneous context C and chart intent Q, the task generates source values, derived chart-ready quantities, and an executable visualization.
  • The ERV pipeline separates extracting intent-relevant evidence, reasoning over it to derive chart-ready data, and visualizing the result.
  • The auditable intermediate state J = (Dsrc, Dder) records extracted source data and derived chart-ready data before rendering.
  • Compared with endpoint-only benchmarks, the task emphasizes retrieval, reasoning, real documents, and intermediate-data evaluation.
  • Exposing J enables stage-by-stage evaluation that distinguishes extraction, reasoning, and visualization failures.

3 Benchmark: DEEPCHART

DEEPCHART is an expert-annotated benchmark built from complex academic, financial, and ecosystem documents to test chart generation across evidence extraction, quantitative reasoning, and visualization. Its 1,482 instances include auditable references for every stage.

  • DEEPCHART spans Academic, Finance, and Ecosystem domains, each stressing different evidence-localization, aggregation, derivation, or multimodal reasoning bottlenecks.
  • Each instance pairs model-facing context and query inputs with hidden intermediate data, executable program, and rendered reference-chart artifacts.
  • Experts collect and filter public papers, supplementary materials, 10-K filings, and market or startup ecosystem reports for dispersed quantitative evidence.
  • Contexts are answerable and leakage-controlled, with text or image representations and variants that probe the effects of context length.
  • 1,482 context-query instances comprise the resulting benchmark, with fixed queries and references across multiple text-domain context-scale variants.
  • Reference construction covers source extraction, reproducible transformation into derived quantities, and executable chart generation from Dsrc and Dder.
  • Five expert annotators verify inputs, source data, derived data, code, and charts; ambiguous or unverifiable instances are reviewed, revised, or discarded.

4 Evaluation Framework

DEEPCHART evaluates chart generation stage by stage using source-data fidelity, derived-data fidelity, visual accuracy, and execution rate. These measures separate numerical recovery, reasoning, visual specification, and rendering success.

  • The evaluation reports F1,src for extraction, F1,der for reasoning, VAS for visualization, and ER as an auxiliary rendering measure.
  • F1,src measures recovery of query-relevant numerical evidence through normalized numeric matching against reference source values.
  • F1,der evaluates derived chart-ready quantities and captures aggregation, normalization, ratio, and cross-year comparison errors.
  • Both fidelity metrics perform value-level one-to-one matching, measuring numerical recovery rather than full semantic alignment.
  • VAS averages VLM-judged pass rates over instruction compliance, data-mapping topology, and presentation quality.
  • ER is the percentage of test instances whose generated visualization program successfully produces a valid chart image.

5 Experiments

DEEPCHART evaluates zero-shot chart generation across text and multimodal domains using stage-specific metrics and context scales. Results expose failures in extraction, reasoning, visualization, and execution that endpoint-only evaluation can miss.

  • Experimental setup: DEEPCHART evaluates Academic and Finance under Normal/Long text settings and Ecosystem under native report-level multimodal inputs.The benchmark also includes a separate Finance-Ultra-Long stress test using full 10-K filings.
  • Experimental setup: Stage-specific metrics measure source-data fidelity, derived-data fidelity, visual accuracy, and execution rate across the ERV pipeline.These metrics separately assess extraction, reasoning, visualization, and rendering success.
  • Main benchmark results: 0.782 average ER in Ecosystem contrasts with 0.447 average VAS, showing that executable programs can still produce visually unfaithful charts.Visual faithfulness also exceeds data faithfulness there: average VAS is 0.447 versus F1,src 0.149 and F1,der 0.276.
  • Main benchmark results: In Academic, average F1,src reaches 0.691/0.601 under Normal/Long while average F1,der remains 0.236/0.208.The gap indicates that recovering source values does not ensure correct chart-ready derivation.
  • Domain-specific bottlenecks: Finance performance declines from Normal to Long, with average VAS falling 0.421 to 0.384 and average F1,der 0.249 to 0.217.The paper attributes these consistent drops to greater retrieval and derivation difficulty from additional filing context.
  • Impact of Ultra-Long Context: Ultra-Long Finance inputs reduce average VAS from 0.421 to 0.269, F1,src from 0.385 to 0.167, and F1,der from 0.249 to 0.139.Possible mechanisms include input truncation and dilution from irrelevant filing content; longer context alone does not guarantee faithfulness.
  • Comparison of Rendering Modalities: Rendering backend effects differ by model family: HTML raises proprietary-model ER from 0.735 to 0.957 and VAS from 0.498 to 0.566, but lowers open-weight VAS from 0.427 to 0.368.In free choice, models select Python in 1,591 of 1,600 trials (99.4%), despite HTML benefiting proprietary models.

6 Related Work

Prior work covers chart authoring, chart understanding, document-grounded visualization, and LLM-based data-analysis agents. DEEPCHART is situated among efforts that increasingly integrate natural-language, document, multimodal, and analytical contexts.

  • Chart generation and evaluation: Earlier chart-generation systems recommend or synthesize visualizations from structured tables, analytic specifications, instructions, and analysis intents.Examples include MatPlotBench, Text2Chart, C2/ChartUIE, and PlotCraft, which evaluate instruction following, executable code, and rendering quality.
  • Chart generation and evaluation: Chart-understanding research studies how models read, reproduce, or verify existing charts, complementing chart-authoring benchmarks.Related tasks include chart understanding, chart-to-code, and chart grounding.
  • Document-grounded visualization and data analysis: Document-grounded visualization work generates charts or infographics from textual and document contexts, while data-analysis agents address cleaning, coding, reasoning, tool use, and evidence integration.These lines of work include ChartifyText, Text2Vis, Infogen, Doc2Chart, DV-World, and long-context or multimodal analysis agents.

7 Conclusion

DEEPCHART is an expert-annotated benchmark for faithful chart generation from task-conditioned contexts derived from scientific papers, financial filings, and ecosystem reports. Its Extract–Reason–Visualize formulation supports stage-by-stage evaluation, revealing that visually plausible charts can still fail along the underlying data path.

  • DEEPCHART benchmarks faithful chart generation using task-conditioned contexts from scientific papers, financial filings, and ecosystem reports.
  • The Extract–Reason–Visualize formulation enables stage-by-stage evaluation of extraction, reasoning, and visualization.
  • Zero-shot experiments show that current models can produce visually plausible charts while failing along the underlying data path.
  • The findings highlight the need for better evidence localization, quantitative reasoning, and traceable visualization.

Limitations

DEEPCHART covers three representative domains but does not exhaust all chart-generation scenarios. Its normalized value-level matching metrics are approximate indicators of numerical faithfulness, and expert verification limits scalability in favor of annotation quality and auditability.

  • DEEPCHART covers Academic, Finance, and Ecosystem domains but does not exhaust all data-science chart-generation scenarios.
  • Normalized value-level matching makes F1src and F1der robust to formatting variation but only approximate indicators of numerical faithfulness.
  • Expert verification of source values, derived quantities, executable code, and rendered charts prioritizes annotation quality and auditability over very large scale.

A Benchmark Statistics

The appendix describes benchmark statistics by reporting overall composition, protocol-specific input scale, evidence-entry distributions, and chart-family coverage across domains.

  • Overall benchmark statistics report domains, queries, instances, chart types, and query types.
  • Text input scale is measured in tokens, whereas multimodal input scale is measured in report pages.
  • The statistics include average, median, and maximum evidence entries per instance.
  • Chart-family coverage is broken down by domain, with chart types having fewer than four queries grouped as “Other types”.

B Experimental Details

The experiments use a two-stage pipeline that first generates executable extraction-and-derivation code and then generates executable visualization code from the predicted intermediate state. This design evaluates both chart rendering and support from an explicit source-data and reasoning path under zero-shot conditions.

  • The first generation stage takes task context and chart intent as input and produces executable Python for source extraction and derivation.
  • The second generation stage uses the predicted intermediate state to generate executable visualization code and a final chart image.
  • The two-stage design evaluates whether rendered charts are supported by an explicit source-data and reasoning path.
  • All generation experiments use a zero-shot setting with provider-default decoding parameters unless otherwise specified.
  • Format-level cleanup removes code fences or non-code preambles, while semantic errors in values, quantities, and encodings are not manually corrected.

B.2 Evaluation Details

DEEPCHART evaluates extraction and reasoning through value-level matching, assesses visual outputs with rubric-based VAS, and records executable rendering success. Finance-Ultra-Long uses a fixed subset shared across models for cost-controlled comparison.

  • Data fidelity evaluation: F1,src and F1,der recursively extract numeric leaves from predicted and reference JSON, normalize numeric strings, and apply one-to-one value matching.The matching intentionally ignores JSON keys and nesting structure because schemas vary across models.
  • Data fidelity evaluation: Structural or key-level alignment is not required, while semantic organization errors affecting rendered charts are reflected through VAS data-mapping topology.This separates formatting choices from extraction and reasoning correctness.
  • Visual evaluation: VAS uses cached binary rubrics covering instruction compliance, data-mapping topology, and presentation quality, judged by a deterministic VLM.Each satisfied rubric item contributes one point, and reported VAS is the macro-average across evaluated instances.
  • Execution evaluation: Execution Rate counts outputs only when the generated program finishes successfully and produces a valid PNG image above the minimum file-size threshold.Python programs have a 120-second timeout; HTML outputs are rendered through headless Chromium for backend analysis.
  • Evaluation protocol: Finance-Ultra-Long evaluates a fixed 50-instance subset shared by all models and metrics, covering 16 indicators, 31 bar-chart tasks, and 19 line-chart tasks.The subset averages 708K input tokens, closely matching the full split’s 706.6K-token average.

B.4 VAS Reliability Validation and Judge Ablation

VAS reliability is tested on a stratified 180-chart subset through judge-model ablation and human-alignment experiments. Scores remain consistent across judges and closely agree with human visual assessments.

  • Validation design: The validation set contains 180 charts, with 60 each from Academic, Finance, and Ecosystem domains, sampled evenly across low, medium, and high VAS strata.Within each domain, eligible charts are partitioned by VAS tertiles and 20 charts are sampled from each stratum.
  • Judge-model ablation: Judge-model ablation keeps binary rubric items fixed while comparing gpt-5.2, gpt-4o-mini, qwen3.5-plus, and qwen3-vl-flash-2025-10-15.Judges receive only the generated chart image and corresponding rubric items, without generator or score information.
  • Judge-model ablation: VAS remains stable across judges, with item-level accuracy ranging from 0.773 to 0.898 and Cohen’s κ from 0.415 to 0.755.Chart-level Pearson correlations range from 0.650 to 0.875, while Spearman correlations range from 0.392 to 0.778.
  • Judge-model ablation: The strongest judge agreement is between gpt-5.2 and qwen3.5-plus: 89.8% item-level agreement, κ=0.755, Pearson r=0.875, and Spearman ρ=0.778.The primary qwen3-vl-flash judge also has positive chart-level Pearson correlations with the other judges: 0.722, 0.724, and 0.685.
  • Human alignment: Against human reference labels, the primary judge achieves 94.1% accuracy, 92.5 macro-F1, and κ=0.852 across 2,339 item-level judgments.Automatic VAS correlates with human VAS at Pearson r=0.939 and Spearman ρ=0.878.
Loading 2608.26757v1…