Source-linked AI summary
ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
Jovana Kondic, Pengyuan Li, Dhiraj Joshi, Isaac Sanchez, Ben Wiesel, Shafiq Abedin, Amit Alfassy, Eli Schwartz, Daniel Caraballo, Yagmur Gizem Cinar, Florian Scheidegger, Steven I. Ross, Daniel Karl I. Weidele, Hang Hua, Ekaterina Arutyunova, Roei Herzig, Zexue He, Zihan Wang, Xinyue Yu, Yunfei Zhao, Sicong Jiang, Minghao Liu, Qunshu Lin, Peter Staar, Luis Lastras, Aude Oliva, Rogerio Feris
TL;DR
Chart understanding requires jointly reasoning over visual, numerical, and linguistic information, but existing datasets provide limited multimodal supervision. ChartNet addresses this gap with a code-guided, million-scale dataset of aligned chart representations and specialized subsets. Fine-tuning consistently improves performance across chart understanding tasks and model sizes, often surpassing larger systems and GPT-4o.
Problem
Current VLMs only partially integrate visual, numerical, and linguistic information, while existing datasets are often limited in size, scope, tasks, or multimodal coverage.
Method
ChartNet uses code-guided synthesis to generate aligned chart images, plotting code, data tables, descriptions, reasoning QA, and specialized human, real-world, grounding, and safety subsets.
Results
Fine-tuning on ChartNet consistently improves chart reconstruction, data extraction, summarization, and multi-stage reasoning across models, often surpassing larger systems and GPT-4o.
Takeaways & Limitations
ChartNet provides code-aligned multimodal supervision that transfers across chart understanding tasks and supports robust, interpretable, numerically grounded reasoning.
Abstract
from arXiv · showhide
Understanding charts requires models to jointly reason over geometric visual patterns, structured numerical data, and natural language -- a capability where current vision-language models (VLMs) remain limited. We introduce ChartNet, a high-quality, million-scale multimodal dataset designed to advance chart interpretation and reasoning. ChartNet leverages a novel code-guided synthesis pipeline to generate 1.5 million diverse chart samples spanning 24 chart types and 6 plotting libraries. Each sample consists of five aligned components: plotting code, rendered chart image, data table, natural language summary, and question-answering with reasoning, providing fine-grained cross-modal alignment. To capture the full spectrum of chart comprehension, ChartNet additionally includes specialized subsets encompassing human annotated data, real-world data, safety, and grounding. Moreover, a rigorous quality-filtering pipeline ensures visual fidelity, semantic accuracy, and diversity across chart representations. Fine-tuning on ChartNet consistently improves results across benchmarks, demonstrating its utility as large-scale supervision for multimodal models. As the largest open-source dataset of its kind, ChartNet aims to support the development of foundation models with robust and generalizable capabilities for data visualization understanding. The dataset is publicly available at https://huggingface.co/datasets/ibm-granite/ChartNet
1. Introduction
ChartNet addresses limitations in chart-understanding data by providing large-scale, multimodally aligned supervision spanning visual, numerical, structural, and linguistic information. Experiments report consistent gains across chart tasks and models.
- Current VLMs only partially integrate visual, numerical, and linguistic information when interpreting charts.
- Existing chart datasets are often small, narrow, single-task, or missing modalities such as plotting code, grounding annotations, and reasoning traces.
- ChartNet spans human annotations, grounding, safety data, and real-world charts, broadening coverage for training and evaluation.
- ChartNet provides 1.5 million multimodal tuples containing images, executable plotting code, data, text, and reasoning-based QA, alongside specialized subsets.
- Fine-tuning on ChartNet consistently improves chart reconstruction, data extraction, and summarization across vision-language models.
2. Related Work
Prior chart datasets address individual tasks but often lack broad chart diversity and multimodal completeness. ChartNet instead combines diverse chart types, plotting libraries, modalities, and tasks through code-space generation and filtering.
- Open-source multimodal models generally trail leading proprietary models, while high-quality chart-comprehension training data remains scarce.
- Existing datasets cover question answering, captioning, chart-to-code, and reasoning, but fail to capture the full diversity of real-world charts.
- ChartNet features 24 chart types, multiple plotting libraries, rigorous filtering, human annotations, and tasks spanning chart-to-code, extraction, captioning, reasoning, grounding, and safety.
- Unlike image-space synthesis, ChartNet performs data generation and augmentation in code space.
3. ChartNet Data Generation Pipeline
ChartNet scales chart generation by reconstructing seed images as executable code, iteratively augmenting that code, rendering outputs, and filtering visual defects. The pipeline then derives aligned tables, descriptions, and reasoning QA.
- A VLM converts seed chart images into approximate executable plotting code, which becomes the structured representation for subsequent generation.
- The pipeline uses code, visual context, and tabular data to generate data representations, grounded descriptions, and QA pairs with chain-of-thought reasoning.
- The pipeline iteratively rewrites plotting code to vary chart types, styles, representations, data values, and labels while preserving relevance.
- Generated scripts are executed to render chart images, and successful executions are paired with their outputs before quality filtering.
- 77% of generated code snippets execute successfully, while 36.5% of rendered images contain some visual error before filtering.
- After quality filtering, charts with readability-affecting issues decrease from 14.9% to 5.9% in a manually annotated sample of 3157 charts.
4. The ChartNet Dataset
ChartNet combines a large aligned synthetic corpus with human-verified, real-world, grounding, and safety subsets. These components support multimodal chart reasoning across numerical, visual, textual, and spatial demands.
- The core dataset contains 1.5M aligned synthetic tuples linking chart images, plotting code, tabular data, descriptions, and reasoning-based QA.
- Specialized subsets cover human-annotated data, real-world charts, grounding, and safety.
- 96,643 synthetic chart samples received rigorous human verification and annotation across aligned images, descriptions, and tabular data.
- The real-world subset contains 30K charts from recognized publishers and spans economics, technology, geopolitics, environmental science, and societal trends.
- Real-world charts include captions, sub-captions, key data highlights, and analytical summaries to support visual reasoning, textual grounding, and insight extraction.
- Grounding QA pairs use geometry-aware annotations for chart elements such as axes, ticks, gridlines, legends, and patches.
- Safety data pairs sensitive charts with adversarial questions covering discrimination, hate, violence, political bias, and substance abuse.
5. Experiments
ChartNet is evaluated through a held-out suite spanning four chart-understanding tasks, using models across scales and comparisons with open-source, specialized, and proprietary systems. Public-benchmark evaluation further tests transfer beyond the synthetic training distribution.
- Evaluation setup: ChartNet fine-tuning trains models on Chart-to-Code, Chart-to-Table, Chart-to-Text, and Chart QA with CoT Reasoning.Experiments include ultra-compact, small, and larger model scales.
- Evaluation setup: The held-out ChartNet evaluation suite contains 2,000 multimodal chart tuples covering four tasks.Each tuple includes an image, plotting code, data table, natural-language summary, and reasoning-based QA.
- Task definitions: Chart-to-Code measures executable reconstruction, data fidelity, code similarity, and structural or image correspondence to the source chart.The task asks models to generate executable plotting scripts from chart images.
- Task definitions: Chart-to-Table evaluates whether models can recover plotted data as CSV from chart images, using similarity to the ground-truth table.Minor formatting differences are disregarded in the similarity score.
- Task definitions: Chart-to-Text assesses summaries for coverage of key takeaways, trends, comparisons, visual elements, and chart style.Generated summaries are compared with references verified by the ChartNet generation pipeline.
- Task definitions: Chart QA with CoT Reasoning scores final answers using fuzzy matching against gold references, reporting average fuzzy accuracy.Models produce separate reasoning and answer sections for complex chart questions.
- Baselines and generalization: Evaluation compares open-source VLMs, ChartGemma, GPT-4o, and ChartNet-finetuned variants, while public benchmarks include ChartCap and ChartMimic-v2.Most metrics are automatically computed using GPT-4o as judge, except for Chart QA with CoT Reasoning.
6. Results & Discussion
ChartNet fine-tuning yields consistent gains across chart reconstruction, data extraction, summarization, and reasoning, including transfer to public benchmarks. Smaller ChartNet-tuned models frequently outperform much larger off-the-shelf systems, including GPT-4o in several comparisons.
- Overall results: Substantial and consistent gains appear across all chart-understanding tasks after ChartNet fine-tuning, regardless of model scale.The results attribute this pattern to limited exposure to high-quality multimodal chart supervision in existing VLMs.
- Chart reconstruction: Chart reconstruction improves in execution, data fidelity, and code or image similarity, with small models reaching 90%+ on most metrics.SmolVLM-256M and Granite-Docling-258M gain functional reconstruction capability, while LLaVA-7B improves by up to +42.4 points.
- Chart data extraction: 70.3% is achieved by the best Granite-Vision-2B model for chart data extraction, while LLaVA-7B improves by +41.8 and surpasses GPT-4o’s 46.7% accuracy.The results connect this performance to alignment between code-generated charts and their CSV data.
- Chart summarization: +9.5 to +31.4 gains occur in chart summarization, and finetuned Granite-Vision-2B reaches 83.9%, surpassing GPT-4o and larger open-source baselines.The summaries are constructed jointly from code and rendered visuals.
- Chart QA with CoT Reasoning: +15.17 is LLaVA-7B’s largest improvement in chart QA with CoT Reasoning, reaching 70.3% and outperforming ChartGemma and larger baselines including GPT-4o.Every evaluated model shows steady accuracy improvements on this multi-stage reasoning task.
- Comparison with off-the-shelf models: ChartNet-tuned 2B and 7B models consistently exceed 20B–72B off-the-shelf models in nearly every metric, with especially large gaps over GPT-4o in reconstruction and extraction.The findings suggest code-aligned multimodal supervision can be more effective than simply scaling model size for tightly coupled chart information.
- Public-benchmark generalization: Granite-Vision-2B improves from 1.6 to 12.4 BLEU on ChartCap and from 30.8 to 58.4 on ChartMimic-v2 after fine-tuning.Gains also occur for ultra-compact SmolVLM-256M and transfer across summarization and chart-to-code tasks.
7. Conclusion
ChartNet addresses the shortage of large-scale, high-fidelity supervision aligning chart images, code, numeric data, text, and reasoning. Experiments report consistent gains across models, tasks, and architectures, including performance beyond much larger systems.
- ChartNet aligns images, plotting code, numeric data, textual descriptions, and reasoning traces in over one million multimodal tuples.
- The dataset supports chart-to-code reconstruction, data extraction, summarization, and multi-step reasoning.
- Experiments show consistent gains across model sizes and architectures, often surpassing much larger open-source systems and GPT-4o.
- ChartNet provides an open foundation for numerical reasoning, visualization understanding, document intelligence, and code-aligned multimodal modeling.
A.1. Data Distribution of the Core Dataset
ChartNet covers diverse chart types and plotting packages through code-guided augmentation, while acknowledging execution and rendering challenges for complex representations.
- ChartNet samples one of 24 chart types uniformly during code-guided chart augmentation.
- Higher-complexity chart types are less likely to execute successfully because they contain more code issues.
- Rendering errors such as overlapping labels and obscured data are more common for certain chart types and are flagged during filtering.
- Even chart types and plotting packages representing less than one percent of the dataset still correspond to thousands of charts.
A.2. QA with Long CoT Reasoning
ChartNet constructs long-chain-of-thought supervision through staged question generation, planning, chart description, reasoning, modality bridging, and answer distillation, with filtering for informative charts and grounded annotations.
- Question generation: The pipeline generates challenging questions requiring comparisons, trends, anomalies, intersections, or hypothetical aggregations rather than trivial lookups.
- Planning and captioning: A pseudo-CoT first separates a high-level solving plan in <SUMMARY> from a detailed chart description in <CAPTION>.
- Reasoning and answers: The next stage produces explicit step-by-step reasoning from the caption, plan, chart code or CSV, and image, followed by a concise conclusion.
- Modality bridging: Modality bridging creates an answer-agnostic textual surrogate that preserves visual and quantitative evidence needed for reasoning and conclusion recovery.
- Long-form CoT: GPT-OSS distills long-form reasoning with detailed checks, calculations, assumptions, and ambiguity resolution separated from the final answer.
- Quality control: Filtering retains charts with clear semantic and quantitative cues and non-trivial reasoning requirements while discarding low-information visuals.
- Grounding QA: Grounding-based QAs combine structural retrieval templates with reasoning-based approaches, covering multi-step patterns such as extrema, changes over time, distributions, and rankings.
A.6. Safety
ChartNet’s Safety subset evaluates robustness to adversarial chart prompts in sensitive domains, pairing safe and unsafe responses across curated categories and visual formats.
- The Safety subset targets robustness under safety-critical conditions.
- It pairs charts with adversarial prompts about health, finance, and social issues, alongside safe and unsafe responses for preference-based optimization.
- The subset contains 7,000 training samples and 600 test samples across 18 safety categories.
- Charts span bar, line, pie, scatter, and other types across plotting libraries, visual styles, and complexity levels.
- Filtering combines textual safety checks with visual saliency gating to enforce semantic safety and visual quality.
- Data retrieval: Retrieval QA templates ask about chart elements, legends, positions, labels, and colors.
- Reasoning QA: Reasoning templates cover sums, differences, averages, medians, ratios, inequalities, and comparisons across series or time points.
C. Human–LLM Agreement Evaluation
The evaluation tests whether GPT-4o can serve as an automated judge for chart data extraction by comparing its ratings with human judgments. GPT-4o aligns with human ratings and preserves the same model ranking.
- Evaluation setup: GPT-4o-as-judge was evaluated on chart data extraction, where reconstructed tables are compared against ground truth.The study focuses on this task because it is described as ChartNet’s most challenging task.
- Evaluation setup: Human annotators rated outputs from Granite Vision finetuned on ChartNet, GPT-4o, and LLaVA using a shared correctness rubric.The evaluation used 100 randomly sampled chart–table pairs.
- Human agreement: Krippendorff’s α = 0.81 indicates high agreement between the two independent human annotators.This agreement supported using a single annotator as the human reference for the remaining models.
- Rating alignment: Pearson r = 0.86 for Granite Vision and r = 0.62 for LLaVA show alignment between human ratings and GPT-4o-as-judge.GPT-4o was slightly more lenient on low-quality outputs but remained tightly aligned on higher-quality predictions.
- Model ranking: Humans and GPT-4o independently ranked the models identically: Granite Vision first, GPT-4o second, and LLaVA last.This consistency supports GPT-4o’s suitability as an automated evaluator for chart data extraction.