PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild

Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, Shafiq Joty

arXiv:2407.04172v2cs.AIcs.CLcs.CV

TL;DR

Existing chart instruction-tuning methods rely on underlying data tables and weakly aligned vision-language backbones. ChartGemma instead uses chart-image-generated instructions with PaliGemma. It reports state-of-the-art results across five benchmarks and more faithful, realistic summaries of complex charts.

  • Problem

    Existing chart instruction-tuning methods use underlying or synthetic data tables and weakly aligned backbones, while real-world charts require visual trend and pattern understanding.

    Source: §1 · p. 1

  • Method

    ChartGemma is an instruction-tuned chart-understanding model built over PaliGemma and trained with instructions generated directly from chart images.

    Source: §1 · p. 2

  • Results

    ChartGemma achieves state-of-the-art results across five benchmarks spanning chart summarization, question answering, and fact-checking, and produces more faithful, realistic summaries of complex charts.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The paper presents ChartGemma as an effective model for understanding and reasoning over real-world charts.

    Source: §6 · p. 8

    ChartGemma is prone to hallucinations, occasionally producing factually incorrect statements or erroneous code.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Given the ubiquity of charts as a data analysis, visualization, and decision-making tool across industries and sciences, there has been a growing interest in developing pre-trained foundation models as well as general purpose instruction-tuned models for chart understanding and reasoning. However, existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts, ignoring the visual trends and patterns in chart images, and use weakly aligned vision-language backbone models for domain-specific training, limiting their generalizability when encountering charts in the wild. We address these important drawbacks and introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images, thus capturing both high-level trends and low-level visual information from a diverse set of charts. Our simple approach achieves state-of-the-art results across $5$ benchmarks spanning chart summarization, question answering, and fact-checking, and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries. We release the code, model checkpoints, dataset, and demos at https://github.com/vis-nlp/ChartGemma.

1 Introduction

Existing chart models rely on table-derived instructions and weakly aligned vision-language backbones, limiting their fit to complex real-world charts. ChartGemma instead uses image-generated instructions with PaliGemma and reports strong results across chart tasks.

1 Introduction

  • Existing instruction-tuning methods draw on underlying or synthetic data tables and chart-specific or weakly aligned backbones, constraining real-world chart understanding.Tables can miss nuanced visual details, while weakly aligned backbones limit generalizability.Source: §1 · pp. 1–2
  • ChartGemma generates instruction-tuning data directly from chart images, preserving visual features and avoiding dependence on available underlying tables.This allows use of a broader range of charts available across the web.Source: §1 · p. 2
  • ChartGemma achieves state-of-the-art results across five benchmarks and produces more faithful, realistic summaries of complex charts than other methods.The benchmarks span chart summarization, question answering, and fact-checking.Source: §1 · pp. 2–3
  • ChartGemma is built over PaliGemma, whose larger alignment dataset supports a model that is smaller than existing chart-understanding models.The paper evaluates it across chart summarization, question answering, and fact-checking.Source: §1 · p. 2

2 Chart Instruction Data Generation

The data-generation pipeline builds a diverse chart corpus and creates instructions directly from chart images, including tasks that target visual trends and relations. The resulting dataset emphasizes visual attributes and shows higher sampled instruction accuracy than ChartInstruct.

2 Chart Instruction Data Generation

  • The instruction-tuning dataset is generated by first curating a diverse chart corpus spanning visual styles and elements, then generating visual instruction-tuning data directly from the charts.Source: §2 · p. 3

2.1 Assembling the Chart Corpus

  • The corpus contains 122,857 charts drawn from synthetic, curated, and in-the-wild sources, combining controlled examples with broader stylistic variety.Examples include PlotQA, Statista, and WebCharts.Source: §2.1 · p. 3

2.2 Visual Chart Instructions

  • Predefined tasks cover chain-of-thought reasoning, summarization, fact-checking, chart-to-Markdown, and program-aided design.The training mix also includes fact-checking data and ChartQA questions paired with executable code.Source: §2.2 · p. 3
  • Open-ended instructions include explaining time-series trends, describing chart elements, and critically comparing visual information.These tasks are intended to extend applicability to diverse real-world settings.Source: §2.2 · pp. 3–4

2.3 Key Dataset Characteristics

  • The dataset-characteristics analysis examines both visual attributes and the quality of generated chart instructions.The authors describe these as two critical elements for distinguishing the dataset from prior work.Source: §2.3 · p. 4
  • The dataset emphasizes chart lines, shapes, colors, trends, types, and positions that occur in real-world scenarios.The authors state that these visual attributes enhance visual reasoning capabilities.Source: §2.3 · p. 4
  • 82% of 100 sampled synthesized instructions accurately reflected chart content, compared with 61% accuracy reported for ChartInstruct.Another 8% were partially correct; the authors attribute the quality improvement to using chart images rather than automatically generated, often erroneous tables.Source: §2.3 · p. 4

3 Modeling and Methodology

ChartGemma uses PaliGemma’s vision and language components, with full attention over input image and text tokens. It directly fine-tunes the backbone on instruction data, freezing the vision encoder and updating the language model.

3.1 Architecture

  • PaliGemma provides ChartGemma’s SigLIP vision transformer and Gemma-2B language model.Gemma-2B is a decoder-only transformer language model.Source: §3.1 · p. 4
  • Visual features are mapped into the language model’s embedding space, then combined with text embeddings; full attention covers input tokens while output tokens use a causal mask.The authors say this improves image contextual understanding, particularly for complex object relationships.Source: §3.1 · p. 4

3.2 Training Setup

  • Unlike methods that first align the vision encoder and language model, ChartGemma directly fine-tunes its backbone on instruction data in one stage.PaliGemma was trained end-to-end on 10 billion image-text pairs spanning varied styles.Source: §3.2 · pp. 4–5
  • ChartGemma generally outperforms or matches baselines on ChartQA, ChartFC, and ChartCheck while being significantly smaller.The comparison covers closed-ended generation benchmarks.Source: Table 2 · p. 5
  • Instruction-tuning freezes the vision encoder and fine-tunes only the LLM, reducing computational complexity and improving stability with small batches.Source: §3.2 · p. 5

4 Experiments, Results, and Analyses

ChartGemma performs strongly across closed-ended and open-ended chart tasks, with human evaluations supporting its summarization quality. Error analysis identifies challenges with high-resolution charts, code generation, and complex visual styles, while training reaches its best observed performance at epoch 2.

4.1 Experimental Setup

  • Five benchmarks cover chart question answering, fact-checking, open-ended question answering, and summarization, alongside a manually curated set of 100 unseen web charts.Evaluation uses relaxed accuracy for ChartQA, accuracy for ChartFC, and GPT4 ratings of informativeness and factual correctness for open-ended tasks.Source: §4.1 · p. 5

4.2 Performance on closed-ended tasks

  • ChartGemma performs best among chart VLLMs on average ChartQA and on the ChartFC and ChartCheck test splits, but trails 13B ChartAssistant on augmented ChartQA.The authors attribute its ChartCheck results to zero-shot generalization to more realistic charts and report a significant improvement over ChartAssistant on ChartQA’s human-generated split.Source: §4.2 · pp. 5–6

4.3 Performance on open-ended tasks

  • ChartGemma outputs generally receive higher GPT4 scores than ChartInstruct, with the authors reporting stronger informativeness and factual correctness in long-form answers and summaries.The authors suggest image-derived instruction data and PaliGemma’s strong alignment may contribute to these outcomes.Source: §4.3 · pp. 6–7, Figure 3 · p. 6

4.4 Human Evaluation on Summarization

  • Human evaluation found ChartGemma matched or outperformed ChartInstruct-LLaMA2 on all metrics, was more informative and significantly more factually correct, and was equally well structured.Two annotators rated outputs on a 1–5 scale; their agreement had Cohen’s Kappa of 0.538.Source: §4.4 · p. 7

4.5 Error Analysis and Challenges

  • Errors arise when resizing very large charts makes text unreadable, when generated code uses undeclared variables, and on visually diverse charts compared with Pew or Statista charts.The 896×896 PaliGemma variant is significantly slower than the 448×448 version and is described as impractical for consumer-level machines and GPUs.Source: §4.5 · p. 7

4.6 Convergence of ChartGemma

  • ChartGemma reaches its best observed downstream performance at epoch 2, and the authors attribute its quick convergence to PaliGemma’s strong alignment.They leave testing whether this backbone generalizes with many more training samples as future work.Source: §4.6 · p. 8, Figure 5 · p. 8

5 Related Work

Prior chart models span fine-tuning, chart-specific pretraining, and instruction-tuned VLMs. The authors distinguish their approach from instruction-tuning methods that rely on chart tables and weakly aligned backbones.

Chart Representation Learning

  • Chart models have used fine-tuning from language or vision-language models, chart-specific pretraining objectives, and more recently instruction-tuning of pretrained VLMs.Source: §5 · p. 8

Chart Modeling Benchmarks

  • Chart benchmarks cover closed-ended question answering, open-ended explanation and summarization, chart-to-table conversion, and multimodal fact-checking.Source: §5 · p. 8

Instruction-tuning across modalities and for charts

  • Chart instruction-tuning methods synthesize training data from underlying chart tables, which the authors say miss nuanced details of complex real-world charts.Source: §5 · p. 8

6 Conclusion and Future Work

ChartGemma combines image-generated instruction-tuning data with a strongly aligned backbone, while future work targets more varied instructions and chart-relevant evaluation.

6 Conclusion and Future Work

  • ChartGemma addresses table-derived training data and weakly aligned backbones with image-based instruction tuning and a state-of-the-art backbone, achieving improvements with fewer parameters.Error analyses and human studies report more realistic, informative, and factually correct outputs than contemporaries.Source: §6 · p. 8
  • Future work aims to create a more diverse instruction-tuning dataset using human-written instructions that capture chart nuances.Source: §6 · pp. 8–9
  • The authors also aim to develop a more generalized benchmark for complex chart visuals, with chart-relevant evaluation metrics.Source: §6 · p. 9

Limitations

The authors identify constraints in data generation, input resolution, evaluation reproducibility, and output reliability.

Limitations

  • Instruction-tuning data relies on a proprietary LLM, which could restrict use in some commercial environments.Source: Limitations · p. 9
  • The vision encoder is capped at 448x448 resolution; increasing resolution raises processing time quadratically.Source: Limitations · p. 9
  • Informativeness and Factual Correctness evaluation depends on GPT-4, whose updates or deprecation challenge reproducibility.Source: Limitations · p. 9
  • The model can hallucinate factually incorrect statements or erroneous code, so the authors advise guardrails and caution in deployment.Source: Limitations · p. 9

Ethics Statement

The authors describe risks from generated outputs and outline safeguards and data-sharing practices for the study and dataset.

Ethics Statement

  • Autoregressive responses can be misleading or inaccurate, and generated code is not guaranteed to be free of malicious content.Source: Ethics Statement · p. 9
  • The authors recommend strict safety guidelines to mitigate risks from inaccurate outputs and potentially malicious generated code.Source: Ethics Statement · p. 9
  • Base datasets are publicly available as datasets or URLs with public licenses, and chart images came from publicly available research papers screened for offensive content.Source: Ethics Statement · p. 9
  • The human evaluation collected no personal identification information, and the authors report no ethical issues or unwanted biases from the study.Source: Ethics Statement · p. 9

Appendices

The appendices describe chart-corpus sourcing, benchmark-data augmentation, and prompt templates for two instruction-tuning tasks.

A.1 Chart Corpora Collection

  • Chart corpora are grouped into three categories by source and generation method, with category statistics and sources reported in Table 4.Source: §A.1 · p. 12
  • Instruction-tuning task instructions are augmented with training sets from existing benchmark datasets.Source: §A.1 · p. 12

A.3 Prompt Templates for Instruction-tuning Data Generation

  • Prompt templates cover program-aided design and open-ended tasks, drawing inspiration from ChartInstruct and Gemini Flash's ChartQA prompt.Source: §A.3 · p. 12

A.2 Instruction Dataset Analysis

The appendix reports human evaluation favoring ChartGemma on informativeness and factual correctness, alongside details on instruction data, evaluation, training settings, and error patterns.

A.2 Instruction Dataset Analysis

  • The instruction-tuning data combines closed-ended response generation with open-ended answering, drawing on benchmark-inspired and open-ended chart-understanding tasks.The dataset’s instructions accurately reflect chart content about 82% of the time and are partially correct 8% of the time.Source: §A.2 · p. 12

B.1 Hyperparameter settings

  • The appendix directs readers to Table 7 for the hyperparameter settings and training details used for instruction tuning and benchmark fine-tuning.Source: §B.1 · p. 12

B.2 Prompt templates for evaluation

  • GPT4 evaluation prompts are provided for summarization on Chart2Text and the Web set, and for open-ended answering on OpenCQA.The prompt examples are shown in Figures 10 and 11, respectively.Source: §B.2 · p. 12

B.3 GPT4 evaluation on open-ended generation tasks

  • The appendix reports GPT4 evaluation results for informativeness, factual correctness, and relevance across Chart2Text, OpenCQA, and the curated Web chart set.The cited passage points to Table 8 but provides no scores.Source: §B.3 · p. 12

B.4 Human Evaluation Study

  • On the Web chart set, human evaluation found ChartGemma significantly better than ChartInstruct-LLaMA2 in informativeness and factual correctness, while matching it in summary structure.The study used two English-proficient volunteers recruited from the authors’ research lab.Source: §B.4 · p. 12

B.5 Error Analysis

  • Figure 13 shows typos and coding errors among ChartGemma’s outputs.Source: §B.5 · p. 12

B.6 Sample Outputs from ChartGemma

The appendix collects ChartGemma examples across chart tasks, including trend summaries and numerical comparisons, alongside task-data, prompt, and evaluation materials.

Instruction data

  • Figures 6 and 7 show diverse visual and open-ended instruction tasks, while Table 4 describes generated-example counts by task and distinct chart counts.The table caption notes that some charts are used in multiple tasks.Source: Figure 6 · p. 13, Table 4 · p. 13, Figure 7 · p. 14

Data-generation prompts

  • Figures 8 and 9 show Gemini Flash-1.5 prompts for generating program-aided design and open-ended instruction-tuning data, respectively.Source: Figure 8 · p. 14, Figure 9 · p. 14

Sample answers

  • For the British Pound, a sample answer describes depreciation against the US Dollar, a May–August drop, and a slight late recovery; another says Greater Vancouver’s 2015 index was around 180 versus Canada’s around 170.Source: §B.6 · p. 15

Generated-answer examples

Evaluation examples

  • The appendix includes GPT4 evaluation prompts for Chart2Text/Web summarization and OpenCQA answering, plus table captions describing GPT4 and human evaluation measures and a chart-captioning comparison.The supplied captions identify evaluation scope and measures but do not give score values.Source: Figure 10 · p. 16, Figure 11 · p. 16, Table 8 · p. 17

Model outputs and errors

Related papers

Loading 2407.04172v2…