PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartLlama: A Multimodal LLM for Chart Understanding and Generation

Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, Hanwang Zhang

arXiv:2311.16483v1cs.CVcs.CL

TL;DR

Chart-focused multimodal LLMs lack specialized instruction-tuning data for interpreting complex charts. The paper introduces a GPT-4-based three-stage synthesis process and trains ChartLlama on the resulting dataset, reporting stronger performance across chart benchmarks and broader chart capabilities, while multilingual OCR remains a limitation.

  • Problem

    Existing multimodal LLMs and chart datasets provide limited support for complex chart understanding because detailed chart information and comprehensive annotations are scarce.

  • Method

    The paper uses GPT-4 to generate tabular data, render chart figures, and create instruction-tuning data, then trains ChartLlama on the resulting corpus.

  • Results

    ChartLlama outperforms previous methods across ChartQA, Chart-to-text, and chart-extraction benchmarks and supports broader chart-related capabilities.

  • Takeaways & Limitations

    The results support using flexible synthetic data generation to expand chart types, tasks, and multimodal chart-comprehension capabilities.

  • Takeaways & Limitations

    ChartLlama’s current vision encoder lacks multilingual OCR, restricting its utility for charts containing non-English text.

Abstract

from arXiv · show

Multi-modal large language models have demonstrated impressive performances on most vision-language tasks. However, the model generally lacks the understanding capabilities for specific domain data, particularly when it comes to interpreting chart figures. This is mainly due to the lack of relevant multi-modal instruction tuning datasets. In this article, we create a high-quality instruction-tuning dataset leveraging GPT-4. We develop a multi-step data generation process in which different steps are responsible for generating tabular data, creating chart figures, and designing instruction tuning data separately. Our method's flexibility enables us to generate diverse, high-quality instruction-tuning data consistently and efficiently while maintaining a low resource expenditure. Additionally, it allows us to incorporate a wider variety of chart and task types not yet featured in existing datasets. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset. ChartLlama outperforms all prior methods in ChartQA, Chart-to-text, and Chart-extraction evaluation benchmarks. Additionally, ChartLlama significantly improves upon the baseline in our specially compiled chart dataset, which includes new chart and task types. The results of ChartLlama confirm the value and huge potential of our proposed data generation method in enhancing chart comprehension.

1. Introduction

Chart-focused multimodal models are needed because existing systems and datasets provide limited support for complex chart understanding and generation. The paper addresses this gap with a flexible GPT-4-based data pipeline and ChartLlama, which supports broader chart tasks and outperforms prior methods.

  • Motivation: Multimodal LLMs can interpret visual information, but their performance on specialized visual representations such as charts remains limited.Charts are widely used to communicate complex data in research and industry.
  • Motivation: Existing chart datasets mainly support simple question answering or captioning because they lack detailed chart information and comprehensive annotations.Manual web-based annotation also constrains dataset quality.
  • Approach: The proposed approach uses GPT-4 to generate chart data and figures, then create rich multimodal instruction-tuning data for chart understanding and generation.The method is designed to improve accuracy and diversity through separate data, figure, and instruction-generation stages.
  • Contributions: The resulting benchmark emphasizes flexibility, scalability, quality, and diversity, while supporting a broader range of chart types and tasks.The dataset is publicly available according to the paper’s stated contributions.
  • Results: ChartLlama outperforms previous methods on multiple existing benchmarks and supports multi-chart inference, chart de-rendering, and chart editing.The paper reports these capabilities alongside broader chart-type support.

2. Related work

Prior work established strong general-purpose and multimodal language-model foundations, but chart-focused evaluation still exposes difficulties with complex visual reasoning. Existing chart datasets differ in task design and annotation quality, motivating a broader data-generation approach.

  • Language and multimodal models: Open-source LLMs such as Llama and Llama 2 build on large-scale pretraining and instruction datasets, while multimodal models extend these foundations with visual inputs.The related work positions LLaVA, MiniGPT, BLIP2, and mPLUG-Owl as representative multimodal efforts.
  • Dataset and pipeline: The proposed dataset is organized across diverse task types and chart types, with a three-stage pipeline for generating chart data, figures, and instruction data.Figure 2 summarizes distributions, while Figure 3 presents the three stages.
  • Chart understanding: Current multimodal LLMs still struggle with complex chart-related problems in evaluations such as GPT-4V reports and HallusionBench.This motivates specialized chart-understanding resources.
  • Chart benchmarks: Chart benchmarks include simple question-answering datasets such as ChartQA and PlotQA, which differ in annotation quality and evaluation characteristics.The supplied passage identifies human annotation for ChartQA and lower-quality generated questions and answers for PlotQA.

3. Method

The method constructs chart instruction data through three connected stages: controlled data generation, GPT-4-assisted chart rendering, and instruction-answer generation. This separation provides precise chart metadata and supports diverse chart forms and tasks.

  • Pipeline: The pipeline comprises data collection, chart figure generation, and instruction data generation.These three steps are presented as interconnected stages in the proposed approach.
  • Chart data generation: GPT-4 generates tabular data from specified themes, distributions, dataset sizes, and trends to create controlled, diverse source data.Additional guidance is needed to avoid repetitive or meaningless generated data.
  • Chart data generation: The method uses themes and trend descriptions to guide GPT-4 toward meaningful data suitable for instruction-tuning questions and answers.The paper notes that improperly designed prompts can produce repetitive data that deviates from real-world distributions.
  • Chart figure generation: GPT-4 converts the generated data into charts using plotting libraries, function documentation, and randomly sampled successful code examples.The approach is intended to improve chart diversity, visual appeal, and generalization across chart types.
  • Instruction data generation: The final stage uses chart descriptions, raw data, characteristics, and plotting code to generate varied instruction-answer data beyond basic question answering and summarization.These inputs help produce questions about different features of both the data and its visual representation.

4. Experiment

The experiments evaluate ChartLlama on traditional chart-understanding benchmarks and newly introduced description, reconstruction, editing, and generation tasks. Across these evaluations, ChartLlama generally outperforms prior models and LLaVA-1.5, including on novel chart types and qualitative generation tasks.

  • Evaluation setup: The evaluation covers seven tasks, combining ChartQA, Chart-to-text, and Chart-extraction with four additional tasks designed to test scalability across tasks and chart types.The new tasks include detailed description, chart-to-chart reconstruction, and chart generation or modification tasks.
  • Traditional tasks: ChartLlama achieves the best performance on both human and augmented ChartQA splits, despite evaluating zero-shot after training on the proposed dataset.The comparison includes prior methods trained on larger datasets and methods fine-tuned on ChartQA training data.
  • Novel charts and qualitative results: On novel chart types, ChartLlama improves significantly over Unichart and Baseline*, while qualitative examples show more accurate colors, values, chart types, and generated chart details than LLaVA-1.5.In the chart-to-chart and editing examples, LLaVA-1.5 does not reliably preserve chart attributes; in text-to-chart examples, it cannot draw funnel charts.
  • Traditional tasks: ChartLlama consistently outperforms previous approaches on Chart-to-text and performs best on chart extraction in ChartQA.The authors attribute the Chart-to-text improvement partly to handling longer texts without the meaningless repetitions seen in earlier systems.
  • New tasks: ChartLlama significantly improves over LLaVA-1.5 on detailed description and shows consistent gains across three chart generation and modification tasks.The results are reported in Tables 3 and 4, with GPT-based evaluation used for the proposed new tasks.

5. Conclusion

The paper proposes a three-step process for synthesizing chart images and instruction-tuning data, then trains a multimodal LLM on the resulting dataset. Experiments show strong performance, while multilingual OCR remains a limitation for non-English charts.

  • The synthesis pipeline generates chart data, chart figures, and instruction data in three separate steps.This design improves the controllability and diversity of generated data.
  • Experiments on traditional and newly constructed datasets validate the multimodal LLM’s outstanding performance.The diverse instruction-tuning data also supports capabilities absent from previous models.
  • The trained model can extend its comprehension of instructions and figures to new chart categories or tasks.
  • The current vision encoder cannot handle multilingual OCR, restricting utility for charts containing non-English text.The authors propose developing a vision encoder with multilingual OCR proficiency.

Appendix

The appendix presents traditional-task results and visual examples comparing ChartLlama with prior models. These examples emphasize ChartLlama’s ability to follow long instructions, calculate answers, and avoid redundant or hallucinated outputs.

  • Table 5 compares the proposed work with three open-source models and a benchmark-specific Baseline*.The baseline is trained on the training split of each respective benchmark.
  • Visualization examples: ChartLlama follows long instructions and performs calculations correctly in the shown ChartQA examples.

A. Model architecture

The model architecture discussion describes LLaVA-1.5’s vision encoder, projection layer, and parameter-efficient adaptation choices. An ablation table evaluates how prompt components affect second-stage chart generation.

  • Vision encoder: LLaVA-1.5 uses CLIP’s ViT-L/14@336px vision encoder and padding instead of center cropping.Padding preserves all information in the provided image during preprocessing.
  • Prompt ablation: Table 6 ablates in-context examples, documentation, and both components when evaluating second-stage chart-generation results.The first row reports the successful rate of the proposed second-stage data-generation method.
  • Projection layer: Its projection layer replaces a single linear layer with a two-layer MLP, resulting in improved performance.
  • LoRA layer: LoRA layers can achieve performance comparable to full fine-tuning, with LLaVA-1.5 requiring a rank above 128.

B. Dataset Scale

The training and dataset-scale appendix describes pretraining and fine-tuning, visual examples, and standardized prompts for three data-generation stages. Together, these materials characterize the data used to develop ChartLlama.

  • Training stages: Training is divided into pretraining to initialize the vision projector and fine-tuning to align the LLM with instructions.
  • Training stages: Pretraining uses approximately 558k image-caption pairs from LAION-CC-SBU to train the projection layer.
  • Visualization examples: Chart-extraction and chart-to-text visualizations illustrate comparisons with Unichart and LLaVA-1.5.
  • Training stages: Fine-tuning uses 665k instruction-following pairs, supplemented with VQA, OCR, and region-level perception datasets.
  • Generation prompts: Figures 9–11 provide standard prompts for chart-data, chart-figure, and instruction-data generation.Fixed templates are combined with bracketed variables to improve diversity and controllability.

D. Ablation Study on the Conditions of Generation Prompt for ChartLlama

The ablation tests how prompt conditions affect successful chart-figure generation. Including both in-context examples and function documentation substantially improves plotting success.

  • Combining in-context examples with function documentation significantly improves the successful rate of plotting figures.The ablation removes both conditions and retests generation success.

E. Filtering Mechanism

The generation pipeline applies basic checks to filter invalid samples, while acknowledging that some erroneous figures remain difficult to process and screen automatically.

  • Generated samples are retained only when their data format meets the required marker structure and their plotting code executes successfully.Samples with scripts that fail to run are discarded.
  • GPT-4 generates diverse raw tabular data and detailed content descriptions during the chart-data stage.
  • GPT-4 generates chart-drawing code from prompted instructions using Python packages.
  • The authors identify more effective automatic screening as a future way to reduce contamination by poor-quality samples.
  • The pipeline uses five GPT-4 evaluation prompts covering chart-to-text, detailed description, chart-to-chart, text-to-chart, and chart-editing tasks.Each task has distinct scoring criteria and reference information.

G. Comparison with Multi-modal LLMs

Comparisons with multimodal LLMs show that instruction-following training and preservation of language capabilities matter for performance on traditional and newly proposed chart tasks.

  • Traditional Tasks: Some models show low performance because they were not trained on corresponding instruction-following tasks, producing outputs incompatible with the evaluation framework.
  • Traditional Tasks: Qwen-VL underperforms on specially generated charts despite being trained on ChartQA.This comparison highlights a performance gap between ChartQA test splits and the specially generated charts.
  • Traditional Tasks: LLaVA-1.5 improves significantly after training on the designated dataset, despite initially performing poorly.
  • Novel Tasks: Most evaluated models cannot generate executable Python code on the newly proposed tasks, with LLaVA-1.5 as the exception.
  • Novel Tasks: ChartLlama uses fewer training parameters to preserve a degree of generalization ability and basic language capabilities.The authors connect this choice to shorter training, fewer parameters, moderate dataset scale, and pure text data in LLaVA-1.5.

H. More Qualitative Results

Qualitative comparisons show ChartLlama handling chart questions, extraction, and description better than baseline multimodal models, while the evaluation prompts score generated outputs against task-specific references.

  • ChartQA: ChartLlama handles longer chart questions and calculations that challenge Unichart and identifies chart regions that LLaVA-1.5 misses.The examples attribute these differences to instruction following, language understanding, and chart-specific training.
  • Chart Extraction: ChartLlama avoids Unichart’s reversed year order and repetitive trailing characters when converting charts into structured data.
  • Detailed-description evaluation supplies ground-truth descriptions and raw data alongside predictions for GPT-4 scoring.
  • Chart Description: ChartLlama performs best on chart-to-text among Unichart, LLaVA-1.5, and ChartLlama.Unichart produces incorrect values and repetitions, while LLaVA-1.5 suffers from OCR errors and hallucinations.
  • Chart-to-chart evaluation compares predicted Python code with ground-truth code and scores it from 0 to 5.
  • Text-to-chart evaluation provides raw data, instructions, reference code, and predicted code, then returns average and criterion-level scores.
  • Chart-editing evaluation uses original code, editing instructions, and generated code to produce average and criterion-level scores.
Loading 2311.16483v1…