PaperlayerView exact revision on arXiv

Source-linked AI summary

UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning

Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, Shafiq Joty

arXiv:2305.14761v3cs.CL

TL;DR

Existing pretrained language and vision-language models do not explicitly model chart structure, despite charts’ importance for data analysis and reasoning. UniChart addresses this gap with a chart-specific encoder-decoder model, a 611K-chart corpus, and low- and high-level pretraining tasks. It reports state-of-the-art performance across downstream chart tasks, alongside improved efficiency, while limitations remain in compositional math, factuality, and generalization to unseen web charts.

  • Problem

    Existing pretrained models often focus on language, image-text pairs, or layout rather than chart-specific visual encoding and relationships among chart elements.

  • Method

    UniChart combines a chart image encoder and text decoder, pretrained on over 611K diverse charts with low-level extraction and high-level reasoning objectives.

  • Results

    UniChart achieves state-of-the-art performance across evaluated chart tasks and is more than 11 times faster with 28% fewer parameters than MatCha.

  • Takeaways & Limitations

    Chart-specific pretraining objectives and a diverse chart corpus provide a broad pretrained model for chart comprehension and reasoning tasks.

  • Takeaways & Limitations

    The model struggles with compositional mathematical operations, may hallucinate factual statements, and shows a performance drop on unseen WebCharts styles.

Abstract

from arXiv · show

Charts are very popular for analyzing data, visualizing key insights and answering complex reasoning questions about data. To facilitate chart-based data analysis using natural language, several downstream tasks have been introduced recently such as chart question answering and chart summarization. However, most of the methods that solve these tasks use pretraining on language or vision-language tasks that do not attempt to explicitly model the structure of the charts (e.g., how data is visually encoded and how chart elements are related to each other). To address this, we first build a large corpus of charts covering a wide variety of topics and visual styles. We then present UniChart, a pretrained model for chart comprehension and reasoning. UniChart encodes the relevant text, data, and visual elements of charts and then uses a chart-grounded text decoder to generate the expected output in natural language. We propose several chart-specific pretraining tasks that include: (i) low-level tasks to extract the visual elements (e.g., bars, lines) and data from charts, and (ii) high-level tasks to acquire chart understanding and reasoning skills. We find that pretraining the model on a large corpus with chart-specific low- and high-level tasks followed by finetuning on three down-streaming tasks results in state-of-the-art performance on three downstream tasks.

1 Introduction

UniChart addresses the limited chart-specific structure modeling of existing pretrained models with a chart-focused model, corpus, and pretraining objectives. It achieves state-of-the-art results across chart-related tasks while improving efficiency.

  • Motivation: Existing language and vision-language pretrained models do not explicitly model charts’ graphical marks, text, layout, and communicative goals.Charts encode data through elements such as bars and lines, requiring pattern, trend, outlier, and reasoning capabilities.
  • Approach: UniChart combines a chart image encoder with a text decoder for chart comprehension and reasoning.The decoder generates expected outputs from encoded chart images and task prompts.
  • Approach: The model is pretrained on 611K diverse charts using low-level extraction tasks and high-level understanding and reasoning tasks.The corpus spans varied topics and visual styles, while objectives target visual elements, data, and downstream-oriented skills.
  • Results: UniChart outperforms state-of-the-art models on ChartQA and OpenCQA, performs better on chart summarization, and achieves state-of-the-art Chart-to-Table results.These findings include both human and automatic summarization evaluations.
  • Results: UniChart is more than 11 times faster than MatCha and uses 28% fewer parameters.The reported efficiency improvements accompany the downstream performance results.

2 Related Work

Related work covers vision-language and multimodal document understanding models, chart-specific downstream tasks, and prior chart pretraining. Existing approaches generally lack explicit focus on chart understanding, motivating UniChart’s broader chart-specific objectives.

  • Vision-language Pretraining: Vision-language models commonly encode text with BERT-based methods and images with region or patch-based visual representations.These approaches support general multimodal tasks but may neglect crucial image regions or chart-specific structure.
  • Document Understanding: Multimodal document understanding jointly addresses textual content, layout, and visual elements using encoder-only or encoder-decoder architectures.Encoder-only methods typically rely on OCR and specialized embeddings for layout and visual features.
  • Chart Pretraining: MatCha achieved state-of-the-art chart question answering and summarization but focused mainly on data-table generation and textual reasoning tasks.The paper identifies limited emphasis on visual reasoning and text generation as distinctions from UniChart.
  • Chart Pretraining: UniChart uses a larger corpus and chart-specific objectives spanning visual reasoning and text generation for broader chart-related task coverage.This positioning distinguishes it from prior general image-text pretraining and MatCha.
  • Chart Tasks: Chart-related tasks include factoid and open-ended question answering, chart summarization, and chart-to-table generation.The paper evaluates these tasks because they require interaction between language and vision and have public datasets.

3 Chart Pretraining Corpus

The pretraining corpus combines real-world chart datasets, extracted SVG charts, augmented charts, and image-only charts to cover diverse formats, styles, topics, and data availability. It contains over 611K charts, with most providing underlying data tables.

  • Corpus Construction: The corpus was built from online sources and public datasets, including charts with and without underlying data tables.This construction targets diversity in chart styles, topics, and storage formats.
  • Corpus Construction: Existing datasets, SVG extraction, and data augmentation supplied charts with underlying data tables.SVG elements enabled recovery of chart geometry and data values, while public tables supported chart generation.
  • Corpus Construction: Data augmentation decomposed suitable tables and generated charts with D3 and Vega-Lite using heuristics for chart-type selection.Tables were limited to a maximum of eight rows to fit reasonable screen space.
  • Visual Diversity: Visual diversification varied chart colors, shapes, positions, distances, line types, and legend shapes, producing 189,836 charts.The augmentation process was designed to resemble real-world visual variation.
  • Image-only Charts: Image-only chart datasets remained useful for pretraining because chart elements and textual content could be extracted with object detection and OCR.These datasets included sources with high-quality summaries.
  • Summary Generation: Because many charts lacked informative summaries, the authors generated coherent text with InstructGPT and distilled it into Flan-T5 XL.The process began with 3,700 summaries produced by text-davinci-003 before broader generation.
  • Dataset Analysis: The final corpus contains over 611K charts, with bar charts comprising 58.51%, line charts 32.94%, and pie charts 9.39%.Over 80% have underlying data tables, while about 20% are image-only; topics include technology, economy, politics, health, and society.

4 Method

UniChart uses a chart image encoder and text decoder, trained with chart-specific objectives that recover data and visual elements while developing reasoning and generation abilities. It is evaluated through finetuned and zero-shot downstream tasks.

  • Architecture: UniChart is a unified pretrained model for chart comprehension and reasoning.The method section introduces its architecture and pretraining objectives.
  • Architecture: The chart image encoder interprets textual elements, visual elements, and the layout connecting them.It builds upon Donut’s document image understanding architecture.
  • Architecture: Donut’s OCR-free encoder avoids external OCR modules and improves training time and memory efficiency.Its Swin Transformer image encoder processes chart image features.
  • Pretraining Objectives: Low-level objectives retrieve chart data, including data-table generation and estimation of graphical-mark scales from plot-area proportions.Scale targets are derived from bounding-box heights and rounded to two decimal places.
  • Pretraining Objectives: Chart summarization pretraining improves textual description generation using summaries produced directly or through knowledge distillation from GPT models.This objective supports generating textual descriptions from chart images.
  • Downstream Tasks: The model is finetuned or evaluated across factoid QA, complex QA, chart summarization, and chart-to-table, with additional zero-shot evaluation on WebCharts.ChartQA, OpenCQA, Chart-to-Text, and Chart-to-Table provide the downstream settings.

5 Evaluation

UniChart is evaluated against established vision-language and chart-specific baselines across four chart benchmarks, using automatic and human-centered measures. It generally achieves leading performance, with efficiency advantages and persistent challenges on dense charts, complex arithmetic, and factual correctness.

  • 5.1 Baselines & Evaluation Metrics: UniChart is compared with T5, VL-T5, VisionTapas, Pix2Struct, and MatCha across chart question answering, summarization, and table-generation benchmarks.The evaluation covers ChartQA, Chart-to-Text, OpenCQA, and Chart-to-Table, with WebCharts evaluated zero-shot.
  • 5.2 Main Results: UniChart outperforms MatCha and VL-T5 on ChartQA and Chart-to-Text (Pew), but performs slightly lower on Chart-to-Text (Statista).The advantage is especially prominent for human-written ChartQA questions, which require visual and numerical reasoning.
  • 5.2 Main Results: UniChart achieves a higher BLEU score than VL-T5 on OpenCQA and surpasses MatCha on two datasets, including zero-shot WebCharts.These results are reported as evidence of explanatory-answer generation and generalization across diverse visual styles.
  • 5.4 Ablation Study: Removing pretraining objectives slightly decreases performance, with the largest effect observed when Numerical Reasoning is removed.This ablation identifies numerical reasoning pretraining as particularly important for the model’s numerical abilities.
  • 5.3 Human and ChatGPT Evaluation: Human and ChatGPT evaluations find UniChart summaries more informative and factually accurate than MatCha, with high-level patterns and trends especially prominent.The finetuned model produces fewer factual errors than MatCha and the zero-shot version; ChatGPT ratings are roughly consistent with human scores.
  • 5.6 Time and Memory Efficiency: UniChart is faster than MatCha and uses 201M parameters versus 282M, partly because its sliding-window local attention scales linearly with image size.The speed gap is more evident for tasks requiring long output sequences, such as Chart-to-Text.
  • 5.5 Error Analysis and Challenges: Manual analysis shows difficulties with densely populated charts, complex arithmetic calculations, and factual correctness in generated summaries.The authors suggest higher resolution or model size, external program executors, and further factuality improvements as possible directions.

6 Conclusion

The paper presents UniChart as a general-purpose chart model trained with chart-specific objectives on a large, diverse chart collection. Its evaluations report state-of-the-art performance across four downstream tasks and improved time and memory efficiency, while also identifying opportunities for improvement.

  • 6 Conclusion: UniChart is designed for a broad range of chart-related tasks using chart-specific pretraining tasks and a large, diverse collection of charts and summaries.The paper combines human and ChatGPT evaluations to assess the method.
  • 6 Conclusion: UniChart achieves state-of-the-art results on four downstream tasks while improving time and memory efficiency.The conclusion presents these results as evidence of the model’s effectiveness across chart-related benchmarks.
  • 6 Conclusion: The evaluation identifies opportunities for improvement and positions the model and pretraining data as resources for future chart-understanding research.The authors encourage further exploration of this relatively new area.

Limitations

UniChart achieves state-of-the-art performance on several benchmarks but remains limited by compositional mathematical reasoning, hallucinations, and unresolved training choices.

  • UniChart struggles with questions requiring compositional mathematical operations despite its strong ChartQA performance.
  • The model may hallucinate factually incorrect statements on Chart-to-Text and OpenCQA text-generation tasks.
  • WebCharts performance drops noticeably relative to tasks used for finetuning, motivating more generalizable models for diverse web charts.
  • Limited computing resources prevented investigation of downstream-task hyperparameter tuning and whether further pretraining could improve performance.

Ethics Statement

The authors describe dataset-use permissions and warn that generative outputs could facilitate misinformation or contain harmful content.

  • The collected datasets were used under the respective websites’ stated licenses and attribution requirements.
  • The generative models may be abused to misinform the public through factually incorrect responses.
  • The authors cannot guarantee that generated text will not contain hate speech or harmful content.

A.1 Data Augmentation

The data augmentation process uses D3 and Vegalite to generate visually varied charts resembling real-world web charts.

  • D3 and Vegalite were the primary visualization libraries used during data augmentation.
  • Augmented charts vary colors, element dimensions, shapes, and backgrounds to resemble real-world charts found on the Web.

A.2 Data Augmentation by Knowledge Distillation

UniChart expands chart-summary supervision by generating summaries from chart tables or OCR-extracted text, using instruction-tuned language models and ChatGPT.

  • 3,700 charts with underlying tables were selected because their data could serve as chart representations for language models.
  • InstructGPT generated summaries from a demonstration table-caption pair followed by each target chart’s data table.
  • Flan-T5 XL was finetuned on the 3,700-sample dataset and then generated about 470K summaries for charts lacking associated summaries.
  • ChatGPT generated more elaborate summaries for Statista and Pew Research charts than the original benchmark summaries in most cases.
  • For Pew charts without tables, OCR extracted layout-preserving text from visual elements before ChatGPT generated summaries.

A.3 Dataset Analysis

The paper analyzes the chart corpus, pretraining setup, evaluation criteria, and model behavior on challenging examples. It also reports efficiency comparisons and examines how pretraining objectives affect ChartQA performance.

  • Corpus analysis: The corpus analysis reports linguistic differences across datasets, with PlotQA and PewResearch charts often containing longer textual elements than augmented and Beagle charts.
  • Training details: Pretraining used two image resolutions, beginning with 512x512 for 300K steps and continuing at 960x960 for 100K steps.The model was initialized from base Donut weights to reduce computational resource requirements.
  • Ablation study: Removing Numerical Reasoning caused a substantial ChartQA decline, while removing Data Table Generation produced relatively weak ChartQA performance.These ablations indicate that both numerical abilities and understanding underlying chart tables are important to the benchmark outcome.
  • Evaluation: Human evaluation assessed summary informativeness, factual correctness, and semantic levels covering visual, statistical, perceptual, and context-specific content.The study included 150 random samples from the Chart-to-Text Statista test split.
  • Error analysis: The challenging examples include visual numerical reasoning questions, an overpopulated chart that confused the model, and a factual error in a finetuned UniChart summary.
  • Time efficiency: UniChart showed much faster average inference times than MatCha on ChartQA, Chart-to-Text Pew, and Chart-to-Text Statista samples.Inference time was measured on 10 random samples from each benchmark using CPU experiments on Google Colab.
Loading 2305.14761v3…