PaperlayerView exact revision on arXiv

Source-linked AI summary

MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, Dong Yu

arXiv:2311.10774v2cs.CLcs.AI

TL;DR

LMMs remain limited on chart understanding because charts contain abstract, data-specific structures that differ from natural scenes. The paper introduces MMC-Instruction, trains MMCA, and proposes MMC-Benchmark; MMCA outperforms existing open-source methods, while the benchmark remains challenging even for GPT-4V.

  • Problem

    Chart understanding remains difficult for LMMs because charts contain abstract elements and require interpreting graph types, axes, data points, patterns, and trends.

  • Method

    The paper constructs the 600k-instance MMC-Instruction dataset, instruction-tunes MMCA on diverse chart tasks, and introduces a human-annotated nine-task MMC-Benchmark.

  • Results

    MMCA outperforms existing open-source state-of-the-art methods, while MMC-Benchmark exposes substantial chart-understanding challenges for existing LMMs, including GPT-4V.

  • Takeaways & Limitations

    MMC-Instruction and MMC-Benchmark provide an instruction-tuning and evaluation framework for advancing multimodal chart understanding.

  • Takeaways & Limitations

    The study uses a 7-billion-parameter model because limited computing resources prevent evaluating larger variants such as 13-billion-parameter models.

Abstract

from arXiv · show

With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chart image understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal Chart Instruction (\textbf{MMC-Instruction}) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we develop MultiModal Chart Assistant (\textbf{MMCA}), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the need for a comprehensive evaluation of LMM chart understanding, we also propose a MultiModal Chart Benchmark (\textbf{MMC-Benchmark}), a comprehensive human-annotated benchmark with nine distinct tasks evaluating reasoning capabilities over charts. Extensive experiments on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the most recent GPT-4V model. Our work provides an instruction-tuning methodology and benchmark to advance multimodal understanding of charts. Code and data are available at https://github.com/FuxiaoLiu/MMC.

1 Introduction

Chart understanding remains difficult for LMMs because charts encode abstract, data-specific elements unlike natural scenes. The paper addresses this gap with MMC-Instruction, MMCA, and MMC-Benchmark.

  • Chart images contain abstract elements such as trend lines and color-coded legends, making them unlike natural scene images.
  • Existing open-source LMMs often struggle with graph types, axis labels, data points, patterns, and trends.
  • 600k chart-text alignment and instruction-tuning instances form the MMC-Instruction dataset, covering diverse styles, tasks, and chart types.
  • MMCA uses unified instruction tuning on MMC-Instruction and achieves state-of-the-art performance on chart question-answering benchmarks.
  • MMC-Benchmark provides human-annotated evaluation across nine chart-understanding tasks and two quantitative evaluation formats.
  • Experiments show that MMC-Benchmark challenges existing LMMs, including GPT-4V, particularly on chart-to-datatable and chart-to-JSON tasks.

2 Related Work

Related work spans general multimodal LMMs and chart-text understanding. Prior chart datasets and models often depend on synthetic templates or task-specific fine-tuning.

  • Multimodal research includes tool-selection systems using ChatGPT and end-to-end models that train LLMs for visual task completion.
  • Chart-text models train high-resolution image encoders for text recognition but require downstream-specific fine-tuning rather than open-domain multitask understanding.
  • Earlier chart datasets primarily use synthetic data with template-generated questions and answers.

3 MMC-Instruction

MMC-Instruction combines chart-text alignment data with GPT-4-generated instruction-tuning data across diverse chart tasks. Quality control filters and expert checks are applied to the resulting corpus.

  • 3.1 Chart-Text Alignment Data: The alignment corpus covers charts from academic and public sources across topics including computer science, business, health, and biology.
  • 3.1 Chart-Text Alignment Data: The scientific corpus uses arXiv source files and filters out unparsable files and captions with insufficient text.
  • 3.1 Chart-Text Alignment Data: 210k chart-text pairs are collected from scientific chart-caption data and selected public datasets.
  • 3.2 Chart Instruction-Tuning Data: 200k instruction-tuning instances are generated by prompting GPT-4 from chart descriptions in a Human-question and AI-answer format.
  • 3.2 Chart Instruction-Tuning Data: The instruction data includes chart information extraction, chart reasoning, scientific chart understanding, chart-to-datatable, and chart-to-JSON tasks.
  • 3.2 Chart Instruction-Tuning Data: Scientific chart understanding combines paper abstracts, titles, captions, and figure-related paragraphs to provide textual context.
  • 3.2 Chart Instruction-Tuning Data: Chart-to-datatable and chart-to-JSON transform visual chart information into structured tabular or JSON formats.
  • 3.2 Chart Instruction-Tuning Data: Quality control removes overly long or unsuitable instances, including chart-to-JSON examples lacking a title key, and samples 500 instances for expert checking.

4 MMC-Benchmark

MMC-Benchmark addresses limited evaluation of open-ended chart understanding with a human-annotated, diverse benchmark. It combines nine tasks with GPT-4-assisted generation evaluation and multiple-choice evaluation.

  • Existing chart benchmarks often do not evaluate open-ended questions across unbounded chart types.
  • MMC-Benchmark is presented as the first human-annotated benchmark for evaluating LMM chart comprehension.
  • The benchmark covers nine tasks, including extraction, reasoning, contextual and multiple-chart understanding, classification, stock analysis, chart-to-datatable, and chart-to-JSON.
  • Its two evaluation methods are GPT-4-based free-format generation assessment and GPT-4-independent multiple-choice question answering.
  • The benchmark uses manually constructed instruction-answer pairs and includes scientific-chart sources for contextual and multiple-chart understanding.
  • GPT-4-assisted evaluation reaches 0.90 Cohen’s kappa agreement with human evaluation on 300 sampled test instances.

5 MultiModal Chart Assistant (MMCA)

MMCA is built on mPLUG-Owl and continuously trained in two stages using chart-text alignment and chart instruction-tuning data. The stages separately adapt visual representations and instruction-following capabilities for chart understanding.

  • Architecture: MMCA fine-tunes mPLUG-Owl 7B on MMC-Instruction’s chart-text alignment and chart instruction-tuning data.The architecture combines a CLIP vision encoder, visual abstractor, and Vicuna language model.
  • Stage 1: Chart Text Alignment: Stage 1 freezes the language decoder and trains visual components on chart-text alignment data to map chart features into the LLM’s word-embedding space.This stage runs for one epoch.
  • Stage 2: Chart Instruction Tuning: Stage 2 freezes the visual encoder and abstractor, then applies LoRA to the language model using chart instruction-tuning data.This stage runs for three epochs and targets instruction following for chart understanding.
  • Training Overview: The overall MMCA architecture is continuously trained in two stages.The two-stage training structure is depicted in Figure 2.

6 Experiments

Experiments evaluate MMCA against open-source LMMs, GPT-4V, and non-LLM systems on MMC-Benchmark and public benchmarks. MMCA leads open-source models across all nine MMC-Benchmark tasks, while remaining weaknesses center on text-rich and multi-image reasoning.

  • MMC-Benchmark: MMCA performs better than existing open-source models on all nine MMC-Benchmark tasks.The benchmark reports both GPT-4-evaluated free-format generation and multiple-choice understanding results.
  • Error Patterns: Current LMMs understand cross-modality relationships better than text-layout information, with weaknesses attributed to text recognition, scientific knowledge, and mathematical reasoning.Strong image text recognition alone does not guarantee high MMC-Benchmark performance.
  • Error Patterns: Chart-to-datatable and chart-to-JSON tasks are especially difficult because one missing data value makes a prediction incorrect.The authors speculate that these tasks require strong OCR to output every chart value correctly.
  • Error Patterns: Multiple Chart Understanding performs worse overall than Contextual Chart Understanding, possibly because training data with multiple input images is limited.This is presented as an attributed phenomenon rather than a demonstrated causal result.
  • Public Benchmarks: MMCA outperforms existing LMMs on ChartQA, DocVQA, and TextVQA, including methods fine-tuned on the corresponding datasets.MMCA achieves this without fine-tuning on those corresponding datasets, while Pix2Struct and Donut are compared as OCR-free methods.
  • GPT-4V Evaluation: GPT-4V surpasses open-source LMMs on seven of nine MMC-Benchmark tasks but performs poorly on Chart to Datatable and Chart to Json.The reported failure involves misrecognizing chart data values.
  • GPT-4V Error Analysis: An expert analysis of 100 randomly sampled GPT-4V errors identifies language bias as one root cause, accounting for 35% of analyzed errors.The cited analysis describes language priors or parametric memory overriding chart information.

7 Conclusion

The paper addresses chart understanding in LMMs through a large-scale instruction-tuning dataset, a human-annotated benchmark, and the MMCA model. MMCA outperforms existing open-source state-of-the-art methods.

  • Conclusion: The paper introduces MMC-Instruction, a diverse chart instruction-tuning dataset with topics, language styles, chart types, and open-ended answers.The dataset is described as large-scale and designed for chart understanding.
  • Conclusion: The paper introduces MMC-Benchmark, a human-annotated benchmark for quantitative evaluation of LMM chart-understanding abilities.The benchmark is presented as part of the paper’s comprehensive evaluation approach.
  • Conclusion: The paper proposes MMCA, an instruction-tuned LMM that outperforms existing open-source state-of-the-art methods.MMCA is the model contribution built around the proposed chart-understanding resources.

8 Limitations

The study’s scope is constrained by using a 7-billion-parameter multimodal model and current computational resources. The authors identify larger models as future work.

  • Limitations: The study uses a 7-billion-parameter multimodal model because access to higher-end computing resources is limited.The authors note that 13-billion-parameter variants could further enhance the findings.

9 Ethical Considerations

The authors require strict copyright, licensing, privacy, and ethical compliance in dataset construction and annotation.

  • Sources prohibiting copying or redistribution are explicitly excluded from the data.
  • Annotators must avoid collecting questions containing private information while complying with privacy laws and ethical standards.

A.1 MMC-Benchmark

MMC-Benchmark evaluates chart understanding across diverse tasks using both GPT-4-based generation assessment and rule-based multiple-choice evaluation. Experiments show broad weaknesses in existing models, with structured chart-to-data tasks especially difficult and GPT-4V affected by perception and language-bias errors.

  • Evaluation: Generation Ability Evaluation uses GPT-4 to assess predictions against questions and reference answers, while multiple-choice evaluation uses rule-based answer extraction.Regular expressions and response-processing workflows extract numbers and conclusion phrases from long responses for answer matching.
  • Benchmark design: MMC-Benchmark includes diverse chart topics, plot types, and examples spanning nine chart-understanding tasks.The listed tasks include information extraction, reasoning, contextual understanding, topic and type classification, stock analysis, multiple-chart understanding, chart-to-datatable, and chart-to-json.
  • GPT-4V error analysis: GPT-4V errors comprise Perception Error (39%), Language Bias (35%), Reasoning Error (15%), and Lack of Knowledge (11%).Perception errors involve misinterpreting visual information, whereas language bias reflects strong language priors conflicting with chart content.
  • Open-source-model error analysis: Open-source-model errors prominently include Not Following Instructions (27%) and a weak vision encoder (29.6%).The analysis attributes weak vision encoders partly to limited modeling of spatial interactions such as trend lines and color-coded legends.
  • Task results: Chart-to-DataTable and Chart-to-Json are extremely difficult for current LMMs, including GPT-4V, because missing any chart value makes the prediction incorrect.MMCA produces more accurate responses in the required output formats than the cited baselines.
  • Benchmark difficulty: Existing models score lower on MMC-Benchmark than on benchmarks such as ChartQA because its questions are more diverse, open-ended, and language-demanding.
Loading 2311.10774v2…