MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

• Chart images contain abstract elements such as trend lines and color-coded legends, making them unlike natural scene images.Source: §1 · p. 1
• Existing open-source LMMs often struggle with graph types, axis labels, data points, patterns, and trends.Source: §1 · p. 1
• 600k chart-text alignment and instruction-tuning instances form the MMC-Instruction dataset, covering diverse styles, tasks, and chart types.Source: §1 · p. 1
• MMCA uses unified instruction tuning on MMC-Instruction and achieves state-of-the-art performance on chart question-answering benchmarks.Source: §1 · pp. 2–3
• MMC-Benchmark provides human-annotated evaluation across nine chart-understanding tasks and two quantitative evaluation formats.Source: §1 · p. 2
• Experiments show that MMC-Benchmark challenges existing LMMs, including GPT-4V, particularly on chart-to-datatable and chart-to-JSON tasks.Source: §1 · p. 3

• Multimodal research includes tool-selection systems using ChatGPT and end-to-end models that train LLMs for visual task completion.Source: §2 · p. 3
• Chart-text models train high-resolution image encoders for text recognition but require downstream-specific fine-tuning rather than open-domain multitask understanding.Source: §2 · p. 3
• Earlier chart datasets primarily use synthetic data with template-generated questions and answers.Source: §2 · p. 3

3.1 Chart-Text Alignment Data

• The alignment corpus covers charts from academic and public sources across topics including computer science, business, health, and biology.Source: §3.1 · pp. 3–4
• The scientific corpus uses arXiv source files and filters out unparsable files and captions with insufficient text.Source: §3.1 · p. 3
• 210k chart-text pairs are collected from scientific chart-caption data and selected public datasets.Source: §3.1 · p. 4

3.2 Chart Instruction-Tuning Data

• 200k instruction-tuning instances are generated by prompting GPT-4 from chart descriptions in a Human-question and AI-answer format.Source: §3.2 · p. 4
• The instruction data includes chart information extraction, chart reasoning, scientific chart understanding, chart-to-datatable, and chart-to-JSON tasks.Source: §3.2 · pp. 4–5
• Scientific chart understanding combines paper abstracts, titles, captions, and figure-related paragraphs to provide textual context.Source: §3.2 · pp. 4–5
• Chart-to-datatable and chart-to-JSON transform visual chart information into structured tabular or JSON formats.Source: §3.2 · p. 5
• Quality control removes overly long or unsuitable instances, including chart-to-JSON examples lacking a title key, and samples 500 instances for expert checking.Source: §3.2 · p. 5

• Existing chart benchmarks often do not evaluate open-ended questions across unbounded chart types.Source: §4 · p. 5
• MMC-Benchmark is presented as the first human-annotated benchmark for evaluating LMM chart comprehension.Source: §4 · p. 5
• The benchmark covers nine tasks, including extraction, reasoning, contextual and multiple-chart understanding, classification, stock analysis, chart-to-datatable, and chart-to-JSON.Source: §4 · p. 5
• Its two evaluation methods are GPT-4-based free-format generation assessment and GPT-4-independent multiple-choice question answering.Source: §4 · p. 5, §4.2 · p. 6
• The benchmark uses manually constructed instruction-answer pairs and includes scientific-chart sources for contextual and multiple-chart understanding.Source: §4.1 · p. 5
• GPT-4-assisted evaluation reaches 0.90 Cohen’s kappa agreement with human evaluation on 300 sampled test instances.Source: §4.2 · p. 6

Architecture

• MMCA fine-tunes mPLUG-Owl 7B on MMC-Instruction’s chart-text alignment and chart instruction-tuning data.The architecture combines a CLIP vision encoder, visual abstractor, and Vicuna language model.Source: §5 · p. 6

Stage 1: Chart Text Alignment

• Stage 1 freezes the language decoder and trains visual components on chart-text alignment data to map chart features into the LLM’s word-embedding space.This stage runs for one epoch.Source: §5 · p. 6

Stage 2: Chart Instruction Tuning

• Stage 2 freezes the visual encoder and abstractor, then applies LoRA to the language model using chart instruction-tuning data.This stage runs for three epochs and targets instruction following for chart understanding.Source: §5 · p. 6

Training Overview

• The overall MMCA architecture is continuously trained in two stages.The two-stage training structure is depicted in Figure 2.Source: Figure 2 · p. 6

MMC-Benchmark

• MMCA performs better than existing open-source models on all nine MMC-Benchmark tasks.The benchmark reports both GPT-4-evaluated free-format generation and multiple-choice understanding results.Source: §6.3.1 · p. 6, Table 4 · p. 7, Table 5 · p. 7

Error Patterns

• Current LMMs understand cross-modality relationships better than text-layout information, with weaknesses attributed to text recognition, scientific knowledge, and mathematical reasoning.Strong image text recognition alone does not guarantee high MMC-Benchmark performance.Source: §6.3.1 · pp. 6–7
• Chart-to-datatable and chart-to-JSON tasks are especially difficult because one missing data value makes a prediction incorrect.The authors speculate that these tasks require strong OCR to output every chart value correctly.Source: §6.3.1 · p. 7
• Multiple Chart Understanding performs worse overall than Contextual Chart Understanding, possibly because training data with multiple input images is limited.This is presented as an attributed phenomenon rather than a demonstrated causal result.Source: §6.3.1 · p. 7

Public Benchmarks

• MMCA outperforms existing LMMs on ChartQA, DocVQA, and TextVQA, including methods fine-tuned on the corresponding datasets.MMCA achieves this without fine-tuning on those corresponding datasets, while Pix2Struct and Donut are compared as OCR-free methods.Source: §6.3.2 · p. 7, Table 6 · p. 7

GPT-4V Evaluation

• GPT-4V surpasses open-source LMMs on seven of nine MMC-Benchmark tasks but performs poorly on Chart to Datatable and Chart to Json.The reported failure involves misrecognizing chart data values.Source: §6.3.3 · p. 8

GPT-4V Error Analysis

• An expert analysis of 100 randomly sampled GPT-4V errors identifies language bias as one root cause, accounting for 35% of analyzed errors.The cited analysis describes language priors or parametric memory overriding chart information.Source: §6.3.3 · p. 8, Figure 3 · p. 8

Conclusion

• The paper introduces MMC-Instruction, a diverse chart instruction-tuning dataset with topics, language styles, chart types, and open-ended answers.The dataset is described as large-scale and designed for chart understanding.Source: §7 · p. 9
• The paper introduces MMC-Benchmark, a human-annotated benchmark for quantitative evaluation of LMM chart-understanding abilities.The benchmark is presented as part of the paper’s comprehensive evaluation approach.Source: §7 · p. 9
• The paper proposes MMCA, an instruction-tuned LMM that outperforms existing open-source state-of-the-art methods.MMCA is the model contribution built around the proposed chart-understanding resources.Source: §7 · p. 9

Limitations

• The study uses a 7-billion-parameter multimodal model because access to higher-end computing resources is limited.The authors note that 13-billion-parameter variants could further enhance the findings.Source: §8 · p. 9

• Sources prohibiting copying or redistribution are explicitly excluded from the data.Source: §9 · p. 9
• Annotators must avoid collecting questions containing private information while complying with privacy laws and ethical standards.Source: §9 · p. 9

Evaluation

• Generation Ability Evaluation uses GPT-4 to assess predictions against questions and reference answers, while multiple-choice evaluation uses rule-based answer extraction.Regular expressions and response-processing workflows extract numbers and conclusion phrases from long responses for answer matching.Source: §A.1 · p. 12, §A.2.3 · p. 12

Benchmark design

• MMC-Benchmark includes diverse chart topics, plot types, and examples spanning nine chart-understanding tasks.The listed tasks include information extraction, reasoning, contextual understanding, topic and type classification, stock analysis, multiple-chart understanding, chart-to-datatable, and chart-to-json.Source: §A.1 · p. 12, Figure 6 · p. 13, Figure 7 · p. 14

GPT-4V error analysis

• GPT-4V errors comprise Perception Error (39%), Language Bias (35%), Reasoning Error (15%), and Lack of Knowledge (11%).Perception errors involve misinterpreting visual information, whereas language bias reflects strong language priors conflicting with chart content.Source: §A.2.4 · p. 12

Open-source-model error analysis

• Open-source-model errors prominently include Not Following Instructions (27%) and a weak vision encoder (29.6%).The analysis attributes weak vision encoders partly to limited modeling of spatial interactions such as trend lines and color-coded legends.Source: §A.2.5 · pp. 12–13

Task results

• Chart-to-DataTable and Chart-to-Json are extremely difficult for current LMMs, including GPT-4V, because missing any chart value makes the prediction incorrect.MMCA produces more accurate responses in the required output formats than the cited baselines.Source: §A.2.6 · p. 13

Benchmark difficulty

• Existing models score lower on MMC-Benchmark than on benchmarks such as ChartQA because its questions are more diverse, open-ended, and language-demanding.Source: §A.2.6 · p. 13