Source-linked AI summary
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Martin Eisenschlos
TL;DR
Visual-language models struggle with complex charts and plots, which require both visual structure understanding and numerical reasoning. MatCha further pretrains Pix2Struct with chart derendering and math-reasoning tasks, achieving strong gains on chart benchmarks and transferring improvements to broader visual-language domains. The paper notes that complex reasoning remains an open challenge and that visual language covers systems beyond those studied.
Problem
Existing vision-language models perform poorly on complex visual-language data such as charts and plots, despite the prevalence of these data.
Method
MatCha further pretrains Pix2Struct using chart derendering, rendered math reasoning, and continued screenshot parsing.
Results
MatCha establishes new state of the art on 5 of 6 chart-domain setups and improves Pix2Struct on tasks beyond charts, including textbook QA and DocVQA.
Takeaways & Limitations
The learned chart and reasoning knowledge transfers beyond the chart domain to broader visual-language benchmarks.
Takeaways & Limitations
Error analysis finds room for improvement on queries requiring complex reasoning, and the value of purely end-to-end math calculation in weight space remains debatable.
Abstract
from arXiv · showhide
Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models' capabilities in jointly modeling charts/plots and language data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks.
1 Introduction
Visual language combines text with complex visual structure, yet existing vision-language models struggle with charts and plots. MatCha addresses this gap by combining chart derendering and mathematical reasoning pretraining, yielding strong benchmark and cross-domain gains.
- Motivation: Visual language integrates text with lines, shapes, colors, orientations, scales, angles, and spatial relationships that support information extraction and reasoning.Charts, plots, and diagrams are common examples.
- Motivation: Existing vision-language models perform poorly on ChartQA and PlotQA because they must jointly model language and symbols.Pix2Struct is a stronger image-to-text starting point, so MatCha further pretrains it.
- Approach: MatCha introduces chart derendering for layout understanding and mathematical reasoning for numerical reasoning.The tasks decode chart rendering code or tables and answers to image-rendered math questions.
- Results: MatCha surpasses prior methods on ChartQA and PlotQA, beating the prior no-gold-table state of the art by as much as 20%.It also improves chart-to-text summarization and most evaluated domains beyond charts and plots.
- Contributions: The contributions include pretraining tasks, broad improvements and state-of-the-art results, cross-domain transfer, and ablation analyses.Reported transfer domains include textbook VQA and Widget Captioning.
2 Related Work
Prior visual-language research largely emphasizes natural images or simpler synthetic environments, while real-world charts and documents require more complex extraction and reasoning. Related approaches use OCR-aware objectives, end-to-end pretraining, or reasoning-focused synthetic tasks.
- Research focus: Vision-language research has predominantly focused on natural images, with relatively little attention to complex real-world visual language.Synthetic datasets often use simpler visual systems and relatively naive questions.
- Research focus: Real-world visual-language datasets require substantially more complex information extraction and reasoning than many synthetic benchmarks.This distinction motivates methods specialized for charts and plots.
- OCR-based methods: OCR-aware methods inject recognized text and positions through alignment or prediction objectives, while ChartBERT relies on OCR text and positions.These methods differ from fully end-to-end visual-language pretraining.
- End-to-end methods: Donut, Dessurt, and Pix2Struct are end-to-end pretrained models, with Pix2Struct providing a generic checkpoint for visual-language tasks.MATCHA uses the Pix2Struct architecture and continually pretrains its checkpoint.
- Reasoning pretraining: Reasoning-oriented pretraining adds numerical or program-based tasks because naive language-modeling objectives do not directly teach difficult reasoning skills.Prior work uses templates, synthetic programs, executors, and other synthetic objectives.
- Scope: Visual language overlaps with visually situated language but also includes cases where visual objects and patterns carry most of the meaning.The distinction matters because text-heavy document models do not cover the full visual-language scope.
3 Method
MatCha injects layout understanding and numerical reasoning into Pix2Struct through chart derendering, math reasoning, and continued screenshot parsing. Chart data and code pairs are collected from notebooks and synthetic or converted tables, while math datasets provide complementary reasoning formats.
- Design: The method targets layout understanding and basic mathematical operations through chart derendering and math reasoning pretraining tasks.These tasks are combined with the original Pix2Struct screenshot-parsing objective.
- Chart derendering: Chart derendering trains the model to generate a chart’s underlying data table or rendering code from its image.Code controls layout properties, while tables determine numbers and groupings.
- Chart data: Chart-code pairs come from licensed GitHub IPython notebooks, while chart-table pairs come from converted Wikipedia tables and additional generated sources.The collection process independently accumulates chart-code and chart-table pairs.
- Chart data: The chart-code snippets can be noisy because earlier notebook blocks may contribute relevant content and snippets may include irrelevant code.This is an explicit limitation of the collected supervision.
- Data controls: Pretraining excludes test tables and charts for PlotQA and ChartQA to avoid information leakage.Ablations report that both chart-table sources are useful and that diversity improves performance.
- Math reasoning: Math reasoning addresses operations such as sorting, min/max, and averaging that chart derendering alone does not provide.The model learns these capabilities from textual math datasets rendered as images.
- Math reasoning: MATH supplies categorized synthetic questions, while DROP supplies reading-comprehension questions requiring number extraction and computation.The two datasets are described as complementarily helpful.
- Joint pretraining: Continued screenshot parsing is retained to prevent catastrophic forgetting, and the final objective mixes all pretraining tasks.The retained task predicts simplified HTML from partially masked website screenshots.
4 Experiment
The experiments evaluate MATCHA across chart, plot, summarization, and broader visual-language tasks using mixed pretraining, task-specific fine-tuning, and established metrics. MATCHA achieves strong chart-domain results and transfers improvements to Pix2Struct tasks beyond charts.
- Experimental Setups: The pretraining mixture combines math reasoning, chart derendering, and screenshot parsing, with chart-to-code downweighted because its noisy data caused training instability.The overall mixture is 40% math reasoning, 40% chart derendering, and 20% screenshot parsing.
- Experimental Setups: Evaluation covers ChartQA, PlotQA, Chart-to-Text, and additional Pix2Struct tasks spanning documents, user interfaces, and other visual-language domains.ChartQA and PlotQA include subsets differing in extractive difficulty and numerical reasoning requirements.
- Main Results: MATCHA beats Pix2Struct by 8.2% on ChartQA without gold data tables and outperforms all models on PlotQA v2, where questions require numerical reasoning.On PlotQA v1, VL-T5 with an underlying data table performs about 4% better than MATCHA.
- Main Results: MATCHA improves on Pix2Struct for both Chart-to-Text sets and achieves state-of-the-art performance on Pew, while underperforming PaLI-17B on Statista.The Chart-to-Text metric is BLEU4.
- Main Results: Across all datasets, MATCHA outperforms the strongest baseline without gold-table access, Pix2Struct, by approximately 10% on average.The comparison covers the chart and plot-domain benchmarks summarized in the main results.
- Results on Pix2Struct Tasks: MATCHA transfers beyond charts, outperforming Pix2Struct by 2.3% on average across Pix2Struct tasks and by 1.6% when ChartQA is excluded.Improvements are reported for AI2D, Widget Captioning, and DocVQA among other tasks.
5 Analyses and Discussions
The analyses isolate how MATCHA’s pretraining components affect ChartQA and identify strengths in extraction and reasoning alongside persistent numerical and attribute-related weaknesses.
- Ablation Study: Removing chart derendering lowers average performance by ≈4%, compared with 2.4% for math reasoning and 1.6% for screenshot parsing.Chart derendering matters more on the augmented set, while math reasoning matters more on the human set.
- Ablation Study: DROP removal decreases performance by 1.7%, versus 0.5% for MATH removal, suggesting DROP’s reading-comprehension format better matches visual-language QA.DROP jointly requires information extraction and numerical reasoning.
- Ablation Study: Removing real-world chart-table pairs drops overall performance by 2%, with nearly 3% lost on the human set versus 1.2% on the augmented set.The findings indicate synthetic data handles extractive questions better, while diverse real-world layouts benefit reasoning.
- Ablation Study: Removing chart-to-code pretraining reduces ChartQA performance by 1.1%, mainly on the human set where reasoning is more complex.Chart-to-code pretraining supplies information about visual layout and attributes beyond the data table.
- Fine-grained Analysis and Error Analysis: MATCHA improves Pix2Struct in every analyzed ChartQA category and beats PaLI in data extraction and math reasoning, but lags PaLI on plot attributes.Plot attributes cover less than 10% of examples, limiting their effect on overall ChartQA performance.
- Fine-grained Analysis and Error Analysis: After excluding annotation errors, 48.3% of MATCHA’s errors involve math reasoning, 43.4% data extraction, and 8.0% plot attributes.Sophisticated reasoning and high-precision numerical computation remain especially difficult.
- Fine-grained Analysis and Error Analysis: MATCHA correctly handles examples requiring averages and complex coreference resolution, while all models fail an example requiring very accurate numerical computation.The case studies contrast successful multi-step reasoning with a precision-sensitive failure.
- Ablation Study: Continuing Pix2Struct pretraining improves ChartQA from 56.0 to 57.0, whereas MATCHA pretraining improves it from 56.0 to 64.2.The comparison separates the effect of continued original-objective training from MATCHA’s additional components.
6 Conclusion
MATCHA enhances an image-to-text transformer with chart understanding and reasoning pretraining. It achieves state-of-the-art results across chart tasks and improves transfer to visual-language tasks beyond charts.
- Conclusion: MATCHA learns to predict chart data tables and rendering code while decoding answers to image-rendered math questions.The method injects chart understanding and numerical reasoning into an image-to-text transformer.
- Conclusion: MATCHA establishes new state-of-the-art results on 5 of 6 setups across three chart-domain benchmarks covering QA and summarization.The evaluated benchmarks include chart tasks in both question answering and summarization.
- Conclusion: MATCHA improves upon Pix2Struct on visual-language tasks beyond charts, including textbook QA and DocVQA.This indicates transfer of learned knowledge outside the pretraining domain.
- Conclusion: Ablations find chart derendering essential for extractive questions and math pretraining important for queries requiring complex reasoning.The component effects differ according to the type of downstream question.
Limitations
The paper identifies limitations in complex mathematical reasoning, plot-attribute grounding, experimental reproducibility, and the broad scope of visual language.
- MATCHA still has room for improvement on queries requiring complex reasoning.
- MATCHA underperforms PaLI on plot attributes, possibly because it lacks massive-scale grounded image-text pretraining with rich semantics.Chart-to-code pretraining offers only limited plot-attribute grounding because many features use plotting-package defaults rather than explicit code.
- Reported experimental numbers come from a single run, although the authors acknowledge that multiple runs would strengthen the evaluation.The authors cite compute cost as a barrier, despite collecting data across multiple scenarios to assess robustness.
- Visual language is an umbrella term that includes systems beyond the chart and plot domains studied here, such as comics and manga.
Ethics Statement
The authors state that MATCHA was not trained on sensitive private information and that its data were synthetically created or publicly available under permissive licenses.
- MATCHA uses synthetic or publicly available web data and was not trained on sensitive private information.The authors characterize the model as low risk for generating harmful content.
A More Details on Datasets Used
The datasets include web-derived chart-table pairs and selected synthetic math questions targeting arithmetic and comparison-based numerical reasoning.
- Chart-table pairs were collected from Statista, Pew, Our World in Data, and OECD.
- The math-question data exclude overly complex questions and focus on basic arithmetic and comparison modules.
- Arithmetic modules include addition, subtraction, division, multiplication, mixed operations, and their multiple-operation variants.
- Comparison modules include closest-value, kth-biggest, pairwise, and sorting tasks, including composed variants.
B Details of Baselines
The baseline suite includes table-to-text and visual table models, with OCR variants used when gold chart tables are unavailable.
- T5 encodes a linearized table and optional query to decode an answer or summary.When available, the gold table serves as input.
- VisionTaPas extends TaPas with visual input through a ViT model and cross-modal fusion layers.
- T5-OCR, VL-T5-OCR, and VisionTaPas-OCR replace gold tables with tables extracted from chart images by OCR.
- The reported baseline performance numbers come from prior work by Masry et al. and Kantharaj et al.