PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA

Zhuowan Li, Bhavan Jasani, Peng Tang, Shabnam Ghadar

arXiv:2403.16385v2cs.CVcs.CL

TL;DR

Chart VQA models struggle with complex reasoning questions, while human-written chart annotations are scarce and costly to collect. The paper trains an LLM data generator to decompose questions into steps and derive answers with tools, then uses the generated data to train chart VQA models. The approach achieves state-of-the-art accuracy on ChartQA and PlotQA, with ChartQA human-written-question accuracy rising from 37.8% to 53.9%.

  • Problem

    Chart VQA models struggle with complex reasoning, while complex human-written questions are costly to collect and scarce in existing datasets.

    Source: §1 · p. 1

  • Method

    An LLM data generator decomposes chart questions into sub-questions, answers them step by step, and combines answers with external tools, using template-generated training data.

    Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 4

  • Results

    The augmented models achieve state-of-the-art accuracy on ChartQA and PlotQA, with relaxed accuracy on ChartQA human-written questions improving from 37.8% to 53.9%.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The results indicate that step-by-step reasoning ability can be distilled into downstream chart VQA models through LLM-generated data.

    Source: §5 · p. 8

    Template-based QA generation relies on rich SVG annotations and cannot generalize to unlabeled images; available annotations may contain missing or incorrect values.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Understanding data visualizations like charts and plots requires reasoning about both visual elements and numerics. Although strong in extractive questions, current chart visual question answering (chart VQA) models suffer on complex reasoning questions. In this work, we address the lack of reasoning ability by data augmentation. We leverage Large Language Models (LLMs), which have shown to have strong reasoning ability, as an automatic data annotator that generates question-answer annotations for chart images. The key innovation in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data generator learns to decompose the complex question into step-by-step sub-questions (rationales), which are then used to derive the final answer using external tools, i.e. Python. This step-wise generation procedure is trained on synthetic data generated using a template-based QA generation pipeline. Experimental results highlight the significance of the proposed step-by-step generation. By training with the LLM-augmented data (LAMENDA), we significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets. In particular, our approach improves the accuracy of the previous state-of-the-art approach from 38% to 54% on the human-written questions in the ChartQA dataset, which needs strong reasoning. We hope our work underscores the potential of synthetic data and encourages further exploration of data augmentation using LLMs for reasoning-heavy tasks.

1. Introduction

Chart VQA reasoning requires combining visual perception with operations such as matching chart legends to bars and calculating values, yet complex human annotations are scarce. The paper uses LLM-generated, step-by-step question-answer data to improve chart VQA, reporting stronger results on challenging human-written questions.

1. Introduction

  • Chart reasoning combines text recognition, matching legends to visual marks, and calculation, as in summing the 2010 Democrat and Republican values.The paper describes this as requiring multimodal perception and reasoning.Source: §1 · p. 1
  • Complex human-written ChartQA questions are scarce per image, and contemporary models struggle with them; about half of reported errors are reasoning failures.Collecting human annotations for chart reasoning is costly.Source: §1 · pp. 1–2
  • Synthesize Step-by-Step has the generator form a question, decompose it into sub-questions, answer them, and combine the answers using tools such as Python.The generator is trained using a template-based corpus of rationales.Source: §1 · p. 2
  • 38% to 54% accuracy on ChartQA human-written questions; the augmented models also achieve state-of-the-art accuracy on ChartQA and PlotQA.The reported gain is on questions requiring complex reasoning.Source: §1 · p. 2

2. Related Work

Related work spans OCR-dependent and OCR-free models for text-heavy images, chart-VQA datasets with different image and question properties, and multimodal LLM methods. The paper situates its approach among tool-using reasoning systems and LLM data generation, emphasizing step-by-step generation with templates and tools.

2. Related Work

  • OCR-dependent models use external OCR as input or a training objective, while recent models such as Donut, Dessurt, and Pix2Struct move away from OCR.MATCHA [35] and DEPLOT [34] tune Pix2Struct for chart-specific tasks.Source: §2 · p. 2
  • ChartQA [40] uses real images and human-written questions, while PlotQA is synthetic but open-vocabulary; earlier synthetic datasets use templatic questions.Earlier datasets DVQA, FigureQA, and LeafQA are described as easy for prior models.Source: §2 · p. 2
  • Multimodal LLM work aligns image features to pretrained LLM inputs using lightweight modules, including projection layers in LLaVA [36] and BLIP-2 [30].Source: §2 · p. 2
  • Prior reasoning methods decompose visual questions or use LLMs to plan tool calls, while LLM data-generation work includes distillation; this paper focuses on step-by-step generation with templates and tools.The related work distinguishes its focus from data-generation approaches that learn from logits or generated data directly.Source: §2 · p. 2

3. Method

The method builds a chart-QA generator whose synthetic outputs train downstream chart VQA models. It combines chart inputs with stepwise executable rationales, tuned using template-generated data.

3.1. Architecture

  • The generator concatenates projected CLIP ViT image features, a DEPLOT-predicted linearized chart table, and a prompt for MPT-7B QA generation.A trainable linear layer projects image features into the LLM input space.Source: §3.1 · p. 3

3.2. Straight-forward data generation

  • Straightforward generation trains on chart question-answer pairs and has the LLM emit questions and answers directly.The generated answers can be noisy, which the step-by-step method is designed to improve.Source: §3.2 · p. 3

3.3. Synthesize step-by-step

  • Synthesize Step-by-Step generates questions with executable rationale programs, then runs their extractive and calculation steps in Python to derive answers.The generator itself can answer extractive steps; this was more effective and simpler than using an off-the-shelf VQA model. A second stage of full-model fine-tuning after projection-layer training did not improve performance.Source: §3.3 · pp. 3–4

3.4. Tuning corpus: template-based QA generation

  • Manual question templates and domain-specific reasoning programs are instantiated against SVG annotations to produce question-answer pairs and stepwise rationales.This pipeline depends on rich SVG annotations and cannot generalize to unlabeled images; available annotations may also contain missing or incorrect values.Source: §3.4 · p. 4

3.5. Evaluation of generated data

  • Generated QA quality is assessed by its downstream chart-VQA accuracy gain, with LAMENDA used to train the MATCHA evaluation model.The pipeline uses SVG-derived template QA to tune the generator, which produces rationale-bearing questions whose answers are executed before the data trains MATCHA.Source: §3.5 · p. 5

4. Experiment

Experiments show that step-by-step LAMENDA data improves chart VQA performance on ChartQA and PlotQA, with gains particularly evident on reasoning questions. Ablations compare generation strategies and question types, while analysis identifies remaining failure cases.

4.1. Set up

  • The study evaluates ChartQA and PlotQA, tuning its generator on 357k template QA pairs and producing additional QAs for training images from both datasets.ChartQA includes human-annotated and machine-augmented questions; PlotQA is a large synthetic dataset.Source: §4.1 · p. 5

4.2. Main results

  • Step-wise answer generation reaches 52.55% accuracy versus 47.76% for direct answer generation, and step-by-step synthesis outperforms direct generation with both VQA-predicted and LLM-generated answers.The authors suggest LLM-generated answers may benefit from access to predicted data tables.Source: §4.2 · p. 6
  • On ChartQA validation, combining step-by-step, template-generated, and additional chart-captioning data reaches 58.65% accuracy, a 10.89% improvement over baseline.The combined-data model also improves more on human-written questions (+15.41%) than augmented extractive questions (+6.36%).Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Comparison with SoTA

  • MATCHA reaches 72.6% relaxed accuracy; on ChartQA human-written questions its relaxed accuracy rises from 38% to 54%, and generated data lifts overall relaxed accuracy 8.58% for Pix2Struct and 8.92% for MATCHA.The comparison reports ChartQA test-split results for models with generated data.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Generalization

  • On PlotQA, MATCHA with generated data achieves 92.89% relaxed accuracy, with gains of 3.46% at low resolution and 0.92% at high resolution.The data generator was trained on ChartQA template questions and applied to PlotQA training images.Source: §4.4 · p. 7, Table 4 · p. 7

4.5. Ablations and analysis

  • Among 960 ChartQA human-written questions, LAMENDA helps 22%, while both models remain wrong on 45%; gains concentrate on reasoning, but generated questions can still be ambiguous or irrelevant.In a sample of helped questions, 70% were reasoning, 17% count, 10% color, and 0% extractive.Source: §4.5 · pp. 7–8, Figure 4 · p. 8
  • Each of seven generated question types improves performance by less than 3%; combining them yields 7.03% improvement, with math calculation contributing the most at 2.76%.The categories include color, spatial reasoning, counting, minimum/maximum, average, comparison, and other math calculations.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusion

The paper introduces Synthesize Step-by-Step, using LLMs to augment chart-reasoning training data by decomposing complex questions into easier sub-questions. On two standard benchmarks, the authors report significant improvements, including higher relaxed accuracy on ChartQA’s human-written questions.

5. Conclusion

  • 53.9% versus 37.8% relaxed accuracy on ChartQA human-written questions is the reported improvement from training with augmented data.The authors report significant improvements on two standard benchmarks.Source: §5 · p. 8
  • Synthesize Step-by-Step uses LLMs to augment chart-reasoning training data by decomposing complex questions into easier sub-questions.The authors describe this step-by-step generation as critical for synthesizing good-quality data.Source: §5 · p. 8
  • The authors present LLM-generated data as a way to distill strong step-by-step reasoning ability into downstream models.Source: §5 · p. 8

A. Data scaling analysis

The data-scaling analysis finds that improvements are largest at the initial 25% of LAMENDA data and continue with smaller margins as more data is used. The analysis uses a shortened 5k-iteration training schedule due to time limitations.

A. Data scaling analysis

  • Improvements are large initially at 25% of LAMENDA data, then continue with smaller margins as the data amount increases.Source: A. Data scaling analysis · p. 12
  • The analysis trains MATCHA with 25%, 50%, 75%, and 100% of LAMENDA data for a shortened 5k iterations due to time limitations.The setting and data match row 5 in Table 2.Source: A. Data scaling analysis · p. 12
  • Table 6 reports results for different amounts of generated data.Source: Table 6 · p. 12

B. Template-based QA generation

Template-generated QA data improves on the baseline but falls below the LLM generator, and its reliance on ground-truth SVG metadata limits generalization to images without SVG. The authors also report that template diversity matters for performance.

B. Template-based QA generation

C. Prompts

The prompts are used to control the LLM-based data generator’s production of questions and answers.

C. Prompts

  • Prompts enable controllable generation of questions and answers by the LLM-based data generator.Source: C. Prompts · p. 12

D. Dataset statistics

The appendix reports dataset statistics for ChartQA, PlotQA, and chart captioning datasets, and says the generator produced QA pairs for images in the captioning datasets.

D. Dataset statistics

E. Additional results

The appendix supplies full results underlying Tables 4 and 5, including relaxed and strict accuracy.

E. Additional results

  • Table 11 gives full results, including relaxed and strict accuracy, for the comparison with state-of-the-art systems on PlotQA test split in Table 4.Source: E. Additional results · p. 12
  • Table 12 gives full results, including relaxed and strict accuracy, for the question-type analysis in Table 5.Source: E. Additional results · p. 12

F. Examples comparing MATCHA with and without LAMENDA

The appendix reports that MATCHA with generated data reaches state-of-the-art performance on PlotQA, and includes a ChartQA question-type analysis and a qualitative example of improved visual-and-arithmetic answering.

F. Examples comparing MATCHA with and without LAMENDA

  • The appendix includes lists of templates and LLM-generator prompts, plus statistics for ChartQA, PlotQA, and chart captioning datasets.Source: Table 7 · p. 13, Table 8 · p. 14, Table 9 · p. 14
  • With generated data, MATCHA achieves state-of-the-art performance on the PlotQA test split.Source: Table 11 · p. 15
  • On ChartQA validation, each question type helps under strict accuracy, and combining all types yields a further gain.Source: Table 12 · p. 15
  • In the illustrated visual-understanding-and-arithmetic question, baseline MatCha fails, while MatCha fine-tuned with the paper’s data answers correctly.Source: Figure 6 · p. 16
Loading 2403.16385v2…