PaperlayerView exact revision on arXiv

Source-linked AI summary

Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs

Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, Abhanshu Sharma

arXiv:2403.12596v1cs.CL

TL;DR

Smaller VLMs remain limited in complex reasoning, motivating the transfer of reasoning capabilities from LLMs. The paper combines improved chart representations with synthetic reasoning data and multitask fine-tuning, achieving state-of-the-art ChartQA performance against a model 10x larger. Its training-data synthesis depends on chart-table representations.

  • Problem

    Complex reasoning remains limited in smaller VLMs, despite advances in LLM reasoning capabilities.

    Source: §1 · p. 1

  • Method

    The approach improves learned chart representations and fine-tunes VLMs on synthetic reasoning traces using a multitask setup.

    Source: §1 · pp. 1–2

  • Results

    The method significantly improves PaLI-3 and achieves state-of-the-art ChartQA performance against PaLI-X, which has 10x more parameters.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The authors conclude that synthetic data generation with multitask training transfers reasoning capabilities from larger LLMs to smaller VLMs.

    Source: §8 · p. 9

    Synthetic-data generation requires chart tables, and replacing gold tables with inferred tables or OCR output may affect final model quality.

    Source: §9 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) are achieving increasingly strong performance on multimodal tasks. However, reasoning capabilities remain limited particularly for smaller VLMs, while those of large-language models (LLMs) have seen numerous improvements. We propose a technique to transfer capabilities from LLMs to VLMs. On the recently introduced ChartQA, our method obtains state-of-the-art performance when applied on the PaLI3-5B VLM by \citet{chen2023pali3}, while also enabling much better performance on PlotQA and FigureQA. We first improve the chart representation by continuing the pre-training stage using an improved version of the chart-to-table translation task by \citet{liu2023deplot}. We then propose constructing a 20x larger dataset than the original training set. To improve general reasoning capabilities and improve numerical operations, we synthesize reasoning traces using the table representation of charts. Lastly, our model is fine-tuned using the multitask loss introduced by \citet{hsieh2023distilling}. Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B without using an upstream OCR system, while keeping inference time constant compared to the PaLI3-5B baseline. When rationales are further refined with a simple program-of-thought prompt \cite{chen2023program}, our model outperforms the recently introduced Gemini Ultra and GPT-4V.

1 Introduction

Chart question answering requires linking visual chart properties with textual content, and complex reasoning remains limited in smaller VLMs. The paper proposes better chart representations and fine-tuning on synthetic reasoning traces generated by more capable LLMs.

  • The approach combines better learned image representations with fine-tuning on synthetic datasets containing reasoning traces generated by more capable LLMs.Source: §1 · pp. 1–2
  • The paper also explores an online setup for numerical reasoning refinements.Source: §1 · p. 1
  • ChartQA questions can require extracting information, grouping or computing intermediate quantities, and reasoning with the results.Source: §1 · p. 1
  • The paper applies a rationale-generation multitask framework to multimodal tasks, unlike standard distillation that predicts a rationale before the answer.Source: §1 · p. 2
  • The recipe achieves state-of-the-art ChartQA performance with significantly improved PaLI-3 performance and 10x fewer parameters than prior work.Source: §1 · p. 2

2 Related Work

The related work positions the approach among VLM architectures and chart-understanding methods, including chart-to-table pipelines, OCR-assisted systems, and synthetic-data approaches.

VLM landscape

  • PaLI-3 pairs a ViT-3B vision backbone with a UL2-2B language backbone in an encoder-decoder architecture.Source: §2 · p. 2

Existing approaches for chart understanding

  • UniChart also pre-trains a chart-image encoder, while another approach translates charts into tables before querying an LLM.Source: §2 · pp. 2–3

The role of upstream OCR systems

  • OCR systems can extract unstructured chart text, but this work assumes access only to the chart image.Source: §2 · p. 3

Improving chart reasoning with synthetic data

  • Unlike chart-summary generation, this work uses LLMs to create additional QA pairs with rationales alongside programmatic examples.Source: §2 · p. 3

3 Dataset

The dataset section describes ChartQA and its chart tables, then details LLM- and program-generated training examples. The resulting dataset is roughly 20x larger than the original training set.

3.1 Brief description of ChartQA

  • ChartQA combines a human set with a more simplistic machine-generated augmented set, and charts come from four sources with tables gold except for inferred Pew tables.Source: §3.1 · p. 3

3.2 Synthetic Generation Methods

  • LLMs generate rationales from chart tables, questions, and answers, while programmatic methods generate arithmetic QA pairs; the added data emphasizes diverse questions requiring multiple quantities.Source: §3.2 · p. 3
  • PaLM 2-S produces rationales for training examples using four-shot prompts, and grounding justifications in answers typically reduces hallucinations.Source: §3.2 · p. 3
  • ExtraQA uses a one-shot prompt to add examples targeting validation-set error types, with only lightweight structure filtering and no verification for hallucinations or fluency.Source: §3.2 · pp. 3–4
  • ArithmeticQA uses templates and fixed operations such as median, maximum, and minimum to generate numerical or comparative questions with rationales, using ChartQA human examples and their tables.Source: §3.2 · p. 4

3.3 Resulting Dataset

  • The resulting dataset is roughly 20x larger than the original training set.Source: §3.3 · p. 4
  • Using examples from both PaLM 2-S and PaLM 2-L improves performance, but ablations do not show either model is better; the role of sampling strategies remains uninvestigated.Source: §3.3 · p. 4

4 Method

The method adapts PaLI-3 by improving chart representations through chart-to-table pretraining, then fine-tuning with synthetic data and a multi-task answer-and-rationale objective.

4.1 Pre-training: Chart2Table Mixture

  • The chart-to-table mixture continues pretraining with the ViT unfrozen, combining synthetic charts with paired chart-table examples from several existing datasets.Sources include chart-table pairs from DVQA, TaTA, and the Benetech challenge, alongside other chart-to-table data.Source: §4.1 · pp. 4–5, Table 3 · p. 5
  • The synthetic dataset is roughly 20x larger than the original dataset.Table 2 describes the synthetic dataset overview.Source: Table 2 · p. 5

4.2 Fine-tuning: Multi-task Loss

  • Fine-tuning treats answers and rationales as independent, separately prefixed tasks and balances their losses with λ.Predicting rationale followed by answer instead increases inference time and gives rationale tokens equal importance to answer tokens during training.Source: §4.2 · p. 5
  • The first multimodal application of this setup keeps inference time constant while improving quality.The authors report both outcomes for their experiments.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate ChartPaLI-5B’s training recipe on ChartQA, FigureQA, and PlotQA, then examine remaining reasoning errors and program-of-thought refinement. ChartPaLI-5B achieves state-of-the-art performance among fine-tuned VLMs on ChartQA and outperforms PaLI-3 across the tested regimes on FigureQA and PlotQA.

5.1 Setup

  • The recipe continues pre-training PaLI-3 with ViT unfrozen, then freezes the ViT encoder for fine-tuning on synthetic ChartQA data.Pre-training uses 6K steps; fine-tuning uses 5K steps.Source: §5.1 · pp. 5–6

5.2 Results on ChartQA

  • Chart-to-table pre-training with the Chart2Table mixture improves PaLI-3’s chart representation and slightly increases its ChartQA performance.The reported improvement also persists as the dataset scales.Source: §5.2 · p. 6, Table 4 · p. 6
  • Rationales improve quality in the multitask setup, particularly on human-written questions, while their singletask effect is not significant.The authors hypothesize that multitask training guides internal reasoning without requiring rationale-token prediction at inference.Source: §5.2 · p. 6, Table 5 · p. 6
  • More synthetic QA pairs improve over original ChartQA performance, and rationales continue to improve quality in both setups; at high data volumes, singletask and multitask differences are no longer significant.The increase is predominantly in the augmented set, which is also synthetically constructed.Source: §5.2 · p. 6

5.3 Results on FigureQA and PlotQA

  • ChartPaLI-5B outperforms PaLI-3 in all tested operation regimes; results include roughly 96%+ on FigureQA and roughly +47.1% on PlotQA v2 at convergence.Evaluation covers zero-shot, quick adaptation, and convergence; PlotQA zero-shot is excluded because training-subset images occur in pre-training.Source: §5.3 · p. 7

5.4 Errors and Challenges

  • Remaining challenges include extracting and operating on intermediate quantities, incorrect arithmetic, color reasoning, multi-value condition checks, and question-prefix outputs that can resemble rationales.The synthetic data lacks color metadata, and the authors describe arithmetic computation as frequently incorrect despite correct value extraction or inference.Source: §5.4 · pp. 7–8

5.2 Results on ChartQA

  • ChartPaLI-5B achieves state-of-the-art performance among fine-tuned VLMs on ChartQA.The best-performing fine-tuned setup is named ChartPaLI-5B.Source: §5.2 · p. 7, Table 7 · p. 8

5.5 Refinement with Program of Thoughts

  • Program-of-thought refinement uses PaLM 2-S to turn arithmetic in rationales into code, with self-consistency over five sampled rationales improving results particularly on the human set.The reported refinement does not affect the augmented set; the authors suggest its arithmetic may be absent or simple enough for the fine-tuned VLM.Source: §5.5 · pp. 8–9

6 Performance Overview

The method significantly outperforms prior models specialized on ChartQA. With rationale generation, it matches Gemini Ultra and outperforms previous approaches.

6 Performance Overview

  • The authors evaluated Fuyu-8B on ChartQA themselves because its authors had not reported benchmark results.Other prior-work results were extracted from the referenced papers.Source: §6 · p. 9
  • Our method significantly outperforms prior models specialized on ChartQA.The authors also note that ChartLlama-13B concurrently uses synthetic data, but with a different approach.Source: §6 · p. 9
  • With rationale generation, the method matches Gemini Ultra and outperforms previous approaches.VLM-generated rationales provide a text representation of the chart conditioned on the question for larger LLMs to operate on.Source: §6 · p. 9

7 Future Work

The authors identify gaps in the training mixtures and in the coverage of complex reasoning examples as areas for future work.

7 Future Work

  • The training mixtures lack examples that use color to construct reasoning examples.The authors suggest bootstrapping color-related examples by having a smaller model generate extraction questions and then combining them.Source: §7 · p. 9
  • Very complex reasoning examples are limited, especially those requiring semantic chart-element identification and numerical computation.Source: §7 · p. 9

8 Conclusion

The proposed recipe improves VLM reasoning capabilities, and its application to PaLI-3 establishes a new ChartQA state of the art. Refining predicted rationales with program-of-thoughts further improves the reported comparison with larger models.

8 Conclusion

  • On ChartQA, the method applied to PaLI-3 significantly outperforms the 10x larger PaLI-X and establishes a new state of the art.Source: §8 · p. 9
  • The composite solution with program-of-thought rationale refinement outperforms Gemini Ultra and GPT-4V on ChartQA.The refinement uses PaLM 2-S.Source: §8 · p. 9
  • Synthetic data generation coupled with a multitask setup transfers reasoning capabilities from larger LLMs to smaller VLMs.The authors also report that the pre-training stage improves downstream performance.Source: §8 · p. 9

9 Limitations

The authors flag dependencies on chart tables and PaLI-3, along with risks and limits in synthetic-data generation and reasoning coverage. The appendix describes the prompts and generated examples used in the approach.

9 Limitations

  • Synthetic data generation requires chart tables, and replacing gold tables with inferred tables or OCR output is likely to affect final model quality.The final model itself works on pixels only.Source: §9 · p. 9
  • PaLI-3 is proprietary, although the authors say the recipe and training methodology should apply broadly to open-source models.They regard an externally available open-source VLM as a better choice.Source: §9 · p. 9
  • LLM-based synthetic dataset construction carries hallucination risks, and the authors cannot guarantee that trained model outputs cannot be abused.They call for additional care when releasing models or datasets openly.Source: §9 · p. 9
  • Prompt creation relied on human inspection of model errors, while the LLM capabilities used for synthetic data still have limitations.Source: §9 · p. 9

A Prompts for PaLM-2

  • ChartQA-Rationale-S uses PaLM 2-S to predict rationales from the table, answer, and question; ChartQA-ExtraQAR-S/L use PaLM 2-S/L with an extended one-shot prompt.Source: A Prompts for PaLM-2 · p. 12
  • The program-of-thought prompt was manually built by inspecting validation errors in which the VLM computed numeric values incorrectly.Source: A Prompts for PaLM-2 · p. 12

B Generated Examples

  • A generated example uses only the table representation to produce a question, answer, rationale, and three to five additional questions.Source: B Generated Examples · p. 12
  • For mean questions, a function processes selected values from a markdown table and outputs the elements, including a reasoning trace for computing the mean.Source: B Generated Examples · p. 12

C Model Outputs

The examples show both successful value extraction and handling of quantities, alongside limitations in color matching, complex comparisons, and enumerating values. They also include cases where answers and rationales disagree or contain arithmetic errors.

  • The analysis highlights examples of intermediate-value extraction, quantity handling, and missing-value inference, while also flagging weak color matching and difficulty with equality and complex enumeration.Source: C Model Outputs · p. 12, Figure 10 · p. 14, Figure 11 · p. 15
  • In one example, the rationale subtracts 648 Other religions admissions from 2778 Muslim admissions and gives the answer 2130.Source: C Model Outputs · p. 14
  • For the brightest yellow bar question, the baseline answer is 56 and ours is 22, while the rationale identifies a maximum of 18.Source: C Model Outputs · p. 14
  • The wage example's answer is marked inexact but correct within 5% of ground truth, although its rationale reports a difference of 16815 for 44690 minus 38370.Source: C Model Outputs · p. 15, Figure 15 · p. 15
  • For the Poland comparison, the answer is Malta, but the rationale instead names Cyprus and says its difference from Poland is 0.3.Source: C Model Outputs · p. 15

D Synthetic Dataset: Statistics

The synthetic examples cover Data Retrieval and Compositional questions, but not Visual or Visual and Compositional questions. The reported counts are organized by chart type and source, and the chart images remain the original ChartQA images.

Loading 2403.12596v1…