PaperlayerView exact revision on arXiv

Source-linked AI summary

SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials

Wonjoong Kim, Sangwu Park, Yeonjun In, Seokwon Han, Chanyoung Park

arXiv:2405.00021v3cs.CVcs.AIcs.CL

TL;DR

Chart question answering calls for logical interpretation, but charts mix useful and irrelevant information that can challenge chart-to-table extraction. SIMPLOT extracts chart essentials and uses chart-oriented prompting; the paper reports improved performance and applicability across tasks and models.

  • Problem

    Chart question answering requires logical interpretation, and irrelevant chart information can challenge chart-to-table extraction.

    Source: §1 · pp. 1–2

  • Method

    SIMPLOT extracts essential chart information for table-based reasoning and uses a human-oriented prompt to guide chart reasoning.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    The paper reports that SIMPLOT improves performance, addresses concerns raised by existing works, and applies to various tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    SIMPLOT is described as model-agnostic and applicable to existing chart-reasoning models that use chart-to-table extraction.

    Source: §1 · p. 2

    SIMPLOT is expected to have limited performance on unseen out-of-distribution data.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recently, interpreting complex charts with logical reasoning has emerged as challenges due to the development of vision-language models. A prior state-of-the-art (SOTA) model has presented an end-to-end method that leverages the vision-language model to convert charts into table format utilizing Large Language Model (LLM) for reasoning. However, unlike natural images, charts contain a mix of essential and irrelevant information required for chart reasoning, and we discover that this characteristic can lower the performance of chart-to-table extraction. In this paper, we introduce SIMPLOT, a method designed to extract only the elements necessary for chart reasoning. The proposed method involves two steps: 1) training to mimic a simple plot that contains only the essential information from a complex chart for table extraction, followed by 2) performing reasoning based on the table. Our model enables accurate chart reasoning without the need for additional annotations or datasets, and its effectiveness is demonstrated through various experiments. Furthermore, we propose a novel prompt mimicking how human interpret charts for more accurate reasoning. Our source code is available at https://github.com/sangwu99/Simplot.

1 Introduction

Chart question answering requires logical interpretation that conventional natural-image VQA does not address, while existing approaches face extraction and reasoning limitations. SIMPLOT targets these problems with essential-information extraction and chart-oriented prompting.

1 Introduction

  • Charts require mathematical reasoning that differs from conventional natural-image VQA, a comparatively less explored area.The introduction notes that models proficient with natural images can struggle with chart formats and logical interpretation.Source: §1 · p. 1
  • Earlier chart-reasoning approaches rely on fixed-format rules or OCR and key-point modules, while end-to-end vision-language methods require downstream-task fine-tuning.The introduction associates OCR and key-point approaches with annotation costs and dependence on module performance.Source: §1 · p. 1
  • Deplot can miss relevant chart text, extract irrelevant details, and overlook visual information unavailable in its extracted table.Examples include missing a counting unit or “Northern America,” extracting “CC BY,” and failing to answer questions about bar order or line color.Source: §1 · p. 2
  • SIMPLOT combines row-column rendering and essential-information filtering to address Deplot’s limitations in chart-to-table extraction.The paper reports that rendering row and column information provides explicit supervision, while simplifying charts improves table-extraction performance.Source: §1 · p. 2
  • SIMPLOT introduces a human-oriented chart instruction prompt intended to guide LMMs toward reasoning that mimics human chart interpretation.The paper also presents guidelines for using chart text in reasoning and says the prompt is designed specifically for chart reasoning.Source: §1 · p. 2
  • Experiments are reported to validate SIMPLOT’s effectiveness, and the method is described as applicable beyond Deplot to other chart-to-table reasoning models.The paper characterizes SIMPLOT as model-agnostic for existing models that use chart-to-table extraction.Source: §1 · p. 2

2 Proposed Method: SIMPLOT

SIMPLOT trains chart-to-table extraction around charts containing only reasoning-relevant information, then uses the extracted table and original chart for LMM reasoning. Its training aligns a student encoder for original charts with a teacher learned from simple charts.

2.1 Preprocessing Stage

  • Offline preprocessing creates a simple chart with essential elements and renders row and column information onto charts to support table extraction.At inference, an LMM extracts the row and column information for rendering.Source: §2.1 · pp. 3–4

2.2 Training Stage: Chart-to-Table Extraction

  • Training has two phases: a teacher encoder and table decoder learn from simple charts, then a student encoder learns from original charts through teacher distillation.The student is trained to map original-chart representations into the representation space of simple charts.Source: §2.2 · p. 4

2.2.1 Phase 1: Learning with Essential Part from Simple Chart

  • Simple charts greatly outperform original charts for chart-to-table extraction when used for both training and inference.Figure 3 compares extraction accuracy for original charts in green and simple charts in red.Source: Figure 3 · p. 4, §2.2.1 · p. 4

2.2.2 Phase 2: Encoding Original Chart into Representation Space of Simple Chart

  • Phase 2 aligns the student’s original-chart representations with corresponding teacher representations from simple charts, using shuffled-value negatives and a triplet loss.A table-generation cross-entropy loss is also included, and the final loss balances the triplet and table losses.Source: §2.2.2 · pp. 4–5

2 Proposed Method: SIMPLOT

  • SIMPLOT uses an LMM to reason from an extracted table and the original chart, combining tabular values with visual information absent from the table.The framework also uses prompts for reasoning.Source: Figure 2 · p. 3, §2.3 · p. 5

2.3 Inference Stage: Reasoning with Extracted Table

  • The inference-stage LMM receives both the extracted table and original chart, so it can address precise values and visual attributes not represented in the table.The proposed Human-oriented chart instruction mimics how people interpret charts to support more accurate reasoning.Source: §2.3 · p. 5

3 Experiments

Experiments evaluate SIMPLOT on chart-to-table extraction and chart question answering, then test its components, applicability, and performance in more challenging settings. SIMPLOT leads among table-based reasoning models, though its relative performance varies across ChartQA subsets.

3 Experiments

  • Experiments use ChartQA and PlotQA; chart QA is evaluated with Relaxed Accuracy, and chart-to-table extraction with the proposed Relative Distance metric.Relative Distance is derived from Relative Mapping Similarity because vanilla RMS does not measure extraction performance accurately in some cases.Source: §3 · p. 6

3.1.1 Results on Chart-to-Table Extraction

  • SIMPLOT outperforms Deplot and UniChart on ChartQA chart-to-table extraction, while GPT-4V performs poorly on line charts whose values must be inferred from images.The authors argue even minor extraction errors in values or row and column names can lead to incorrect answers.Source: §3.1.1 · p. 6, Table 1 · p. 6

3.1.2 Results on Chart Reasoning

  • SIMPLOT performs best among table-based ChartQA reasoning models, but is sometimes slightly lower than other models on the augmented set.Its strong performance on the more complex human-authored set without additional fine-tuning contrasts with models that score higher on augmented questions.Source: Table 2 · p. 7, §3.1.2 · p. 7

3.2 Further Analysis

  • Both row-column rendering and distillation from a simple chart improve extraction, while the human-oriented prompt improves question-answering performance.The ablations are reported in Table 3.Source: §3.2 · p. 7, Table 3 · p. 7
  • Applying SIMPLOT to Deplot and UniChart improves extraction and question answering; adding images at inference also improves vanilla Deplot and UniChart, and the proposed prompt benefits Deplot with images.SIMPLOT is described as model-agnostic and can be applied to existing table-based reasoning models.Source: §3.2 · pp. 7–8, Table 4 · p. 8
  • On 85 manually filtered challenging QA pairs requiring at least two rows and two columns, the gap between SIMPLOT and Deplot+img+prompt grows as questions become harder.The authors argue ChartQA's simpler questions, often answerable from one row, are insufficient to assess SIMPLOT's table-extraction capability fully.Source: §3.2 · p. 8, Table 5 · p. 8
  • SIMPLOT outperforms Deplot on MultiChartQA's comparative and sequential tasks, and extracted tables support other tasks without additional training.The paper notes that MultiChartQA uses chart formats different from those used to train SIMPLOT, so training on that dataset could potentially improve performance.Source: Table 6 · p. 8, §3.2 · p. 9
  • In a case study, SIMPLOT captures chart counting units such as “million/billion” that Deplot misses.Figure 6 illustrates the difference in extracted tables.Source: §3.2 · p. 9

4 Related Work

Prior chart-processing approaches include rule-based systems and OCR or key-point detection, while later work uses vision-language models and chart-to-table reasoning. The cited literature describes table-based LLM reasoning as supporting downstream tasks without additional training.

4 Related Work

  • Early rule-based chart processing lacked flexibility across chart types, while OCR and key-point approaches depend on annotations and module performance.The passage cites Balaji et al. (2018), Choi et al. (2019), Poco and Heer (2017), Zhou et al. (2023), and Xue et al. (2023).Source: §4 · p. 9
  • Recent end-to-end vision-language models require task-specific fine-tuning, whereas chart-to-table methods pair extracted tables with LLM question answering for downstream tasks without additional training.The passage cites Liu et al. (2022a) and Xia et al. (2023) for chart-to-table reasoning.Source: §4 · p. 9

5 Conclusion

SIMPLOT extracts chart information relevant to reasoning and pairs this with a chart-specific prompt that mimics human interpretation. Experiments show improved performance and applicability to other tasks.

5 Conclusion

  • SIMPLOT extracts only chart information needed for reasoning and uses textual information within charts to support accurate chart reasoning.The conclusion also states that it addresses concerns raised by existing works and improves performance.Source: §5 · p. 9
  • The proposed chart-reasoning prompt guides an LMM to imitate how humans interpret charts.The authors report that extensive experiments demonstrate improved performance.Source: §5 · p. 9
  • SIMPLOT can be applied to other tasks, according to the conclusion.Source: §5 · p. 9

Limitations

The authors expect limited performance on unseen out-of-distribution data, while reporting baseline gains on OpenCQA without training and a one-round table-extraction approach for downstream tasks.

Limitations

  • SIMPLOT is expected to have limited performance on unseen out-of-distribution data, a boundary the authors also attribute to other chart-reasoning methods.Source: Limitations · p. 10
  • The authors state that one round of table-extraction training can support various downstream tasks on OOD charts, unlike methods requiring separate training for each task.Source: Limitations · p. 10
  • On OpenCQA, where tables were not provided, inference without training outperformed existing baselines, which the authors say indicates some generalization to OOD charts.Source: Limitations · p. 10

Ethics Statement

The paper describes prior vision-language and chart-reasoning approaches, then details SIMPLOT’s row-column rendering and human-oriented prompt. It focuses evaluation on ChartQA because it contains complex real-world charts.

Ethics Statement

  • The ethics statement says SIMPLOT was not trained on private or sensitive data and uses publicly accessible ChartQA and PlotQA datasets.The authors also acknowledge potential misuse to mislead the public and warn that outputs are not guaranteed to be accurate.Source: Ethics Statement · p. 10

A Complete Related Work

B Row-column Rendering

  • For training, row and column labels from the ground-truth table are rendered onto simple and original charts; at inference, an LMM extracts the labels for rendering onto the original chart.Source: B Row-column Rendering · p. 13

C Human-oriented Prompt for Chart Reasoning

  • Human-oriented chart instruction prompts an LMM to follow human-like chart perception, beginning with universal chart elements and adding chart-type-specific directions.The prompt also aligns chart positions and outlines with the generated table; instructions for bar, line, and pie charts are described.Source: C Human-oriented Prompt for Chart… · pp. 13, 15

D.1 Dataset Description

  • The paper mainly evaluates on ChartQA because it contains complex real-world charts and three chart types, whereas PlotQA and DVQA contain simpler charts.The paper also reports additional PlotQA experiments and says its QA pairs are used for testing, not training.Source: §D.1 · p. 15

D.2 Compared Methods

The comparison section groups baselines and describes SIMPLOT’s advantages over table-based alternatives. It also explains a relative-distance metric intended to assess value mapping when column labels differ or are arbitrary.

D.2 Compared Methods

  • The baselines span vision-language pretrained models, supervised chart-reasoning models, and table-based reasoning; the table-based comparison uses GPT-4V for methods with extracted tables.Source: §D.2 · pp. 15–16
  • SIMPLOT improves table extraction and downstream-task accuracy by using chart textual information and chart-specific prompts, while retaining LLM reasoning after table extraction.The comparison includes Deplot and a table-extraction variant of UniChart; the original UniChart requires direct training for each downstream task.Source: §D.2 · p. 16

D.3 Details regarding Proposed Metric

  • Relative Distance evaluates value mapping by matching row-and-column keys, then measuring numeric-value distance; this avoids penalizing semantically equivalent or arbitrary column names.The authors note that RMS can register a loss from differing column names even when QA is unaffected.Source: §D.3 · pp. 16–17
  • Examples show that extracted column labels may differ from ground truth without affecting reasoning, while unclear image labels can lead the LMM to generate arbitrary columns.Source: Figure 9 · p. 17, Figure 10 · p. 17

E Additional Experiments with PlotQA

On PlotQA, SIMPLOT performs better across chart types, although its gains are smaller than on real-world ChartQA. Experiments also find benefits from pairing images with tables and little sensitivity to λ.

E Additional Experiments with PlotQA

  • SIMPLOT outperforms other methods across chart types on PlotQA, though its gains are smaller than on real-world ChartQA.The authors note that PlotQA contains only simple charts, whereas SIMPLOT is designed for real-world charts with unnecessary and noisy information.Source: E Additional Experiments with PlotQA · p. 17, Table 11 · p. 17

F Effectiveness of Utilizing Image with Extracted Table

G Hyperparameter Analysis

  • Changes in RD were minimal across tested λ values, indicating that SIMPLOT performed consistently across those settings.Figure 11 reports table-extraction performance with varying λ.Source: G Hyperparameter Analysis · p. 18

H Effectiveness of Negative Sample

Negative samples yield a slight table-extraction improvement, while case studies illustrate more accurate row selection across chart types and the Human-oriented chart instruction's role in locating chart components.

H Effectiveness of Negative Sample

  • Negative samples slightly improve table-extraction performance, with value accuracy important even when the overall difference appears minor.The samples share positive examples' chart format but contain different values, helping distinguish visually similar charts in representation space.Source: H Effectiveness of Negative Sample · p. 18, Table 13 · p. 18

I More Case Studies

  • SIMPLOT extracts relevant rows more accurately than Deplot in the shown examples, including pie and bar charts.Deplot omits relevant rows, adds redundant or irrelevant rows, and can fail to produce values; SIMPLOT produces accurate tables across these examples.Source: I More Case Studies · p. 18
  • Human-oriented chart instruction lets the LMM locate the line component “Favor” precisely, producing a correct answer where the unprompted LLM answers incorrectly.The case study presents the instruction as imitating human chart interpretation.Source: I More Case Studies · p. 19

J Evaluations on more Challenging Questions

The authors argue that simple ChartQA questions can mask extraction differences, while a challenging multi-row, multi-column question distinguishes the methods and highlights the importance of accurate tables.

J Evaluations on more Challenging Questions

  • Similar QA performance can occur even when Deplot extracts a less accurate table, because a simple question may not depend on the erroneous part.The authors use this observation to argue that more complex questions could better differentiate the methods.Source: J Evaluations on more Challenging… · p. 19
  • On a question requiring multiple rows and columns, Deplot fails while SIMPLOT answers correctly using its more accurate table.The example asks for the sum of China's 1955 birth rate and 1965 death rate; the authors say it shows accurate extraction is crucial for challenging questions and ChartQA is insufficient for precise reasoning-model evaluation.Source: J Evaluations on more Challenging… · p. 19

K Error Analysis

The error analysis identifies line breaks in chart labels and color recognition as failure cases. The implementation reports training time and model and prompting settings.

K Error Analysis

  • A line break within a chart item can split it into separate entities during row extraction, producing inaccurate rows.The authors suggest refining rows and columns using contextual information before rendering as one possible remedy.Source: K Error Analysis · p. 19, Figure 18 · p. 21
  • Color recognition remains a failure case: the LMM identifies Jamaica's orange color as red and answers incorrectly.The authors suggest broadening the range of similar colors considered together as a possible solution.Source: K Error Analysis · p. 19, Figure 19 · p. 21

Hardware.

  • Training used an NVIDIA RTX A6000, with phase 1 taking about 2 hours per epoch for 7 epochs and phase 2 about 4 hours per epoch for 9 epochs.Source: Hardware · p. 19

Parameters.

  • SIMPLOT has 374M parameters; GPT-4's exact parameter count is unreported, and LMM prompting uses temperature 0.1.Source: Parameters · p. 19
Loading 2405.00021v3…