PaperlayerView exact revision on arXiv

Source-linked AI summary

DePlot: One-shot visual language reasoning by plot-to-table translation

Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun

arXiv:2212.10505v2cs.CLcs.AIcs.CV

TL;DR

Chart reasoning requires extensive extraction and reasoning, while prior systems need large supervision and struggle with complex human-written queries. The paper introduces DePlot to translate charts into linearized tables and pass them to LLMs for one-shot reasoning. On human-written ChartQA queries, DePlot+LLM achieves a 29.4% improvement over the end-to-end SOTA comparison reported in the conclusion.

  • Problem

    Chart reasoning requires detailed information extraction and numerical reasoning, while prior end-to-end systems require large task datasets and remain limited on complex human-written queries.

  • Method

    DePlot is an end-to-end image-to-text Transformer that converts plots into linearized tables, which an off-the-shelf LLM uses for one-shot reasoning.

  • Results

    29.4% improvement over end-to-end SOTA is achieved by one-shot DePlot+LLM on the ChartQA human-query set.

  • Takeaways & Limitations

    DePlot translates visual-language reasoning into plot-to-table conversion followed by language-model reasoning, enabling a one-shot ChartQA approach with less supervision than finetuned SOTA methods.

  • Takeaways & Limitations

    DePlot’s effectiveness depends on accurate, diverse, in-domain plot-table data and does not extend to visual language lacking a clear latent textual representation.

Abstract

from arXiv · show

Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24.0% improvement over finetuned SOTA on human-written queries from the task of chart QA.

1 Introduction

Visual-language reasoning on charts requires both detailed information extraction and strong reasoning, while prior end-to-end systems need extensive supervision and remain limited on complex human-written queries. DePlot addresses this by translating plots into linearized tables for one-shot LLM reasoning.

  • Chart QA requires extracting relevant image information, organizing it sensibly, and reasoning over the extracted entries.
  • MATCHA reaches 38.2% accuracy on human-written ChartQA queries despite finetuning on large amounts of task data.
  • DePlot decomposes visual-language reasoning into plot-to-table translation followed by one-shot reasoning over the translated table with an LLM.
  • DePlot is an end-to-end image-to-text Transformer trained on synthetic and web-crawled charts paired with their underlying data tables.
  • The proposed table-matching metric evaluates textual and numeric entries with relative-error tolerance while remaining invariant to transpositions and row or column permutations.
  • 29.4% improvement over the fully supervised second-best method is achieved on human-written queries with one-shot supervision.

2 Background

Visual-language reasoning differs from ordinary image reasoning because it combines detailed chart information extraction with numerical reasoning. DePlot connects an end-to-end chart extractor to LLM-based table reasoning, outperforming prior chart-extraction and supervised reasoning approaches in supported settings.

  • Natural-image multimodal methods generally capture basic visual information, whereas chart reasoning requires detailed extraction and numerical reasoning.
  • DePlot plus LLMs outperforms supervised SOTA by 29.4% on complex human-written queries by assigning extraction and reasoning to separate pretrained models.
  • One-shot GPT-3 table reasoning can approach SOTA performance on table QA datasets trained with at least thousands of examples.
  • Prior plot and chart information-extraction systems commonly combine OCR, detection or segmentation, and hand-crafted rules, often specializing in particular graph types.
  • DePlot, as an end-to-end neural model, outperforms ChartOCR by very large margins on plot-to-table conversion.
  • The paper formulates chart extraction as plot-to-table translation and proposes a metric that evaluates table structure and values rather than only matching number sets.

3 Standardizing the Plot-to-table Task

The paper standardizes plot-to-table translation and introduces RMS, a table-matching metric designed to evaluate textual and numeric reconstruction robustly across table rearrangements. It also presents DePlot as an end-to-end, chart-type-agnostic extraction model and validates RMSF1 against human judgments.

  • Task Definition: Plot-to-table translation is formulated as the underlying data-extraction goal for chart information extraction, with unified task formats and evaluation.The formulation targets the recovered data table rather than intermediate chart objects such as coordinates, bounding boxes, or keypoints.
  • Metric Design: RNSS evaluates only unordered numeric entries, ignoring positions, nonnumeric content, large relative errors, and precision–recall distinctions.These limitations motivate a more comprehensive table-similarity metric.
  • Metric Design: RMS represents tables as unordered mappings from row and column headers to values, matching entries using textual and numeric similarities.Its matching procedure computes pairwise key similarities, finds a minimal-cost correspondence, and derives precision, recall, and RMSF1.
  • Metric Design: RMSF1 is invariant to row and column permutations and table transpositions while allowing penalized small textual or numeric errors.The metric evaluates both the original and transposed tables and retains the higher score.
  • DePlot Model: DePlot is an end-to-end model that handles line, dot, bar, and pie charts without type-specific engineering or hybrid components.Its finetuning corpus combines synthetic and real-world plot-table pairs from three sources.
  • Human Evaluation: RMSF1 correlates more strongly with human judgments than RNSS on 50 predicted plot-table pairs evaluated by six annotators.The comparison uses Pearson’s r and Spearman’s ρ correlations between metric scores and human table-quality ratings.

4 Prompting LLMs for Reasoning

The prompting approach converts charts into linearized tables, concatenates those tables with questions, and uses one-shot in-context examples for LLM reasoning. It supports Chain-of-Thought or Program-of-Thought prompting and optionally self-consistency.

  • Prompt Construction: DePlot outputs a chart’s textual form as a linearized table, which is concatenated with the question to construct an LLM prompt.A one-shot example is prepended following the typical in-context learning paradigm.
  • Reasoning Strategies: Prompts use either Chain-of-Thought or Program-of-Thought reasoning formats adapted for reasoning over tabular data.The paper also evaluates self-consistency, which selects the majority-voted answer across sampled reasoning paths.

5 Experiment

The experiments evaluate plot-to-table translation and downstream QA on ChartQA and PlotQA, comparing DePlot with prior systems and one-shot DePlot+LLM configurations. DePlot+LLM is especially strong on human-written ChartQA queries but underperforms finetuned SOTA on synthetic PlotQA queries.

  • Experimental Setup: The evaluation covers plot-to-table conversion and downstream QA on ChartQA and PlotQA, using RNSS and RMSF1 for conversion and 5%-tolerance exact match for QA.ChartQA includes augmented and human-written sets, while PlotQA is synthetic and includes v1 and v2.
  • Plot-to-table Translation: DePlot outperforms ChartOCR by significant margins on both plot-to-table metrics.The gap is especially large on RMSF1, which captures structural errors in row and column organization.
  • Plot-to-table Translation: DePlot also outperforms PaLI and MATCHA for plot-to-table conversion, while higher PaLI input resolution substantially improves extraction accuracy.The comparison suggests that visual-language-specific initialization and task-specific finetuning benefit plot-to-table accuracy.
  • Downstream Tasks: 29.4%: DePlot+LLM improves over MATCHA on human-written ChartQA queries, increasing accuracy from 38.2% to 67.6%.The best setup combines FlanPaLM and Codex with self-consistency.
  • Downstream Tasks: DePlot+LLM underperforms finetuned SOTA overall on synthetic PlotQA queries, despite significantly outperforming finetuned SOTA on human-written chart QA.The authors attribute the contrast to the restricted, templatic nature of synthetic queries and the greater supervision available to finetuned models.
  • Downstream Tasks: Without demonstrations, the models generally fail, while few-shot prompting performs similarly to one-shot prompting; aligned conversational LLMs can process DePlot tables zero-shot.The latter observation is reported for ChatGPT, GPT-4, and Bard and is described as potentially improving benchmark performance.

6 Analyses and Discussions

The analyses examine DePlot+LLM’s numerical-reasoning strengths, visual-attribute failures, and generalization to charts outside its training domain.

  • Case studies: DePlot+FlanPaLM correctly answers a numerical ChartQA case that finetuned MATCHA answers incorrectly.Both chain-of-thought and program-of-thought prompting produce the correct answer.
  • Case studies: DePlot+LLM fails when a query depends on bar color, because plot-to-table translation discards visual attributes.The system can obtain the right answer for the wrong reason when color information is unavailable.
  • PlotQA analysis: 66.6% vs. 91.5%: DePlot+LLM underperforms finetuned MATCHA on PlotQA.The paper attributes this gap partly to templatic synthetic queries and partly to lost visual information.
  • Out-of-distribution analysis: The evaluation uses charts from the same domain as DePlot’s training corpus, limiting conclusions about cross-domain generalization.The authors note that few public resources pair charts with their associated tables.
  • Out-of-distribution analysis: On 10 TaTa charts, DePlot reconstructs underlying tables with an average 78% RMSF1 score.These charts were sourced from dhsprogram.com, and choropleth maps were excluded because they were absent from training.

7 Conclusion

The conclusion presents DePlot+LLM as a two-step visual-language reasoning method: translate plots into linearized tables, then use an LLM for one-shot reasoning. It reports standardized plot-to-table evaluation and a 29.4% ChartQA improvement over end-to-end SOTA.

  • Method: DePlot+LLM converts plots into linearized tables and combines the conversion model with an off-the-shelf LLM for one-shot reasoning.The conversion model is an image-to-text Transformer, while the LLM performs reasoning over the translated table.
  • Evaluation: The paper standardizes plot-to-table conversion with a table-similarity metric covering structure and numeric values while ignoring row and column permutations.The metric supports comparison across plot-to-table systems.
  • Results: +29.4%: one-shot DePlot+LLM improves performance over end-to-end SOTA finetuned with thousands of examples on human-written ChartQA queries.The conclusion identifies this as the result on the ChartQA human-query set.
  • Future direction: The analyses identify visual-attribute encoding as a direction for future improvement.The conclusion specifically highlights encoding attributes such as color that may be lost during table translation.

Limitations

The paper’s limitations concern dependence on accurate plot-to-table conversion, restricted representation scope, and omitted visual layout information.

  • Data and robustness: DePlot’s performance depends strongly on accurate plot-to-table conversion, which generally requires large, diverse, in-domain plot-table data.The robustness of conversion on out-of-domain web charts remains insufficiently understood.
  • Scope: DePlot does not support visual language lacking a clear latent textual representation, such as textbook figures made with specialized software.Its approach is suited to graphics with structured representations that can be translated into tables.
  • Representation: The current approach ignores visual layout information, including element orientation and color.The authors propose incorporating these attributes into the decoding target in future work.

Ethics Statement

The ethics statement characterizes DePlot as low risk as an information-extraction model but notes potential risks when combined with LLMs.

  • Risk assessment: DePlot is described as low risk because it converts graphic information into textual tables.The stated risk assessment concerns the information-extraction component itself.
  • Deployment: When combined with LLMs, DePlot+LLM may generate toxic content, so real-world deployment requires caution and precautions such as filtering.The paper compares this potential risk with standalone LLM use.
  • Data: Training and evaluation data are synthetically created or publicly available web data with permissive licenses.The statement describes the data sources and licensing conditions.

A Details of Baselines

The baselines span text-only, multimodal, OCR-based, detector-based, and large vision-language models for chart understanding tasks.

  • Baseline models: T5 encodes a linearized table, and optionally a query, to decode an answer or summary.With a gold table, T5 does not directly use the chart image.
  • Baseline models: VL-T5 and VisionTaPas incorporate chart images through visual encoders or crossmodal fusion layers.VisionTaPas is modified from TaPas with a ViT and crossmodal fusion layers.
  • OCR baselines: OCR variants of T5, VL-T5, and VisionTaPas replace gold tables with OCR-extracted data tables from chart images.The OCR variants retain their corresponding base architectures while changing the table source.
  • Other baselines: CRCT detects textual and visual chart elements before processing them with a multimodal Transformer, while PaLI is a 17B-parameter vision-language model.CRCT is described as the best-performing PlotQA model, and PaLI as SOTA on multiple vision-language tasks.

B Human Evaluation Questions

Human evaluation scores plot-table pairs across structural completeness, label correctness, and numerical accuracy. Annotators rate each aspect from 1 to 5, and the final score averages the three ratings.

  • Scoring: The final table score is the average of the three human judgment ratings.The evaluation uses a 1-to-5 scale, with higher scores indicating better quality.

D More Case Study

The case studies show DePlot+LLM handling several numerical and table-based questions, while exposing failures when reasoning depends on visual attributes or imperfect plot-to-table alignment.

  • Successes: DePlot+LLM successfully solves line-plot questions requiring numerical reasoning that MATCHA fails.Table 8 presents a line-plot example characterized by this contrast.
  • Successes: DePlot+LLM computes a 6.8-point difference between Independents and Republicans in 2011.The calculation uses 58.0 minus 51.2 from the 2011 row.
  • Table questions: The prompting examples cover identifying extrema, ranking groups, and retrieving the year associated with a minimum value.The examples identify Republicans as having the overall worst performance, Independents as second-highest in 2007, and 2007 as Republicans’ minimum year.
  • Table questions: The table QA prompt represents chart data with year rows and party columns containing favor rates.The example table includes Democrats, Republicans, and Independents across seven years.
  • Python-based reasoning: DePlot+LLM is described as especially suitable for complex numerical computations such as averaging multiple values.Table 9 illustrates this use case.
  • Python-based reasoning: The Python prompt supports averages, sums, differences, threshold checks, and relative increases over selected table entries.Examples calculate an average difference, a multi-year average, a sub-50% party, a sum, a percentage increase, and a cross-year difference.
  • Failures: The framework cannot answer questions about color or other visual attributes because those properties are lost during modality translation.The current textual table encoding does not represent these visual attributes.
  • Failures: Plot-to-table conversion can fail when line-plot dots do not align one-to-one with x-axis labels, producing extrapolated values while ignoring dots.Table 10 also reports a DEPLOT+LLM error from identifying the wrong graph as the red graph, whereas MATCHA’s error appears numerical.
Loading 2212.10505v2…