PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

Mengsha Liu, Daoyuan Chen, Yaliang Li, Guian Fang, Ying Shen

arXiv:2403.11236v2cs.CL

TL;DR

Chart summarization systems still face chart-content mismatches and reasoning errors. ChartThinker combines chain-of-thought reasoning and context retrieval with a large chart dataset. The authors report performance above 8 state-of-the-art methods across 7 evaluation metrics, while noting that Content Similarity alone may misrepresent summary quality.

  • Problem

    Existing visual-language models can omit or fabricate chart content and misinterpret a chart’s intended message.

    Source: §1 · p. 1

  • Method

    ChartThinker combines chain-of-thought reasoning and context retrieval, with chart parsing, chart-description pretraining, and instruction fine-tuning.

    Source: §1 · p. 2

  • Results

    ChartThinker outperforms 8 state-of-the-art methods across 7 evaluation metrics in chart summarization tasks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors’ ablations underscore the role of chain of thought in reasoning and context retrieval in semantic understanding.

    Source: §5 · p. 9

    Content Similarity alone can underrate detailed, accurate summaries when they extend beyond the standard answer.

    Source: §4.4.2 · p. 9

Abstract

from arXiv · show

Data visualization serves as a critical means for presenting data and mining its valuable insights. The task of chart summarization, through natural language processing techniques, facilitates in-depth data analysis of charts. However, there still are notable deficiencies in terms of visual-language matching and reasoning ability for existing approaches. To address these limitations, this study constructs a large-scale dataset of comprehensive chart-caption pairs and fine-tuning instructions on each chart. Thanks to the broad coverage of various topics and visual styles within this dataset, better matching degree can be achieved from the view of training data. Moreover, we propose an innovative chart summarization method, ChartThinker, which synthesizes deep analysis based on chains of thought and strategies of context retrieval, aiming to improve the logical coherence and accuracy of the generated summaries. Built upon the curated datasets, our trained model consistently exhibits superior performance in chart summarization tasks, surpassing 8 state-of-the-art models over 7 evaluation metrics. Our dataset and codes are publicly accessible.

1. Introduction

Chart summaries can reduce the effort of describing charts, but existing visual-language models may omit or fabricate chart content and misread a chart’s intended message. ChartThinker combines chain-of-thought reasoning and context retrieval with a large, diverse chart dataset, and the paper reports state-of-the-art results on chart benchmarks.

Motivation

  • Automatic chart summarization aims to explain charts and summarize key takeaways, reducing the effort of manually writing descriptions.Manual descriptions can be time-consuming and prone to errors.Source: §1 · p. 1
  • Existing visual-language models can omit or fabricate chart content and misinterpret the chart’s intended message.The paper identifies these as matching and reasoning errors, respectively.Source: §1 · pp. 1–2

Contributions

  • ChartThinker combines chain-of-thought reasoning with context retrieval to generate chart descriptions.The method also pre-trains on chart-description pairs and fine-tunes on question-answer pairs; a chart-parsing module combines extracted data with the prompt.Source: §1 · p. 2
  • The Chart-Sum-QA contribution includes 595,955 chart-caption pairs and 8 million instruction-question pairs spanning diverse topics and visual styles.The abstract describes broad topic and style coverage as supporting better matching from the training-data perspective.Source: §1 · p. 2
  • ChartThinker’s extensive automatic and human evaluations demonstrate state-of-the-art performance on various chart benchmarks; the authors release their dataset and codes to facilitate further research.Source: §1 · p. 2

2. Related Work

Prior chart-summarization approaches include template-based generation, neural models, and visual-language models; the paper positions its work as an integration of chain-of-thought reasoning and context retrieval. It also surveys developments in prompt engineering and contextual visual-language models.

Chart Summarization

  • Template-based chart generation can produce similar descriptions, while neural architectures and attention mechanisms were adopted to improve detail understanding.The cited examples include ResNet-based encoders and attention mechanisms.Source: §2 · p. 2
  • Prior work integrated chart information with language models to improve the logical coherence and accuracy of chart summaries.The passage also notes chart inverse-rendering and mathematical-reasoning pretraining for chart understanding.Source: §2 · p. 2

Visual-Language Models

  • Context-aware visual-language models include Flamingo’s few-shot capabilities, BLIP-2’s query transformer, and instruction-following through context examples.The passage also surveys Mini-GPT4 and LLaVA as multimodal model developments.Source: §2 · p. 3

Prompt Engineering

  • Prompt-engineering methods include chain-of-thought examples, multiple reasoning paths, tree-structured expansion, and graph-structured thought.The surveyed methods also include dynamic reasoning paths, response blueprints, and actionable programs.Source: §2 · pp. 3–4

Our Position

  • The authors identify a gap in holistic integration, positioning their work as combining chain of thought with context retrieval for chart summarization.They describe prior LLM work as often emphasizing factual accuracy, logical coherence, or training architecture separately.Source: §2 · p. 4

3. Methodology

ChartThinker combines a large chart-summary and instruction dataset with chart and prompt encoding, chart parsing, and a context-enhanced chain-of-thought generator. Its generation process uses retrieved chart context alongside thought chains to target comprehensive, logically consistent summaries.

3.1. Dataset Construction

  • Chart-Sum-QA contains 595,955 chart-summary data points and 8,170,000 instruction-question pairs for pre-training and fine-tuning.The instruction-question pairs combine generated chart QA with other QA datasets; the data is split into 80% training, 10% validation, and 10% testing.Source: §3.1 · p. 4

3.2. Image and Text Encoder

  • The image encoder represents chart text, data points, visual elements, and trends, while the text encoder processes the prompt for the LLaMA2 decoder.The chart image encoder produces a fixed-length feature vector; the prompt encoder produces a token sequence.Source: §3.2 · p. 5

3.3. Chart Parsing Module

  • The chart parsing module combines OCR-extracted chart text and numbers with Deplot tables to provide text-number pairs and other text alongside the prompt.OCR does not extract positions, while Deplot can struggle with numerical extraction and irrelevant chart elements.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4. Context-Enhanced CoT Generator

  • The Context-Enhanced CoT Generator combines thought chains with retrieved chart context to target comprehensive summaries and improve their quality and logical consistency.Its retrieval library contains 1,000 chart-text pairs; examples are selected by image-feature similarity, and higher-ranked examples receive more weight.Source: §3.4 · pp. 5–6

4. Experiment

The experiment compares ChartThinker with transformer and visual-language baselines, evaluates chart-summary quality automatically and by human judgment, and describes its fine-tuning setup.

4.1.1. Baselines

  • Eight baselines span text-to-text, chart-specific, and large visual-language models, including encoder-decoder and multimodal systems.The comparisons include T5, Chart2text, Field-Infusing Model, BART, LLaMA-Adapter-v2, MiniGPT-4, mPLUG-Owl, and LLaVA.Source: §4.1.1 · p. 6

4.1.2. Automatic & Human Evaluation

  • Automatic evaluation uses BLEU, BLEURT, CIDEr, Content Selection, and perplexity; human raters score matching degree and reasoning correctness on a 1–5 scale.Three evaluators independently rate 200 summaries, with the final score averaged across evaluators.Source: §4.1.2 · pp. 6–7

4.1.3. Fine-tuning Implementation

  • Fine-tuning uses LoRA rank 16 on qproj, kproj, and vproj, with Alpha 32, dropout 0.05, and a learning rate of 2e-4.The setup also uses paged_adamw_8bit, gradient accumulation of 8, and evaluation every 20 steps.Source: §4.1.3 · p. 7

4.2. Main Results

ChartThinker leads the reported benchmark comparisons, with strong automatic results against classic models and visual-language encoder-decoder models, and favorable human ratings for matching and reasoning.

4.2.1. Benchmark Results

  • ChartThinker shows the best overall performance against classic transformer models and outperforms encoder-decoder visual-language models, with a 1.61 BLEU increase over contemporary SOTA methods.The classic-model comparison reports best overall performance; the encoder-decoder comparison reports the BLEU improvement.Source: §4.2.1 · p. 8
  • Instruction fine-tuning for each chart is associated with the best PPL score and more accurate descriptions of chart values.The authors attribute improved performance to using a directive dataset during fine-tuning.Source: §4.2.1 · p. 8

4.2.2. Human Evaluation Results

  • Human evaluators favored ChartThinker for chart-summary matching and reasoning correctness, reporting fewer data omissions and fabrications and more accurate chart inferences.The evaluation compared ChartThinker with eight baselines on the chart-to-text test set.Source: §4.2.2 · p. 8, Table 5 · p. 8

4.3. Ablation Studies

Ablations associate the model’s components and training datasets with different aspects of summary quality, while the discussion notes limits in training-regime comparability and Content Similarity evaluation.

4.3. Ablation Studies

  • Removing chart parsing reduces underlying-data description accuracy, while removing context retrieval or CoT weakens coherence, reasoning, and content comprehensiveness.The authors link context retrieval removal to loss of contextual examples and CoT removal to loss of stepwise generation and final integration.Source: §4.3 · pp. 8–9
  • Excluding the chart-description dataset lowers BLEU by 1.46, CIDEr by 0.39, and CS by 2.65% on the chart-to-text pew test set.The authors also report reduced BLEURT and higher PPL, noting difficulty with unfamiliar chart layouts.Source: §4.3 · p. 9
  • Without instruction-dataset fine-tuning, chart-summary matching weakens, unrelated content can occur, and BLEU and CS scores drop.The authors attribute this to reliance on pretraining parameters without adaptation to chart tasks.Source: §4.3 · p. 9

4.4.1. Training Paradigms and Task-Specific Optimization

  • ChartThinker’s multimodal pretraining is unsupervised and not task-specific, unlike the supervised OCR systems used by some text-only models.The authors also note that Paddle OCR is not specialized for chart summarization, unlike other referenced OCR models.Source: §4.4.1 · p. 9

4.4.2. Limitations of Content Similarity (CS) as a Single Metric.

  • ChartThinker’s detailed, accurate summary receives a 32.29% CS score versus 38.74% for the less descriptive OCR-T5, illustrating that CS can misrepresent accuracy.The authors caution that valid content extending beyond a standard summary may receive an inferior CS score.Source: §4.4.2 · p. 9, Table 6 · p. 9

5. Conclusion

The paper presents Chart-Sum-QA and ChartThinker as contributions to chart summarization, reporting favorable automated and human evaluations and component ablations.

  • The paper introduces Chart-Sum-QA, a comprehensive chart-summarization dataset, and ChartThinker, a method for training visual-language models to use contextual information.Source: §5 · p. 9
  • Automated and human evaluations report that ChartThinker outperforms eight SOTA methods across seven evaluation metrics.Source: §5 · p. 9
  • Ablations support the effectiveness of the model components and dataset, highlighting CoT for reasoning and context retrieval for semantic understanding.The paper also releases its dataset, codes, and empirical results.Source: §5 · p. 9

A. Generated Summary Example

The section presents ChartThinker-generated summaries through chart examples, reporting values and trends across varied topics.

  • California has the most newly enrolled foreign students among the listed states, at 59,801, followed by New York at 41,040.Source: Figure 5 · p. 13
  • Political advertising revenue for six publicly held local TV station companies fluctuated sharply from 2012 to 2018, ending at 1.3 billion dollars.Source: Figure 6 · p. 13
  • Women’s share of U.S. representatives rose from 2.5% in 1966 to 24.1% in 2018, while adult internet use increased from 14% in 1995 to 87% in 2014.Source: Figure 7 · p. 13, Figure 13 · p. 15
  • For adults without a spouse or partner present, the chart reports higher unpartnered percentages in 2017 than 2007 for the under-35, 35–54, and 55–64 age groups, but a lower percentage for those over 65.Source: Figure 8 · p. 13
  • In a December 2019 survey, 76% of Latino adults had not heard the term Latinx, while 3% said they used it.Source: Figure 9 · p. 14
  • Harassment of Jews peaked at 39% in December 2013, while retransmission-fee revenue rose from 0.1 billion dollars in 2006 to 12.5 billion in 2022.Source: Figure 10 · p. 14, Figure 14 · p. 15
  • The examples also report that half of U.S. adults seldom or never discuss religion with non-family and show survey responses on who should lead in solving national problems.Source: Figure 11 · p. 14, Figure 12 · p. 14

B. Retrieval Library

The Retrieval Library organizes chart examples into four stages, moving from chart type and overview to axes meaning and numerical trend. It contains 1,000 charts and supplies top-K relevant examples during stage-specific training.

  • The first retrieval stage helps the model identify a chart’s type from examples in the library.Source: B. Retrieval Library · p. 15
  • The second stage uses chart-title-derived overall captions and retrieved examples to produce a chart overview summary.Source: B. Retrieval Library · p. 15
  • The third stage explains the horizontal and vertical axes and their relationships to deepen chart understanding.Source: B. Retrieval Library · p. 15
  • The fourth stage teaches the model to describe chart data trends in natural language and produce a numerical trend description.Source: B. Retrieval Library · p. 15
  • The library contains 1,000 charts, with 250 charts and associated textual descriptions assigned to each of four stages; training retrieves the top K relevant examples within the respective stage.Source: B. Retrieval Library · p. 15

C. Comparison of examples generated by various models

The paper presents generated-summary examples and describes a manual evaluation of summaries from ChartThinker and four baselines. Annotators assessed chart matching and reasoning correctness using randomized, three-rater scoring.

Loading 2403.11236v2…