Source-linked AI summary
Chart-to-Text: A Large-Scale Benchmark for Chart Summarization
Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty
TL;DR
Charts can contain important insights that are difficult and time-consuming to infer manually. Chart-to-text introduces a 44,096-chart benchmark with two datasets, neural baselines for table-available and image-only settings, and evaluations showing fluent but error-prone summaries that struggle with complex trends.
Problem
Inferring important chart insights requires substantial cognitive and perceptual effort, while prior resources lacked large-scale datasets and strong baselines for chart-specific visual features.
Method
The paper constructs two datasets totaling 44,096 charts and adapts state-of-the-art models for settings with and without underlying data tables.
Results
The best models usually produce fluent summaries and reasonable BLEU scores but suffer hallucinations, factual errors, and difficulty explaining complex chart patterns and trends.
Takeaways & Limitations
Chart-to-text provides a research benchmark for developing chart summarization models and evaluation measures while exposing distinctive challenges of the task.
Takeaways & Limitations
More diverse chart-summary sources and cross-domain experiments remain necessary to evaluate model generalizability.
Abstract
from arXiv · showhide
Charts are commonly used for exploring data and communicating insights. Generating natural language summaries from charts can be very helpful for people in inferring key insights that would otherwise require a lot of cognitive and perceptual efforts. We present Chart-to-text, a large-scale benchmark with two datasets and a total of 44,096 charts covering a wide range of topics and chart types. We explain the dataset construction process and analyze the datasets. We also introduce a number of state-of-the-art neural models as baselines that utilize image captioning and data-to-text generation techniques to tackle two problem variations: one assumes the underlying data table of the chart is available while the other needs to extract data from chart images. Our analysis with automatic and human evaluation shows that while our best models usually generate fluent summaries and yield reasonable BLEU scores, they also suffer from hallucinations and factual errors as well as difficulties in correctly explaining complex patterns and trends in charts.
1 Introduction
Chart summarization aims to help people extract important insights from charts without the substantial cognitive and perceptual effort that manual interpretation can require. Chart-to-text addresses limited data and modeling resources with a broad benchmark, neural baselines, and evaluations of remaining challenges.
- Chart interpretation can be challenging and time-consuming because extracting key insights requires substantial cognitive and perceptual effort.
- Chart summarization generates natural-language explanations of charts and their key takeaways.
- Summaries can help authors communicate chart insights, help readers identify important points, and improve accessibility for visually impaired people using screen readers.
- Existing table-to-text approaches do not directly explain visually salient chart features such as trends, correlations, and outliers.
- Large-scale data and strong neural baselines were previously limited; an earlier dataset contained 8,305 charts from one source and only two chart types.
- Chart-to-text introduces two datasets totaling 44,096 charts, state-of-the-art baselines, and automatic, human, and qualitative evaluations.
2 Related Work
Prior chart summarization systems used templates or data-driven models, but existing approaches provide limited variation or fail to address charts’ visual communication properties. Chart-to-text builds on data-to-text and image-captioning methods while targeting these gaps.
- Chart Summarization: Early chart summarization systems used planning-based architectures and templates to generate captions for bar and line charts.
- Chart Summarization: Template-based generation can limit generality, insight variety, grammatical styles, and lexical choices compared with data-driven models.
- Chart Summarization: Recent chart systems used transformer or LSTM encoder-decoder models, but prior datasets contained only 8,305 or 306 chart summaries.
- Data2text Generation: Data-to-text generation summarizes tables across domains including sports, weather, recipes, biographies, and open-domain tasks.
- Image Captioning: Image-captioning methods commonly exploit large-scale datasets, object detection, or OCR, whereas charts encode quantitative data through marks such as bars and lines.
3 Chart-to-text Datasets
The benchmark combines Statista and Pew charts with complementary annotation and extraction procedures, producing broad coverage of chart types, topics, summary lengths, and semantic content. Its datasets support both structured-data and chart-image settings for chart-to-text research.
- Data Collection: The benchmark draws on two sources selected for sufficiently large numbers and varieties of charts with textual descriptions.
- Data Collection: Statista contributed 34,811 charts from 34,810 publicly accessible webpages, including screenshots, data tables, labels, titles, and human-written descriptions.
- Data Collection: Pew contributed 9,285 charts from 3,999 pages, but only 143 charts had underlying data tables.
- Data Annotation: Pew annotation handled missing tables and ambiguous chart-paragraph relations through chart-image OCR, text-role classification, candidate paragraphs, and relevance scoring.
- Data Annotation: The text-role classifier achieved 95.0% overall precision and 97.6% precision for title classification on its test set.
- Dataset Analysis: Bar charts dominate both datasets, comprising 87.9% of Statista charts and 67.9% of Pew charts; line charts account for 10.2% and 26.4%, respectively.
- Dataset Analysis: Pew summaries are about twice as long as Statista summaries, while complex charts generally receive longer summaries than simple charts.
- Dataset Analysis: Statistical and comparative information is most common, whereas Pew summaries more often report trends and causal relations requiring greater perceptual and cognitive effort.
4 Chart-to-text Baseline Models
The paper evaluates chart-to-text with image captioning, data-to-text, and combined vision-text baselines across settings with and without underlying data tables. Models encode chart images, tabular records, metadata, or OCR text before generating summaries.
- Problem Variations: Two problem variations use either an available underlying data table or only chart images and metadata when the table is unavailable.The image-only setting is described as more challenging and realistic because most online charts lack underlying data tables.
- Model Categories: Three model categories comprise image captioning, data-to-text generation, and combined vision-text systems that first extract chart text with CRAFT OCR.These categories respectively use chart images, underlying tables, or OCR-derived text as inputs for summary generation.
- Image Captioning Models: SAT uses a ResNet50 image encoder and unidirectional LSTM decoder, with dataset-specific Barlow Twins pretraining for the chart-image encoder.Directly applying an ImageNet-pretrained ResNet50 to chart images produced poor experimental results, motivating separate pretraining for each dataset.
- Data-to-Text Models: Chart2text embeds ordered data records with positional encodings and adds an auxiliary objective predicting whether each record appears in the output sequence.The model is an adapted Transformer based on a data-to-text architecture.
- Data-to-Text Models: Field-Infusing Model combines LSTM-encoded cell values with row and column information before a 3-layer Transformer encoder-decoder generates summaries.For Pew, it also incorporates OCR-text bounding-box information into the table representations.
- Sequence-to-Sequence Models: BART and T5 use flattened table or ordered OCR text as sequence-to-sequence inputs, with T5 also supporting bounding-box positional embeddings for OCR tokens.BART concatenates the title with row-wise table content, while T5 uses a chart-to-text prefix and has an OCR variant incorporating spatial information.
5 Evaluation
The evaluation measures chart-summary quality across automatic metrics, datasets, model inputs, and human judgments. Results show that pretraining and table access improve performance, while image-based systems remain vulnerable to extraction errors, hallucinations, and difficulties with complex visual reasoning.
- Automatic evaluation: BLEU, CIDEr, BLEURT, Content Selection, and Perplexity evaluate overlap, meaning, content selection, and fluency.BLEURT and Content Selection are averaged at the sentence level, while Perplexity uses GPT-2 Medium.
- Automatic evaluation: Large-scale unsupervised pretraining significantly boosts performance, while image captioning produces fluent text but captures less relevant chart information.The image captioning model has low Perplexity but a low Content Selection score.
- Automatic evaluation: TAB-BART and TAB-T5 generate well-structured, relevant summaries from available tables, whereas OCR-based models are slightly less effective because OCR introduces input noise.Chart2text and Field-Infuse extract table information but produce lower-quality text without large-scale pretraining.
- Automatic evaluation: 77.31% automatic data-extraction accuracy leaves TAB_OCR models less effective at selecting relevant information than models using ground-truth tables.TAB_OCR models use automatically generated tables, whereas TAB models use ground-truth data tables.
- Automatic evaluation: Performance drops significantly on Pew, but OCR-based vision-plus-text models improve over the vision-only model when underlying tables are unavailable.Pew contains charts with ill-defined structure and no available data tables.
- Human evaluation: Human evaluation found TAB-T5 significantly better than OCR-T5 on factual correctness, coherence, and fluency, while model fluency remained comparable to gold summaries.The gap was especially pronounced for factual correctness and was worse for OCR-T5.
- Error analysis and challenges: Error analysis identifies hallucinations, factual errors, and failures to explain complex visual patterns and trends as persistent challenges.OCR-based models more often misassociate chart values, while broader visual-style diversity remains an open generalizability challenge.
6 Conclusion
The paper presents two large-scale chart-summarization datasets and several state-of-the-art baselines. Its evaluations highlight both their promise and unique challenges for chart summarization.
- The paper presents two large-scale datasets for chart summarization.
- It provides several state-of-the-art baselines and evaluation measures.
- Evaluation reveals unique challenges for the chart-summarization task.
- The benchmark is intended to support model and metric development in chart summarization.
Ethical Considerations
The authors describe ethical safeguards for dataset construction and annotation, while warning that fluent but inaccurate model outputs could spread misinformation.
- The datasets use publicly available charts whose publishers grant academic publication rights.
- Pew content may be used when attributed to the Center or not attributed to a different party.
- Annotators were paid $0.10–$0.15 per chart based on estimated task time and candidate count.Compensation was based on a minimum wage of $7.25 per hour and an estimated one-minute task duration.
- Annotations were anonymized to protect annotator privacy.
- Uncorrected fluent outputs containing hallucinations and factual errors may mislead the public.
A.1.1 Example Webpage from Statista
A Statista webpage pairs a chart image with description text, whose opening portion summarizes the chart while later text supplies irrelevant background.
- The webpage contains a chart image and accompanying description text.
- The highlighted opening text provides a succinct chart summary.
- The remaining text contains irrelevant background information, such as Facebook’s history.
A.1.2 Annotation of x-axis Labels in Statista
The Statista annotation interface supports labeling the x-axis labels of charts.
- Figure 6 shows the user interface for labeling x-axis labels in the Statista dataset.
A.1.3 Identify Candidate Paragraphs in Pew
The Pew candidate-paragraph process combines chart proximity and textual overlap to identify paragraphs relevant to each chart. The broader benchmark also includes dataset analysis, model baselines, transfer experiments, chart-type evaluations, and image-based table extraction.
- Candidate paragraph identification: The Pew dataset uses Mechanical Turk annotation to select relevant paragraphs, with quality controls requiring high approval rates, extensive prior task completion, and a sample task.Annotators received $0.10–$0.15 per chart, depending on the number of candidate paragraphs.
- Transfer results: Transfer pretraining on Statista, Pew, or ToTTo yields only marginal improvement in final performance.The transfer experiment uses T5-based models with the same pretraining and fine-tuning procedure.
- Performance by chart type: TAB-T5 summarizes simple, frequent chart types more effectively than complex, infrequent types such as pie charts.The comparison is reported on the Statista test set by chart type.
- Automatic data extraction: The image-based task extends ChartOCR with textual-label detection, OCR, color-based association, and structured-table recovery.Detected values are linked to x-axis labels and legend labels to reconstruct the chart’s fully structured data table.
- Dataset analysis: The benchmark spans diverse topics and chart styles, with topic distributions reported across its two datasets.Additional examples cover a diverse range of chart types and styles.