Source-linked AI summary
Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model
Jason Obeid, Enamul Hoque
TL;DR
Chart interpretation can be difficult, especially when users must compare many marks or have limited access to visual information. The paper addresses this gap with a chart-summary dataset and a transformer-based neural model using data variable substitution. The resulting summaries are more informative, concise, and coherent, while the model improves content selection over the base model.
Problem
Large publicly available datasets pairing chart data with human-generated summaries were unavailable, despite challenges in interpreting charts and limitations of existing chart text.
Method
The paper constructs a corpus of 8,305 chart-summary pairs and extends a transformer-based data-to-text model with data variable substitution.
Results
The model generates more informative, concise, and coherent summaries than the base model, with human evaluations showing improvements of at least 1 point in those dimensions.
Takeaways & Limitations
The approach provides a data-driven method for automatically generating natural-language chart descriptions that convey chart insights.
Takeaways & Limitations
Fact hallucination still occurs, including incorrect data-variable indices and irrelevant tokens, particularly for domains with low coverage.
Abstract
from arXiv · showhide
Information visualizations such as bar charts and line charts are very popular for exploring data and communicating insights. Interpreting and making sense of such visualizations can be challenging for some people, such as those who are visually impaired or have low visualization literacy. In this work, we introduce a new dataset and present a neural model for automatically generating natural language summaries for charts. The generated summaries provide an interpretation of the chart and convey the key insights found within that chart. Our neural model is developed by extending the state-of-the-art model for the data-to-text generation task, which utilizes a transformer-based encoder-decoder architecture. We found that our approach outperforms the base model on a content selection metric by a wide margin (55.42% vs. 8.49%) and generates more informative, concise, and coherent summaries.
1 Introduction
Chart understanding can be difficult and time-consuming, motivating summaries that convey chart insights. The paper introduces a chart-summary corpus and a transformer-based neural model for generating such summaries.
- Interpreting charts can be difficult and time-consuming when users must compare many graphical marks to infer insights.
- Chart summaries can help users identify insights and support chart comprehension, including for visually impaired people and those with low cognitive abilities.
- Existing chart NLG systems commonly use statistical or template-based methods to produce textual data facts.
- The proposed transformer-based model learns from chart-summary pairs and uses data variable substitution to generate more factually correct statements.
- The paper introduces a corpus of 8,305 charts paired with underlying data tables and human-written summaries.
- The study evaluates the model against a baseline, while also releasing its source code and dataset publicly.
2 Related Work
Prior chart-description systems largely relied on planning, statistical, or template-based generation. This paper instead positions data-driven learning over chart-summary pairs as a way to improve generality and linguistic variation.
- Earlier systems used planning-based approaches to map chart data and marks into caption structure and textual descriptions.
- Automatic insight-generation systems typically infer potentially important data facts statistically and present them with natural-language sentences and charts.
- Template-based chart-generation methods may lack generality and provide fewer grammatical and lexical variations than data-driven models.
- Data-to-text research has used sequence-to-sequence models for jointly learning content selection and surface realization across domains.
- Transformer models were selected as the base because prior work found them more fluent and coherent than sequence-to-sequence counterparts.
3 Chart Summarization Dataset
The paper introduces a large-scale chart-summarization dataset pairing chart images, underlying data, metadata, and human-written summaries. It includes bar and line charts whose concise summaries typically describe and compare salient chart information.
- The dataset addresses the lack of large publicly available collections of charts paired with human-generated summaries.
- The corpus contains 8,305 charts collected from Statista with data tables, titles, axis labels, and concise summaries.
- 7,726 of 8,305 charts lacked x-axis labels, so regular-expression and named-entity methods identified temporal labels and annotators supplied remaining labels.Human annotators handled 1,353 remaining missing labels by examining chart-related text and the chart.
- The dataset includes simple and complex bar charts as well as simple and complex line charts.Complex charts include stacked or grouped bars and multiple-line charts.
- Chart summaries average 5.2 sentences and 113.4 tokens, while data tables average 32.3 cells and summaries commonly describe and compare salient points.Common salient information includes extremes such as highest and lowest values.
4 The Chart-to-Text Model
The model adapts a transformer encoder-decoder for chart data by adding chart-aware inputs, positional embeddings, content selection, and data-variable substitution. These changes help select records for mention and ground generated tokens in chart data.
- Base Model: The base model encodes record features, applies binary content selection, and decodes a summary from the encoder representation.Its input features are entity, type, value, and information.
- Our Proposed Approach: The proposed approach modifies record inputs to include chart-specific information and uses the encoder’s binary prediction to select mentioned records.The decoder predicts each next token using the encoder output and previously generated summary tokens.
- Our Proposed Approach: Positional embeddings are reintroduced because chart data often contains ordered temporal or ordinal dimensions.
- Our Proposed Approach: The substitution process targets hallucinations by grounding generated references in chart data rather than treating them as ordinary tokens.The paper identifies factual errors from irrelevant predicted tokens as a particular concern for cross-domain chart data.
- Our Proposed Approach: Data-variable substitution replaces references to chart data, titles, and axis labels during training and reverses the mapping during testing.The seven variable categories include subjects, dates, axis labels, titles, table cells, trends, and scales.
- Training: The model uses two training-label types: record labels indicate whether records appear in summaries, and token labels indicate whether summary tokens occur in records.
5 Evaluation
The evaluation compares the proposed model with a baseline using automatic metrics, human judgments, factuality checks, and qualitative analyses. Across these evaluations, the model generally produces more informative, concise, coherent, and factually correct summaries, while remaining errors include hallucinated or irrelevant tokens.
- 5.1 Automatic Evaluation: The automatic evaluation compares the proposed model with a baseline using BLEU and a content selection metric based on records shared with the gold summary.The study also conducts human evaluation and qualitative analysis to examine summary quality and trade-offs.
- 5.1 Automatic Evaluation: The proposed model slightly improves BLEU and outperforms the baseline by a wide margin on content selection.The authors associate this improvement with generating summaries in terms of data variables rather than directly memorized data values, helping address hallucination.
- 5.2 Human Evaluation: The proposed model scores at least over 1 point higher than the baseline on informativeness, conciseness, and coherence in human evaluation.The evaluation samples 40 charts across four chart types and uses three respondents per statistic.
- 5.2 Human Evaluation: The baseline performs slightly better on fluency, scoring 3.78 compared with 3.73 for the proposed model.The human evaluation assesses informativeness, conciseness, coherence, and fluency on a 5-point Likert scale.
- 5.2 Human Evaluation: Over 50% of the proposed model’s statements were judged factually correct, compared with 22% for the baseline.Partially correct statements accounted for 15.56% of the proposed model’s outputs and 7.24% of the baseline’s.
- 5.3 Qualitative Analysis: Positional embeddings produced summaries with more meaningful discourse structure and salient information in a qualitative comparison.The analysis also found that the baseline could repeat memorized summaries, whereas the proposed model still occasionally generated irrelevant tokens and incorrect variable indices.
- 5.4 Error Analysis: Hallucination remains a common error, especially when the model predicts an incorrect data-variable index or an irrelevant token.The latter error commonly occurs for chart domains with low coverage in the training set, producing differences in fluency and irrelevant-token frequency across domains.
6 Conclusion and Future Work
The paper introduces a transformer-based neural approach and dataset for automatic chart summarization, reporting summaries that are more informative, concise, coherent, and factually correct than the base model. Future work targets broader data coverage, additional comparisons, and accessible chart-summarization applications.
- The approach generates more informative, concise, coherent, and factually correct chart summaries than the base model.
- The authors plan larger datasets spanning more diverse domains and additional chart types to improve model quality and generalizability.
- Future evaluation will compare the method with additional models developed for data-to-text generation.
- The authors envision interactive chart-summarization applications that improve accessibility for blind and visually impaired people through audio.