PaperlayerView exact revision on arXiv

Source-linked AI summary

VisText: A Benchmark for Semantically Rich Chart Captioning

Benny J. Tang, Angie Boggust, Arvind Satyanarayan

arXiv:2307.05356v1cs.CVcs.HCcs.LG

TL;DR

Automatic chart-captioning methods struggle to describe perceptual and cognitive features such as complex trends and patterns, despite captions’ value for comprehension and accessibility. VisText introduces a 12,441-pair benchmark with image, data-table, and scene-graph representations plus semantically layered captions, and evaluates text-based, image-guided, and prefix-tuned models. The models generate coherent, semantically rich captions and perform on par with state-of-the-art chart-captioning models, while the study identifies six error categories and computational constraints.

  • Problem

    Existing chart-captioning approaches struggle to generate captions that express perceptual and cognitive features such as complex trends and patterns.

  • Method

    VisText provides 12,441 chart-caption pairs with image, data-table, and scene-graph representations, and the authors train text-based, image-guided, and semantic prefix-tuned models.

  • Results

    The models generate coherent, semantically rich captions and perform on par with state-of-the-art chart-captioning models across machine translation and text generation metrics.

  • Takeaways & Limitations

    VisText supports chart captions that describe construction, statistics, trends, and complex patterns, with semantic customization for differing user needs.

  • Takeaways & Limitations

    Computational constraints required truncating inputs to 1,024 characters and outputs to 512 characters, likely affecting scene-graph models disproportionately.

Abstract

from arXiv · show

Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts' construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph -- a hierarchical representation of a chart's visual elements akin to a web page's Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work.

1 Introduction

VisText addresses the difficulty of automatically generating chart captions that capture perceptual and cognitive features, not merely chart construction. It contributes a 12,441-pair benchmark and models that produce semantically rich captions.

  • Motivation: Captions can improve chart-data recall and comprehension, while often providing the only access to data for people with visual disabilities.Manual chart captioning is laborious, so charts are rarely captioned in practice.
  • Motivation: Existing automatic methods largely describe chart construction rather than complex trends and patterns preferred by both sighted and blind readers.These methods include templates, image captioning, and data-to-text translation.
  • Contribution: VisText provides 12,441 chart-caption pairs with rasterized images, backing data tables, scene graphs, and captions spanning construction, statistics, perceptual features, and cognitive features.The scene graph is a hierarchical representation analogous to a web page’s DOM.
  • Approach: The authors train text-based, image-guided, and semantic prefix-tuned captioning models using VisText’s varied chart representations and semantic content.These models are intended to support captions with customized semantic content.
  • Results: The models generate coherent, semantically rich captions, perform on par with state-of-the-art chart captioning models, and exhibit six identified error categories.Evaluations use machine translation and text generation metrics alongside qualitative analysis.

2 Related work

Prior chart-captioning approaches use heuristics, image captioning, or data-to-text translation, but their datasets and outputs contain too little semantically rich chart insight. Consequently, generated captions often emphasize statistics over complex trends and patterns.

  • Heuristic-Based Chart Captioning: Heuristic systems extract propositions, summary statistics, or data facts and assemble standardized natural-language descriptions.These descriptions lack the variation and linguistic construction associated with semantically rich captions.
  • Chart Captioning as Image Captioning: Neural image-captioning systems adapt visual captioning pipelines to chart images, including chart-type classification and textual-content detection.The cited approaches differ in pipeline complexity, but share chart images as input.
  • Chart Captioning as Text Translation: Data-to-text systems translate time series or tabular structures into captions using recurrent or transformer architectures.These methods are closely related to VisText’s text-based captioning models.
  • Limitations of Prior Work: Existing datasets contain insufficient training examples of captions expressing perceptual and cognitive insights that distinguish charts from data tables.As a result, generated captions are largely limited to reporting statistics, which readers prefer less than complex trends and patterns.

3 The VisText Dataset

VisText was designed to address shortcomings in chart representations and caption coverage by combining 12,441 diverse charts with multiple modalities and semantically layered captions. Its dataset pairs chart structure with crowdsourced descriptions of statistical, perceptual, and cognitive content.

  • Dataset Motivation: Existing datasets trade off perceptual fidelity and chart semantics by representing charts mainly as images or data tables.Images capture trends and outliers but not rich element relationships, while tables may not uniquely determine a chart.
  • Chart Representations: VisText provides 12,441 crowdsourced English captions for bar, line, and area charts, represented as rasterized images, data tables, and scene graphs.Scene graphs preserve perceptual fidelity and chart semantics in a computationally tractable representation.
  • Chart Representations: Scene graphs encode the position, content, and semantic role of visual elements such as marks, titles, axes, and gridlines.They are extracted from rendered charts and flattened through depth-first traversal for language-model inputs.
  • Caption Generation: VisText captions distinguish L1 construction, L2 statistics and relations, and L3 perceptual and cognitive phenomena, while leaving L4 domain-specific insights for future work.Captions begin with L1 content and then present L2 and L3 content.
  • Caption Generation: L1 content is generated algorithmically, whereas L2 and L3 content is crowdsourced from participants who describe observed trends and patterns without repeating L1 information.Invalid or unclear descriptions and problematic content were removed through verification and heuristics.
  • Dataset Analysis: 95% of coded statements expressed L2 or L3 content, comprising 39.7% L2 and 55.7% L3 statements.The analysis manually coded 230 crowdsourced captions and found 2.4% L1 statements and 2.2% uncategorizable statements.

4 Chart Captioning Models

VisText supports text-based, multimodal, and semantically customizable chart captioning models. The modeling setup uses textual chart representations, chart images, and semantic prefix-tuning to vary caption content.

  • Text-Based Captioning: The authors fine-tune pretrained ByT5 models to translate textual chart representations into natural-language captions.ByT5’s byte encoding supports noisy inputs and reduces preprocessing and out-of-dictionary issues.
  • Image-Guided Captioning: The models compare scene-graph and data-table inputs while adapting image-guided translation to combine chart images with textual representations.The multimodal variants include image-scene-graph and image-data-table models.
  • Image-Guided Captioning: Image-guided models use visual information complementary to textual inputs, including affordances relevant to perceiving trends.All image-guided models use VL-T5 and extracted visual features from chart images.
  • Semantic Customization: Semantic prefix-tuning treats L1 and L2/L3 caption generation as separate tasks so users can vary the semantic content of outputs.Models are trained with and without prefixes specifying the desired semantic level.

5 Evaluation and Results

VisText models are evaluated with quantitative caption-quality metrics and qualitative analysis of semantically rich statements. Text-based models generally capture chart construction well, while image inputs and image-guiding provide limited benefit; qualitative analysis reveals recurring factual and generation errors.

  • Quantitative Model Performance: All non-image models generate high-quality L1 captions containing chart titles, axis names, and axis scales.Relation Generation measures how often these chart fields appear in generated captions.
  • Quantitative Model Performance: Scene-graph models perform similarly to data-table models across evaluative metrics, making scene graphs a potentially preferred representation for future chart captioning.Scene graphs can be more easily extracted from web-based charts.
  • Quantitative Model Performance: Image-guided captioning does not significantly improve performance because chart images and textual representations often contain highly similar information.The captions rarely reference visual information absent from the scene graph or data table.
  • Quantitative Model Performance: Prefix-tuning does not significantly change performance but enables models to produce L1, L2/L3, and combined captions for semantic customization.Models with and without prefix-tuning are exposed to the same charts.
  • Qualitative Caption Evaluation: Approximately half of analyzed L2 and L3 statements are factually accurate, while generated captions often remain natural and include compound or complex sentences.The qualitative analysis sampled 512 statements and found recurring errors in the remaining statements.
  • Qualitative Caption Evaluation: The qualitative error analysis identifies identity, value, direction, stability, repetition, and nonsensical errors.Reported examples include off-by-one category identities, incorrect numerical values, trend-direction mistakes, repeated claims, and misinterpretations.

6 Discussion

VisText supports precise, semantically descriptive chart captions from multiple representations and enables captions customized to diverse user needs. However, factual inaccuracies make human intervention important, motivating mixed-initiative chart-authoring systems that can verify and correct generated captions.

  • 6 Discussion: VisText contains 12,441 charts with semantically rich captions and rasterized image, data table, and scene graph representations.The dataset supports textual, visual, and structured chart inputs.
  • 6 Discussion: Models fine-tuned on textual representations and image-guided inputs generate precise, semantically descriptive captions.Semantic prefix-tuned models also produce captions customized to diverse user needs.
  • 6 Discussion: VisText models perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics.
  • 6 Discussion: Mixed-initiative chart-authoring systems would let authors correct generated captions and support interactive verification using chart representations.The proposed systems could link caption segments to table rows or chart regions.
  • 6 Discussion: Chart captioning models are prone to factual inaccuracies that can have severe harmful consequences.

Limitations

The study is constrained by computational limits that especially affect scene-graph models and by a narrow chart-type scope. These limitations define boundaries for interpreting results and motivate future work on more complex visualizations, where scene graphs may remain more faithful to visual form.

  • Limitations: Memory limits forced the use of the smallest-parameter T5 and ByT5 variants, while transformer sequence complexity affected model performance.The quadratic relationship between sequence length and time/space complexity was especially consequential for byte-level sequences.
  • Limitations: Scene-graph inputs averaged 948 characters versus 426 for data tables, causing more incomplete sequences and truncation of long titles or axis names.Raw scene graphs contained richer visual information but were unavailable to the models after reduction.
  • Limitations: VisText is scoped to univariate bar, area, and line charts, leaving multivariate and other complex chart forms for future work.The initial scope was chosen because single-series line charts commonly expose data trends.
  • Limitations: As visualizations become more complex, scene graphs may be more fruitful than data tables because they directly encode visual form.VisText provides raw visualization specifications to support this future work.

Ethics Statement

The paper highlights risks from incorrect automated captions, including misleading readers and mismatches with accessibility needs. It recommends verification, author review, and interpretability work while noting that current semantic labels cover only one visualization-consumption task.

  • Incorrect captions can influence readers’ takeaways even when they conflict with the depicted data, with risks amplified by charts’ persuasive force.
  • Authors recommend identifying generated captions, enabling interactive verification, or using them as drafts in mixed-initiative authoring systems.
  • Automated captioning receives mixed reactions from people with disabilities because research priorities may not match their captioning needs.
  • Existing quantitative metrics can penalize contextually varying captions even when qualitative analysis judges them high quality.
  • VisText’s semantic labels derive from natural-language descriptions for visualization consumption and do not cover the broader range of visualization tasks.

B.2 Evaluation Details

The evaluation combines machine-translation, language-model, relation-generation, and distance metrics with claim-level qualitative analysis. Splitting captions into independent claims allows factual precision to be measured without treating an entire compound sentence as wrong after one mistake.

  • Models are evaluated with BLEU, Perplexity, Relation Generation, ROUGE, Word Mover’s Distance, and Translation Edit Rate.
  • Relation Generation evaluates chart title, axis names, and axis scales, while it is not separately reported for L2/L3 captions.
  • Qualitative evaluation iteratively samples captions until additional examples yield no marginal information about error types.
  • Evaluating independent, mutually exclusive claims enables more precise factual-precision estimates than sentence-level scoring.A single compound sentence can contain several claims, so one incorrect claim need not invalidate the others.

C Ablation Studies

Ablation studies test transformer backbones and stochastic L1 caption generation. Results show that changing the random seed produces nearly identical metrics, supporting diversity in the L1 dataset rather than seed-sensitive performance.

  • Ablation studies measure the effects of transformer architecture and stochastic L1 caption generation.
  • Transformer Backbone: The backbone comparison includes ByT5-small, T5-small with and without prefix-tuning, and BART-base.
  • L1 Caption Generation: Nearly identical metrics across two random seeds indicate that stochastic L1 generation captures diverse captions.

D Implementation Details

The implementation trains text-based and image-guided captioning models with specified model backbones, training schedules, input limits, and repeated runs. Image-guided models extend VLT5 and use visual features extracted from chart images.

  • Implementation code and package versions are documented in the project repository and accompanying tables.
  • Text-based models use 300M-parameter ByT5-small, fine-tuned for 50 epochs with Adam and a 5e−05 learning rate.
  • Text inputs are truncated to 1024 tokens and output captions to 512 tokens to fit GPU memory.
  • Each model is trained three times with and without prefix-tuning, reporting means and standard deviations.
  • Image-guided models extend VLT5 and extract visual features from chart images using Bottom-Up Feature Extraction with 36 bounding boxes per image.

D.3 Ablation Models

The ablation models use the default models’ parameters except for the parameter under investigation. Training runs use 16 virtual CPU cores, 128GB of memory, and eight Titan XP GPUs.

  • Ablation models retain the default models’ parameters while varying only the parameter of interest.
  • Training uses 16 virtual CPU cores on Xeon E5 hypervisors with 128GB of memory.
  • Training uses PCI pass-through access to eight NVIDIA Titan XP GPUs with 12GB of memory each.

E Additional VisText Dataset Details

The dataset details describe template-based Level 1 caption generation and a crowdsourced protocol for collecting descriptions of chart trends and patterns. The study uses eligibility, consent, attention checks, and open-ended responses to structure and assess participation.

  • L1 Caption Generation Process: Level 1 captions are randomly generated from three title templates and six axis templates.
  • L1 Caption Generation Process: The caption generator randomly includes axis scales and substitutes selected words with synonyms within templates.
  • Crowdsourced Study Protocol: Participants on Prolific complete visualization-description tasks designed to elicit trends and patterns without repeating information already provided.
  • Crowdsourced Study Protocol: Most participants could complete the study within 14 minutes, based on pilot runs used to calibrate task duration.
  • Crowdsourced Study Protocol: Each task includes an attention check on chart type, and submissions failing more than two checks are flagged for manual review.
  • Eligibility and Consent: Participants must be sighted, comfortable reading English and interpreting charts, over 18, and have JavaScript enabled.
Loading 2307.05356v1…