PaperlayerView exact revision on arXiv

Source-linked AI summary

Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content

Alan Lundgard, Arvind Satyanarayan

arXiv:2110.04406v1cs.HCcs.CL

TL;DR

Visualization descriptions may improve access to meaningful chart information, but their usefulness is difficult to evaluate because their semantic content and reader reception are poorly understood. The paper develops a four-level semantic model from 2,147 sentences and evaluates it with blind and sighted readers, finding significant differences in preferred content. It concludes that accessible captions should attend to reader preferences and communicate trends and statistics more richly.

  • Problem

    Visualization descriptions may improve accessibility, but their usefulness is difficult to evaluate because their semantic content and reader reception are poorly understood.

  • Method

    The paper develops a four-level semantic-content model through grounded theory analysis of 2,147 sentences and evaluates description usefulness with blind and sighted readers.

  • Results

    Blind and sighted readers differ significantly in which semantic content they rank as most useful, while both generally prefer mid-level content.

  • Takeaways & Limitations

    Accessible captions should reflect reader preferences and communicate chart trends and statistics rather than only design elements or high-level insights.

  • Takeaways & Limitations

    The model focuses on English sentences and canonical chart types, so extensions to other languages and bespoke charts may be warranted.

Abstract

from arXiv · show

Natural language descriptions sometimes accompany visualizations to better communicate and contextualize their insights, and to improve their accessibility for readers with disabilities. However, it is difficult to evaluate the usefulness of these descriptions, and how effectively they improve access to meaningful information, because we have little understanding of the semantic content they convey, and how different readers receive this content. In response, we introduce a conceptual model for the semantic content conveyed by natural language descriptions of visualizations. Developed through a grounded theory analysis of 2,147 sentences, our model spans four levels of semantic content: enumerating visualization construction properties (e.g., marks and encodings); reporting statistical concepts and relations (e.g., extrema and correlations); identifying perceptual and cognitive phenomena (e.g., complex trends and patterns); and elucidating domain-specific insights (e.g., social and political context). To demonstrate how our model can be applied to evaluate the effectiveness of visualization descriptions, we conduct a mixed-methods evaluation with 30 blind and 90 sighted readers, and find that these reader groups differ significantly on which semantic content they rank as most useful. Together, our model and findings suggest that access to meaningful information is strongly reader-specific, and that research in automatic visualization captioning should orient toward descriptions that more richly communicate overall trends and statistics, sensitive to reader preferences. Our work further opens a space of research on natural language as a data interface coequal with visualization.

1 INTRODUCTION

Visualization descriptions can improve access to chart information, but accessibility practices remain uncommon and their semantic content is difficult to evaluate. This paper introduces a four-level semantic model and finds that blind and sighted readers differ in which content they find useful.

  • Visualizations can communicate critical public-health information efficiently while excluding people with disabilities when their information is difficult to read and understand.
  • Accessible chart captions and alt text remain rare, while automatic captioning methods often lack accessibility-guideline consultation and empirical evaluation with disabled readers.
  • The model categorizes semantic content into four levels: construction properties, statistical relations, perceptual and cognitive phenomena, and domain-specific insights.
  • 2,147 sentences supported the model, which was evaluated through rankings of descriptions by 30 blind and 90 sighted readers.
  • 3,600 ranked descriptions revealed significant reader-group differences, although both groups generally preferred mid-level semantic content.
  • The findings suggest that captions for blind readers should convey chart trends and statistics rather than only low-level design elements or high-level insights.

2 RELATED WORK

Related work spans accessible media, HCI, computer vision, and natural language processing, but these literatures use divergent methods and terminology. Automatic captioning approaches differ according to whether they process rasterized charts or structured chart representations.

  • Visualization-description research spans accessible media, HCI, computer vision, and NLP, whose efforts have largely remained siloed and use terms such as “caption” inconsistently.
  • Automatic captioning methods process either rasterized chart images with CV and NLP or structured specifications of chart construction.
  • Rasterized-image approaches commonly use an encoder-decoder pipeline in which a vision model supplies salient features to a language model generating captions.
  • Structured chart representations preserve construction information and can support structured rendering through SVG or scenegraph APIs.
  • Structured output representations can lose input semantics such as relationships among axes, legends, and nested components.
  • Visualization toolkits can use structured input to add ARIA attributes and simplify template-based caption generation.

2.2 Accessible Media and Human-Computer Interaction

Accessible-media and HCI research includes expert-authored guidelines and non-expert description tasks. These approaches can produce richer descriptions, but human-authored content requires attention to potential bias.

  • Automatic-methods researchers rarely collaborate directly with disabled people or consult accessibility guidelines, unlike some HCI and accessible-media efforts.
  • STEM graphics guidelines provide best practices for conveying chart content across printed, audio, tactile, and web-based media.
  • Crowdsourcing uses remote non-experts for tasks that remain infeasible for automatic methods, including online accessibility and remote description services.
  • Non-expert visualization-description tasks may involve free-form writing or filling expert-authored sentence templates.
  • These approaches can yield richly detailed, natural-sounding descriptions and training data, but human-authored descriptions may contain biases.

2.3 Natural Language Hierarchies and Interfaces

Prior work frames visualization language through linguistic levels and natural-language interfaces. This paper distinguishes semantic content from syntactic complexity and focuses on language conveyed by visualizations rather than language used to construct them.

  • Linguistics-inspired research examines how natural language describes and interacts with visualizations.
  • Demir et al.’s hierarchy organizes propositions and syntactic complexity, whereas this model orders the semantic content conveyed by sentences.
  • The semantic model is agnostic to sentence length, clause count, and conjunctions.
  • Kosslyn’s framework distinguishes syntactic, semantic, and pragmatic levels for analyzing charts, informing the paper’s linguistics-inspired model construction.
  • Natural Language Interfaces construct and explore visualizations, while this work primarily studies sentences conveyed by visualizations.
  • The model may help provide textual alternatives to inline charts for users who prefer not to see graphics in conversational interfaces.

3 CONSTRUCTING THE MODEL: EMPLOYING THE GROUNDED THEORY METHODOLOGY

The authors used grounded theory to build a corpus and inductively refine semantic-content categories for visualization descriptions. A larger survey produced 582 descriptions comprising 2,147 sentences across varied visualization types, domains, and difficulty levels.

  • Grounded theory process: Grounded theory guided iterative data collection, open coding, axial coding, and refinement of semantic-content categories.The methodology was chosen for constructing theory inductively in a domain lacking a dominant theory.
  • Grounded theory process: Preliminary data combined descriptions from journalistic visualizations, student-authored descriptions, and richly descriptive author-written descriptions.The visualizations covered chart types including bar, line, and scatter plots, and domains such as public health, climate change, and gender equality.
  • Survey corpus: 582 descriptions comprising 2,147 sentences were collected in the larger online survey used to test and refine the preliminary categories.The survey followed the initial inductive analysis and formed the larger corpus.
  • Survey corpus: 50 visualizations were curated across chart type, dataset topic, and visual-conceptual difficulty.The set included bar, line, and scatter plots spanning academic, business, and nonbusiness data journalism domains.
  • Survey design: 120 participants each described five randomly selected visualizations, producing at least 10 authored descriptions per visualization.Participants were explicitly prompted to include trends and insights beyond basic chart elements and encodings.
  • Survey results: The corpus contained 9.1% Level 1, 35.1% Level 2, 42.9% Level 3, and 12.9% Level 4 sentences.The authors interpret this fairly balanced distribution across Levels 2–4 as evidence that prompting captured a breadth of visualized content.

4 A FOUR-LEVEL MODEL OF SEMANTIC CONTENT

The paper presents a four-level model classifying visualization-description content from construction properties through statistical relations, perceptual phenomena, and domain-specific context. The levels describe conveyed semantics independently of the computational methods used to generate them.

  • Level 1: Elemental and Encoded Properties: Level 1 describes elemental and encoded visualization properties, including chart type, titles, legends, channels, axis labels, and scales.These sentences read construction details directly from the chart or its specification without synthesis or interpretation.
  • Level 1: Elemental and Encoded Properties: Level 1 content can be generated from structured visualization specifications using formal grammars or templates.When structured specifications are unavailable, computer-vision and natural-language methods may instead operate on rasterized visualizations.
  • Level 2: Statistical Concepts and Relations: Level 2 reports dataset-based statistical concepts and relations, including means, standard deviations, extrema, correlations, and comparisons.Unlike Level 1, this content requires access to the visualization’s backing dataset.
  • Level 3: Perceptual and Cognitive Phenomena: Level 3 conveys perceptual and cognitive phenomena such as complex trends, multifaceted patterns, and noteworthy exceptions that may not be apparent through non-visual analysis.Its examples use natural-sounding language to synthesize overall patterns and identify gaps or exceptions.
  • Computational considerations: Level 3 is characterized as perceiver-dependent, whereas Levels 1 and 2 are perceiver-independent.The authors associate current Level 3 generation primarily with human perception and cognition, despite possible future machine capabilities.
  • Level 4: Contextual and Domain-Specific Insights: Level 4 concerns contextual and domain-specific knowledge and experience related to the visualized data.The model’s levels are indexed to semantic content, not to the computational means by which sentences may be generated.

5 APPLYING THE MODEL: EVALUATING THE EFFECTIVENESS OF VISUALIZATION DESCRIPTIONS

The model was applied in a mixed-methods evaluation to compare the usefulness of visualization descriptions for blind and sighted readers. The evaluation examined whether descriptions conveying different semantic-content levels communicate meaningful information to different reader groups.

  • Evaluation: The evaluation asked 30 blind and 90 sighted readers to rank descriptions authored at varying semantic-content levels.The authors used the model to structure comparisons of description usefulness across reader groups.

5.1 Evaluation Design

The evaluation used representative visualizations and unlabeled, randomized descriptions isolating each semantic-content level. Readers ranked usefulness in an open-ended, holistic task intended to capture their own interpretations of usefulness.

  • Stimuli: 15 visualizations represented three chart types, three topic categories, and three difficulty levels.The set included five visualizations for each dimension: type, topic, and difficulty.
  • Ranking task: Each visualization was paired with four unlabeled descriptions, one for each semantic-content level, presented in random order.The descriptions were ranked from 1, least useful, to 4, most useful.
  • Stimuli: Exemplar descriptions were assembled from participant-authored sentences so each text chunk contained content from only one semantic level.This addressed the fact that earlier participant descriptions often mixed levels or omitted some levels.
  • Ranking task: Participants ranked usefulness in contextualized scenarios, while blind readers were told that the visualization was inaccessible through screen-reader technology.The study intentionally left usefulness open rather than defining it as efficiency, informativeness, or clarity.

5.2 Participants

The study included 30 blind and 90 sighted readers, recruited through different channels and completing differently timed, compensated tasks.

  • The participant groups comprised 30 blind readers and 90 sighted readers recruited through Prolific, the blind community, and Twitter.
  • Blind readers were expected to use screen readers, while sighted readers qualified through English proficiency and absence of color vision deficiency.
  • Blind readers completed an approximately 1-hour task at $50 per hour, whereas sighted readers completed an approximately 20-minute task at $10-12 per hour.
  • Blind participants used VoiceOver, JAWS, NVDA, or other screen readers, with VoiceOver and JAWS each used by 10 readers.
  • Readers ranked semantic content from least useful to most useful on a four-level scale in the study summarized by Table 4.

5.3 Quantitative Results

Blind and sighted readers differed significantly in how they ranked the four semantic content levels, with each group favoring different levels.

  • 1,800 rankings per reader group were aggregated in Table 4 heatmaps showing how often each semantic level received each rank.
  • p < 0.001 in Friedman’s tests for both groups, rejecting equal mean ranks across the four semantic content levels.
  • Sighted readers ranked Levels 3 and 4 as most useful, while Levels 1 and 2 were least useful in pair-wise comparisons.
  • Blind readers ranked Levels 2 and 3 as most useful, while Levels 1 and 4 were least useful.
  • Level 1 was the only level with a bimodal ranking distribution, despite being ranked least useful by most readers in both groups.

5.4 Qualitative Results

Qualitative feedback showed that blind readers encountered visual barriers frequently and preferred factual descriptions combining statistical and trend information, while sighted readers expressed weaker but positive support.

  • 5.4.1 Descriptions Are Important to Both Reader Groups: All blind readers reported encountering inaccessible visualizations, most often in educational materials, social media, and newspaper websites.
  • 5.4.1 Descriptions Are Important to Both Reader Groups: Blind readers strongly agreed that visual-only public information was inaccessible and that textual visualization descriptions were important.
  • 5.4.1 Descriptions Are Important to Both Reader Groups: A blind participant described undescribed visualizations as unavailable, linking inaccessible data with feeling excluded, vulnerable, and devalued.
  • 5.4.1 Descriptions Are Important to Both Reader Groups: Sighted readers were divided about barriers to reading visualizations but supported natural language descriptions overall.
  • 5.4.2 Higher-Level Semantic Content Is Not Always More Useful: 63% of blind readers opposed subjective interpretations, contextual information, or editorializing in descriptions.
  • 5.4.3 Trends and Statistics Are Particularly Useful: Blind readers generally preferred Levels 2 and 3, especially descriptions combining statistics with overall perceptible trends in natural-sounding language.
  • 5.4.4 Combinations of Content Levels Are Likely Most Useful: 40% of blind readers said the most useful descriptions would combine content from multiple semantic levels.
  • 5.4.4 Combinations of Content Levels Are Likely Most Useful: The authors argue that automatic captioning should communicate overall trends and statistics while remaining sensitive to reader preferences.

6 DISCUSSION AND FUTURE WORK

The four-level model supports descriptive, evaluative, and generative work, reframing natural language as an interface into visualizations and data alongside graphics.

  • The model has practical implications for accessible data representations and theoretical implications for visualization’s relationship with natural language.
  • 6.1 Natural Language As An Interface Into Visualization: Natural language can serve as an interface into visualization by exposing the semantic content visualizations convey.
  • 6.1 Natural Language As An Interface Into Visualization: The grounded-theory model provides descriptive power, while the reader study provides evaluative power by revealing divergent group preferences.
  • 6.1 Natural Language As An Interface Into Visualization: Future work can use the model’s generative power to develop multimodal interfaces and accessible data representations tailored to reader needs.
  • 6.2 Natural Language As Coequal With Visualization: Natural language may be coequal and complementary with visualization, with statistics or explanations sometimes better conveyed through language and clusters through visual representation.
  • 6.2 Natural Language As Coequal With Visualization: The paper frames its contribution as reversing Bertin’s direction: using visualization as an occasion for a return to language.
  • 6.2 Natural Language As Coequal With Visualization: Because the model focuses on English sentences and canonical chart types, extensions to other languages and bespoke charts may be warranted.
Loading 2110.04406v1…