PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Natural Language Interfaces for Data Visualization: A Survey

Leixian Shen, Enya Shen, Yuyu Luo, Xiaocong Yang, Xuming Hu, Xiongshuai Zhang, Zhiwei Tai, Jianmin Wang

arXiv:2109.03506v2cs.HC

TL;DR

V-NLI research has expanded, but the field lacks a comprehensive framework for organizing its systems and identifying knowledge gaps. This survey reviews 57 systems using seven visualization-pipeline stages and proposes directions to guide future research.

  • Problem

    The growing V-NLI literature lacks a comprehensive organization of systems, knowledge gaps, and research challenges for the expanding community.

  • Method

    The survey exhaustively reviews relevant work from 2000–2021 and classifies V-NLIs across seven information-visualization pipeline stages.

  • Results

    The survey collects 57 V-NLI papers and provides a systematic overview of the characteristics currently supported and studied in V-NLI.

  • Takeaways & Limitations

    The resulting overview and proposed future directions are intended to guide follow-up research on natural-language interfaces for data visualization.

  • Takeaways & Limitations

    The survey is scoped to interfaces that accept natural-language queries and automatically output appropriate visualizations.

Abstract

from arXiv · show

Utilizing Visualization-oriented Natural Language Interfaces (V-NLI) as a complementary input modality to direct manipulation for visual analytics can provide an engaging user experience. It enables users to focus on their tasks rather than having to worry about how to operate visualization tools on the interface. In the past two decades, leveraging advanced natural language processing technologies, numerous V-NLI systems have been developed in academic research and commercial software, especially in recent years. In this article, we conduct a comprehensive review of the existing V-NLIs. In order to classify each paper, we develop categorical dimensions based on a classic information visualization pipeline with the extension of a V-NLI layer. The following seven stages are used: query interpretation, data transformation, visual mapping, view transformation, human interaction, dialogue management, and presentation. Finally, we also shed light on several promising directions for future work in the V-NLI community.

1 INTRODUCTION

Visualization-oriented natural language interfaces (V-NLIs) complement traditional WIMP interaction by letting users express analysis intents in natural language rather than translating them into tool-specific operations. This survey organizes the growing V-NLI research landscape using seven stages of an information visualization pipeline and identifies directions for future work.

  • Motivation: Traditional WIMP interfaces impose a steep learning curve because users must translate analysis intents into tool-specific visualization operations.This limitation is contrasted with V-NLI, which lets users express their analysis intents in their own terms.
  • Motivation: V-NLIs emerged as a complementary input modality that uses natural-language queries for data visualization alongside traditional WIMP interaction.Their development was enabled by advances in Natural Language Processing and supporting NLP toolkits.
  • Challenges: Designing V-NLIs is challenging because human language is ambiguous and underspecified, conversational context is difficult to maintain, and system capabilities lack discoverability.The introduction presents these challenges as motivations for the development of numerous academic and commercial systems.
  • Survey scope: The survey addresses the growing V-NLI literature by organizing existing work, categorizing the research landscape, identifying knowledge gaps, and helping newcomers understand community challenges.It surveys seven information-visualization-pipeline stages: query interpretation, data transformation, visual mapping, view transformation, human interaction, dialogue management, and presentation.

2 SURVEY LANDSCAPE

This survey focuses on visualization-oriented natural language interfaces that automatically produce visualizations from natural-language queries. It reviews relevant work from 2000–2021 and organizes V-NLI research through a seven-stage visualization pipeline.

  • Scope: V-NLIs accept natural-language queries and automatically output appropriate visualizations through keyboard, speech, or selected-text input.The survey scope includes visualization-oriented interfaces rather than general natural-language systems.
  • Related fields: Related fields, including visualization recommendation, are discussed alongside V-NLI when they share overlapping pipeline stages or technologies.The survey uses these related discussions to broaden coverage and explain their importance and relationship to V-NLI.
  • Survey methodology: The review covered relevant VIS, HCI, NLP, and DMM journals and conferences from 2000–2021.The initial Google Scholar search using “natural language” AND “visualization” returned 1436 papers.
  • Survey methodology: After screening titles, abstracts, and, when necessary, full texts, the authors analyzed 57 V-NLI papers and collected 283 related works.The review used progressively detailed screening to determine relevance to V-NLI.
  • Analytical framework: The survey classifies V-NLI systems using seven stages: query interpretation, data transformation, visual mapping, view transformation, human interaction, dialogue management, and presentation.These stages extend the classic information visualization pipeline with a V-NLI layer.

3 CLASSIFICATION OVERVIEW

The survey extends the classic information visualization pipeline with a V-NLI layer and classifies systems across seven stages from query interpretation to presentation. The stages can overlap within one utterance, support iteration, and may be interpreted contextually.

  • Framework: The classification framework adds a V-NLI layer to the information visualization pipeline and develops categorical dimensions around visualization generation.The framework is based on the pipeline presented by Card et al. and focuses on how V-NLI facilitates visualization generation.
  • Pipeline stages: Query interpretation parses natural-language queries into data attributes and analytic tasks while addressing underspecified utterances.Semantic and syntax analysis first discovers hierarchical query structures.
  • Pipeline stages: Data transformation generates alternative data subsets or derivations through data-plane operations such as aggregation and pivoting.Because most analyzable raw data is tabular, the survey names this stage transformed data.
  • Pipeline stages: Visual mapping connects information extracted from natural-language queries to visual structures through spatial substrates, graphical elements, and graphical properties.Layout configuration is part of the spatial substrate, while graphical elements include points, lines, surfaces, and volumes.
  • Pipeline stages: View transformation renders visual structures into views, but it is rarely involved in V-NLI systems.Common forms include navigation, animation, and visual distortion such as a fisheye lens.
  • Pipeline stages: Human interaction, dialogue management, and presentation connect visualization states, contextual conversation, and generated visual displays across the pipeline.A single utterance can span multiple stages, users can iterate over stages, and utterances can be interpreted contextually.

4 QUERY INTERPRETATION

Query interpretation underpins subsequent V-NLI stages by parsing language, inferring users’ analytic tasks and relevant data attributes, and supplying defaults for underspecified utterances. Existing systems commonly use NLP parsing and task-keyword matching, but struggle with visualization-specific language and vague queries.

  • Query interpretation parses natural-language input, infers analytic tasks and data attributes, and supplies defaults for underspecified utterances.It is described as the foundation for all subsequent stages.
  • Semantic parsers apply NLP steps such as tokenization, part-of-speech tagging, entity recognition, stemming, dependency parsing, and N-gram generation.FlowSense additionally tags visualization-relevant utterances, including column names, node labels, node types, and dataset names, using placeholders.
  • Most V-NLI systems infer analytic tasks by matching query tokens with predefined task keywords, while NL4DV supports Correlation, Distribution, Derived Value, Trend, and Filter.The survey notes that integrated tasks remain limited and are often rule-based, motivating broader hierarchical and data-type-specific task models.
  • Systems extract explicitly named and implicitly referenced data attributes, including attributes identified through values or aliases.One approach uses BERT embeddings for natural-language queries and table headers, while another uses a transformer-based sequence-to-sequence model.
  • Most V-NLIs handle specification-oriented queries more reliably than underspecified or vague queries involving synonyms, abbreviations, or terminology variations.Defaults, ambiguity widgets, and personalized inference are presented as possible responses, but complex combinations of vague modifiers remain unsupported.

5 DATA TRANSFORMATION

This section examines how V-NLI systems transform raw data into focused data for visual insights, including aggregation, binning, and grouping. It also discusses the relationship between V-NLI and natural-language database interfaces.

  • Data transformation operations: V-NLI data transformation commonly applies aggregation, binning, and grouping to quantitative data.Examples include DeepEye’s visualization-language aggregation functions, Arklang’s finite aggregation operators, and Eviza’s regular spatial binning.
  • Automated data insights: V-NLI systems can directly calculate and visualize results when transformation information is extracted from natural-language queries.Without that information, inspecting datasets and constructing meaningful visualizations remains time-consuming.
  • Automated data insights: Because meaningful analysis may otherwise require time-consuming dataset inspection, practical systems should automatically recommend data insights.Systems such as DataSite and Voder generate data facts during visual data exploration.
  • NLI4DB and V-NLI: NLI4DB translates natural-language queries into executable database queries, while V-NLI can visualize their results.Not every dataset query requires visualization; some questions are best answered with exact values.
  • NLI4DB and V-NLI: V-NLI and NLI4DB can complement each other through visualizations containing answers, exact value responses, and shared benchmark resources.WikiSQL and Spider are identified as NLI4DB benchmarks that could support the design of V-NLI benchmarks.

6 VISUAL MAPPING

V-NLIs perform visual mapping by configuring spatial substrates, selecting graphical elements, and applying graphical properties. Existing systems primarily use 2D layouts, traditional marks, and color, size, and shape, while implicit mark inference and broader encodings remain challenging.

  • Spatial substrate: V-NLIs map data onto spatial substrates by specifying layout configurations such as x- and y-axis assignments, but unspecified mappings create a very large search space.Some systems accept queries such as showing GDP on the y axis and year on the x axis grouped by Country Code.
  • Spatial substrate: Most V-NLIs are limited to 2-dimensional space along x and y axes, although 3-dimensional and hyperdimensional representations are possible.Examples of broader representations include adding a z axis and using parallel coordinates.
  • Graphical elements: V-NLIs can explicitly interpret graphical elements such as scatterplots, but mark information is usually inaccessible and may be missed when expressed implicitly.Systems may recognize “create a scatterplot of mpg and cylinders,” yet can fail to infer a scatterplot from “point.”
  • Graphical properties: Color, size, and shape are the most commonly applied graphical properties in V-NLIs, whereas orientation and texture are rarely used.Color also requires discretion because it can be unfriendly to people with color-vision deficiencies and carry different cultural meanings.

7 VIEW TRANSFORMATION

View transformation is a rarely used stage in V-NLI systems, with only a few systems supporting natural-language control, mainly for viewpoint navigation. Future work could address animation, data-GIFs, and visual distortions.

  • 7 VIEW TRANSFORMATION: View transformations include location probes, viewpoint controls, and distortions that reveal information, scale or translate views, or modify visual structure.These transformations can be applied after visual mapping when visualization specifications are rendered through libraries such as D3.
  • 7 VIEW TRANSFORMATION: Only a few V-NLI systems support natural-language control over view transformations, and their support is mainly limited to viewpoint navigation.
  • 7 VIEW TRANSFORMATION: Future improvements can target animation, data-GIFs, and visual distortions.

8 HUMAN INTERACTION

V-NLI human interaction helps users express intents through ambiguity resolution, system discoverability, and multimodal input. Research highlights limitations in heuristic ambiguity handling, template-based suggestions, speech input, and gesture integration.

  • Ambiguity Resolution: Ambiguity widgets return decision rights to users when vague natural language causes failures in intent recognition or attribute extraction.They involve detecting ambiguity and presenting widgets, alongside approaches that infer defaults for underspecified utterances.
  • Ambiguity Resolution: Current ambiguity resolution mainly relies on heuristics, while probabilistic frameworks and richer widgets could address broader ambiguities and improve user experience.The discussion separates algorithmic ambiguity detection from user-facing widget presentation.
  • System Discoverability: System discoverability helps users understand available operations and phrase queries correctly through autocompletion and command suggestions matched to visualizations and datasets.These mechanisms provide interpretable hints for analytical question formulation and information sense-making.
  • System Discoverability: Discoverability remains underdeveloped: utterance realization is template-based, suggestion timing and selection need research, and existing tools primarily support keyboard input.Speech-based discoverability remains open for further investigation.
  • Multimodal Interaction: Multimodal V-NLI combines natural language with direct manipulation, speech, pen, touch, keyboard, and gestures to support more natural and engaging visual exploration.Examples include Orko for graph exploration and Valletto’s speech-based conversational interface with multitouch gestures and conventional input.
  • Multimodal Interaction: Speech input faces triggering difficulties, missing assistive features, and transcription errors, while gesture recognition and tracking remain limited areas of research.Typing and speech are common NL-query modalities, but speech and gesture interaction require further attention.

9 DIALOGUE MANAGEMENT

Dialogue management enables V-NLI users to iteratively refine visualizations and investigate interests through conversational queries. Key challenges include maintaining visualization-state coherence and resolving references to visual elements across language and direct-manipulation interactions.

  • Dialogue Management: Conversational V-NLIs let users refine existing visualizations and progressively explore aspects of interest through follow-up queries.Users may pose queries based on prior queries, enabling deeper investigation of a chart.
  • Dialogue Management: Evizeon supports back-and-forth visual analysis by filtering charts for “measles in the UK” and adding detail to the “orange spike” in a follow-up query.The system applies pragmatic principles to interpret references to visual properties in the chart.
  • Dialogue Management: A conversational transitions model maintains coherent visualization states by representing transitions such as retain, shift, continue, and reset.The model describes how visualization states transition during analytical conversations.
  • Dialogue Management: Conversational V-NLI systems should infer users’ transitional goals and respond to each visualization state rather than relying only on visualization recommendation engines.Existing VisRec-based systems are described as insufficient for analytical conversation.
  • Dialogue Management: Users often prefer additional content beyond exact answers, motivating insight-driven conversational interfaces.At the language level, existing systems mainly use NLP toolkits for co-reference resolution but lack detailed visualization-element modeling.
  • Co-reference Resolution: Co-reference resolution identifies entities mentioned by pronouns or multiple noun phrases, including visual elements referenced after direct manipulation.Multimodal systems can link follow-up queries to selected or related visual elements and data, while broader modality support could include eye-gaze and gesture.

10 PRESENTATION

V-NLI systems produce visualizations ranging from traditional charts to maps, networks, and infographics, while increasingly using natural language as an output modality. Presentation research includes annotations, narrative storytelling, chart captions, and visual question answering, but remains limited by template-based methods, narrow chart coverage, and dataset-dependent interpretation.

  • V-NLI outputs include traditional charts as well as richer forms such as maps, networks, and infographics, with natural language increasingly complementing visual presentation.
  • Annotation: Annotations explain and emphasize dataset insights by generating natural language statements and mapping them appropriately to visualizations.ADVISor generates annotated visualizations to answer natural-language questions about tabular data.
  • Annotation: Annotation methods can turn data into stories, but their scalability is weak because most current systems are template-based.Advanced NLG models, richer data types and annotation forms, and an annotation specification language are proposed directions.
  • Narrative storytelling: Narrative storytelling combines visual and generated textual elements to communicate data insights, yet existing systems often target one information type and rely on templates.Combining timelines, maps, and other information types, alongside learning-based approaches and dedicated datasets, could improve story expressiveness and quality.
  • Chart captions: Chart-description systems have evolved from predominantly rule-based methods toward pipelines that parse charts and generate captions, but they mostly support simple chart types.Further directions include broader chart-type coverage, improved language quality and descriptive ability, and large datasets spanning diverse domains and chart types.
  • Visual Question Answering: Visual Question Answering answers questions about charts or infographics by encoding and fusing visualizations and questions, but interpretation is limited by visualization-agnostic models and dataset diversity.Visualization-specific models and transferring existing models to other areas are identified as promising directions.

11 RESEARCH OPPORTUNITY … 11.3 Interaction

The survey identifies open research opportunities in V-NLI across knowledge, models, interaction, presentation, datasets, and applications. The covered challenges include missing domain and dataset semantics, reliance on basic NLP methods, and limited interaction support for discoverability and user history.

  • 11 RESEARCH OPPORTUNITY: The survey organizes V-NLI’s research challenges and opportunities into knowledge, model, interaction, presentation, dataset, and application aspects.This organization is guided by an information visualization pipeline proposed by Card et al.
  • 11.1 Domain Knowledge: None of four evaluated state-of-the-art open-source V-NLIs recognized that nitrogen dioxide and NO2 share meaning or that age relates to birth year.The systems included both academic and commercial V-NLIs, highlighting the need for domain knowledge when extracting query attributes.
  • 11.2 NLP Model: Most existing V-NLI systems rely on hand-crafted grammar rules or typical NLP toolkits, motivating use of advanced models such as ELMO, BERT, GPT-3, and CPM-2.The paper notes that several state-of-the-art NLP models have reached human performance on specific tasks and have been applied to data visualization.
  • 11.2 NLP Model: Existing attribute-matching technologies remain at the letter-matching level, while V-NLIs need deeper semantic matching that considers dataset semantics alongside query semantics.Some systems use semantic parsing toolkits such as SEMPRE, but query semantics alone are insufficient.
  • 11.3 Interaction: Open-ended textboxes limit discoverability because users may not know valid inputs or supported chart types, whereas real-time prompts can clarify features and correct errors.Prior practices report this benefit for prompts during query typing.
  • 11.3 Interaction: Although conversational V-NLIs analyze queries in context, few systems use interaction history to address varying user preferences among multiple suitable chart types.The paper identifies user interaction history as an underused source for satisfying query intent.

11.4 Presentation · 11.5 Dataset · 11.6 Application

The survey identifies future directions for V-NLI presentation, datasets, and applications. It calls for natural language as both input and output, immersive visualization support, dedicated training datasets, and broader real-world integration.

  • 11.4 Presentation: V-NLI systems should use natural language as both input and output to democratize access to visual analysis.The proposed direction combines visual question answering, natural language generation, and annotation in a hybrid system.
  • 11.4 Presentation: Existing V-NLI systems mostly support 2D visualization, while immersive natural-language interfaces for data visualization remain unavailable.Prior work has augmented static visualizations with virtual content, but no system was identified as supporting immersive NLI for data visualization.
  • 11.4 Presentation: The survey suggests enriching the data visceralization pipeline with natural-language interaction for immersive analytics.Lee et al. introduced data visceralization and a conceptual pipeline, providing a related reference for this direction.
  • 11.5 Dataset: Large-scale data collection can facilitate community development, but general NLP datasets cannot be directly applied as V-NLI training samples.The survey notes that visualization researchers have begun collecting large datasets for V-NLI, summarized in Table 8.
  • 11.6 Application: V-NLI research is being extended to real-world applications that integrate visual analysis with natural-language interfaces.Examples include spatially uncertain mobile trajectories, medical imaging and biomedical research, audio travel guidance, and educational path visualization.
  • 11.6 Application: A query engine uses intuitive natural-language input to convert, store, and retrieve spatially uncertain mobile trajectories.This application was proposed by Huang et al.
  • 11.6 Application: A V-NLI was proposed to support medical imaging and biomedical research.This application was proposed by Leo John et al.
  • 11.6 Application: Because V-NLI can be integrated as a visualization-system module, the survey anticipates further application opportunities.PathViewer illustrates this integration by using flow-diagram and NLP ideas to visualize students’ intermediate steps.

12 CONCLUSION

Visualization-oriented natural language interfaces have developed rapidly over the past two decades as a complementary input modality for visual analytics. This survey fills a gap by comprehensively reviewing V-NLI characteristics and proposing promising future-work directions.

  • V-NLIs have developed rapidly over the past two decades as a complementary input modality for visual analytics.
  • The survey fills the community’s lack of a comprehensive overview of related V-NLI research.
  • It identifies characteristics currently concerned with and supported by V-NLI and proposes promising directions for future work.
Loading 2109.03506v2…