PaperlayerView exact revision on arXiv

Source-linked AI summary

NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries

Arpit Narechania, Arjun Srinivasan, John Stasko

arXiv:2008.10723v3cs.HC

TL;DR

Visualization NLIs are difficult to build because developers must combine NLP, analytic-task interpretation, and visualization design. NL4DV provides a Python toolkit that converts a tabular dataset and natural-language query into a JSON analytic specification with inferred attributes, tasks, and Vega-Lite charts. The paper demonstrates its use for prototyping and integrating visualization NLIs, while noting that broader benchmarking and more reliable or customizable inference remain future work.

  • Problem

    Building visualization NLIs requires developers to implement non-trivial NLP and visualization-interpretation pipelines, often without specialized NLP expertise.

  • Method

    NL4DV processes a tabular dataset and natural-language query, infers attributes and analytic tasks, and returns structured JSON plus ordered Vega-Lite specifications.

  • Results

    NL4DV supports visualization NLI prototyping through applications including Jupyter visualizations, Vega-Lite editing, ambiguity widgets, and multimodal interaction.

  • Takeaways & Limitations

    Developers can use NL4DV to create new visualization NLIs or add natural-language querying to existing visualization systems.

  • Takeaways & Limitations

    Initial validation used datasets with 300–6000 rows and up to 27 attributes, while formal benchmarking on a large labeled query corpus remains future work.

Abstract

from arXiv · show

Natural language interfaces (NLIs) have shown great promise for visual data analysis, allowing people to flexibly specify and interact with visualizations. However, developing visualization NLIs remains a challenging task, requiring low-level implementation of natural language processing (NLP) techniques as well as knowledge of visual analytic tasks and visualization design. We present NL4DV, a toolkit for natural language-driven data visualization. NL4DV is a Python package that takes as input a tabular dataset and a natural language query about that dataset. In response, the toolkit returns an analytic specification modeled as a JSON object containing data attributes, analytic tasks, and a list of Vega-Lite specifications relevant to the input query. In doing so, NL4DV aids visualization developers who may not have a background in NLP, enabling them to create new visualization NLIs or incorporate natural language input within their existing systems. We demonstrate NL4DV's usage and capabilities through four examples: 1) rendering visualizations using natural language in a Jupyter notebook, 2) developing a NLI to specify and edit Vega-Lite charts, 3) recreating data ambiguity widgets from the DataTone system, and 4) incorporating speech input to create a multimodal visualization system.

1 INTRODUCTION

Visualization NLIs must interpret natural-language queries into attributes, analytic tasks, and suitable visualizations, but building this pipeline requires substantial NLP and visualization expertise. NL4DV addresses this need with a high-level toolkit and JSON-based output for prototyping and integration.

  • Motivation: Visualization NLI developers must implement query interpretation in addition to interface and visualization rendering.Existing NLP tools do not remove the need to understand and assemble the underlying techniques.
  • Query challenges: Queries vary from explicitly specifying attributes, tasks, and chart types to implicitly expressing them through values or terms such as “average.”The IMDb examples illustrate progressively more difficult interpretation requirements.
  • Query challenges: Ambiguous queries can leave both the intended attribute and analytic task unclear, requiring systems to detect ambiguity and infer suitable visualizations.For example, “rating” may map to multiple attributes, while a scatterplot can suggest correlation.
  • NL4DV contribution: NL4DV accepts a tabular dataset and natural-language queries, inferring data attributes and analytic tasks before returning ordered Vega-Lite specifications.Its built-in mappings connect attributes, tasks, and visualizations.
  • NL4DV contribution: The toolkit formalizes inferred information as a JSON-based analytic specification that developers can parse programmatically.This supports building new visualization NLIs and adding natural-language querying to existing systems.

2 RELATED WORK

Prior visualization NLIs use parsing and interaction techniques to map natural language to visualization actions, while visualization toolkits and grammars simplify chart construction. NL4DV builds on these foundations as an interface-agnostic toolkit that exposes inferred attributes and tasks in structured form.

  • Prior visualization NLIs: Visualization NLIs support capabilities for specifying or interacting with visualizations, answering data questions, and controlling systems through natural language.The related systems surveyed differ in interface and capability scope.
  • Prior visualization NLIs: Articulate, DataTone, FlowSense, Eviza, and Evizeon apply parsing or semantic techniques to support visualization generation and interaction.DataTone additionally uses GUI widgets to resolve detected query ambiguities.
  • NL4DV’s position: These systems commonly infer data attributes and analytic tasks to determine user intent and generate responses.NL4DV adopts lexical and dependency parsing while formalizing the inferred information in JSON.
  • NL4DV’s position: Unlike custom engines that translate queries directly into system actions, NL4DV is interface-agnostic and provides programmatically parseable structured output.Developers can use the output with different interfaces and system components.
  • Visualization grammars: Vega-Lite provides NL4DV’s underlying visualization grammar because its concise declarative specifications support visualization design and prototyping.This choice also aligns with Vega-Lite’s use in web-based and Python-based visual analysis.

3 NL4DV OVERVIEW

NL4DV encapsulates query processing and visualization recommendation in a toolkit pipeline that converts natural-language queries into structured analytic information and chart specifications. Its design goals emphasize accessible, modular, ambiguity-aware, and customizable NLI development.

  • Pipeline overview: A typical visualization NLI pipeline collects a query, infers attributes and analytic tasks, recommends visualization specifications, and renders them through a library.NL4DV encapsulates the query processor and recommendation components.
  • Design goals: NL4DV is designed to flatten the NLP learning curve by exposing high-level functions rather than requiring developers to implement query-processing mechanics.The primary target users are developers without NLP experience.
  • Design goals: Its modular output lets developers use default Vega-Lite recommendations or connect inferred attributes and tasks to other visualization components.The formalized representation is intended for programmatic integration.
  • Design goals: NL4DV marks whether information was inferred explicitly or implicitly and highlights ambiguities in its response.This helps developers decide how to use the output and whether to provide interface cues such as ambiguity widgets.
  • Design goals: The toolkit supports dataset-specific aliases and values so domain terminology can participate in attribute and task inference.This goal addresses domain-specific datasets and vocabulary.

4 NL4DV DESIGN AND IMPLEMENTATION

NL4DV exposes a high-level query-analysis API that infers dataset metadata, attributes, analytic tasks, and visualizations from natural-language queries. Its pipeline parses queries, matches phrases to dataset concepts, and packages results into modular visualization-oriented output.

  • API and output: A single analyze_query(query) call returns an attributeMap, taskMap, and visList of relevant visualization specifications.The response is a JSON object that developers can use to render charts or access inferred attributes and tasks separately.
  • Data interpretation: NL4DV initializes with a tabular dataset and derives attribute types, value ranges, and domains as metadata for query interpretation.This metadata supports analytic-task inference and visualization-specification generation.
  • Attribute inference: NL4DV matches query N-grams against attributes, aliases, and values using syntactic and semantic similarity, mapping matches meeting a 0.8 threshold to attributes.The attributeMap records query phrases, inference types, and ambiguous matches; value references can produce implicit attribute inferences.
  • Task inference: NL4DV identifies Correlation, Distribution, Derived Value, Trend, and Filter tasks, distinguishing visualization-determining base tasks from filters.The task-inference stage uses parsed phrase relationships, such as mapping “relationship” to Correlation and linking it to referenced attributes.
  • Query parsing: The query parser converts units, assigns POS tags, builds a dependency tree, removes selected stopwords, stems terms, and generates N-grams.These parsed representations are used during the remaining stages of query interpretation.
  • Visualization generation: NL4DV compiles inferred visualizations into a visList whose Vega-Lite specifications, inference types, attributes, and tasks can be rendered or passed to other recommendation engines.This modular output supports integration with alternative rendering libraries and custom visualization recommendation systems.

5 EXAMPLE APPLICATIONS

NL4DV supports several application patterns, from rendering natural-language-specified charts in notebooks to integrating natural language into editors, ambiguity widgets, and multimodal systems. Across these examples, developers parse NL4DV’s returned analytic structures and visualization specifications to add natural-language capabilities.

  • Using NL4DV in Jupyter Notebook: NL4DV renders the first visualization in its visList through a Python wrapper function in Vega-Lite-capable environments such as Jupyter Notebook.The render vis(query) function automatically renders the first visualization returned for the query.
  • NL-Driven Vega-Lite Editor: The Vega-Lite editor uses NL4DV’s visList to present alternative chart designs, including colored and colored-plus-faceted scatterplots.For a query about college debt and earnings, the user selects the faceted scatterplot as the active chart.
  • Recreating Ambiguity Widgets in DataTone: NL4DV can reproduce DataTone-like ambiguity widgets by exposing ambiguous attributes and values through its returned attributeMap and taskMap.In the Olympics example, “medals” maps to four possible attributes, while “hockey” and “skating” are ambiguous values for Sport.
  • Adding NL Input to an Existing Visualization System: NL4DV augments existing visualization systems with natural-language input, including speech commands for specifying charts and filtering points in the multimodal MMPLOT system.Speech input is converted to text, then taskMap and visList are parsed to apply filters and update chart specifications.
  • Adding NL Input to an Existing Visualization System: Implementing the example applications would typically require hundreds of lines of additional code, whereas NL4DV requires one analyze query(query) call and a few parsing lines.This lets developers focus more on interface design and user experience.

6 DISCUSSION AND FUTURE WORK

The discussion identifies evaluation gaps, experiments with follow-up queries, and future improvements to interpretation, query coverage, and customization. It also reports initial testing on tabular datasets and response times.

  • 6.1 Evaluation: Initial validation used datasets with 300-6000 rows and up to 27 attributes, with response times of 1-18 sec. (mean: 3 sec.).These timings were reported from a MacBook Pro with a 6-core 2.9GHz processor and 16GB RAM running MacOS Catalina version 10.15.5.
  • 6.1 Evaluation: The authors identify formal benchmarking on a large corpus of natural-language queries and longitudinal usability studies as needed evaluation.They propose collecting labeled visualization-specification utterances and gathering feedback from visualization and NLP developers.
  • 6.2 Supporting Follow-up Queries: NL4DV does not currently support follow-up queries, but an experimental dialog parameter analyzes filtering and encoding changes using prior responses.With dialog=true, conversational centering identifies missing attributes, tasks, or visualization details in follow-up queries.
  • 6.3 Improving Query Interpretation and Enabling Additional Query Types: Future interpretation work targets better attribute-type inference and more reliable task detection beyond the current lexicon- and dependency-based approach.The authors cite uncertainty in task keywords and conflicts between keywords and existing data attributes as motivating cases.
  • 6.3 Improving Query Interpretation and Enabling Additional Query Types: NL4DV is primarily geared toward visualization-specification queries, leaving question answering and visualization-formatting tasks for future support.Adding these query types would require new task categories, parameters, computed answers, and view changes in the output format.
  • 6.4 Balancing Simplicity and Customization: The toolkit simplifies use through defaults but does not yet support custom NLP models for attribute and task inference.Future customization must also ensure that outputs from custom models compile into NL4DV’s specification or extend it with additional information.

7 CONCLUSION

NL4DV supports prototyping visualization NLIs by converting datasets and natural-language queries into JSON-based analytic specifications. Example applications show its use across notebooks, web systems, existing visualization tools, and open-source development resources.

  • 7 CONCLUSION: NL4DV generates a JSON-based analytic specification containing attributes, tasks, and visualizations inferred from a dataset and natural-language query.The specification supports visualization creation in Jupyter notebooks, web-based visualization NLIs, and existing visualization tools.
  • 7 CONCLUSION: Example applications demonstrate NL4DV for natural-language visualization in Jupyter notebooks, web-based NLIs, and existing visualization systems.The authors provide NL4DV and the example applications as open-source software.
Loading 2008.10723v3…