Source-linked AI summary
Data2Vis: Automatic Generation of Data Visualizations Using Sequence to Sequence Recurrent Neural Networks
Victor Dibia, Çağatay Demiralp
TL;DR
Creating visualizations with expressive grammars is difficult because users must select fields, transformations, and visual mappings. Data2Vis addresses this by translating dataset specifications into Vega-Lite specifications with an attention-based LSTM sequence model. Qualitative evaluations show valid and varied generated visualizations, while the authors identify field-selection and training-data limitations.
Problem
Users must manually choose data attributes, transformations, and visual mappings, making expressive visualization authoring difficult for people with limited time or visualization expertise.
Method
Data2Vis uses an attention-based encoder–decoder LSTM to translate JSON dataset specifications into Vega-Lite visualization specifications.
Results
Qualitative evaluations show that Data2Vis learns valid Vega-Lite vocabulary and syntax, produces univariate and multivariate visualizations, and generates a richer variety of charts than Voyager 2 under the comparison setting.
Takeaways & Limitations
The paper presents Data2Vis as a baseline for learning visualization-generation rules from examples and as a foundation for visualization generation at scale.
Takeaways & Limitations
About 15–20% of tests exhibit limitations such as phantom fields or unintuitive field selections, and the authors relate these issues partly to small and insufficiently diverse training data.
Abstract
from arXiv · showhide
Rapidly creating effective visualizations using expressive grammars is challenging for users who have limited time and limited skills in statistics and data visualization. Even high-level, dedicated visualization tools often require users to manually select among data attributes, decide which transformations to apply, and specify mappings between visual encoding variables and raw or transformed attributes. In this paper we introduce Data2Vis, a neural translation model for automatically generating visualizations from given datasets. We formulate visualization generation as a sequence to sequence translation problem where data specifications are mapped to visualization specifications in a declarative language (Vega-Lite). To this end, we train a multilayered attention-based recurrent neural network (RNN) with long short-term memory (LSTM) units on a corpus of visualization specifications. Qualitative results show that our model learns the vocabulary and syntax for a valid visualization specification, appropriate transformations (count, bins, mean) and how to use common data selection patterns that occur within data visualizations. Data2Vis generates visualizations that are comparable to manually-created visualizations in a fraction of the time, with potential to learn more complex visualization strategies at scale.
1 INTRODUCTION
Visualization tools trade speed against expressivity, while grammars and higher-level tools still demand substantial specification effort. Data2Vis frames visualization design as translation from data specifications to Vega-Lite specifications and demonstrates automated generation.
- Motivation: Visualization tools span a speed–expressivity spectrum, with expressive APIs requiring more programming effort and dedicated tools favoring ease and speed.Declarative grammars occupy an intermediate position but can still be difficult to learn and tedious to specify.
- Motivation: Users may need to choose attributes, statistical transformations, and mappings between visual encodings and raw or summarized data.These decisions can be daunting for users with limited time or statistics and visualization expertise.
- Motivation: Rule- and heuristic-based automation is limited by the need to enumerate rules explicitly and does not directly exploit expertise encoded in existing visualizations.The paper motivates learning visualization rules from examples containing both data and design context.
- Data2Vis: Data2Vis formulates visualization design as sequence-to-sequence translation from data specifications to visualization specifications.It trains an LSTM-based neural translation model on a corpus of Vega-Lite specifications.
- Data2Vis: The model is demonstrated as a way to ease visualization authoring for novices and help experienced users jump-start design.The work also integrates Data2Vis into a publicly available web application.
2 RELATED WORK
Prior work develops visualization grammars, automated design tools, and neural synthesis or translation methods. Data2Vis extends these lines by learning end-to-end data-to-visualization generation rather than relying on manually specified rules or constraints.
- Declarative Visualization Specification: Vega-Lite is a concise, high-level grammar built on Vega, with lower expressivity, and its corpus pairs datasets with corresponding visualization specifications.Data2Vis trains on this Vega-Lite corpus.
- Declarative Visualization Specification: Declarative grammars compose marks, scales, encodings, and guides to support customization, but greater expressivity can reduce creation speed and increase learning difficulty.Data2Vis aims to bridge this speed–expressivity gap.
- Automated Visualization: Data2Vis casts automated visualization as a learning problem and generates visualizations directly from data through an end-to-end approach.This differs from recent recommendation methods that use feature extraction and manually specified constraints.
- Automated Visualization: Earlier automated visualization approaches depend on manually created rules and heuristics, which can be voluminous, tedious to update, and incomplete for edge cases.A learning approach is presented as potentially improving system maintenance and scalability.
- Deep Neural Networks for Synthesis: Deep neural networks have been used for synthesis tasks including drawings, images, music, and code, as well as for machine translation between natural and domain-specific languages.Data2Vis applies this translation perspective to formal data and visualization languages.
3 PROBLEM FORMULATION
Data2Vis treats visualization generation as translation between a JSON dataset sequence and a Vega-Lite specification sequence. Its encoder–decoder architecture uses bidirectional recurrent states and attention to generate target tokens from source context.
- Problem Formulation: The input sequence contains dataset fields and values in JSON format, while the output is a valid Vega-Lite visualization specification.This establishes visualization generation as a sequence-to-sequence translation task.
- Problem Formulation: The encoder–decoder system is jointly trained to maximize the probability of producing the correct target sequence for a source sequence.The encoder represents the source and the decoder produces the translation.
- Sequence Modeling: Bidirectional recurrent units encode each input position using information from both preceding and following tokens.The forward RNN reads the sequence in order, while the backward RNN reads it in reverse.
- Sequence Modeling: Attention lets the decoder focus on relevant source-sequence aspects while generating each output token.It also supports mappings between source and target sequences of different lengths.
- Tokenization: Data2Vis uses character-level tokenization because its source and target sequences consist primarily of symbols rather than learnable word groups.The approach is designed for formal specification sequences.
4 MODEL
Data2Vis uses an attention-based encoder-decoder RNN to translate source data sequences into visualization specifications. The model uses bidirectional encoding, recurrent decoding, and LSTM units selected over GRUs for better validity results.
- The model uses an attention-based encoder-decoder architecture to generate target visualization sequences from source data sequences.A bidirectional RNN encoder produces source states, while an RNN decoder predicts target tokens using prior tokens, recurrent state, and an attention-derived context vector.
- The decoder’s context vector is a weighted average of source states that captures information relevant to predicting each target token.
- A 2-layer bidirectional RNN encoder and 2-layer RNN decoder each use 512 LSTM units.
- LSTM cells produced better valid JSON and valid Vega-Lite specifications than GRU cells in the authors’ experiments.
5 DATA AND PREPROCESSING
Data2Vis preprocessing and training target field selection, data types, and transformations while reducing the burden of character-level sequence modeling. The training corpus contains 4300 Vega-Lite examples derived from 11 datasets, with fixed optimization settings.
- Learning objectives: The model must select fields, distinguish data types, and apply transformations appropriate to each field when generating visualizations.Supported transformations include aggregate, bin, calculate, filter, and timeUnit at the view level, with aggregate, bin, sort, and timeUnit also available at the field level.
- Preprocessing: Character-based modeling requires longer sequences and substantial hidden layers and parameters to capture long-term dependencies.The preprocessing strategy scaffolds learning by reducing vocabulary and sequence length requirements.
- Training data: 4300 Vega-Lite visualization examples based on 11 distinct datasets form the training dataset.The examples include six visualization types and three transforms.
- Training data: The sample dataset contains area, bar, circle, line, point, and tick visualizations, plus aggregate, bin, and timeUnit transforms.
- Training: The model is trained end-to-end with negative log likelihood, stochastic gradient descent, Adam, and a fixed learning rate of 0.0001.Input dropout is 0.5, and source and target sequences are capped at 500 symbols.
6 RESULTS
The evaluation uses qualitative inspection on held-out Rdatasets data, attention weights, beam search, and comparison with Voyager 2. Results indicate that Data2Vis generates valid, varied specifications using data types, transformations, and common selection patterns.
- Qualitative evaluation: The held-out Rdatasets evaluation produced valid univariate and multivariate visualizations, suggesting that the model captures aspects of visualization generation.Rdatasets contains 1147 datasets originally distributed with R and its add-on packages.
- Qualitative evaluation: The model learned Vega-Lite vocabulary and syntax, including quotes, brackets, symbols, keywords, and appropriate string and quantitative field types.
- Qualitative evaluation: Qualitative results suggest that the model applies appropriate bin and aggregate transformations, including means on numeric fields, and combines common data-selection patterns for bivariate plots.
- Beam search: Beam search with beam width=15 generated diverse plots spanning bar, area, and line chart types while focusing on two fields and subsetting by sex.The decoder outputs parallel beam results to explore alternative visualization candidates rather than only the highest-probability translation.
- Attention analysis: Attention weights were nonmonotonic and emphasized characters associated with visualization fields, assigned data types, and applied transformations.The model placed strong weights on “num0” and its value “8” while generating a quantitative field specification.
- Prototype: The authors developed a web prototype supporting dataset import, visualization generation, and visualization update for analyst exploration.
7 DISCUSSION
Data2Vis demonstrates a neural approach to generating visualization specifications and can support both novice authoring and expert exploration, while remaining limited in field selection, transformation complexity, and training-data coverage.
- Impact and Use Case: The model learns visualization patterns that generalize across real-world datasets, including categorical variables such as gender, sex, and location.
- Impact and Use Case: Data2Vis can jumpstart expert exploration with valid initial specifications that analysts can iteratively correct, although outputs may be insufficient for unusual data or complex transforms.
- Limitations: About 15-20% of tests exhibit limitations such as phantom fields, unintuitive field choices, or low-information selections.
- Limitations: The model mainly generates univariate and bivariate plots and cannot apply complex transforms or use multiple variables.
- Limitations: The relatively small and insufficiently diverse training data limits generalization and prevents learning field-distribution properties such as means, ranges, and categories.
- Impact and Use Case: Beam search generates multiple visualization specifications from one dataset, addressing the fact that data can map to multiple valid visualizations.
8 CONCLUSION
The paper frames visualization generation as sequence-to-sequence translation from data specifications to visualization specifications and demonstrates deep-model generation of univariate and bivariate plots. It presents Data2Vis as a baseline for future deep-learning systems that generate visualizations at scale.
- 8 CONCLUSION: Visualization is formulated as a sequence-to-sequence translation problem that maps data specifications to visualization specifications.
- 8 CONCLUSION: Data2Vis demonstrates the efficacy of a deep sequence-to-sequence model for generating univariate and bivariate plots.
- 8 CONCLUSION: The paper identifies initial failure conditions, proposes remediation ideas, and outlines an agenda for future work.
- 8 CONCLUSION: The authors position the problem formulation and model as an appropriate baseline for future deep-learning research on automated visualization generation.
- 8 CONCLUSION: The approach sets the stage for systems that learn to generate visualizations at scale and may support guided visual data exploration.