PaperlayerView exact revision on arXiv

Source-linked AI summary

AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization

Aoyu Wu, Yun Wang, Xinhuan Shu, Dominik Moritz, Weiwei Cui, Haidong Zhang, Dongmei Zhang, Huamin Qu

arXiv:2102.01330v2cs.HCcs.GR

TL;DR

Visualizations are increasingly treated as data processed by AI, but research questions, techniques, and representation formats span diverse disciplines. This survey organizes that literature through a taxonomy, reviews common AI4VIS tasks and approaches, and identifies research questions for managing and exploiting visualization data.

  • Problem

    AI4VIS research spans diverse disciplines with differing questions, techniques, and visualization representations, requiring a comprehensive cross-field foundation.

  • Method

    The survey analyzes an interdisciplinary corpus of 98 papers from 10 computer-science research areas and organizes the literature with a classification of visualization data, motivations, and AI approaches.

  • Results

    The survey presents a taxonomy that helps readers find relevant literature, understand current techniques, and identify areas for future research.

  • Takeaways & Limitations

    Treating visualizations as an emerging data format frames AI4VIS as a basis for new theories, problems, techniques, and applications.

  • Takeaways & Limitations

    Reverse engineering visualization programs remains computationally expensive, lacks robustness, and is currently impossible for bespoke charts.

Abstract

from arXiv · show

Visualizations themselves have become a data format. Akin to other data formats such as text and images, visualizations are increasingly created, stored, shared, and (re-)used with artificial intelligence (AI) techniques. In this survey, we probe the underlying vision of formalizing visualizations as an emerging data format and review the recent advance in applying AI techniques to visualization data (AI4VIS). We define visualization data as the digital representations of visualizations in computers and focus on data visualization (e.g., charts and infographics). We build our survey upon a corpus spanning ten different fields in computer science with an eye toward identifying important common interests. Our resulting taxonomy is organized around WHAT is visualization data and its representation, WHY and HOW to apply AI to visualization data. We highlight a set of common tasks that researchers apply to the visualization data and present a detailed discussion of AI approaches developed to accomplish those tasks. Drawing upon our literature review, we discuss several important research questions surrounding the management and exploitation of visualization data, as well as the role of AI in support of those processes. We make the list of surveyed papers and related material available online at ai4vis.github.io.

1 INTRODUCTION

The survey frames visualizations as an emerging AI-processed data format and organizes AI4VIS research around what visualization data is, why AI is applied, and how it is processed.

  • Motivation: Visualizations are increasingly created, shared, collected, and reused with AI, extending their role beyond artifacts designed solely for people.The paper presents this trend as evidence that visualizations are becoming a new data format processed by AI.
  • What is visualization data: Visualization data comprises digital representations with multimodal content, including visual encodings, encoded data, text, and images.These characteristics create challenges for designing tailored AI approaches.
  • Research landscape: AI4VIS research spans diverse communities with different motivations, research questions, techniques, and visualization formats.The survey therefore draws on literature from multiple computer science disciplines.
  • Why apply AI: The survey identifies visualization generation, enhancement, and analysis as three common goals for applying AI to visualization data.These goals are further classified into subcategories.
  • How to apply AI: The survey outlines seven common tasks and discusses corresponding AI approaches for each task.The task organization supports a structured discussion across research areas.

2 RELATED SURVEYS

Earlier surveys address automated visualization creation and recommendation, while this survey positions AI4VIS across broader research communities and visualization-data applications.

  • Survey scope: The paper motivates a broader synthesis because AI4VIS research spans different communities, goals, techniques, and visualization representations.This scope extends beyond surveys focused on particular automation tasks.
  • Automated visualization design: Prior work classifies automated visualization design systems as knowledge-based, data-driven, or hybrid approaches.A later survey systematically reviews visualization and infographic recommendation using this classification.
  • Related perspectives: Related surveys also examine visualization automation through database and communication-oriented research perspectives.These efforts form part of the literature that the present survey situates within AI4VIS.

3 METHODOLOGY

The survey uses a relation-search corpus and a what-why-how classification to organize interdisciplinary AI4VIS literature, while explicitly limiting scope and claiming neither comprehensiveness nor exhaustiveness.

  • Definition and scope: The survey focuses on AI techniques or systems that input or output visualization data, especially charts and infographics.It excludes scientific visualizations, chart-type-specific research, and work primarily analyzing human interaction data.
  • Search methodology: The corpus is built by manually traversing citation and reference networks, beginning with 2020 IEEE Visualization Conference papers and related surveys.Breadth-first traversal is used to reduce over-focus on one research line.
  • Corpus: 98 papers from 10 research areas form the interdisciplinary corpus, with Visualization contributing 34 of 98 papers.Human-computer Interaction and Databases are the next prominent areas, alongside several other computer science fields.
  • Corpus: The surveyed literature increases steadily over the last decade, with a surge since 2018 and a peak at 2020.Figure 2 reports publication counts by research area and year.
  • Limitations: The manually searched corpus is not claimed to be comprehensive or exhaustive.The authors describe it as sufficient for contextualizing the research landscape and identifying future opportunities.
  • Coding and classification: The classification organizes surveyed work around what visualization data is and its representation, why AI is applied, and how AI approaches are structured.The what axis covers internal and feature representations, while the how axis introduces seven common tasks.
  • Coding and classification: The task abstraction decomposes system papers into tasks and reconciles inconsistent vocabularies across disciplines.The supplemental website documents labels and supporting quotes for the categories.

4 DATA: WHAT IS VISUALIZATION DATA

The survey formalizes visualization data through its content formats and computational representations, emphasizing how these choices affect machine analysis. It organizes visualizations as graphics, programs, or hybrids, alongside internal and feature representations for AI tasks.

  • Content formats and representations: Visualization data is categorized into graphics, programs, and hybrids, while internal and feature representations support computation and machine learning.Internal representations remove unnecessary information for computation, whereas feature representations convert visualizations into mathematically convenient inputs.
  • Graphics: Raster graphics are common and easy to share but lose visualization semantics, requiring reverse engineering before automated analysis.Reverse engineering reconstructs information such as chart type, visual encoding, and underlying data, but robustness and accuracy remain open challenges.
  • Graphics: Vector graphics are less lossy than raster graphics because formats such as SVG describe visual elements as scalable shapes with styles.The low-level descriptions reduce the difficulty of reverse engineering compared with raster graphics.
  • Programs: Imperative languages offer flexible, open-ended visualization creation, but their irregularities and ambiguities hinder efficient machine extraction of semantics.The survey observes few efforts analyzing imperative programs because their unstructured nature complicates machine analysis.
  • Programs: Declarative languages encode visualization construction through semantic components, prescribed structures, sensible defaults, and constraints.Specifications such as Vega-Lite describe elements including data encodings, axes, and legend properties.
  • Feature representation: Visualization data is multimodal, combining graphics, programs, text, and underlying data as features for machine learning.Program features can preserve high-level visualization information, while text and multimodal feature fusion can improve chart detection and classification.
  • Feature representation: High-level visualization tasks remain relatively dissatisfactory when representations or features do not capture visualization-specific information.The survey identifies limitations in assessment and visual question answering, and argues that visualization-specific feature learning and multimodal fusion remain open directions.

5 GOAL: WHY APPLY AI TO VISUALIZATION DATA

The survey organizes why AI is applied to visualization data into generation, enhancement, and analysis, using whether visualizations are inputs or outputs and whether they are single or many. These goals span creating visualizations, retargeting existing ones, and exploiting visualization collections.

  • Goal taxonomy: The taxonomy identifies three goals: visualization generation, enhancement, and analysis.Generation produces visualizations, enhancement processes an input visualization, and analysis organizes and exploits a visualization collection.
  • Taxonomy design: The goal taxonomy uses visualization input/output and single/many visualization axes, merging output-single and output-many categories when their subcategories overlap.The matrix counts papers by goal, and papers may have multiple goals.
  • Visualization Generation: Visualization generation aims to ease time-consuming authoring by producing visualizations from data, anchors, designs, or contextual information.The survey distinguishes data-based, anchor-based, design-based, and context-based generation.
  • Visualization Enhancement: Visualization enhancement retargets existing visualizations through added information, natural-language descriptions, question answering, or interactive overlays.Examples include encoding information in visualization images, summarizing visualizations, answering questions about charts, and improving legibility.
  • Visualization Analysis: Visualization analysis manages and exploits collections through retrieval, usage and design-pattern mining, and guidance for examining chart ensembles.Mined patterns can support visualization recommendation, while chart ensembles can become objects of visual data analysis.

6 TASKS: HOW TO APPLY AI TO VISUALIZATION DATA

The survey abstracts AI4VIS into seven recurring tasks and examines how they connect across visualization goals. Transformation is often foundational, while reverse engineering remains constrained by sequential failures and bespoke chart types.

  • Task distribution: Recommendation dominates Visualization research, whereas Computer Vision and AI emphasize transformation and reasoning.Recommendation accounts for 17/34 Visualization papers; transformation and reasoning account for 8/16 and 7/16 papers in Computer Vision and AI research.
  • Transformation: Transformation converts visualization content between modalities, with reverse engineering reconstructing programs or data from graphics.It commonly precedes enhancement and analysis, supporting downstream querying and reasoning.
  • Reverse engineering: Reverse engineering decomposes graphics into semantic elements before composing those elements to extract visual encodings or underlying data.Decomposition uses computer vision and machine learning, while composition mainly uses visualization-domain heuristics.
  • Open challenges: Reverse engineering remains an open challenge for bespoke visualizations because deriving both visual encoding and underlying data is difficult.Sequentially dependent stages can propagate a single failure through the pipeline, while semi-automatic remedies remain chart-type-specific.

6.2 Assessment

Assessment measures visualization quality through scores or rankings and often supports generation and recommendation. The survey highlights generalizability, explainability, and subjective quality as persistent challenges.

  • Task definition: Assessment outputs either a numerical quality score or a relative ranking.Its main motivation is improving visualization design, including providing cost functions for automatic generation.
  • Scoring methods: Scoring-based methods are preferred for downstream optimization because they represent absolute quality rather than only relative quality.Voyager and Draco map rankings to numerical scores, while other approaches learn scores or rankings from empirical data.
  • Challenges: Machine-learning assessment approaches face poor generalizability and limited explainability.Many assume available datasets or specifications and support only limited chart types; Draco combines design constraints with a learned weighting function.
  • Open questions: Objective visualization qualities dominate existing work, while subjective qualities such as aesthetics remain underexplored.Crowdsourced judgments of subjective quality can be inconsistent and inaccurate.
  • Future direction: Future assessment systems are envisioned to produce human-readable explanations and useful design guidelines alongside quality judgments.

6.3 Comparison

Comparison estimates similarity or other metrics between visualizations and supports generation, querying, and collection management. Existing methods rely heavily on engineered features and distances, while querying increasingly uses structural representations to resolve ambiguity.

  • Role of comparison: Comparison supports anchor-based recommendation and query-by-example by providing metrics between visualizations.It is studied on both visualization programs and graphics.
  • Difference-based methods: Difference-based comparison models chart changes as operations with numerical costs, but under-specified operations make path search and exhaustive coverage difficult.GraphScape and Dziban explicitly represent edits and their costs, while extending operation coverage is labor-intensive.
  • Distance-based methods: Distance-based comparison converts visualizations into feature vectors and computes distances, making feature choice and distance-function choice central challenges.Features can come from graphics, text, data, or specifications.
  • Multi-source similarity: Combining multiple feature sources through concatenation or weighted distances leaves unresolved questions about how overall similarity should be measured.The survey describes both concatenated feature vectors and weighted hybrid distances.
  • Open challenges: Comparison remains dominated by hand-crafted features and distances rather than models fine-tuned on user feedback.ScatterNet learns scatterplot similarity from crowdsourced feedback, but its generalization to other statistical charts is unclear.
  • Querying: Querying retrieves visualizations relevant to user needs, often relying on transformation and comparison and operating on programs when image-space semantics are insufficient.Queries may use keywords, natural language, or structural constraints.
  • Querying: Structural queries improve retrieval by adding fields such as chart type, labels, legends, marks, and encodings to keyword-based search.Vega-Lite specifications can support broad constraints and query-by-example.
  • Querying challenges: Visualization indexing remains restricted to certain types, and extending it to diverse visualizations and raster graphics remains difficult.The survey also identifies limited empirical understanding of users’ information-seeking intentions.

6.5 Reasoning

Reasoning interprets visualizations to derive semantic information such as insights and summaries, using both visualization images and programs. The survey identifies deep-learning progress alongside major gaps in datasets and visualization-specific models.

  • Challenges and Methods: Reasoning derives high-level semantic information from visualizations rather than merely extracting visual encodings or producing numerical assessment scores.Common outputs include insights and text summaries.
  • Challenges and Methods: Reasoning supports visualization enhancement and may rely on reverse engineering to improve algorithm performance.
  • Discussion and Open Questions: Off-the-shelf CNNs were not currently good models for human graphical perception, while visualization reasoning remains under-explored because datasets and tailored feature-learning models are limited.Visualization details can substantially affect encoded data and conveyed meanings.
  • Challenges and Methods: Chart summarization methods range from short interpretive descriptions to explanations of high-level insights conveyed by charts.
  • Challenges and Methods: Visual question answering methods decode charts into data tables and query them, or use end-to-end deep learning to answer natural-language questions.

6.6 Recommendation

Recommendation automates visualization creation by selecting data, encodings, or both. Existing systems use enumeration, heuristics, optimization, and machine learning, but large design spaces and limited task coverage remain open challenges.

  • Challenges and Methods: Visualization recommendation includes data recommendation, encoding recommendation, and hybrid recommendation that decides both data and encodings.
  • Challenges and Methods: Data recommendation selects fields, transformations, or insights, with enumeration and template-based data facts serving as early approaches.Enumeration can impose a heavy burden on users.
  • Challenges and Methods: Insight recommendation commonly defines insight taxonomies with assessment metrics, while QuickInsights uses a unified formulation of insights and scoring metrics across types.
  • Challenges and Methods: Encoding recommendation covers data and non-data encodings, using ranking, heuristic rules, optimization, and machine-learning models.
  • Challenges and Methods: Hybrid recommendation combines data and encoding recommendation sequentially or formulates recommendation as an end-to-end optimization problem.
  • Discussion and Open Questions: Recommendation remains challenged by complex multi-objective optimization, limited machine-learning coverage, and top-k outputs that may not fit all users.

6.7 Mining

Mining analyzes collections of visualizations to discover design and data patterns. The survey highlights interactive exploration and recommender integration, while noting that current methods often rely on simple statistics and clustering.

  • Mining: Mining visualization data targets design patterns and data patterns, supporting visualization analysis of accumulated online visualization collections.
  • Mining Design Patterns: Design mining derives principles from existing artifacts, including common color ramps and other usage patterns.
  • Mining Design Patterns: Simple statistics and clustering can produce patterns that require interpretation through interactive interfaces or downstream recommendation applications.
  • Mining Data Patterns: Data-pattern mining explores encoded patterns in visualization ensembles using chart-specific systems and chart-type-independent 2D projections.
  • Discussion and Open Questions: Future work includes mining semantic design patterns, automating visualization design-space construction, and applying advanced mining with human-in-the-loop analytics.

7 FUTURE RESEARCH OPPORTUNITIES

The survey identifies interoperability, reverse engineering, representation choice, model design, large-scale management, and human participation as major future research issues. It frames these challenges around visualizations as heterogeneous data that must serve both machines and people.

  • Visualization Standards and Interoperability: Inconsistent visualization content formats impede interoperability, motivating a common standard that covers existing partial formats.
  • Visualization Standards and Interoperability: Reverse engineering remains computationally expensive and lacks robustness, and the survey states that bespoke charts are currently impossible to reverse engineer.
  • Representation and Machine Learning: Programs are compact and computationally inexpensive, but may not generalize beyond specific chart types or parameter values observed during training.
  • Representation and Machine Learning: Visualization images challenge machine learning because local graphical details and text such as labels are critical to interpretation.
  • Representation and Machine Learning: The survey proposes incorporating empirical visualization knowledge into machine-learning models tailored to visualization research.Draco is cited as an example.
  • Big Visualization Data: Big visualization data raises issues in storing and managing heterogeneous, often unstructured visualizations and in mining and analyzing them at scale.
  • Human-Visualization Ecosystem: Future AI systems must address the tradeoff between human- and machine-friendliness and support mixed-initiative, human-centered creative practices.

8 DISCUSSION AND LIMITATION

The survey’s taxonomy is inductively organized but not exhaustive, with scope limited by simplifying assumptions and uncovered research areas. It also identifies extensions involving auxiliary data, intermediate-stage processing, and visualization collection summarization.

  • The taxonomy was constructed inductively by observing existing work and iteratively generalizing its classification.
  • Dependencies among tasks can support sequential system pipelines for complex problems without violating mutual exclusiveness.Assessment and comparison metrics may serve as optimization functions for recommendation.
  • The authors do not claim exhaustiveness because inductive collection may omit research questions, including tasks studied in adjacent fields.Image compression and style transfer are cited as computer-vision tasks unexplored for visualizations.
  • The survey focuses on charts and infographics, excluding scientific visualizations and work tailored to specific visualization types.The authors identify generalizing the taxonomy to a wider spectrum of visualization data as an open concern.
  • What: Future taxonomies should account for auxiliary data such as ground-truth labels, interaction logs, and analysis provenance.
  • Why: The why taxonomy omits research that exploits visualization data between input and output stages, such as eye-tracking-based learning-curve prediction.This work was excluded because it did not fit the survey’s theme.
  • How: Potential extensions include tasks for processing auxiliary data and visualization collection summarization, which remains limited and visualization-type-specific.

9 CONCLUSION

The survey formalizes visualizations as an emerging data format and reviews AI4VIS research. Its classification helps readers locate literature, understand techniques, and identify future research areas.

  • The survey presents a classification that organizes literature across research areas and helps readers find relevant work.
  • The classification helps readers understand current techniques and identify areas for future research.
Loading 2102.01330v2…