PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative AI for Visualization: State of the Art and Future Directions

Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, Wei Zeng

arXiv:2404.18144v1cs.LGcs.AIcs.HC

TL;DR

Visualization-specific data structures and analytic requirements make GenAI applications challenging to evaluate, curate training data for, and integrate with traditional pipelines. This survey reviews GenAI methods across visualization tasks and generation types, synthesizing their applications and challenges. It concludes that task-specific challenges remain and require further investigation.

  • Problem

    The survey addresses the absence of a comprehensive review of GenAI methods used for visualization, where task-specific applications and challenges span multiple visualization stages.

    Source: §1 · p. 3

  • Method

    The authors collect and categorize 81 GenAI4VIS papers by concrete visualization tasks and the generation methods used for them.

    Source: §1 · p. 3

  • Results

    GenAI methods have been applied across data enhancement, visual mapping, stylization, and interaction, using sequence, tabular, spatial, and graph generation, while task-specific challenges remain.

    Source: §8 · p. 49

  • Takeaways & Limitations

    Future LLM-based NL2VIS work should integrate visualization-domain knowledge, validate semantic correctness, and strengthen user-feedback loops.

    Source: §4.1.5 · p. 28

    LIDA’s sequence-generation and spatial-generation stages remain separate in a linear workflow, and the two models do not share knowledge.

    Source: §7.3 · p. 48

Abstract

from arXiv · show

Generative AI (GenAI) has witnessed remarkable progress in recent years and demonstrated impressive performance in various generation tasks in different domains such as computer vision and computational design. Many researchers have attempted to integrate GenAI into visualization framework, leveraging the superior generative capacity for different operations. Concurrently, recent major breakthroughs in GenAI like diffusion model and large language model have also drastically increase the potential of GenAI4VIS. From a technical perspective, this paper looks back on previous visualization studies leveraging GenAI and discusses the challenges and opportunities for future research. Specifically, we cover the applications of different types of GenAI methods including sequence, tabular, spatial and graph generation techniques for different tasks of visualization which we summarize into four major stages: data enhancement, visual mapping generation, stylization and interaction. For each specific visualization sub-task, we illustrate the typical data and concrete GenAI algorithms, aiming to provide in-depth understanding of the state-of-the-art GenAI4VIS techniques and their limitations. Furthermore, based on the survey, we discuss three major aspects of challenges and research opportunities including evaluation, dataset, and the gap between end-to-end GenAI and generative algorithms. By summarizing different generation algorithms, their current applications and limitations, this paper endeavors to provide useful insights for future GenAI4VIS research.

1. Introduction

GenAI offers new ways to support visualization tasks, but visualization’s data structures and analytic requirements create challenges for applying general-purpose methods. This survey reviews GenAI4VIS research and organizes it by visualization task and method.

1. Introduction

  • GenAI can model designs from real-world data and support visualization operations from data processing and mapping to automatic visualization generation and chart question answering.The authors describe its potential to reduce the workload and complexity of traditional computational methods and broaden the diversity of generated designs.Source: §1 · pp. 2–3
  • Visualization poses distinctive challenges for GenAI: evaluation must consider efficiency and data integrity, training data is difficult to curate, and rule-based pipeline constraints complicate end-to-end integration.These characteristics make it less straightforward to apply general-domain pretrained models to visualization-specific generation.Source: §1 · p. 3
  • The survey addresses the lack of a comprehensive review focused on GenAI for visualization by collecting 81 papers and categorizing methods by the visualization tasks they address.It aims to clarify technical developments and challenges and identify research opportunities.Source: §1 · p. 3
  • The survey proceeds through data enhancement, visual mapping generation, stylization and communication, and interaction, with task subsections covering data and algorithms alongside remaining challenges.It concludes with discussion of dominant challenges and future research opportunities.Source: §1 · pp. 3–4

2. Scope and Taxonomy

The survey defines its scope through four visualization stages and four generation data structures, then relates methods to tasks and reviews prior surveys. It describes a search-and-filter process for collecting the literature.

2.1. Scope and Definition

  • The survey groups methods by data structure: sequences, tables, graphs, and spatial data such as 2D images and 3D models.These categories connect generation algorithms to the data structures used in visualization tasks.Source: §2.1 · pp. 4–5
  • The survey organizes GenAI4VIS into data enhancement, visual mapping generation, stylization and communication, and interaction, excluding generative algorithms based purely on rules or optimization.It defines data enhancement as improving data quality, completeness, or feature representation; stylization as applying aesthetic choices; and interaction as users’ dynamic engagement with visualized data.Source: §2.1 · pp. 6–7
  • Method-task associations are not one-to-one: sequence generation is common in mapping and interaction, tabular and graph generation in data enhancement, and spatial generation in enhancement and stylization.The survey notes graph generation’s potential for mapping and stylization, where graph structures can support visual encoding and layout optimization.Source: §2.1 · p. 7

2.2. Related Survey

  • Unlike broader AI-in-visualization surveys and surveys of specific subareas, this review focuses on concrete GenAI methods, technical advances, challenges, and opportunities for visualization.The authors report that no previous survey had been dedicated to the various GenAI methods used for visualization tasks.Source: §2.2 · p. 8

2.3. Survey Methodology

  • The authors gathered papers from prior surveys and recent work, expanded the set through references, citations, and keyword searches, and manually excluded non-GenAI rule-based or optimization methods.They collected 81 papers; they caution that manual keyword searching and citation traversal may miss relevant application papers with similar methods.Source: §2.3 · pp. 8–9

3.1. Data Inference

GenAI data inference fills in unseen data, including interpolation, augmentation, and super-resolution. The survey reviews graph, tabular, and spatial generation methods and discusses where generated data may fall short.

3.1.1. Graph Generation

  • Graph generation supports chemical-data exploration and graph-structured simulation reconstruction, using GNNs and specialized latent-space traversal for molecular interpolation.GNN-Surrogate reconstructs ocean simulation data from parameters through an adaptive-resolution graph; ChemoVerse finds molecular interpolation paths through regions of interest.Source: §3.1.1 · pp. 10–12

3.1.2. Tabular Generation

  • GANs generate tabular surrogate data, while SERD addresses privacy risk by satisfying differential-privacy conditions with fake entities.Table-GAN adds a classifier to maintain consistency between categorical labels and other attributes in generated records.Source: §3.1.2 · pp. 12–13

3.1.3. Spatial Generation

  • Spatial inference covers imagery, volumetric, and spatial-temporal data, supporting tasks from interpolation and detection to 3D reconstruction and flow extrapolation.DeepOrganNet reconstructs 3D organ models from single-view medical images; STNet addresses spatial-temporal super-resolution of volumetric data.Source: §3.1.3 · p. 13
  • Spatial methods include VAEs, GANs, and disentangled representation learning; VATLD and SCANViz use disentangled learning for data enhancement.VAEs support user-directed emoji interpolation, while VATLD extracts interpretable traffic-light features for generating training examples.Source: §3.1.3 · pp. 14–15, Figure 3 · p. 15

3.1.4. Discussion

  • Generated data reflects learned distributions rather than guaranteed accurate values, and rare outliers may be overlooked or underrepresented.The survey cautions against relying on such generation for tasks requiring accurate data features.Source: §3.1.4 · pp. 15–16
  • Automatic disentanglement cannot guarantee interpretable dimensions, and user refinement may reach only a small subset, leaving other dimensions unaddressed.The choice of β in β-VAE remains heuristic, and extracted dimensions may not represent properties users want to explore.Source: §3.1.4 · p. 16

3.2. Data Embedding

Data embedding uses visualization images to carry data that can later be recovered losslessly. The surveyed methods use QR codes or data images and balance restoration needs against visual quality and embedding capacity.

3.2.1. Spatial Generation

  • Data embedding combines QR codes and visualization images; QR codes provide error correction, while visualization images carry encoded data and can use real-world training datasets.For invertible visualization, data-to-image methods can also represent the raw data to be embedded.Source: §3.2 · p. 16, §3.2.1 · pp. 16–17
  • Chartem, VisCode, and InvVis illustrate data-embedding pipelines; VisCode uses a chart-trained visual-importance network, while InvVis uses concealing and revealing networks.VisCode combines BCEWithLogits loss and SSIM; InvVis uses invertible steganography and discrete wavelet transforms to reduce texture-copying artifacts.Source: Figure 4 · p. 17, §3.2.1 · pp. 17–18

3.2.2. Discussion

  • Restoration evaluation relies on data-image metrics such as RMSE, while embedding capacity trades off against image quality and practical precision requirements remain to be assessed.Pixel-wise metrics cannot fully reflect restoration accuracy when original data are absent from evaluation.Source: §3.2.2 · p. 18

4. Visual Mapping Generation

Visual mapping generation uses GenAI to help non-experts create charts, including generation of visual primitives through sequence, tabular, spatial, and graph-based approaches. The section surveys methods and discusses limits on training data, rules, and end-to-end image generation.

4.1.1. Sequence Generation

  • Sequence methods generate visualization code or actions, spanning Data2Vis sequence-to-sequence code, Table2Charts action prediction, NL2VIS systems, and LLM-based code generation.ADVISor separates SQL generation from rule-based visualization, whereas ncNet generates visualization queries end to end; LIDA scaffolds and executes generated code.Source: §4.1.1 · pp. 19–22

4.1.2. Tabular Generation

  • Tabular methods predict design choices directly or enumerate candidate visualizations for AI scoring and ranking, with expert rules available when learned recommendations are unsatisfactory.VizML predicts five design parameters from 841 dataset-level features; DeepEye classifies and ranks candidates generated from a predefined search space.Source: §4.1.2 · pp. 22–23

4.1.3. Spatial Generation

  • Spatial methods support sketch-to-dashboard generation and dynamic density-map synthesis, using chart recognition and validation for sketches and GAN-based generation for density maps.GenerativeMap uses Perlin-noise synthetic training data, Bi-GAN, enlarged convolution kernels, and Poisson blending for more natural density changes.Source: §4.1.3 · pp. 24–25, Figure 6 · p. 25
  • For volumetric visualization, GAN pipelines can split rendering into opacity-image generation followed by translation into a final rendered image.The opacity GAN takes viewpoint and opacity transfer function as input; the second GAN also uses color-transfer information and the generated opacity image.Source: §4.1.3 · pp. 24–26

4.1.4. Graph Generation

  • KG4VIS uses knowledge-graph embeddings to score candidate visual encodings by combining data-feature, column, and design-choice entities with learned relation vectors.Its graph represents relationships among data features, data columns, and visualization design choices.Source: §4.1.4 · pp. 26–27, Figure 7 · p. 26

4.1.5. Discussion

  • Generation scope can be constrained by limited training data or incomplete rules; Data2Vis covers four chart types, while Table2Charts omits basic operations such as aggregation.Visualization-image generation also rarely uses fully end-to-end methods because stochastic models struggle to maintain regular shapes and rigid structures.Source: §4.1.5 · p. 27
  • The discussion identifies domain knowledge, semantic correctness, and user-feedback loops as future needs for LLM-based NL2VIS, and proposes hybrid models combining LLMs with traditional visualization techniques.The proposed hybrid approach aims to pair natural-language understanding with adherence to visualization best practices.Source: §4.1.5 · p. 28

5.1. Style Transfer

Style transfer applies visual attributes from references to new visualization designs. The survey covers spatial attribute transfer and graph-layout transfer, alongside discussion of domain knowledge and reference-image choices.

5.1. Style Transfer

  • Style transfer uses existing examples as inspiration, representing overall style as a template whose graphical attributes are separated from content and applied to new data.Style can include color, texture, and layout, or focus on a particular attribute such as color.Source: §5.1 · p. 29

5.1.1. Spatial Generation

  • Spatial transfer methods extract color palettes or recognize visual elements, while hybrid attribute transfer uses element recognition and learned comparisons to adapt multiple attributes to new content.Vistylist embeds visual elements in 256-dimensional vectors and compares Euclidean distances; an attention-based approach weighs visual properties.Source: §5.1.1 · pp. 29–30

5.1.2. Graph Generation

  • Graph-layout transfer learns layouts from examples using graph features and a VAE-style encoder-decoder, with a 2D latent map for exploration.The described method uses relative pairwise distances and adjacency matrices as inputs, with GNNs for graph-level representations.Source: §5.1.2 · pp. 30–31

5.1.3. Discussion

  • Transfer may incorporate domain constraints such as categorical-color discriminability or terrain elevation continuity, and reference images may be natural images as well as visualizations.The cited discussion describes these constraints as supporting interpretation and scientific accuracy.Source: §5.1.3 · p. 31

5.2. Chart Embellishment

Chart embellishment uses semantically related imagery to make visualizations more appealing and informative. The section describes prompt-driven image generation and emphasizes control and data integrity.

5.2.1. Spatial Generation

  • Pictorial embellishments add semantically related images to charts or maps to provide context and meaning, such as product images in sales charts.The surveyed generation methods for pictorial visualization mainly use Stable Diffusion-based techniques.Source: §5.2.1 · p. 32
  • Prompt-driven methods generate semantic-related objects for chart components, with some approaches accepting arbitrary text prompts or letting users select a data mask and assess visual distortion.The cited unified approach classifies visual representations into foreground and background.Source: §5.2.1 · pp. 32–33

5.2.2. Discussion

  • Text-to-image advances improve control over layout, text, and vector graphics, but data integrity remains a priority because visualizations must faithfully convey data patterns.Generative AI can also personalize style and reduce time spent searching for or adjusting elements.Source: §5.2.2 · p. 33

5.3. Textual Annotation

Textual annotations guide interaction and interpretation, explain data, and direct attention. The section reviews template-based and language-model approaches, then identifies occlusion, automation, and visual design as research challenges.

5.3. Textual Annotation

  • Annotations support interpretation and interaction by explaining data, directing attention, and making complex information more accessible across text-integrated and contextually interpreted datasets.For data without accompanying narratives, annotations can supply interpretation that raw data lacks.Source: §5.3 · p. 33, §5.3.1 · pp. 33–34

5.3.1. Sequence Generation

  • Sequence-generation approaches include detecting chart elements for template-based descriptions and learning noteworthy visual features to produce chart captions.AutoCaption uses one-dimensional residual neural networks to analyze relationships between visualization elements.Source: §5.3.1 · p. 34, Figure 8 · p. 35
  • LLM approaches generate descriptive captions for generic visualizations and natural-language datasets from Vega-Lite specifications.The surveyed work identifies prompt engineering as an emerging direction for annotation techniques.Source: §5.3.1 · p. 34

5.3.2. Discussion

  • Annotation design still faces chart occlusion; the discussion calls for better placement, automated context-aware annotation, and richer visual cues that complement the composition.The paper also notes that annotations should support pattern highlighting, deeper understanding, and user engagement.Source: §5.3.2 · pp. 35–36

5.4. Timeline & Story

The survey organizes timeline and story generation around spatial storyline layouts, graph-based chart structure, and sequences that connect data facts with narration. It describes layout optimization, SVG-to-graph modeling, and language-model or reinforcement-learning approaches to generating story content and exploratory insights.

5.4.1. Spatial Generation

  • Storyline generation optimizes the spatial layout of story components, with PlotThread using paired layouts and reinforcement learning to imitate and improve intermediate user layouts.PlotThread’s reward measures similarity between the user layout and the generated intermediate layout.Source: §5.4.1 · p. 36

5.4.2. Graph Generation

  • For dynamic charts, a graph neural network pipeline converts SVG elements and their stroke-wise and element-wise relations into representations used to classify chart elements.The graph construction uses five node features: element type, node color, fill color, stroke color, and stroke width.Source: §5.4.2 · p. 37

5.4.3. Sequence Generation

  • Data stories commonly assemble sequences of facts; relational data supports table descriptions and links between visuals and narration, while time-series tools can support continuous exploratory analysis.The survey contrasts continuous insight extraction with traditional tools designed mainly for single-step guidance.Source: §5.4.3 · p. 37
  • Large language models generate animated visuals and audio narrations, while Data Player links narration text with visualizations and recommends animation presets under domain-knowledge constraints.Data Player uses few-shot examples to prompt semantic-link sequence generation from a data table and narration text.Source: §5.4.3 · p. 38
  • Reinforcement-learning systems model exploratory visual analysis as state-action sequences and use familiarity and curiosity rewards to recommend coherent time-series insights.A causalCNN embeds time sequences of different lengths into equal-length representations for the curiosity reward.Source: §5.4.3 · p. 38

6. Interaction

The survey covers GenAI interaction through visualization retrieval and chart question answering, including methods that encode raster or SVG structure and fuse chart images with language. It also identifies opportunities in retrieval-augmented generation, multimodal queries, chart interaction, and visual analytics.

6. Interaction

  • GenAI interaction includes natural-language chart question answering and similarity search for finding desired charts in a collection.Chart question answering lets users seek insights through questions about charts; retrieval targets charts matching user intent.Source: §6 · p. 38

6.1. Visualization Retrieval

  • Visualization retrieval supports downstream uses including design learning, reuse, corpus construction, web mining, and computational journalism.The survey describes GenAI retrieval methods tailored to user intent about visual structure or other features.Source: §6.1 · p. 39

6.1.1. Spatial Generation

  • Raster-chart retrieval methods use triplet autoencoders to combine image- and data-level similarity or contrastive learning to align visualization images with user intent.LineNet contributes LineBench, a corpus of over 115,000 line charts with metadata from four real-world datasets.Source: §6.1.1 · p. 39

6.1.2. Graph Generation

  • For SVG visualization retrieval, graph contrastive learning combines graph and image-level representations, with reported retrieval results more structurally consistent.The approach uses InfoGraph and a GNN encoder to represent graph structure.Source: §6.1.2 · p. 40

6.1.3. Discussion

  • Retrieval-augmented generation is proposed to ground visualization generation in retrieved knowledge, while composed retrieval combines image input with text describing additional intent.The survey suggests RAG could reduce task complexity and improve reliability; multimodal retrieval could extend beyond image-and-text combinations.Source: §6.1.3 · p. 40

6.2. Chart Question Answering

  • Chart question answering aims to answer users’ questions about charts to expedite data analysis and enhance interaction.Questions can be factual or open-ended, visual or non-visual, and simple or compositional.Source: §6.2 · p. 41, §6.2.1 · p. 41

6.2.1. Sequence Generation

  • CQA systems either detect chart elements and structure before answer generation or use unified vision-language models to fuse chart images and text for end-to-end answers.The survey describes PlotQA’s element detection and OCR pipeline, DVQA’s chart-specific vocabulary encoding, and VisionTaPas’s cross-modality encoder.Source: §6.2.1 · pp. 41–43

6.2.2. Discussion

  • CQA remains largely natural-language-centered and often relies on chart-element detection that may be inaccurate on complex, noisy images; more precise vision-language models may simplify this pipeline.The discussion also identifies multimodal interaction and LLM support for broader visual-analytics workflows as future directions.Source: §6.2.2 · pp. 43–44

7. Research Challenges And Opportunities

The survey identifies evaluation criteria, dataset limitations, and the divide between end-to-end GenAI and visualization’s intermediate operations as challenges for GenAI4VIS. It proposes more diverse and reusable data, stronger assessment, and approaches that combine direct data-to-view learning with inspectable and controllable generation.

7.1. Evaluating GenAI for Visualization

  • Evaluation should consider accuracy and fidelity, intent alignment and controllability, robustness and consistency, and bias and ethics alongside traditional efficiency and aesthetic measures.The discussion highlights hallucination assessment for LLM-generated content, especially in accuracy-sensitive visual question answering.Source: §7.1 · pp. 44–45

7.2. Dataset

  • Many GenAI4VIS training datasets lack the diversity of real-world visualizations, motivating study of this gap and dataset supplementation.The survey links diversity to learning broader topics, styles, and design patterns for more versatile and contextually appropriate generation.Source: §7.2 · p. 46
  • Different visualization formats limit dataset reuse, can constrain dataset size, and may contribute to overfitting, motivating robust retargeting across formats.Examples include translating Vega-Lite code to Python or extracting SVG structure from raster images.Source: §7.2 · p. 46
  • Available datasets vary in origin and annotation, and large real-world collections such as VIS30K and VizNet are described as supporting data enhancement rather than visual-mapping tasks.The survey also calls for data-centric studies that diagnose GenAI4VIS models together with their training data.Source: §7.2 · pp. 46–47

7.3. GenAI4VIS vs. Generative Visualization

  • Hybrid GenAI4VIS pipelines can disconnect code generation from stylization, while direct data-to-view learning and inspectable intermediate steps are proposed as complementary directions.In LIDA, the image-based stylization model lacks structural information from the preceding sequence-generation stage; the survey proposes embedding control in models or supporting human intervention.Source: §7.3 · pp. 47–49

8. Conclusion

The survey reviews GenAI applications across data enhancement, visual mapping, stylization, and interaction, using sequence, tabular, spatial, and graph generation methods. It concludes that task-specific challenges, evaluation, dataset concerns, and integration between end-to-end GenAI and visualization pipelines warrant further investigation.

8. Conclusion

  • GenAI methods spanning sequence, tabular, spatial, and graph generation have been applied across data enhancement, visual mapping, stylization, and interaction, but visualization-specific challenges and broader evaluation, dataset, and pipeline-integration issues remain.The survey presents its technical review as a basis for reflecting on existing work and future GenAI4VIS research.Source: §8 · p. 49
Loading 2404.18144v1…