A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization

1 INTRODUCTION

• ML4VIS faces two integration challenges: identifying visualization problems suited to ML and selecting and adapting techniques appropriate to each task.Applying ML to unsuitable problems can add uncertainty or inexplicability without benefit, while technique suitability varies by problem.Source: §1 · p. 1
• Prior work mainly examined visualization assisting ML or ML-based visualization creation, leaving ML applications across visualization problems less fully discussed.The survey identifies itself as the first published survey on ML4VIS.Source: §1 · pp. 1–2
• The survey reviews literature across visualization, HCI, ML, and data mining to examine how ML techniques are employed and adapted for visualization.It frames visualization-related problems as involving visualization creation, interaction, and evaluation.Source: §1 · pp. 1–2
• The authors codify 88 ML4VIS studies into seven processes: Data Processing4VIS, Data-VIS Mapping, Insight Communication, Style Imitation, VIS Interaction, VIS Reading, and User Profiling.These processes are integrated with existing visualization models in an ML4VIS pipeline and mapped to an ML taxonomy.Source: §1 · p. 2
• The survey discusses research challenges and opportunities and provides an interactive browser of its findings.The authors present the survey as a stepping stone for further exploration.Source: §1 · p. 2

2 RELATED WORK

• Related work spans surveys guiding ML-assisted design and theoretical models describing visualization processes.The survey proposes an ML4VIS pipeline based on 88 papers to map how ML can benefit visualization creation, evaluation, and interaction.Source: §2 · p. 2, §2.2 · p. 2

2.1 Guiding ML-assisted Design

• General ML-assisted design guidelines can inform visualization, but visualization-specific needs such as precise data understanding and efficient interaction are rarely addressed in general interface design.Prior ML-assisted visualization design work focused on creation and gave little attention to interaction and perception.Source: §2.1 · p. 2

2.2 Theoretical Models in Visualization

• Earlier work integrated ML into visualization models chiefly to facilitate analysis of complex data, while how ML can benefit visualizations remained unclear.The survey responds to a gap in summarizing ML's contribution to visualization.Source: §2.2 · p. 2

3 SURVEY METHODOLOGY

• Manual analysis of 88 papers across VIS, HCI, ML, and data mining and management was chosen to extract detail beyond high-level automatic analysis.Table 1 identifies relevant venues, while Fig. 1 depicts study counts by field and over time.Source: §3 · pp. 2–3, Table 1 · p. 3, Figure 1 · p. 3

3.1 Survey Scope

• The final corpus contained 88 relevant papers, selected from 259 after applying criteria for ML use, visualization relevance, novel application or improvement, and publication scope.The search focused on articles from 2016 January to 2020 October, supplemented by reference-based search; papers published before 2000 were excluded.Source: §3.1 · pp. 3–4

3.2 Paper Coding

• The authors used three analysis phases: extracting paper descriptions, coding visualization processes, and coding learning-task types.At least two authors coded each paper; process codes were iteratively refined through discussion, while a bottom-up approach identified learning tasks.Source: §3.2 · p. 4

4.1 Data Processing4VIS

• ML-assisted Data Processing4VIS adapts data for visualization, including interactive cleaning after visualization creation and dimensionality reduction that maximizes visually perceived cluster separation.Interactive cleaning detects errors and offers users cleaning options while updating the visualization; the supervised reduction method accounts for human perception.Source: §4.1 · pp. 4–5

4.2 Data-VIS Mapping

• In Data-VIS Mapping, ML recommends visual representations, learns design choices, and can translate JSON-encoded data into Vega-Lite specifications.Other approaches synthesize intermediate visualizations to reduce storage and computation, or support parameter exploration without running expensive creation processes.Source: §4.2 · pp. 5–6, Figure 3 · p. 6

4.3 Insight Communication

• For Insight Communication, ML can interpret supplied or detected insights and map them to visualizations, including infographics generated from text.Some systems use predefined design spaces; Text-to-Viz generates infographics across layout, description, graphic, and color dimensions.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 Style Imitation

• Style Imitation uses ML to reproduce visualization styles, including color ramps learned from expert examples and graph layouts learned from drawings.DeepDrawing maps new graph data to drawings in a learned style, while PlotThread imitates human-designed storyline layouts.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5 VIS Interaction

• In VIS Interaction, ML refines selections and interprets natural inputs such as sketches and language commands to support visualization interaction.LassoNet predicts intended selections from a 2D lasso, viewpoint, and point-cloud characteristics; other systems address ambiguity in sketches or commands.Source: §4.5 · pp. 7–8, Figure 6 · p. 7

4.6 User Profiling

• User Profiling applies ML to interaction logs to predict next actions and infer user characteristics, informing adaptive visualizations for different needs and abilities.Studies use mouse interactions and eye gaze to investigate behaviors, learning, cognitive abilities, analysis goals, and personality.Source: §4.6 · p. 8

4.7 VIS Reading

• VIS Reading uses ML to extract visualization content, answer questions about it, and model human perception of attention, similarity, and readability.Extracted elements can support indexing, restyling, and reuse; some methods embed metadata in bitmap visualizations for later retrieval.Source: §4.7 · pp. 8–9

5 AN ML4VIS PIPELINE

• The pipeline adapts existing visualization models by adding style, replacing users’ internal states with observable outputs, and not separating visual mapping from view transformation.The authors cite end-to-end models that combine mapping and transformation; removing less-defined concepts is intended to better fit ML inputs and outputs.Source: §5 · p. 9
• The pipeline links seven ML-assistable processes through shared inputs and outputs, so ML in one process can benefit connected processes and guide where tools embed ML.The processes are grouped under DATA, VIS, and USER; for example, User Profiling can inform VIS Interaction, whose predictions may reduce interaction latency.Source: §5 · pp. 9–10
• Examples show cross-process loops: infographic attention estimates support Insight Communication and VIS Reading, while predicted user selections connect User Profiling with VIS Interaction.The paper also identifies user-adaptive mapping, communication, and style imitation as possible directions, while noting few studies had achieved them.Source: §5 · p. 10

6 ALIGN ML WITH VIS

• The ML-VIS mapping connects six information types and seven visualization processes with training-data formats and ML tasks to characterize practice and guide exploration.The authors categorize studies by learning task rather than changing model types, to align ML capabilities with visualization needs.Source: §6 · p. 10, §6.2 · p. 11

6.1 Training Data Formats

• Visualization images, natural language, engineered features, sequences, and graphs encode different visualization information, including visual content, user actions, and styles.Image datasets include FigureSeer (60k), DVQA (300k), and FigureQA (100k); natural-language tasks often use small labeled datasets and may combine ML with non-ML methods.Source: §6.1 · pp. 10–11

6.2.1 An Overview

• 44 studies used classification and 25 used regression; unsupervised learning appeared in 27/88, while reinforcement learning appeared in 4 papers and semi-supervised learning in 2.VIS Reading showed a particularly strong classification preference (23/36); Data Processing4VIS and Style Imitation favored unsupervised learning, but each had few studies.Source: Figure 12 · p. 11, §6.2.1 · p. 12

6.2.2 Supervised Learning

• Supervised regression uses numerical labels, chiefly in USER processes, while classification assigns finite classes and appears in all seven processes except Style Imitation.Classification can decompose complex tasks; end-to-end models can jointly predict object types and bounding boxes.Source: §6.2.2 · p. 12

6.2.3 Unsupervised Learning

• Unsupervised generative models create similar samples, dimension reduction supports 2D visualization or feature extraction, and clustering groups similar data or visual objects.GANs are used in Data-VIS Mapping to synthesize scientific visualizations without running expensive rendering, though GAN training can be difficult; clustering complex inputs benefits from representative features.Source: §6.2.3 · pp. 12–13

6.2.4 Semi-supervised Learning

• Only two surveyed studies used semi-supervised learning, both through active learning with users labeling data duplicates or visual patterns to iteratively improve classification.This approach combines a small amount of labeled data with a larger amount of unlabeled data.Source: §6.2.4 · p. 13

6.2.5 Reinforcement Learning

• Reinforcement learning formalizes visualization problems as sequential actions that maximize cumulative reward, illustrated by layout modifications rewarded for similarity to a user-preferred storyline.Only a limited number of ML4VIS studies used it, and the paper calls for further exploration.Source: §6.2.5 · p. 13

7.1 From the Aspect of ML

• Semi-supervised and reinforcement learning are underused despite potential to reduce labeling needs or model sequential decisions; broader use and exploration remain open.The survey also encourages consideration of other ML perspectives, including federated and transfer learning.Source: §7.1 · p. 14
• Most studies construct their own datasets, while dataset size and label reliability can constrain quality and potentially threaten model validity; public high-quality datasets are needed.The paper also notes unclear benchmark tasks, which make peer comparison difficult, and identifies graphic-element extraction, visualization generation, and graph layout as possible starting tasks.Source: §7.1 · p. 14

7.2 From the Aspect of VIS

• The surveyed work centers on static, standard visualizations, with dynamic forms, less common chart types, chart decorations, and multi-view visualizations receiving limited attention.The paper notes dynamic visualizations may require more training data, more complicated models, and greater computational resources.Source: §7.2 · p. 14
• General ML methods may not fit visualization-specific inputs: chart images differ from natural images, and the paper calls for methods tailored to visualization needs.A cited timeline-infographic system required post-processing to address poor performance associated with those differences.Source: §7.2 · p. 14
• Black-box automation can conflict with user expectations, while unsatisfactory results may reduce trust; the survey calls for closer collaboration and systematic usability and cognitive studies.Existing systems support post-hoc refinement, but the paper says further work is needed for users to interpret, modify, and improve ML.Source: §7.2 · p. 15

8 DISCUSSION

• The discussion aims to introduce ML4VIS comprehensively while distinguishing restrictions of ML from limitations of the survey.It presents survey limitations as possible directions for future surveys.Source: §8 · p. 15

8.1 ML is Not a Panacea

• When patterns are simple or training data is hard to obtain, non-ML methods such as rules, statistics, or SVG-tree traversal may be preferable.The paper also describes FlowSense combining predefined rules and ML when labeled examples and computational resources are lacking.Source: §8.1 · p. 15

8.2 Limitations & Future Work

• The survey covers technique and application papers, not evaluation papers, and its analysis may leave other under-explored ML4VIS areas unidentified.The authors suggest reviewing evaluation studies to understand user perceptions and guide application design.Source: §8.2 · p. 15

• The survey analyzes 88 papers on using ML techniques to solve problems related to data visualization.It is guided by questions about which visualization processes ML assists and which ML capabilities are used.Source: §9 · p. 15
• It identifies seven visualization processes benefiting from ML and aligns them with ML learning tasks to relate visualization needs to ML capabilities.The processes are organized in a proposed ML4VIS pipeline that maps their role in general visualizations.Source: §9 · p. 15
• The survey discusses current ML4VIS practices and future research opportunities based on its analysis of the collected studies.Source: §9 · p. 15