Source-linked AI summary
DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing
Priyan Vaithilingam, Elena L. Glassman, Jeevana Priya Inala, Chenglong Wang
TL;DR
Visualization editing can be difficult when GUI options are hard to find and NLIs do not provide immediate, iterative feedback. DynaVis combines natural-language instructions with persistent synthesized widgets; in a study of 24 participants, users preferred it over an NLI-only interface for further edits and editing confidence.
Problem
Visualization editors may need low-level expertise to find options in complex GUIs, while NLIs lack immediate feedback and easy fine-grained exploration.
Source: §1 · pp. 2–3Method
DynaVis uses natural-language edit or widget requests to synthesize persistent GUI widgets for users to make further visualization edits.
Source: §1 · p. 3Results
In a study with 24 participants, users preferred DynaVis over an NLI-only interface for ease of further edits and editing confidence from immediate visual feedback.
Source: §8 · p. 22Takeaways & Limitations
Persistent widgets support composing and revisiting chart-refinement and data-exploration edits.
Source: §1 · p. 3Five participants said dynamic widgets could become overwhelming in long editing sessions and wanted ways to pause or organize them.
Source: §5.4 · p. 18
Abstract
from arXiv · showhide
Users often rely on GUIs to edit and interact with visualizations - a daunting task due to the large space of editing options. As a result, users are either overwhelmed by a complex UI or constrained by a custom UI with a tailored, fixed subset of options with limited editing flexibility. Natural Language Interfaces (NLIs) are emerging as a feasible alternative for users to specify edits. However, NLIs forgo the advantages of traditional GUI: the ability to explore and repeat edits and see instant visual feedback. We introduce DynaVis, which blends natural language and dynamically synthesized UI widgets. As the user describes an editing task in natural language, DynaVis performs the edit and synthesizes a persistent widget that the user can interact with to make further modifications. Study participants (n=24) preferred DynaVis over the NLI-only interface citing ease of further edits and editing confidence due to immediate visual feedback.
1 INTRODUCTION
Visualization edits can be hard to discover in complex GUIs and hard to explore through NLIs alone. DynaVis combines natural language with persistent, dynamically synthesized widgets, and its 24-participant study found a preference over an NLI-only interface.
1 INTRODUCTION
- Long-tailed edits require users to identify the right low-level option and find it in a GUI, while simplified interfaces can restrict customization.For example, rotating axis labels may require knowledge of the visualization grammar and navigating nested property menus.Source: §1 · p. 2
- NLIs can infer editing options from instructions, but lack GUI support for fine-grained exploration, immediate feedback, and quickly undoing or reapplying edits.Users may need to try different values for properties such as line width or color without knowing the exact value in advance.Source: §1 · pp. 2–3
- DynaVis blends natural-language instructions with synthesized GUI widgets that users can interact with for further edits.Users can describe an edit or request a widget directly; in the former case, the widget includes a preset value from the instruction.Source: §1 · p. 3
- The LLM-powered synthesis engine translates natural-language input into widgets and control functions connected to visualization properties.The widgets persist after synthesis, allowing users to repeat and coordinate edits.Source: §1 · p. 3
- In a study with 24 participants, users preferred DynaVis to an NLI-only interface for ease of repeating edits and confidence from immediate visual feedback.The introduction lists a user study evaluating how participants use DynaVis to solve visualization-editing tasks.Source: §1 · p. 3
2 USAGE SCENARIO
The scenario follows Alice as she creates and refines a stock chart, then gives collaborator Alex widgets to explore the data. Natural-language commands and widget requests produce controls for further edits and exploration.
2 USAGE SCENARIO
- DynaVis lets users request edits through natural language or ask directly for a widget, which it dynamically generates for visualization changes.The interface includes a command bar, a working chart, and a panel for managing synthesized widgets.Source: §2 · pp. 3–4
- After Alice asks to move the legend left, DynaVis creates a position menu, and she settles on the top-left corner after trying alternatives.The final choice is one Alice had not initially expected.Source: §2 · p. 4, Figure 3 · p. 4
- A synthesized x-axis label editor lets Alice adjust text size and rotation together, and she settles on size 15 and rotation −45.She tries combinations after an exploratory command specifying size 20 and rotation 60 degrees.Source: §2 · p. 4, Figure 4 · p. 5
- When Alice requests a color widget without specifying colors, DynaVis generates a stock-specific picker so she can try colors with immediate feedback.The widget is tailored to the stock symbols in the current dataset and does not itself edit the chart.Source: §2 · p. 5, Figure 5 · p. 5
- Alex can use generated filters and range controls to compare selected stocks and zoom into a chosen time and price window.The scenario also describes the value of dynamic widgets for authors’ chart refinement and readers’ data manipulation.Source: §2 · pp. 5–6
- Persistent, compositional widgets let users revisit or revert edits, unlike NLI commands that require new instructions for each update.With a conventional GUI, users would need to track their prior edits to repeat or redo them.Source: §2 · p. 6
3 DYNAVIS SYSTEM DESIGN AND IMPLEMENTATION
DynaVis pairs natural-language chart editing with persistent, modular widgets and an LLM-based synthesis pipeline. Its system summarizes data, generates chart specifications and widget code, then validates the outputs before use.
3.1 Dynamic Widgets
- Dynamic widgets pair an HTML interface with a JavaScript callback that updates the chart or data when users interact with its controls.The callback receives the event and current chart, and can return transforms and an updated chart.Source: §3.1 · pp. 6–7
- Widgets make modular edits to declarative Vega-Lite specifications, while data transforms run in listed order and across widgets chronologically.Transform widgets also provide a switch to enable or disable their transformation.Source: §3.1 · pp. 7–8
3.2 Synthesis Framework
- The synthesis framework separates preprocessing, LLM-based generation, and post-processing across a Data Summarizer, Chart Engine, and Widget Engine.The data summary is used in subsequent LLM queries instead of sending the full dataset.Source: §3.2 · p. 8, §3.2.1 · p. 8
3.2.1 Pre-processing: Data Summary and Visualization Pre-processing.
- The Data Summarizer grounds LLM queries with a compact dataset summary built from extracted properties, samples, and LLM-enriched semantic descriptions.It extracts types, statistics, and random column samples, then enriches the summary with dataset and field descriptions and semantic-type predictions.Source: §3.2.1 · pp. 8–9
3.2.2 LLM-based Synthesis of Visualization and Dynamic Widgets.
- The Chart Engine generates or edits Vega-Lite specifications, while the Widget Engine synthesizes HTML and JavaScript widget components from the data summary, chart specification, and user prompt.The chart engine can receive an existing specification; templates and few-shot examples guide widget-code generation.Source: §3.2 · p. 8, §3.2.2 · p. 9
3.2.3 Post-Processing through program analysis.
- Post-processing checks synthesized chart specifications for JSON, syntax, and schema errors, then parses widget code into syntax trees for program analysis.Widget checks include resolving HTML ID conflicts and validating callback names, parameters, and chart-property handling.Source: §3.2.3 · pp. 9–10
3.3 User Interface Implementation
- The web application uses React and TypeScript for its front end, Python for its back end, and GPT-3.5-turbo for responsive interactive synthesis.The authors report that GPT-4 generated widgets more accurately but had too much latency for smooth interaction.Source: §3.3 · p. 10
4 USER STUDY DESIGN
The user study compares DynaVis with a baseline that combines natural-language commands and pre-populated widgets, using visualization-editing tasks and a within-subjects procedure. It measures task outcomes, workload, user ratings, tool preferences, and qualitative feedback.
4 USER STUDY DESIGN
- The study uses a within-subjects design with 24 participants completing visualization-editing tasks in DynaVis and a baseline NLI-based tool.Participants completed one set of five tasks with each tool.Source: §4 · p. 10, §4.2 · p. 10
4.1 Participants
- Participants came from two research universities and varied in visualization and data-analysis experience; none had used Vega-Lite or Vega.Their prior visualization tools included matplotlib, Seaborn, ggplot, Tableau, Excel, and D3.Source: §4.1 · p. 10
4.2 Study Conditions
- The UINL baseline combines natural-language commands with pre-populated basic chart widgets, while UIDW adds custom dynamic widgets that are synthesized after natural-language edit commands.Both conditions retain the same pre-populated widgets; users in UIDW can also add custom widgets directly.Source: §4.2 · pp. 10–11
4.3 Tasks
- Tasks use three datasets and cover concrete edits, exploratory choices, and chart-based questions across stock, unemployment, and weather visualizations.Each task contains five editing subtasks, starting from a base dataset and visualization.Source: §4.3 · pp. 11–12
4.4 Study Procedure
- Task and tool order were counterbalanced, participants received tutorials and practice time, and each task had a 15-minute limit.The study recorded screen and audio, used post-task surveys, and ended with a comparative survey and informal interview.Source: §4.4 · p. 12
4.5 Measurements and Analysis
- Measures include subtask success, app telemetry, post-task workload and experience ratings, comparative tool preference and workload, and coded interview responses.Quantitative metrics were compared using paired t-tests.Source: §4.5 · pp. 12–13, Table 1 · p. 13, Table 2 · p. 13
5 USER STUDY RESULTS
In the user study, participants found UIDW easier for task completion and used dynamic widgets to fine-tune edits. They also reported better AI understanding and described benefits of immediate feedback, repeat editing, and control, alongside challenges for extended use.
5.1 Task Completion
- UIDW participants found tasks significantly easier than UINL participants (6.26 vs. 5.13); failures occurred in both conditions, often because of date-format errors.Task times did not differ significantly; six of seven UINL failures and both UIDW failures occurred on the date-range task.Source: §5.1 · pp. 13–14
5.2 Self-reported Cognitive Task Load Index
- In direct comparisons, 80%–83% of participants reported lower mental demand and feeling less hurried with UIDW, though post-task NASA TLX ratings showed no significant differences.Direct-comparison responses also favored UIDW for perceived success, effort, and frustration.Source: §5.2 · p. 14
5.3 User Behavior
- UIDW participants used 2.67 natural-language commands per task versus 8.4 with UINL, while using dynamic widgets 13.25 times on average.Pre-populated widget use did not differ significantly between conditions.Source: §5.3.1 · p. 15, Figure 12 · p. 15
5.3.2 User Strategy.
- In UIDW, nine participants relied only on widgets, while ten used natural language for broad edits and widgets for fine-grained changes.In UINL, six participants always used commands, while 18 first sought pre-populated widgets.Source: §5.3.2 · p. 15
5.3.3 Prompting Strategy.
- Participants typically prompted for edits with an action, chart property, and value; widget prompts generally omitted the value for later UI adjustment.Filtering prompts varied, and two prompts using incorrect chart-property names led to incorrect charts.Source: §5.3.3 · p. 16
5.4 User Preference
- Participants valued dynamic widgets for repeated edits and immediate visual feedback, while noting that changing widget panels could become overwhelming in long sessions.Some wanted ways to pause widget creation or group and control displayed widgets.Source: §5.4 · pp. 16–18
- UIDW users reported better AI understanding than UINL users, and participants described widgets as providing greater control over edits.Twelve participants said widget prompts were easier and more reliable than natural-language edit commands.Source: §5.4 · p. 17
5.5 Tool Performance
- Each GPT-3.5 query added an average of 1.47 seconds of latency, and commands required an average of 1.16 automatic retries.Retries addressed syntax or semantic errors before correct output was produced.Source: §5.5 · p. 18
6 DISCUSSION AND FUTURE WORK
DynaVis combines natural language and dynamic widgets, and participants used both to communicate intent and make repeated precise edits. The discussion identifies usability and expressiveness boundaries alongside directions for extending and managing the widgets.
Modalities beyond NL and UI widgets
- Participants preferred DynaVis and used both modalities to communicate intent and make repeated, precise visualization edits.The study reports preference for the tool, while the discussion describes interaction through both natural language and widgets.Source: §6 · p. 18
DynaVis design opportunities
- Voice, gestures, direct manipulation, and by-example specifications are proposed as future interaction modalities beyond natural language and widgets.Source: §6 · p. 18
Modalities beyond NL and UI widgets
- Users may struggle to choose which modality to use, so learning how each modality’s benefits and disadvantages fit is an open research need.Source: §6 · p. 18
Static vs. Dynamic UI
- Dynamic interfaces may increase cognitive load during long editing sessions, especially for expert users, unlike static interfaces users may learn over time.Some study participants raised this concern and suggested predictable widget categorization.Source: §6 · pp. 18–19
- NL interfaces do not expose every option continuously: commands may reveal unknown features or shift capability discovery to users, risking mistrust.The discussion notes that more research is needed to address these limitations.Source: §6 · p. 19
Supporting imperative plotting libraries and lower-level visualization grammars
- Declarative high-level grammars support modular widgets but expose fewer options for complex edits, motivating bidirectional editing for lower-level visualization representations.Making only part of a line dashed is given as an example of a task requiring lower-level details.Source: §6 · p. 19
Future applications
- Future work could tailor widgets to accessibility constraints, support user-built task panels, and investigate dynamic widgets in other applications.The discussion also proposes generalizing dynamic widgets beyond visualization editing to software with extensive configuration options.Source: §6 · p. 19
DynaVis design opportunities
- Participants requested ways to categorize and manage widgets, while proposed additions include command history, saved widget combinations, and sharing visualizations with widgets.Suggested management features include categorizing, collapsing, and expanding widgets.Source: §6 · pp. 19–20
7 RELATED WORK
Related work spans visualization authoring, natural-language interfaces, dynamic UI synthesis, and multimodal interaction. DynaVis combines natural-language editing with widgets and uses an LLM rather than fixed rules or a predefined command grammar to synthesize controls for visualization properties.
Visualization authoring tools
- Visualization authoring tools reduce initial chart-creation effort with high-level mappings and smart defaults, but refinement still requires navigating many editing options.DynaVis is presented as complementary to these systems, supporting edits after users specify visualization intent.Source: §7 · p. 20
Natural language interfaces for visualization (V-NLIs)
- Visualization NLIs generate charts from commands, and systems such as Tableau, Power BI, and Google Spreadsheets also support natural-language data queries.The cited commercial tools limit natural-language interaction to queries and corresponding standard charts; some V-NLI authoring systems also support editing.Source: §7 · p. 20
Customizable and dynamic user interfaces
- Prior dynamic interfaces synthesize controls through customization, demonstrations, relaxation, or constrained grammars, while DynaVis uses an LLM for direct manipulation of visualization properties.Bespoke uses demonstrations and parameter-type heuristics; BOLT and EVIZA restrict commands to predefined grammars.Source: §7 · p. 21
- DynaVis’s LLM-based widgets can use user context to reduce sensitivity to command ambiguity, avoid fixed rules or query relaxation, and represent relations across multiple properties.The paper also says the generated UI space is not restricted to a fixed set or kind of interface.Source: §7 · p. 21
Multi-modal user interfaces
- Prior multimodal systems combine natural language, programming, sketching, pen, or touch; DynaVis pairs natural-language intent with interactive widgets and may add direct canvas control.Source: §7 · pp. 21–22
8 CONCLUSION
DynaVis blends natural language with dynamically synthesized widgets for visualization editing. In a study with 24 participants, participants preferred it over an NLI-only interface, citing easier further edits and confidence from immediate visual feedback.
8 CONCLUSION
- In a study with 24 participants, users preferred DynaVis over an NLI-only interface, citing easier further edits and confidence from immediate visual feedback.Source: §8 · p. 22