Visualizationary: Automating Design Feedback for Visualization Designers using LLMs

Introduction

• Interactive editors enable code-free visualization authoring, but effective design still requires visual skills, aesthetic understanding, and experience.Heuristics can constrain creativity, while Grammar of Graphics specifications can produce standardized solutions that reduce outcome diversity.Source: §1 · p. 1
• VISUALIZATIONARY provides automated, actionable perceptual feedback without standardizing design changes, aiming to support designers across expertise levels while preserving creativity.Its analyze-clarify-guide-track (ACGT) workflow analyzes a visualization with perceptual filters, clarifies findings, guides changes, and tracks the design over time.Source: §1 · p. 1
• The web-based system translates chart images into text with a vision-language model, generates guidance with an LLM, and presents findings in a hierarchical interactive report.Designers can upload images of designs made in tools of their choice and drill into findings at different levels of detail.Source: §1 · p. 1
• The evaluation followed 13 novice, intermediate, and expert designers over several days as they created and revised visualizations; three senior researchers assessed the resulting designs.The study reports four successive revisions and quantitative and qualitative feedback, examining effectiveness and shortcomings across expertise, feedback type, and design optimization.Source: §1 · pp. 1–2

2.1 Design Feedback

• Effective feedback must be understandable and actionable; low-level guidance offers concrete improvements, while abstract critiques such as “poor contrast” can be difficult to address.Academic settings more often emphasize user and usage feedback, whereas peer feedback is more prevalent in practice; Munzner’s nested model primarily incorporates the former.Source: §2.1 · p. 2

2.2 Automatic Design Feedback on Visualizations

• Automated visualization feedback includes linting for chart errors, chart recommendation based on data or design knowledge, and emerging accessibility auditing approaches.Visualization linting applies static-analysis ideas to problems such as chart construction errors, visualization mirages [43], and deceptively designed line charts; VizLinter helps rectify Vega-Lite specification problems [13]. Recommendation methods draw on data properties,,, perceptual principles, expert feedback, dataset-visualization pairs [30], and design knowledge; accessibility approaches include Chartability and color-vision-deficiency assessment.Source: §2.2 · p. 2
• Unlike systems that display perceptual metrics, this work explains metric findings and suggests how designers may address visualization shortcomings.Prior quantitative approaches include VisLab, which collects crowdworker feedback for online evaluation, and Perceptual Pat, which reports perceptual metrics across artifact iterations.Source: §2.2 · p. 2

2.3 Large Language Models

• LLMs predict likely word sequences from text training data; pre-training and task-specific input support uses such as question answering, while multimodal models can accept images and text.The passage describes LLMs as having billions of parameters and notes that pre-training commonly uses next-word prediction; services such as ChatGPT separate user sessions and adapt to user inputs, [26], [70].Source: §2.3 · p. 2
• The system uses multimodal LLMs to describe a visualization, then feeds those descriptions and user queries to text-only LLMs for feedback, with a strict preamble intended to reduce hallucination [44].The multimodal models take the visualization and questions as input; the uni-modal models generate design feedback from the resulting descriptions and queries.Source: §2.3 · p. 2

3 AUTOMATED VISUALIZATION FEEDBACK

• Designers submit consecutive visualization versions and receive natural-language feedback to guide iterative improvement.The system is intended to support, not replace, a trained designer’s critical eye.Source: §3 · pp. 2–3

3.1 Target Audience

• The system primarily targets novice and intermediate designers who lack experience critiquing their work objectively, but the authors also find automated feedback useful for experts.It is not intended to replace a trained designer’s critical eye.Source: §3.1 · p. 3

3.2 Design Rationale

• The design rationale emphasizes immediate, real-time, tool-independent, consistent, scalable, and less costly feedback compared with human evaluation.Automation can reduce subjectivity, bias, and human error, handle multiple users simultaneously, and reduce—but not eliminate—the need for costly human evaluators.Source: §3.2 · p. 3

3.3 Analyze-Clarify-Guide-Track (ACGT) Workflow

• The ACGT workflow analyzes screenshots with perceptual filters, clarifies metrics in accessible terms, guides revisions with actionable advice, and tracks iterations and changes over time.Tracking records corresponding quality improvements so designers can understand the impact of earlier recommendations and avoid local optima.Source: §3.3 · p. 3

4 HOW VISUALIZATIONARY WORKS

• Visualizationary provides visualization designers with automated design critiques in natural language.It is implemented as a web-based platform whose interactive front end lets users explore visual feedback, while its back end supports the process using off-the-shelf language models.Source: §4 · p. 3

4.1 System Overview

• The ACGT workflow analyzes uploaded chart images, explains filter results in natural language, suggests ways to address shortcomings, and tracks design changes across iterations.The system comprises a web-based design interface, interactive reports, and an automated backend for extracting visualization data and generating textual feedback.Source: §4.1 · p. 4

4.2 Visualizationary Design Management Interface

• Projects contain visualization revisions and reports; designers upload JPG or PNG screenshots to start analysis, and completed reports appear in a project timeline, typically after a couple of minutes.The interface supports creating, editing, and deleting projects associated with a user.Source: §4.2 · p. 4

4.3 Visualization Design Reports

• Interactive reports organize filter feedback into expandable categories, combining summaries, explanations, suggestions, and raw metrics, with visual indicators for issues and transparency about extracted data and calculations.Designers can skim category-level summaries or expand sections to inspect more detailed feedback and metrics.Source: §4.3 · pp. 4–5

4.4 Feedback System

• Ten visualization filters across five topics combine computer-vision and vision-language models with heuristics and LLM judgment to turn extracted visual characteristics into explanations and actionable guidance.Not every filter produces actionable feedback, and the filter set is described as extensible.Source: §4.4 · p. 5
• Salience filters estimate gaze and flag attention patterns, including text-focused, center-focused, or low salience on attention-relevant regions, while warning that model measurements may be biased or inaccurate.The virtual eyetracker predicts attention during an overview task; the center is defined as the middle third of the chart image.Source: §4.4 · p. 5
• Other filters use DePlot and OCR for title and text feedback, extracted chart data for chart recommendations, and image analysis to advise on color use and color-vision-deficiency risks.The chart recommendation depends on extractable data and has limitations, including inability to recommend treemaps; color-vision-deficiency feedback includes simulated views and suggested palettes.Source: §4.4 · pp. 5–6
• The revision tracker summarizes changes in quantifiable metrics and whether they are beneficial, while archived reports allow comparison of qualitative changes that the tracker does not capture.The tracker provides a high-level overview intended to help designers see improvements over time and avoid local optima.Source: §4.4 · pp. 6–7

4.5 Prompt Engineering for Visualization Feedback

• Prompt templates constrain LLM feedback with metric-specific questions, interpretation conditions, design-knowledge suggestions, and separate prompts that describe changes between revisions.The authors also use grounding context and prompt guidelines to reduce hallucinations, verbosity, and nondeterministic output.Source: §4.5 · p. 7
• The prompt combines filter suggestions with the current version’s output and interpretations and the previous version’s output and interpretations.Its sequence introduces details about each version before adding that version’s output and interpretations.Source: §4.5 · p. 7

4.6 Implementation Notes

• The web application used HTML, CSS, JavaScript, JQuery, Python Flask, server-side Python 3 analysis, and MongoDB.It was hosted on AWS during the user study and used gpt-3.5-turbo as its LLM.Source: §4.6 · p. 7

5.1 Participants

• The study recruited 13 designers with more than a year of visualization experience: 6 novices, 4 intermediates, and 3 experts.Novices had up to 3 years’ experience, intermediates 4–7 years, and experts more than 7 years.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 Task

• Designers created a visualization over 3–5 days, choosing their dataset and tools, and uploaded at least five versions while working independently for at least two hours.Participants could create any type of visualization; they were asked to work independently.Source: §5.2 · p. 8

5.3 Procedure

• The online procedure included a briefing, longitudinal design period, post-study interview, and independent expert assessment.The briefing took about 30 minutes; interviews covered how designs evolved, participants’ experiences with Visualizationary, and their ideal feedback provider.Source: §5.3 · p. 8
• Independent expert participants with data-visualization teaching and grading experience assessed changes from each designer’s first to final design using a 1–5 Likert scale (1 decline, 3 neutral, 5 improvement).Source: §5.3 · p. 8

5.4 Data Collection

• The study collected recorded and transcribed interviews, demographics, and uploaded chart images and design reports.Demographics were gathered with online forms, while Visualizationary automatically collected the images and reports.Source: §5.4 · p. 8

5.5 Apparatus

• Interviews and experiments used participants’ own computers; designers could use their preferred authoring tools if they could upload visualization screenshots.The researchers recommended a display resolution of at least 1920×1080 and Google Chrome because the interface was optimized for that setting.Source: §5.5 · p. 8

6.1 Design Evolution

• Across novice, intermediate, and expert examples, feedback prompted chart changes, salience adjustments, text simplification, and readability fixes.N1 changed chart form and adjusted salience; I2 simplified text and reorganized labels; E1 changed colors, labels, and circle outlines in response to feedback.Source: §6.1 · pp. 9–10

6.2 Feedback Usage Patterns

• Of 68 feedback instances, 33 concerned Scanner Deeply, including 18 about alignment between salience and intended focal points; chart recommendations appeared five times.Implemented changes averaged 1.46 in the first iteration and 1.23 in the last. Participants sometimes declined feedback when it conflicted with their goals, and some recommendations were inappropriate or hard to implement.Source: §6.2 · pp. 11–12

6.3 Design Evaluation

• 3.69 was the average design-improvement score; evaluators rated novice, intermediate, and expert groups 3.34, 3.92, and 4.11, respectively.The scale’s neutral point was 3. Evaluators also noted improvements in data-ink ratio, text placement, and color use, though some novice feedback was not direct or actionable and CVD palette changes and salience adjustments were difficult.Source: Table 4 · p. 12, §6.3 · pp. 12–13, Table 5 · p. 12
• Evaluators’ judgments diverged when design changes were subtle, benefits uncertain, or designers’ intentions were unknown.Some filters also malfunctioned or produced inappropriate recommendations, and evaluators sometimes disagreed about changes across iterations or how to interpret a designer’s objective.Source: §6.3 · p. 13

6.4 Post-Study Interview

• Participants valued the system’s ability to surface overlooked design concerns, while noting that misdiagnosis and nonspecific suggestions could hinder implementation.Eleven of 13 participants used the tracker to monitor iterative changes, especially when salience issues took more than one step to address.Source: §6.4 · p. 13
• Ten of 13 participants said the hierarchical text report helped them stay organized, although participants still found the text volume overwhelming.Some participants also described the LLM’s response as human-like and said it made them feel less isolated during design.Source: §6.4 · p. 14

7 DISCUSSION

• A vanilla LLM supplied with symbolic metrics and visualization-design guidance gave actionable feedback, benefited even experienced designers, and made the ACGT framework effective for communicative visualizations.These are the study’s three central conclusions.Source: §7 · p. 14

7.1 Visualization Design Feedback

• Balancing general communication guidelines with designers’ personal or domain-specific goals remains difficult because nuanced intent requires more extensive modeling.The authors propose future feedback systems that address communicative effectiveness alongside designers’ broader motivations.Source: §7.1 · p. 14
• Visualizationary’s automated checks can assess best practices, accessibility, color vision deficiency, and visual saliency, while designers can set success benchmarks such as comprehension speed.The discussion proposes a subjective evaluator tailored to designers’ styles, which would require knowledge of best practices, accessibility standards, and real-user feedback; LLM-driven agents are a possible route.Source: §7.1 · p. 14
• Intermediate and expert participants used Visualizationary more effectively than novices, possibly because automation and limited foundational design skills hindered novices.Participants’ reliance on autogenerated charts may have weakened or prevented manual design skills and may have affected the study’s Table 4 results.Source: §7.1 · p. 14

7.2 Limitations

• Future improvements include adding filters, predicting task- and intent-sensitive saliency, making filter reasoning and issue locations interpretable, and clarifying feedback language and experimental questions.The discussion gives detecting charts that do not begin at zero, linting for consistency, and moving beyond “if, then” guidelines as examples.Source: §7.2 · p. 14
• LLM hallucinations [31] could contradict designers’ goals or fail to improve visualizations, so the authors identify mitigation methods as future work.Proposed approaches include incorporating user feedback into fine-tuning [33] and detecting self-contradictions.Source: §7.2 · pp. 14–15
• Privacy risks include data sharing with commercial services and model training on user data, but local open-source models can avoid external services and the system does not fine-tune on user data.The authors state that not fine-tuning on user data prevents memorization or leakage by actors with model access.Source: §7.2 · p. 15

• The study found that an off-the-shelf LLM can provide high-quality, actionable visualization-design feedback to novice, intermediate, and expert users.Source: §8 · p. 15
• Visualizationary combines LLMs and visualization guidelines in an automated design workflow called ACGT: analyze, clarify, guide, and track.Source: §8 · p. 15
• The evaluation involved 13 designers of different seniority in a longitudinal design task, pre- and post-study interviews, and assessments of their designs by three expert evaluators.Source: §8 · p. 15