PaperlayerView exact revision on arXiv

Source-linked AI summary

Vibe Analysis: Exploring LLM Adoption by Data Visualization Practitioners

Shani C Spivak, Aditi Krishna, Mahsan Nourani, Melanie Tory

arXiv:2609.31922v1cs.HCcs.AI

TL;DR

Evidence on LLM adoption among visualization practitioners remains limited, despite visualization’s specialized tools, rules, and constraints. This study draws on practitioner interviews to characterize LLM use and vibe analysis, an iterative process across visualization tasks. The findings describe potential speed alongside verification, skill, design, and downstream quality concerns.

  • Problem

    There is no established process definition for LLM-mediated data analysis or visualization, despite Vis work involving specialized tools, rules, heuristics, and constraints.

    Source: Vibe Coding · p. 2

  • Method

    The authors synthesize interview accounts into a conceptual model of iterative LLM-supported work across visualization tasks, including project constraints and external factors.

    Source: Findings · p. 3, Vibe Analysis · pp. 4–5

  • Results

    LLMs may speed visualization work, but may also increase verification burden, weaken skill formation, narrow design exploration, and create downstream maintenance and quality problems.

    Source: Conclusion · p. 13

  • Takeaways & Limitations

    The visualization community is called to prepare users to apply LLMs critically and effectively, with attention to learning, training, and responsible data presentation.

    Source: Conclusion · p. 13

    The participant sample was skewed and not representative of all data workers, and reported LLM use may be over- or under-reported.

    Source: Limitations and Future Work · p. 13

Abstract

from arXiv · show

Large language models (LLMs) are enticing in their promise to support data visualization (Vis) through faster and simpler workflows for data prep, analysis, and visualization creation. Yet LLMs are notoriously error-prone and not built for data visualization tasks. Few studies have explored LLM adoption among Vis practitioners. To fill this gap, we conducted semi-structured interviews with members of the Data Visualization Society, a global community of data visualization designers. Our findings show that Vis designers actively use LLMs for both creative and technical aspects of the visualization process. A new visualization workflow is emerging, a process we call vibe analysis, analogous to vibe coding. Some key challenges raised by participants parallel those of vibe coding, while others are Vis-specific, like gaps in Vis knowledge and chart verification. This work opens up opportunities for research combining LLM-mediated work with Vis tools that incorporate data visualization guidance, constraints, and best practices.

Related Work

Prior work examined analytic-chatbot adoption and generative models for visualization, but this study focuses on visualization practitioners across roles and sectors. It introduces vibe analysis as a practice and examines trust, verification, and Vis-specific challenges.

User Preferences and Adoption of LLMs

  • Earlier research on analytic chatbots described LLM-enabled high-level insights, automated insight discovery, and data generation, while cautioning that these shifts could yield shallower understanding.Qian et al. reported that LLMs reversed the prior bottom-up sequence of data preparation followed by insights.Source: User Preferences and Adoption of LLMs · p. 2
  • Compared with a 21-expert study of generative models, this work focuses specifically on Vis practitioners and examines trust, verification, and LLM-use challenges.The earlier study identified rapid prototyping opportunities and concerns about bias amplification and misleading visualizations.Source: User Preferences and Adoption of LLMs · p. 2
  • The study extends earlier work by examining LLM adoption, practices, and roadblocks across Vis work rather than a single pipeline part or visualization type.The authors note that earlier studies did not identify the approach they call vibe analysis.Source: User Preferences and Adoption of LLMs · p. 2, Vibe Coding · p. 2

Vibe Coding

  • The authors frame vibe analysis as analogous to vibe coding, while highlighting visualization-specific differences in practitioners’ LLM use.Vibe coding research describes practitioners reviewing, refining, and correcting stochastic code generation.Source: Vibe Coding · p. 2
  • Unlike coding, visualization involves domain-specific tools, rules, heuristics, and constraints that may be absent from model training or lay users’ common knowledge.The paper notes that LLMs are also entering visualization work, as they have entered the software development chain,.Source: Vibe Coding · p. 2
  • This study offers a sector- and role-agnostic interview view of LLM use across Vis processes, introducing vibe analysis and describing its characteristics, benefits, and challenges.The authors distinguish this scope from earlier work focused on particular parts or types of visualization.Source: Vibe Coding · p. 2

Methodology

The study used a DVS-distributed screening survey to recruit Vis practitioners for semi-structured interviews about LLM use, challenges, trust, and verification. Researchers analyzed interview transcripts through inductive thematic coding.

Methodology

  • The study asked how practitioners use LLMs in visualization and how they verify LLM responses in that context.The team designed semi-structured interviews to address these questions.Source: Methodology · p. 2

Participants & Interview Protocol

  • After screening survey respondents through the Data Visualization Society, the researchers interviewed 18 practitioners across eight countries.Participants opted in for paid interviews; the study received IRB approval and obtained informed consent.Source: Methodology · p. 3, Participants & Interview Protocol · p. 3
  • Interviews covered Vis expertise, LLM use, trust, verification, challenges, and participant demographics.Two researchers conducted virtual interviews averaging 28 minutes; sessions were recorded and transcribed, with notes used once when a recording failed.Source: Participants & Interview Protocol · p. 3

Qualitative Analysis

  • Researchers used inductive thematic coding, independently open-coded transcripts, reconciled themes, then independently closed-coded and merged results.They adapted Vis pipeline groupings to participants’ descriptions so the reported components reflected interview findings; the analysis was inspired by Crisan et al..Source: Qualitative Analysis · p. 3
  • Figure 1 identifies the LLM tools participants reported using.The supplemental material provides more detailed information about tools.Source: Figure 1 · p. 3

Findings

The findings describe an iterative LLM-supported process spanning data tasks across the visualization pipeline. The authors present a conceptual model informed by interview analysis and existing models.

Findings

  • Participants described an iterative process of using LLMs for data tasks across stages of the visualization pipeline.The reported scope is not comprehensive, and interviewee ratios are descriptive rather than estimates of prevalence among visualization practitioners.Source: Findings · p. 3
  • Figure 2 presents an author-developed conceptual model of vibe analysis informed by interview analysis and inspired by the data storytelling lifecycle and vibe coding cycle [5].The model’s scope reflects participants’ accounts and the cited framing, while other aspects of data analysis and visualization may also be affected by LLM use.Source: Findings · p. 3, Figure 2 · p. 4

Vibe Analysis

Practitioners use LLMs across creative and technical visualization work, in an iterative process the paper calls vibe analysis. Its workflow also depends on project constraints and external factors, including data, tools, legal considerations, time, cost, setting, and medium.

Vibe Analysis

  • Vibe analysis is an iterative, LLM-mediated visualization process in which people prompt for tasks and shift from direct specification toward articulating goals, curating content, and evaluating quality.Participants described using LLMs to brainstorm, wrangle, explore, and visualize data, often in loosely organized sessions aimed at workable overviews or charts.Source: Vibe Analysis · p. 4
  • The conceptual model represents an iterative loop in which users instruct an LLM, the LLM generates code and feedback, and practitioners review results or code, potentially across multiple tools and environments.Any visualization task may be supported by an assistant, and the process may take place in an LLM interface or alongside tools such as Tableau, Power BI, or Excel.Source: Vibe Analysis · pp. 4–5

Visualization Goals

  • LLMs appeared throughout the visualization pipeline, but finishing touches were less common (8/18) than earlier-stage uses; language and storytelling support was reported by 6/18.Participants described using LLMs for planning and ideation, data collection and preparation, exploration, and review and refinement; the authors speculate that finishing-touch use may be lower because of inconsistency and imprecision.Source: Visualization Goals · pp. 5–6
  • LLMs supported both creative and functional visualization work, with reported use highest for design (17/18), data wrangling (13/18), exploration (12/18), and ideation (11/18).Participants also reported use for analysis and insight discovery (5/18 each), programming (8/18), debugging or troubleshooting (6/18), and language or storytelling (6/18).Source: Visualization Goals · p. 5

The Interaction Environment

  • Project constraints include data sources and APIs, schemas, domain knowledge, and architecture such as the applications, languages, and environments used.The model's interaction environment supplies context and a place to review results; the conceptual model also characterizes tools, memory, and tasks as model components.Source: Vibe Analysis · p. 4, The Interaction Environment · pp. 6–7, The Model · p. 7

External Factors

  • External factors such as legal restrictions, ethics, setting, time, cost, and medium may affect project constraints and model use.Participants described privacy restrictions, concerns about misuse or misinterpretation, and switching among free model tiers; medium choices included static charts and interactive pieces.Source: External Factors · p. 7

Vibe Analysis and the Changing Nature of Data Work

Interviewees described changes in how they work with visualization tools and LLMs, along with efficiency, insight-discovery, and capability benefits. Some also reported job-related concerns and pressure to use AI.

Vibe Analysis and the Changing Nature of Data Work

  • 5/18 participants explicitly described shifts in their work, including moving from visualization tools to coding or from coding toward prompting, creation, and selecting useful outputs.The authors characterize this as practitioners abstracting away many fine-grained design and development details.Source: Vibe Analysis and the Changing Nature… · pp. 7–8
  • 14/18 participants reported efficiency benefits, 11/18 insight discovery, and 9/18 expanded capabilities from using LLMs in data work.Reported capability expansion included coding in unfamiliar languages, using tools without training, or developing products participants could not otherwise create.Source: Vibe Analysis and the Changing Nature… · p. 8
  • 5/18 participants felt compelled to use AI at work, citing employer expectations or concern that they would otherwise fall behind; participants also voiced fear of replacement.The paper notes that these perceptions echo sentiments in developer communities and tech-sector leadership,.Source: Vibe Analysis and the Changing Nature… · p. 8

Vis Practitioners’ Strategies

Participants described strategies for managing errors, sensitive data, and trust, including changing prompts, limiting what they share, and checking model outputs. Their approaches ranged from prevention to post-generation verification.

Vis Practitioners’ Strategies

  • Participants adapted prompts by adding context or guidance (9/18), starting over (8/18), breaking tasks into steps (6/18), and refining prompts (6/18).About 4/18 asked the LLM to check its own work, and 3/18 asked for explanations.Source: Vis Practitioners’ Strategies · p. 8
  • To prevent errors, participants planned ahead, prompted specifically, described or sampled data, used version control, or limited LLMs to repetitive tasks and rough drafts.The authors note that these strategies put the onus on users and suggest that applications could explicitly support them.Source: Vis Practitioners’ Strategies · p. 8
  • Most participants did not fully trust models, and 10/18 said they shared outputs after modifying or verifying them, while 4/18 shared only internally and 4/18 openly.Participants described validation as a trust lever, including reviewing code or checking results against data.Source: Vis Practitioners’ Strategies · p. 8
  • For sensitive data, 8/18 participants said they would not use LLMs; others described masking information, sharing schemas, using local or quarantined models, or using proxy data.About 5/18 reported masking or removing personally identifiable information, and 4/18 said they would share only a database schema or description.Source: Vis Practitioners’ Strategies · pp. 8–9
  • Participants checked outputs through visual inspection, spot checks, descriptive statistics, code execution or review, raw-data comparisons, and tests or trusted sources.The most frequently reported methods included looking for visible errors (8/18), spot checks (5/18), and checking descriptive statistics (5/18).Source: Vis Practitioners’ Strategies · p. 9

Challenges with LLMs for Data Analytics

Participants reported errors and other challenges with LLM-supported data analytics, including issues shared with vibe coding and difficulties specific to visualization. The paper also finds that strategies are sparse for societal impacts and broader user concerns.

Challenges with LLMs for Data Analytics

Challenges prevalent to both vibe analysis and vibe coding

  • Common challenges included low-quality outputs, such as plausible-looking but incorrect results, and misunderstood analytic intent compounded by visualization-specific encoding language.Participants also described limited context windows, which could require restarting complex analytic work, and verbose code that slowed presenting or sharing work.Source: Challenges prevalent to both vibe… · pp. 10–11
  • LLMs could produce inconsistent designs and accumulate design debt, complicating analytic reproducibility, comparability, and consistency across tools and audience-specific requirements.The paper notes that design debt can be harder to unwind than code-based technical debt because visualization tools differ in available and compatible design options.Source: Challenges prevalent to both vibe… · p. 10
  • Some educator-participants worried that shortcuts from LLMs could leave users without skills for designing good visualizations, while outputs still required time to review and correct.For visualization work, review includes checking data handling and chart design as well as code.Source: Challenges prevalent to both vibe… · pp. 10–11
  • Models often reverted to simple charts, which participants said could limit creativity and direct their thinking.The paper relates this homogeneity to a common challenge in vibe coding: limited scope and diversity in training and evaluation datasets.Source: Challenges prevalent to both vibe… · p. 11

Challenges more prevalent to vibe analysis than vibe coding

  • Visualization-specific challenges included gaps in Vis training and guidance, and hidden code that made chart outputs harder to test or verify, especially for non-coders.Unlike code often visible in vibe-coding platforms, code behind generated charts was not typically shared by default.Source: Challenges more prevalent to vibe… · p. 11

Strategy Gaps

  • Strategies addressed common pitfalls, model quirks, and system limitations more often than societal impacts or user concerns.The authors note that bias, privacy, and uncertainty cannot be fully mitigated, and suggest clearer disclosure by developers and users when sharing LLM-generated results.Source: Strategy Gaps · pp. 11–12

Discussion

The discussion recommends verification support, Vis-specific guidance, and tool designs that preserve human creativity while limiting downstream design debt.

Discussion

  • Practitioners need verification support through tool integration, training, and resources, since their current checks are practical but anecdotal and no technique works universally.Spot checks, descriptive statistics, and comparisons across models could be formalized or automated with rule-based techniques.Source: Recommendations for tool developers · p. 12, Discussion · p. 12

Recommendations for tool developers

  • Tool developers should use linting-like checks, transparency, and constraints to make verification more structured and prevent unintended changes from minor edits.Design guidance should mitigate hallucinations without supplanting human creativity or limiting novel design.Source: Recommendations for tool developers · p. 12

Recommendations for the Vis Community

  • The Vis community needs training that covers LLM capabilities, pitfalls, verification, and realistic practice, alongside resources for navigating workflow and job changes.Training should balance LLM benefits with an understanding of limitations and risks.Source: Recommendations for the Vis Community · p. 12
  • Design debt arises when visual choices that meet immediate needs create scalability, maintenance, or error problems that later require patches or overhauls.Early decisions should account for audience, integration, maintenance, hosting, and compatibility needs.Source: Recommendations for the Vis Community · p. 13
  • A visualization-focused version repository could help practitioners track changes, restore earlier versions, and improve transparency and attribution.The authors note that not all practitioners are comfortable with existing repository platforms.Source: Recommendations for the Vis Community · p. 13

Limitations and Future Work

The study's participant profile and self-reported practices limit the scope of its findings, and the authors identify broader sampling and reassessment as future work.

Limitations

  • Participants were skewed in country, age, race, and gender, and their self-reported LLM practices may reflect over- or under-reporting.The authors state that participants were not representative of all data workers.Source: Limitations and Future Work · p. 13

Future Work

  • Future research could include workers from other professions with lower visualization expertise or data analytics as a secondary responsibility.The authors describe the current participants as not representative of all data workers.Source: Limitations and Future Work · p. 13

Limitations and Future Work

  • The study focused on LLMs, and future work could reassess findings as the technology advances and investigate tool fit and design debt.The authors also propose studying user preferences for dividing visualization functions between LLMs and traditional tools.Source: Limitations and Future Work · p. 13

Conclusion

The authors call for critical, responsible use of LLMs in visualization and for preparing users to use them effectively. They emphasize that faster work may bring verification, skill, design, and maintenance trade-offs.

Conclusion

  • The authors call on the visualization community to think critically about where and how LLMs are applied and how users are trained to present data.They argue that preparing users may become a more pressing question than whether to use LLMs for visualization.Source: Conclusion · p. 13
  • LLMs may speed up visualization work while increasing verification burden, weakening skill formation, narrowing design exploration, and creating downstream maintenance and quality problems.The authors frame these trade-offs as both a challenge and an opportunity for a community that values clear and accurate data representation.Source: Conclusion · p. 13
  • The closing quote underscores goal-driven exploration and experimentation, including recognizing that some analysis and visualization tasks may not be appropriate to offload.It distinguishes knowing what one can do from knowing what one perhaps should not do.Source: Conclusion · p. 13
Loading 2609.31922v1…