Source-linked AI summary
LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models
Victor Dibia
TL;DR
Automatic visualization must connect data semantics, visualization goals, and specifications while supporting users through a complex authoring process. LIDA addresses this with a modular LLM- and image-generation pipeline, achieving a low visualization error rate and supporting multiple visualization and infographic workflows.
Problem
Automatic visualization systems must handle data semantics, goal enumeration, specification generation, and user control, while existing approaches have interaction, coverage, control, and grammar-specificity limitations.
Method
LIDA uses a four-module pipeline combining natural-language dataset summarization, LLM-generated goals and visualization code, execution and refinement, and image-generated infographics.
Results
LIDA reports a low visualization error rate of VER = 3.5% and supports grammar-agnostic visualization and data-faithful infographic generation.
Takeaways & Limitations
LIDA provides a simplified, general, flexible, and scalable foundation for interactive visualization generation and workflows such as data exploration and data stories.
Takeaways & Limitations
LIDA’s performance may be limited for visualization grammars that are poorly represented in the underlying LLM training data, such as GUI-oriented tools.
Abstract
from arXiv · showhide
Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) such as ChatGPT/GPT-4 and image generation models (IGMs) are suitable to addressing these tasks. We present LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes and filters visualization code and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a python api, and a hybrid user interface (direct manipulation and multilingual natural language) for interactive chart, infographics and data story generation. Learn more about the project here - https://microsoft.github.io/lida/
1 Introduction
LIDA frames automatic visualization as a multi-stage generation problem that combines LLMs and image generation models to address data understanding, goal exploration, visualization generation, and infographic creation.
- Research gap: Automatic visualization requires understanding dataset semantics, enumerating relevant goals, selecting suitable representations, and implementing visualizations.These steps can be tedious, error-prone, and demanding for users with limited visualization experience.
- Research gap: Existing AUTOVIZ systems face limitations including heuristic coverage, substantial user interaction, limited control, and grammar-specific training requirements.These constraints motivate a more general and controllable pipeline.
- Approach: LIDA assembles foundation models into a modular pipeline that represents datasets as natural-language summaries and generates visualization goals and code.The approach applies LLMs to multiple visualization subtasks rather than relying on separate subtask-specific models.
- Approach: The system combines visualization generation with text-conditioned image generation to produce stylized infographics that are generally faithful to data.LIDA also supports conversational control, refinement, and interfaces combining direct manipulation with multilingual natural language.
- Contributions: LIDA introduces visualization error rate and self-evaluated visualization quality as metrics for pipeline reliability and visualization quality.The paper implements the approach as an open-source library with Python, web APIs, and a web interface.
- Contributions: LIDA is presented as a first end-to-end multi-step generation pipeline addressing diverse visualization and infographic subtasks.The authors characterize it as simplified, general, flexible, and scalable compared with existing AUTOVIZ approaches.
2 Related Work
LIDA builds on foundation models for creative multimodal tasks and reframes automated visualization as a modular generation pipeline. It addresses limitations of prior AUTOVIZ systems, including grammar dependence, custom training requirements, limited control, and weak error recovery.
- Foundation models support creative tasks across text, images, audio, and video, motivating their use for visualization and infographic generation.
- LIDA uses LLMs to generate visualization goals and code, then passes rendered visualizations to image-generation models for stylized infographics.
- Prior learning-based AUTOVIZ approaches are limited to a single grammar, require custom models and paired training data, and lack infographic generation, fine-grained control, and robust error recovery.
- LIDA addresses these limitations with a grammar-agnostic pipeline, natural-language control, automatic or semi-automatic goal generation, and data-faithful infographic generation.
- Infographics convey data-driven narratives through visual imagery and embellishments, but creating them can require skills across multiple tools and domains.
3 The LIDA System
LIDA is a four-module system that grounds visualization generation in compact dataset summaries, explores goals, generates executable visualization code, and creates stylized infographics. Its interface supports refinement, explanations, multilingual interaction, and visualization debugging.
- LIDA comprises SUMMARIZER, GOAL EXPLORER, VISGENERATOR, and INFOGRAPHER modules implemented as Python libraries with an optional user interface.
- 3.1 SUMMARIZER: SUMMARIZER extracts types, statistics, and samples, then optionally enriches the compact summary with LLM- or user-provided semantic descriptions and field types.
- 3.2 GOAL EXPLORER: GOAL EXPLORER generates a question, a corresponding visualization, and a rationale from the dataset summary; requiring a rationale yields more semantically meaningful goals.
- 3.2.1 VISGENERATOR: VISGENERATOR uses language- and grammar-specific executable scaffolds, fill-in-the-middle code generation, execution, and filtering to produce visualization specifications and raster images.
- 3.2.1 VISGENERATOR: Filtering can discard non-compiling candidates, select consensus solutions, or select candidates with the highest predicted correctness probability, with the latter two approaches requiring multiple LLM passes.
- 3.3 INFOGRAPHER: INFOGRAPHER applies editable natural-language styles to visualization images through text-conditioned diffusion-based image-to-image generation, with optional post-processing.
- 3.4 USER INTERFACE: The interface displays summaries, goals, specifications, and code scaffolds while supporting natural-language goals, refinement, explanations, accessibility descriptions, and multilingual interaction.
- Together, the modules implicitly address data transformation, encoding, mark selection, styling, layout, and annotation.
4 Evaluation
LIDA evaluates reliability and visualization quality using VER and SEVQ, and studies how summarization affects errors across a benchmark of datasets and grammars.
- Metrics: VER measures the percentage of generated visualizations that produce code compilation errors, with E counting errors and T counting total visualizations.The metric is used to assess pipeline reliability and the impact of system changes.
- Metrics: GPT-4 scores generated visualizations from 1-10 across code accuracy, data transformation, goal compliance, visualization type, data encoding, and aesthetics.SEVQ averages the six dimension scores and provides rationales for the assessments.
- Evaluation setup: LIDA’s benchmark uses 57 Vega datasets, generating 5 goals and 1 visualization per goal across multiple grammars.Temperature is set to 0 with one LLM sample for reproducibility.
- Results: VER = 3.5% across the initial evaluation, indicating a low visualization error rate for LIDA.The study also examines the SUMMARIZER through ablation experiments.
- Results: Including a summary reduces error rate compared with using only field names as the schema summary, while LLM enrichment has less effect on VER.The ablation compares no_enrich, enrich, schema, and no_summary conditions, with variation across grammars; Seaborn has lower VER.
- Results: SEVQ identifies semantic quality issues, such as recommending a bar chart over a requested pie chart and offering automatic repair.The self-evaluation provides a rationale for the recommendation.
5 Conclusion
LIDA frames visualization generation as a multi-stage text and code generation problem addressed with LLMs. It provides grammar-agnostic visualization and infographic generation with controllable workflows and evaluation metrics.
- Conclusion: LIDA formulates visualization generation as a multi-stage text and code generation problem using large language models.The approach targets automatic generation of visualizations and infographics.
- Conclusion: LIDA supports automatic goal generation, conversational visualization refinement, multiple visualization grammars, and infographic generation.The conclusion presents these capabilities as responses to limitations of current automatic visualization systems.
- Conclusion: LIDA introduces VER for reliability and SEVQ for visualization quality in LLM-enabled visualization tools.The metrics are intended to assess pipeline reliability and generated visualization quality.
6 Limitations
LIDA’s performance depends on the representation of visualization grammars in LLM training data and may be limited for complex tasks or underrepresented tools. The authors call for broader benchmarks and further study.
- Low Resource Grammars: LIDA may perform poorly on visualization grammars that are weakly represented in training data, including GUI-based tools such as Tableau and PowerBI.Additional fine-tuning or translation may be needed for such grammars.
- Low Resource Grammars: Complex tasks requiring data transformations beyond a grammar’s expressive capabilities may limit LIDA’s performance.The authors identify task disambiguation, task complexity, and programming-language or grammar choice as future research topics.
- Evaluation scope: More comprehensive benchmarks are needed across a wider variety of datasets and visualization grammars.The current study introduces VER and SEVQ but does not establish broad coverage through those metrics alone.
- Future research: Future work should quantify LLM use of visualization best practices, map failure cases, and study effects on user creativity.These directions include empirical evaluation of model behavior and qualitative study of visualization authoring.
A The LIDA Library
The LIDA library exposes programmatic and web interfaces for a modular visualization workflow. Its UI supports data upload, grammar selection, goal and code generation, execution, evaluation, and refinement.
- Library interfaces: LIDA is implemented as a Python library with Python, web, and command-line interfaces, plus a web-based user interface.The library is distributed through GitHub and pip.
- Workflow: The example workflow generates a data summary, visualization goals, code specifications, and executed visualizations.This figure illustrates the main stages of the LIDA pipeline.
- Self-evaluation: SEVQ evaluates generated visualization code on six dimensions and averages the resulting 1-10 scores.The dimensions are code accuracy, data transformation, goal compliance, visualization type, data encoding, and aesthetics.
- Data upload: Users can select a visualization grammar and upload a dataset; upload triggers goal-generation and visualization-generation tasks.The grammar choice is made in the data-upload section of the UI.
- Visualization generation: The visualization-generation interface accepts natural-language goals and lets users inspect, edit, execute, refine, explain, evaluate, and recommend visualizations.The interface displays both generated code and the resulting visualization.
- Self-evaluation: Self-evaluation can critique a pie chart and recommend a bar chart as an alternative.The critique includes scores across six dimensions and a rationale.
C Design Reflections
LIDA’s design reflects engineering choices across a broad space of foundation-model components, with explicit attention to tradeoffs.
- The authors reflect on design choices and tradeoffs involved in building LIDA with text and image foundation models.
C.1 Prompt Engineering
LIDA’s prompt-engineering choices provide grounding, constrain generation, and improve the reliability and faithfulness of outputs across its modules.
- Prompt engineering aimed to maximize the probability that the LLM would solve each visualization subtask.
- SUMMARIZER: Richer summaries containing qualitative descriptions and semantic types improved generated goals and visualization code.
- GOAL EXPLORER: Few-shot field–rationale examples encouraged exact dataset field names, reducing hallucinated fields and embedding visualization best practices.
- VISGENERATOR: Fill-in-the-middle generation with code scaffolds constrained the model to produce executable visualization code, while scaffold freedom affected supported task complexity.
- Overall prompt design: Low temperature (t = 0) combined with per-grammar scaffolds produced code that correctly compiled into visualization specifications and faithfully addressed subtasks.
- Overall prompt design: The authors also explored multi-task prompts, such as combining summary enrichment into a single LLM call to reduce latency and compute.
C.2 Infographic Generation
LIDA’s infographic generation uses controlled image-to-image diffusion and parsimonious style prompts to preserve the original visualization’s general structure while stylizing it.
- A latent diffusion strength between 0.25 and 0.45 produced stylized images faithful to the original visualization’s general structure.
- Controlled generation minimizes distorted, irrelevant, or superfluous imagery that could distract from the infographic.
C.3 Natural Language Interaction
LIDA combines direct manipulation with natural-language interaction for visualization authoring, refinement, explanation, evaluation, and recommendation. Its infographer conditions data-faithful graphics on generated visualizations and natural-language style tags.
- HYBRID INTERFACE: A hybrid interface pairs direct manipulation, such as selecting fields, with natural-language affordances for visualization creation.
- NL INTERACTION MODES: Natural-language interaction supports refinement, explanation, evaluation, and recommendation operations on generated visualization code.
- The infographer generates data-faithful infographics conditioned on a generated visualization and natural-language style tags that customize chart appearance.