PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language

Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang, Yingcai Wu

arXiv:2311.01920v2cs.HC

TL;DR

Abstract utterances make it difficult to capture user intent and produce appropriate chart specifications. ChartGPT addresses this with sequential sub-task reasoning, fine-tuning on abstract utterance–chart data, and editable intermediate outputs; 23 of 49 charts were obtained on the first attempt.

  • Problem

    Abstract natural-language utterances can be ambiguous or underspecified, making user intent and chart specifications difficult to determine.

  • Method

    ChartGPT decomposes chart generation into sequential sub-tasks, fine-tunes an LLM on abstract utterances and charts, and provides an interface for modifying intermediate outputs.

  • Results

    23 of 49 charts (47%) were obtained on the first attempt in the user study.

  • Takeaways & Limitations

    The interface supports users in exploring and modifying intermediate chart-generation results.

  • Takeaways & Limitations

    Fine-tuning an LLM with specific data may compromise generalizability, despite benefits for stability or new schemas.

Abstract

from arXiv · show

The use of natural language interfaces (NLIs) to create charts is becoming increasingly popular due to the intuitiveness of natural language interactions. One key challenge in this approach is to accurately capture user intents and transform them to proper chart specifications. This obstructs the wide use of NLI in chart generation, as users' natural language inputs are generally abstract (i.e., ambiguous or under-specified), without a clear specification of visual encodings. Recently, pre-trained large language models (LLMs) have exhibited superior performance in understanding and generating natural language, demonstrating great potential for downstream tasks. Inspired by this major trend, we propose ChartGPT, generating charts from abstract natural language inputs. However, LLMs are struggling to address complex logic problems. To enable the model to accurately specify the complex parameters and perform operations in chart generation, we decompose the generation process into a step-by-step reasoning pipeline, so that the model only needs to reason a single and specific sub-task during each run. Moreover, LLMs are pre-trained on general datasets, which might be biased for the task of chart generation. To provide adequate visualization knowledge, we create a dataset consisting of abstract utterances and charts and improve model performance through fine-tuning. We further design an interactive interface for ChartGPT that allows users to check and modify the intermediate outputs of each step. The effectiveness of the proposed system is evaluated through quantitative evaluations and a user study.

I. INTRODUCTION

Natural-language chart creation is intuitive but difficult because abstract utterances leave user intents and visual encodings ambiguous. ChartGPT addresses this with sequential task decomposition, visualization-specific fine-tuning, and an interface for inspecting intermediate results.

  • Motivation: Natural-language interfaces simplify visualization creation for users who lack programming expertise and reduce tedious toolkit interaction for experienced users.Examples include creating an IMDB-rating histogram through a natural-language request.
  • Challenges: Abstract queries often omit data fields, transformations, chart types, and encodings, making intent-to-specification mapping difficult.For example, “type” may refer to genre or rating, while “make the most money” implicitly refers to gross profit.
  • Challenges: Traditional lexical parsers and predefined aliases or rules support inference but are limited by language understanding and difficult maintenance or expansion.The limitation concerns both parser capability and the upkeep of predefined aliases and rules.
  • ChartGPT: ChartGPT decomposes chart generation into sequential sub-tasks so an LLM can produce manageable outputs for complex parameters and operations.The decomposition follows a least-to-most reasoning principle because one incorrect parameter can compromise later operations.
  • ChartGPT: The system fine-tunes FLAN-T5-XL on a dataset of abstract utterances and charts, then exposes intermediate outputs through an interactive interface.The interface lets users explore and modify intermediate chart-generation steps.
  • Evaluation: Quantitative experiments and a comparative user study evaluate the method, while the study also reports usability feedback and future work.The contribution list identifies quantitative experiments, user studies, and a dataset intended to facilitate future research.

II. RELATED WORK

Prior visualization systems use rule-based, machine-learning, parsing, or generic-LLM approaches, but abstract and implicit utterances remain challenging. ChartGPT instead fine-tunes a visualization-specific LLM and decomposes generation sequentially.

  • Visualization Recommendation: Rule-based recommendation systems map data to visual encodings using visualization knowledge and design rules.The related work classifies visualization recommendation techniques into rule-based and machine-learning-based categories.
  • Visualization Recommendation: Machine-learning methods learn from data and adapt to changing conditions, supporting ranking, sequence-to-sequence mapping, and explainability.Examples include DeepEye, Draco-learn, Data2Vis, Table2Charts, and KG4Vis.
  • Natural-Language Interfaces: Natural-language visualization systems use semantic or lexical parsing, ambiguity-resolution widgets, probabilistic grammars, and conversational interaction.Examples include Articulate, DataTone, FlowSense, Eviza, Evizeon, and NL4DV.
  • Natural-Language Interfaces: Existing NLI methods mainly target explicit requests and struggle with incomplete or implicit utterances; parsing capability limits their performance.NL4DV and Ask Data support forms of implicit inference through similarity or syntactic and semantic constraints.
  • Large Language Models: Large language models have demonstrated strong natural-language abilities and have been applied to data transformation, narration generation, web design, and visualization code generation.The related work discusses GPT-3, GPT-4, FLAN-T5, LLaMA, CHAT2VIS, and LIDA.
  • ChartGPT: ChartGPT differs by fine-tuning an open-source visualization-specific LLM, decomposing generation sequentially, and using input-output templates for parsing and applicability.Its approach adopts chain-of-thought reasoning rather than relying only on prompt engineering.

III. BACKGROUND AND PROBLEM FORMULATION

The paper models chart generation as transformations from table data and utterances to formatted visualization specifications and rendered charts. It uses sequential reasoning sub-tasks, shared-model processing, and six steps spanning data and visualization transformations.

  • A. Reasoning Strategies in LMs: The section frames reasoning as breaking a complex task into simpler sub-tasks that are solved from easiest to increasingly complex.Previously solved sub-tasks support later reasoning under the least-to-most strategy.
  • A. Reasoning Strategies in LMs: Chart generation is formulated as a fixed sequence in which one LLM uses the problem context and previous outputs to generate successive answers.The same model handles all sub-tasks, and their answers are consolidated into a complete chart.
  • B. Problem Formulation: The visualization pipeline is grounded in the Information Visualization Data State Reference Model, which describes successive data stages and transformations.This model provides the conceptual basis for the paper’s problem formulation.
  • B. Problem Formulation: The task contains three stages: table data, a formatted visualization specification, and charts.The specification is a text sequence conforming to a visualization grammar that can be parsed and rendered.
  • B. Problem Formulation: The central challenge is transforming table data and user utterances into visualization specifications.Each sub-task is represented as a formatted sequence-to-sequence problem.
  • B. Problem Formulation: Figure 2 shows ChartGPT taking a data table and utterance, applying six sequential LLM-based steps, and producing visualization specifications and charts.Each step uses an LLM fine-tuned on the constructed dataset and leverages outputs from earlier steps.

1) Problem Decomposition:

ChartGPT decomposes chart generation into six successive sub-tasks spanning data transformation and visualization transformation, then formats their outputs into a visualization specification. The proof-of-concept system focuses on selected design choices and common chart types.

  • Data transformation: The pipeline separates data transformation into column selection, row filtering, and optional aggregation.
  • Visualization transformation: Visualization transformation then chooses a chart type, maps fields to visual channels, and adds optional operations such as sorting.
  • Sequential reasoning: Each sub-task is modeled as a sequence-to-sequence problem whose input includes the utterance, table data, and previous answers.
  • Specification construction: The model combines formatted answers for filter, mark, encoding, and sort to construct a Vega-Lite specification, outputting “none” when optional operations are unnecessary.
  • Scope: The proof-of-concept covers seven common chart types and excludes more complex charts and design options beyond its scope.

A. Model Input

ChartGPT uses prompts and a fine-tuning dataset to help the model interpret abstract utterances that omit or blur chart specifications. The dataset is designed to expose diverse domains, chart types, abstraction levels, and reasoning paths.

  • Model input: Model inputs contain table data, the user utterance, and answers from previous sub-tasks, but table data is truncated to column names and two rows because of token limits.
  • Utterance abstraction: Abstract utterances may omit chart types or refer vaguely to fields, creating multiple plausible interpretations and reasoning paths.
  • Fine-tuning motivation: Because a single prompt cannot provide enough examples for these interpretation paths, the authors construct a dataset and fine-tune the model.
  • Dataset requirements: The dataset requirements include diverse domains, data types, chart types, abstraction levels, expressions, and query phrasings.
  • Dataset basis: The dataset is based on nvBench because it supplies charts across 105 domains, although its utterances are mostly explicit.

2) Dataset Construction:

The dataset construction process selects diverse nvBench charts, generates abstract utterances with GPT-3, checks their diversity, and corrects inconsistencies with the original charts. Intermediate chart configurations are converted into formatted sub-task answers for fine-tuning.

  • Chart selection: Charts are randomly selected from nvBench while balancing domains, hardness levels, and chart types after removing multi-table joins.
  • Abstract utterance generation: GPT-3 generates abstract utterances from selected data, original utterances, and chart triplets using manually designed prompts.
  • Diversity checking: The generation process dynamically checks utterance diversity to avoid repetitive polite or verbal phrasing.
  • Abstract utterance correction: Generated utterances are corrected when abstraction removes information needed to remain consistent with the original chart.
  • Step-by-step answer generation: Chart configurations are parsed into intermediate sub-task answers and combined with formatted templates to create expected model outputs.

3) Dataset Statistics:

The constructed dataset combines generated abstract utterances with retained nvBench utterances across diverse tables and charts. A Turing test found that generated utterances were difficult to distinguish from human-created ones and were judged human-created more often than NLV Corpus utterances.

  • Dataset size: The dataset contains 1,916 triplets, 236 data tables, 649 charts, and 1,916 utterances.
  • Dataset composition: Tables span 133 databases, while charts cover seven types; 79% use aggregation, 30% sorting, and 19% filtering.
  • Utterance abstraction: Generated abstract utterances reduce explicit mentions of chart types and aggregations, making the combined dataset resemble NLV Corpus utterances.
  • Turing test: The Turing test used 14 data-analysis-experienced subjects to judge 30 NLV Corpus and 30 generated utterances.
  • Overall results: 56% average judgment error and an overall human-created rating of 0.73 indicate that subjects struggled to distinguish generated from human-created utterances.
  • Comparison: Human-created ratings averaged 0.79 for generated utterances and 0.67 for NLV Corpus utterances, a significant difference with p = 0.03.

D. Model Fine-tuning

ChartGPT fine-tunes FLAN-T5-XL on chart-generation data and produces multiple valid chart candidates through a staged, interactive workflow. Users can inspect and modify intermediate specifications before regenerating charts.

  • Model fine-tuning: The model was fine-tuned on 1,538 training triplets and evaluated on 378 held-out triplets using a 4:1 split.Training used FLAN-T5-XL with AdamW; the reported evaluation loss was 0.10.
  • Chart generation: ChartGPT generates a default top-k set of three charts for each utterance.Candidate filtering removes invalid columns, expressions, aggregations, chart types, encodings, and sort tokens; beam search retains top combinations.
  • Interactive interface: The interface combines table, chart, and detail views so users can inspect data, enter utterances, review charts, and modify specifications.The detail view supports interaction-based edits to chart specifications and results.
  • Usage scenario: In the usage scenario, ChartGPT returned three charts for an abstract movie query, including alternatives based on genre counts and average ratings.The user selected the average-rating chart and identified Documentary as the genre with the highest average IMDB rating.
  • Usage scenario: Users could revise both the natural-language query and intermediate conditions, such as changing Release Year >= 2000 to Release Year >= 2008.The revised result identified Action as having the highest average gross since 2008, after which the user modified the chart configuration.

VI. EVALUATION

The evaluation compares ChartGPT with ncNet and NL4DV using exact consistency and specification similarity. ChartGPT achieves higher top-1 and top-3 performance, while abstract-language ambiguity remains a major source of errors.

  • Evaluation metrics: Consistency requires exact agreement with the ground truth across mark, encoding, aggregation, sort, and filter, while reversed scatterplot axes remain equivalent.This metric treats equivalent scatterplot orientations as consistent.
  • Evaluation metrics: Similarity compares fixed eight-word chart specifications using ROUGE-L and BLEU to capture partial agreement with the ground truth.The sequence includes mark, fields, aggregations, color, filter, and sort; BLEU complements ROUGE-L when axes differ.
  • Evaluation metrics: Invalid results receive zero similarity and consistency scores when they reference nonexistent columns or cannot be parsed into Vega-Lite.The evaluation first validates table relatedness and Vega-Lite parsability.
  • Overall results: ChartGPT outperforms ncNet and NL4DV on both consistency and similarity, with higher scores for its top-1 and top-3 results.The comparison reports top-1 and top-3 results for ChartGPT and NL4DV, but only top-1 for ncNet.
  • Error analysis: The comparison attributes ChartGPT’s advantage to stronger semantic parsing and reasoning about information omitted from abstract utterances.Examples include inferring fields from phrases such as “male and female” or “how old,” and inferring temporal fields and count aggregation.
  • Error analysis: Abstract utterance ambiguity lowers consistency because multiple chart types may be reasonable and models may miss aggregations, filters, or sorting conditions.The analysis distinguishes ambiguity from partial inferences that preserve some fields but omit critical chart details.

VII. USER STUDY

A comparative user study asked visualization-experienced participants to rank charts from ChartGPT, ncNet, and NL4DV. ChartGPT received the strongest rankings, with statistically significant differences in both evaluated user-preference measures.

  • Study design: Twelve participants compared charts from ChartGPT, ncNet, and NL4DV across 15 utterances, 13 data tables, and 42 generated charts.Participants ranked chart quality based on the provided tables and utterances.
  • Study design: The study measured normalized ranking scores and the proportion of cases in which each approach was ranked first.A Friedman test assessed overall differences, followed by post hoc Wilcoxon tests for pairwise comparisons.

B. Usability Study

The usability study examined how participants used ChartGPT to create charts through first attempts, configuration edits, and query rephrasing. Most logged attempts produced charts, although unsupported transformations and designs caused failures.

  • Study design: Twelve participants explored movie or car data and were asked to create at least four desired charts using ChartGPT.They could rephrase inputs, modify intermediate answers or configurations, and regenerate results.
  • Quantitative results: Forty-nine of 53 historical logs produced successfully generated charts, while four failed because users could not obtain satisfactory results.Successful charts were grouped by first-attempt generation, step or configuration adjustment, or input rephrasing.
  • Quantitative results: 23 of 49 charts, or 47%, were obtained on the first attempt.The remaining successful cases involved 13 step or configuration adjustments and 13 input-rephrasing cases.
  • Failure cases: Three failed inputs involved unsupported transformations or visual designs, including gross divided by budget and two side-by-side bar charts.A fourth failed input used contradictory encoding by assigning two fields to the x-axis.

3) Qualitative Feedback:

Participants valued ChartGPT’s flexible semantic understanding, incomplete-intent support, and editable intermediate results. The study also highlighted that users’ design preferences can diverge from common visualization conventions.

  • Incomplete-intent support: ChartGPT supported incomplete intents, letting users begin with only an initial interest in particular data columns and explore progressively.Most subjects entered inputs that did not specify trends, distributions, or relationships.
  • Semantic understanding: Its semantic inference connected indirect language to appropriate data columns, reducing users’ need to phrase queries precisely.For example, “which type of movies earn most” was mapped to Major Genre and Worldwide Gross, while “number of movies over time” favored Release Year over Running Time.
  • Interactive modification: Users could modify intermediate outputs to bring system-generated charts closer to their desired results and preferences.Overall, 10 of 12 subjects used modifications, and seeing initial results gave users a clear direction for changes.
  • Broader implications: The framework’s decomposition-and-fine-tune approach, dataset contribution, and evaluations support applications beyond NL2VIS generation.The authors suggest decomposing other complex downstream tasks, such as long-story writing, into sub-modules.
  • Design preferences: User preferences were not always consistent with common design principles, making modification important for chart design choices.The collected data commonly followed conventions such as scatter plots for two quantitative columns and line charts for trends over time, but study preferences varied.

C. Limitations and Future Work

ChartGPT remains bounded by its supported design space, prompt-length constraints, and the challenge of comparing it fairly with generic LLM-based charting methods. The authors also identify tensions between accuracy, exploratory inspiration, and users’ intentions beyond the system’s capabilities.

  • Support for a larger scope: ChartGPT supports only some chart components and design choices, leaving table reshaping, additional transformations, parameters, and follow-up modifications for future work.The authors specifically identify operations such as pivot and mutate, expanded marks and channels, and follow-up utterances as extensions.
  • Scalability for large input tables: Large input tables can exceed the model’s 580-token maximum prompt length, limiting how comprehensively table information is included.The authors propose reducing input tokens or expanding training data and computational resources; large-table comparison with rule-based methods remains future work.
  • Comparison with generic LLM-based methods: Generic LLM-based methods remain difficult to compare comprehensively because systems such as LIDA lack a predefined design space and support diverse visualization choices.On the test set, LIDA produced 12 Seaborn failures and 67 Altair failures, compared with 7 for ChartGPT.
  • Inspiration versus accuracy: Prioritizing accurate intent capture can underprovide the exploratory content some users want beyond their utterances.Some participants preferred charts including additional data fields, and three requested content that could inspire exploration beyond the query’s scope.
  • Flexibility versus certainty: Natural-language flexibility can exceed the system’s fixed design space, causing misaligned results and confusion when users express unsupported intentions.Observed unsupported intentions included requests that could not be charted and a transformation dividing gross by budget.
Loading 2311.01920v2…