PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Have Visualization Literacy? An Evaluation on Modified Visualizations to Test Generalization in Data Interpretation

Jiayi Hong, Christian Seto, Arlen Fan, Ross Maciejewski

arXiv:2501.16277v1cs.PF

TL;DR

Whether LLMs possess visualization literacy remains underexplored, despite practitioners’ increasing use of LLMs for visualization-related tasks. The paper proposes an evaluation template and finds that GPT-4 and Gemini lack human-level visualization literacy and rely heavily on prior knowledge rather than visualization information.

  • Problem

    Whether LLMs possess visualization literacy remains poorly understood, despite practitioners’ increasing use of LLMs for visualization-related tasks.

    Source: §2.1 · p. 2

  • Method

    The paper proposes a template and methodology for evaluating LLM visualization literacy.

    Source: §1 · p. 1

  • Results

    Both GPT-4 and Gemini lacked visualization literacy relative to humans and relied heavily on prior knowledge rather than visualization information.

    Source: §1 · p. 1, §7 · p. 10

  • Takeaways & Limitations

    Current models are not yet prepared to replace humans in visualization evaluation, though they can be experimented with in diverse scenarios given their cost efficiency.

    Source: §7 · p. 11

    The study used one specific testing method, and exploring other methods could yield different performance.

    Source: §8 · p. 11

Abstract

from arXiv · show

In this paper, we assess the visualization literacy of two prominent Large Language Models (LLMs): OpenAI's Generative Pretrained Transformers (GPT), the backend of ChatGPT, and Google's Gemini, previously known as Bard, to establish benchmarks for assessing their visualization capabilities. While LLMs have shown promise in generating chart descriptions, captions, and design suggestions, their potential for evaluating visualizations remains under-explored. Collecting data from humans for evaluations has been a bottleneck for visualization research in terms of both time and money, and if LLMs were able to serve, even in some limited role, as evaluators, they could be a significant resource. To investigate the feasibility of using LLMs in the visualization evaluation process, we explore the extent to which LLMs possess visualization literacy -- a crucial factor for their effective utility in the field. We conducted a series of experiments using a modified 53-item Visualization Literacy Assessment Test (VLAT) for GPT-4 and Gemini. Our findings indicate that the LLMs we explored currently fail to achieve the same levels of visualization literacy when compared to data from the general public reported in VLAT, and LLMs heavily relied on their pre-existing knowledge to answer questions instead of utilizing the information provided by the visualization when answering questions.

1 INTRODUCTION

The paper proposes a template for assessing whether LLMs have the visualization literacy needed to support visualization evaluation. Its results indicate that GPT-4 and Gemini fall short of human performance and rely heavily on prior knowledge rather than visualization content.

1 INTRODUCTION

  • Because evaluating visualizations requires costly human labor, the template is intended to assess whether LLMs' literacy is comparable to the general public's before they are used in evaluation.The paper presents the template, a series of studies, an exploration of interpretation limitations, and discussion of future potential for using LLMs to evaluate visualizations.Source: §1 · pp. 1–2
  • The evaluation template adapts the 53-item VLAT for GPT-4 and Gemini across 12 visualizations and 8 tasks, primarily measuring accuracy.Source: §1 · p. 1
  • Studies examined performance across models, tasks, and visualizations; prior knowledge versus visual reasoning; answer choices and visualization context; and time and monetary costs against humans.Source: §1 · p. 1
  • Both GPT-4 and Gemini lacked visualization literacy compared with humans, relied heavily on prior knowledge, struggled with open-ended questions, and improved with decontextualization; both were cost-effective, especially Gemini.GPT-4 improved more than Gemini when visualizations were decontextualized.Source: §1 · p. 1

2 RELATED WORK

Related work establishes visualization-literacy tests and chart-question-answering benchmarks, while identifying unresolved concerns about whether LLMs interpret charts or rely on prior knowledge. This study positions its work as a systematic assessment of off-the-shelf LLMs’ visualization literacy.

2.1 Visualization Literacy

  • Visualization literacy is the ability to read and interpret visual representations; prior measures include the 53-item VLAT and 12-item Mini-VLAT [31].VLAT research found a positive correlation between literacy and interpreting unfamiliar visualizations. The paper identifies whether LLMs possess this ability as an open question, defining literacy as the skill to read, understand, and interpret visualization information.Source: §2.1 · p. 2

2.2 Chart Question Answering

  • Chart-question-answering research includes DVQA, ChartQA, FigureQA, and PReFIL, while this study examines whether off-the-shelf LLMs possess visualization literacy rather than rely on prior knowledge [20] [29] [22] [21].Prior CQA work focused largely on datasets and benchmarking, with datasets mostly using real-world data; the paper frames comprehensive analysis of LLM literacy as a contrast [21].Source: §2.2 · p. 2

2.3 Evaluating LLMs

  • LLM evaluation has expanded across programming, planning, computational social science, psychology, and HCI, including studies of human-like responses and replacing human evaluation [43].Hämäläinen et al. found that human-like answers could be affected by malicious inputs.Source: §2.3 · p. 2
  • Visualization-related LLM work has generated captions, images, and chart enhancements, while prior evaluations tested design advice or code-based chart interpretation [26] [17] [47].This study instead evaluates visualization literacy using PNG images, which the passage describes as more common in real-world settings than SVG files and JavaScript code; EvaLLM [34] is a conceptual stack for evaluating generative-AI visualizations.Source: §2.3 · p. 2
  • Prior LLM literacy studies reused VLAT materials, potentially benefiting from training-data exposure, and did not account for answer-choice or option-order effects.Work on misleading-chart identification also used website-collected datasets; the paper identifies whether answers derive from prior knowledge or the presented visualizations as an unresolved question.Source: §2.3 · p. 2

3 PILOT/EXPLORATORY EXPERIMENTS

Pilot experiments exposed weaknesses in chart-data extraction and chart recreation, leading the researchers to test visualization literacy by having models read PNG visualizations directly with data-value labels removed.

Pilot/exploratory experiments

  • In the second exploratory perspective, ChatGPT's chart-recreation code made value and bar-height errors, while API responses gave the highest confidence regardless of accuracy.The task was to recreate charts and produce a new visualization from raw data.Source: §3 · pp. 2–3
  • Even on simple bar charts without value annotations, models failed to extract precise data in all five attempts, relying on labels over bar lengths or positions even when labels were wrong.The errors occurred regardless of whether grid lines were present.Source: §3 · pp. 2–3
  • Models interpreted SVG files more accurately than PNG files, but the researchers focused on PNG because it is more commonly used in everyday contexts.Source: §3 · p. 3
  • The pilot findings led researchers to have LLMs read PNG visualizations directly and remove data-value labels to prevent reliance on those labels.Source: §3 · p. 3

4 ASSESSING LLMS’ VISUALIZATION LITERACY

The section evaluates GPT-4 and Gemini using a modified VLAT assessment, comparing their performance with human results and testing whether visualizations improve their answers. Neither model matched the general public’s visualization literacy, and the findings suggest they often relied on prior knowledge rather than chart information.

4.1.1 Experimental Design

  • To reduce the risk of testing memorized VLAT answers, the authors recreated its charts with randomized values and updated questions and answers, then tested GPT-4 and Gemini using direct image inputs.A pilot found GPT-4 consistently matched VLAT answers for some original questions; the modified assessment aimed to keep the charts out of the models’ training data.Source: §4.1.1 · p. 3, §4.1 · p. 3

4.1.2 Experiment 1: Evaluate LLMs’ Visualization Literacy

  • Each of 53 questions received 120 counterbalanced trials per experiment, yielding 6,360 trials per experiment and 25,440 total across two models and visualization-present/absent conditions.The experiments used separate sessions for each question, randomized question order, and constrained responses to option letters; the no-visualization experiment used GPT-4-turbo-preview and Gemini-pro.Source: §4.1.2 · p. 3, §4.1.3 · p. 4, §4.3 · p. 5

4.2 Analysis Methods

  • They modeled correctness (binary) with logistic regression across visualization type, task type, LLM type, and visualization presence; the model had 629 coefficients and was fitted 1,000 times via bootstrapping.Bootstrap distributions supported coefficient and probability hypothesis tests; the visualization-presence test compared GPT-4 and Gemini probabilities across all 49 visualization/task interactions.Source: §4.2 · pp. 4–5

4.3.2 H1 Visualization Literacy of GPT-4 and Gemini

  • 42 of 49 visualization/task interactions significantly distinguished GPT-4 and Gemini: GPT-4 performed better in 17, Gemini in 25, and 7 showed no difference.Although aggregate coefficients suggested GPT-4 performed better with visualizations, the authors attribute this apparent conflict to Simpson’s paradox, particularly the large GPT-4 advantage on Identify the Hierarchical Structure; without that task, Gemini appeared better overall.Source: §4.3.2 · p. 5

4.3.3 H2 & H3 LLMs in Varied Visualizations and Tasks

  • With visualizations present, positive coefficients supported line, scatterplot, and area charts, while several complex charts also had positive coefficients; only Make Comparisons supported H3 among task types.Gemini had 6 positive task coefficients versus GPT-4’s 2, suggesting it may outperform on Retrieve Value, Find Extremum, Make Comparisons, and Find Clusters. GPT-4 showed stronger alignment for scatterplot correlation and choropleth comparisons, while Gemini did so for scatterplot extrema and bubble-chart correlation.Source: §4.3.3 · pp. 5, 7

4.3.1 RQ1 Visualization Literacy of LLMs vs. Humans

  • Across 53 questions, GPT-4 outperformed humans on 14 and Gemini on 15; neither matched general-public visualization literacy, and they exceeded random-choice accuracy on 25 and 24 questions, respectively.The human baseline came from the original VLAT [24]. The authors suggest reliance on prior knowledge may explain why the models did not consistently benefit from information in the visualizations.Source: §4.3.1 · p. 5, Table 2 · p. 6

4.3.4 H4 LLMs’ Performances with Vis vs. No Vis

  • 18 of 49 visualization/task interactions were significantly affected by visualization presence for GPT-4, and 20 for Gemini, so the authors could not support H4.Because most interactions showed no significant effect, the findings suggest the models mostly relied on their knowledge base despite questions requiring information from the visualization.Source: §4.3.4 · p. 7

5 FURTHER ANALYSIS OF LLMS IN VISUALIZATION UNDER-

The section tests whether answer choices and visualization context affect LLMs’ interpretation of visualizations. Results show that visualizations often improve choice-free responses, while decontextualization can reduce reliance on prior knowledge.

STANDING

  • The section frames two questions: whether answer choices limit LLM performance and whether contextualized visualizations affect interpretation, with the aim of identifying useful choice and context strategies.The authors motivate open-ended questions as practical for visualization reading because LLM answers require no human effort, although existing assessments use choices for efficiency.Source: STANDING · p. 7

5.1.1 Experiment 3: Examine LLMs’ Choice-Free Performance

  • Experiment 3 asked GPT-4 and Gemini the Experiment 1 questions without answer options to test whether choices affected performance across task types.The models were instructed to answer in one word and were tested repeatedly in randomized question order.Source: §5.1 · p. 7, §5.1.1 · p. 7

5.1.2 Experiment 4: Examine LLMs’ Performance without Visualization and Choices

  • Experiment 4 repeated the questions without either choices or visualizations, enabling comparison with Experiment 3’s choice-free responses when visualizations were present.The study repeated each question 120 times, shuffled question order, and recorded response times.Source: §5.1.2 · p. 8

5.1.3 Experiment 5: Examine LLMs’ Performance of Decontextualized Visualization

  • Experiment 5 removed proper nouns from visualizations to test whether decontextualization changed performance, retaining answer choices for comparison with Experiments 1 and 2.After excluding three visualizations, the study used 40 questions and 19,200 trials across two models and visualization-present or -absent conditions.Source: §5.1 · p. 7, §5.1.3 · p. 8

5.2 Analysis Methods

  • Analyses separated choice-free textual and numerical answers from Experiment 5’s accuracy comparisons, using standardized responses, error categories, and task-specific scoring.Single-value numerical responses were scored by relative error; range answers used overlap metrics, while textual matches received binary scores and unexpected responses were counted wrong.Source: §5.2 · p. 8, §5.2.1 · pp. 8–9, §5.2.2 · p. 9

5.3.1 RQ2.1 Results of Experiments 3 and 4

  • With visualizations, GPT-4’s textual-answer accuracy was 38.27% and Gemini’s was 32.39%; without them, the rates were 28.33% and 29.73%, respectively.Across 14,119 valid textual responses, accuracy was 32.49%; the authors conclude that choices guide responses toward expected answers rather than constrain the models.Source: §5.3.1 · p. 9
  • For numerical tasks, visualizations generally improved range performance, while GPT-4 often outperformed Gemini on single-value responses and Gemini sometimes worsened with visualizations.GPT-4 answered all 120 taxi-rating range trials correctly with a visualization, versus 119 for Gemini; for single-value questions, GPT-4’s error improved while Gemini’s worsened with visualizations. Gemini’s area-chart median overlap was approximately 70% with a visualization and 0% in other conditions.Source: Figure 4 · p. 9, §5.3.1 · p. 9

5.3.2 RQ2.2 Accuracy Rates in Experiment 5

  • GPT-4’s accuracy rose from 30.90% on contextualized visualizations to 41.96% on decontextualized ones, while Gemini’s rose from 38.98% to about 43.13%.For GPT-4, the 100% stacked bar chart improved from 4.17% to 36.81%; the authors found accuracy without visualizations closer to random-choice rates for all GPT-4 tasks and most Gemini tasks, suggesting more guessing on decontextualized questions.Source: §5.3.2 · p. 10

6 COSTS DIFFERENCES BETWEEN LLMS AND HUMANS

LLMs were cheaper and faster than human participants in this study, but their limited visualization literacy means they cannot yet replace humans and entails a trade-off for evaluation use.

6 COSTS DIFFERENCES BETWEEN LLMS AND HUMANS

  • For 53 questions, GPT-4 cost about $0.53 and Gemini nothing, versus about $1.80 per human participant; both LLMs were faster than humans’ 25-second limit.The human estimate assumes a $8-per-hour minimum reward rate and about 13 minutes to answer all 53 questions.Source: §6 · p. 10
  • After excluding trials over 100 seconds as outliers, GPT-4 averaged 14.48 seconds per visualization and Gemini averaged 3.01 seconds.GPT-4’s processing time depended mainly on API traffic, while Gemini’s remained relatively stable; the longest GPT-4 trial took 10,018 seconds for one question without a visualization.Source: §6 · p. 10
  • GPT-4 cost around $0.01 per question with a visualization and $0.001 without one; Gemini’s API was free, and the formal study cost approximately $160.The approximately $160 total excludes the pilot study and testing.Source: §6 · p. 10
  • Because each question was tested in a separate session, it remains unclear whether a single session answering all 53 questions would change LLM timing.The authors conclude that LLMs may be cost-effective and time-efficient, but their limited visualization literacy creates a trade-off against recruiting humans.Source: §6 · p. 10

7 DISCUSSION

The evaluated LLMs lack sufficient visualization literacy to replace humans in visualization tasks, in part because prior knowledge can override what a visualization shows. The discussion recommends cautious, task-specific experimentation and further evaluation before relying on LLMs as visualization evaluators.

Discussion

  • Current LLMs lack sufficient visualization literacy: GPT performed relatively well on scatterplot correlation-trend and comparison tasks, but neither model reliably read pie charts or histograms.The authors recommend checking whether a model can accurately read data before using it for initial visualization evaluation.Source: §7 · p. 10
  • The authors recommend assessing LLM literacy on new visualizations against human results and exploring literacy beyond visualization, including cross-modal and broader context understanding.With improved visualization literacy, LLMs could support initial evaluation of new visualizations and systems, potentially offering a cost-effective alternative to crowdsourcing.Source: §7 · p. 10
  • LLMs may answer from pre-existing knowledge instead of the visualization, especially for factual questions without choices, so current models cannot replace humans in visualization tasks.The authors suggest this may help explain a discrepancy with a prior VLAT evaluation; out-of-date knowledge can also lead to incorrect guidance on visualizations with new data.Source: §7 · p. 10
  • LLMs’ reliance on learned facts could be used to flag potential fraud in data and remind visualization designers to reflect on their data sources..Source: §7 · p. 11
  • Designers could experiment with GPT-4 and Gemini on different chart types, while users could select models by task; these recommendations require validation with more visualization and task data.The discussion also suggests trying GPT-4 without visualization context to improve interpretation performance, but says this needs further verification.Source: §7 · p. 11
  • Overalignment can make answers overly broad or conservative, sacrificing specificity and usefulness; addressing it may require training that balances factual accuracy with actionable responses.For determined-range questions, models gave broad responses; the discussion cites Llama2-7b’s 57% refusal rate on innocuous prompts as an example of overalignment.Source: §7 · p. 11

8 LIMITATIONS AND FUTURE WORK

The authors identify limits in how they tested LLM visualization literacy and propose future work on prompting, anonymization, interaction, and broader evaluation. They also present their methodology as a template for reassessing evolving models.

Limitations and future work

  • Future tests could vary session structure and prompt order, and examine whether one-word response limits [38] or Chain-of-Thought Prompting affect accuracy.The study tested questions separately, whereas VLAT presented all 53 at once; the authors suggest shuffled questions in one session and a smaller trial set for alternative prompting methods.Source: §8 · p. 11
  • The study assessed literacy heuristically following VLAT, but not how LLMs interact with visualizations or handle complex visualizations such as dashboards.The authors propose exploring interaction to expand how LLMs read visualizations.Source: §8 · p. 11
  • Future work could test visualization-only prompts and different anonymization levels, from removing proper nouns and category terms to replacing attributes with “x” and “y”.The authors report that LLMs were affected by pre-existing knowledge in most cases and propose investigating whether these prompt and decontextualization changes alter outputs and visualization literacy.Source: §8 · p. 11
  • Future research could examine LLMs’ autonomous visualization reading, why they succeed or fail under specific conditions, and how to improve their visualization-evaluation capabilities.The authors note that investigating why models succeed or fail is challenging because LLMs are opaque.Source: §8 · p. 11
  • The authors expect their methodology to remain useful as LLMs evolve and urge practitioners to reassess models’ visualization literacy regularly.They propose using the methodology as a template for evaluating future models and investigating how to use them in visualization evaluation.Source: §8 · p. 11

SUPPLEMENTAL MATERIAL POINTERS … A.4 Bootstrapped Results

The supplemental material documents the logistic regression model, its hyperparameter selection, and the bootstrap procedure used to analyze visualization-literacy results, alongside links to study resources.

SUPPLEMENTAL MATERIAL POINTERS

  • The study’s visualization, experimental results, figures, code, and analysis and LLM-interaction scripts were released through GitHub and OSF.The code repository is github.com/VADERASU/llm4viz-experiments, and study materials are at osf.io/wcb5g.Source: SUPPLEMENTAL MATERIAL POINTERS · p. 11

A.1 Variables and Interactions

  • The logistic regression model included 629 variables and interactions, combining visualization, task, LLM, and visualization-presence factors.The reported totals were 24 one-way, 133 two-way, 276 three-way, and 196 four-way variables and interactions.Source: §A.1 · p. 14

A.2 Hyperparameter Tuning

  • The tuning evaluated 117 parameter combinations across ten repetitions of ten-fold cross-validation, yielding 11,700 test-set accuracy scores.Candidates varied penalties, solvers, and regularization values.Source: §A.2 · p. 14, Table 3 · p. 14
  • The no-penalty logistic regression model performed best by AUPRC, while penalized models performed comparably at C ≥10; the default lbfgs solver was selected.AUPRC was the primary selection metric because only about 36.1% of the 25,440 samples were correct.Source: §A.2 · p. 14
  • Because 9,185 of 25,440 samples were correct (≈36.1%), the data were imbalanced and AUPRC was used as the primary model-selection metric.The passage reports that metrics followed a similar convergence pattern across penalties.Source: §A.2 · p. 14

A.3 Bootstrapping

  • The authors fitted 1,000 logistic regression models to bootstrap samples of 25,440 data points drawn with replacement from experiments 1 and 2, storing 629 coefficients per model.They analyzed coefficient distributions to test RQ1.1 and hypotheses H1–H3, and used fitted models to estimate combination probabilities for RQ1.1, RQ1.2, H1, and H4.Source: §A.3 · p. 14, Table 4 · p. 14

A.4.1 Coefficients · A.4.3 Visualization Presence Probability Differences

The analyses report statistically significant coefficients except for six, alongside mixed GPT-4–Gemini performance and visualization-presence effects across visualization/task interactions.

A.4.1 Coefficients

  • Mean visualization coefficients are reported in Table 8, while mean task coefficients are reported in Table 9.Source: §A.4.1 · p. 14
  • All coefficients except six were statistically significant, and Table 5 lists the exceptions.The general coefficients, including LLM and visualization-presence variables and their interactions, were statistically significant.Source: §A.4.1 · p. 14, Table 5 · p. 15
  • The general coefficients are visualized as a ridge plot in Figure 5.The passage identifies the plot as showing bootstrapped general coefficients from the logistic regression.Source: §A.4.1 · p. 14, Figure 5 · p. 15
  • Of 49 visualization/task interactions, 17 favored GPT-4, 25 favored Gemini, and 7 showed no difference when visualizations were present.The largest differences favored GPT-4 for Identify Hierarchical Structure on treemaps (95% CI ≈(1.000,1.000)) and Gemini for Make Comparisons on 100% stacked bar charts (95% CI ≈(−0.8895,−0.7170)).Source: §A.4.1 · p. 15
  • GPT-4’s largest reported advantage occurred for Identify Hierarchical Structure on treemaps, with a 95% CI of approximately (1.000,1.000).Source: §A.4.1 · p. 15
  • Gemini’s largest reported advantage occurred for Make Comparisons on 100% stacked bar charts, with a 95% CI of approximately (−0.8895,−0.7170).Source: §A.4.1 · p. 15

A.4.3 Visualization Presence Probability Differences

  • 18 GPT-4 and 20 Gemini visualization/task interactions showed statistically significant differences between visualization presence and absence.The largest reported differences were for Find Correlation/Trend on line charts for GPT-4 (95% LB ≈0.8161) and Retrieve Value on 100% stacked bar charts for Gemini (95% LB ≈0.5573).Source: §A.4.3 · p. 15

A.5 Time Analysis · B EXPERIMENTS WITHOUT CHOICES

GPT-4 generally took longer than Gemini and had less stable response times under continuous requests; the following section reports detailed methods and results for Experiments 3 and 4.

A.5 Time Analysis

  • GPT-4’s response times were relatively unstable compared with Gemini’s under a continuous series of requests.The comparison covers both conditions, with and without choices.Source: §A.5 · p. 15, Figure 6 · p. 15

B EXPERIMENTS WITHOUT CHOICES

A.5 Time Analysis

  • GPT-4 generally took longer than Gemini in both conditions, with and without choices.Figure 6 shows time consumption for each visualization across conditions over time.Source: Figure 6 · p. 15, §A.5 · p. 15

B.1 Numerical Range Analysis and Results

The section presents four interval-overlap metrics and explains how their usefulness differs: some assess similarity or proportional overlap, while another emphasizes containment.

B.1 Numerical Range Analysis and Results

  • Figure 7 presents four metrics; Percentage Overlap can report 100% even when the response range is far too broad or narrow to be useful.This limitation arises when one range is a subset of the other, despite a substantial difference in range size.Source: §B.1 · p. 15
  • The Jaccard Index [14] measures interval similarity from 0 for no overlap to 1 for identical intervals, but may understate proportional overlap when interval lengths differ widely.Source: §B.1 · p. 15
  • The Sørensen-Dice Coefficient expresses overlap relative to the total size of both intervals, with scores from 0 for no overlap to 1 for identical intervals.It is useful when the balance between shared and total interval extent matters.Source: §B.1 · pp. 15–16
  • The Overlap Coefficient [45] compares the intersection with the smaller interval, making it useful for assessing containment or whether a larger interval fully captures a smaller one.Its value ranges from 0 for no overlap to 1 when the smaller interval is completely contained in the larger one.Source: §B.1 · p. 16

C DETAILS ABOUT EXPERIMENT 5

In a follow-up study using decontextualized visualizations, GPT-4 tended to improve over Experiment 1, while Gemini’s performance differences were not significant.

C DETAILS ABOUT EXPERIMENT 5

Related papers

Loading 2501.16277v1…