PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Have Visualization Literacy? An Evaluation on Modified Visualizations to Test Generalization in Data Interpretation

Jiayi Hong, Christian Seto, Arlen Fan, Ross Maciejewski

arXiv:2501.16277v1cs.PF

TL;DR

The paper addresses whether LLMs possess visualization literacy, which matters for assessing their potential role in visualization evaluation. It proposes an evaluation template and tests GPT-4 and Gemini, finding insufficient literacy, reliance on pre-existing knowledge, and model- and visualization-dependent performance.

  • Problem

    Previous research measured humans’ visualization literacy, but evidence about whether LLMs possess this capability remains limited despite their growing use in visualization tasks.

  • Method

    The paper proposes a template for evaluating LLM visualization literacy and applies it to GPT-4 and Gemini across visualization-related experiments.

  • Results

    GPT-4 and Gemini lack sufficient visualization literacy compared with humans, with performance varying across tasks and visualizations and substantial reliance on pre-existing knowledge.

  • Takeaways & Limitations

    LLMs are not yet prepared to replace humans in visualization evaluation, but their cost efficiency supports cautious experimentation with diverse models and scenarios.

  • Takeaways & Limitations

    The study used one specific testing method, separate sessions per question, and prompts placing questions before visualizations, leaving alternative procedures for future work.

Abstract

from arXiv · show

In this paper, we assess the visualization literacy of two prominent Large Language Models (LLMs): OpenAI's Generative Pretrained Transformers (GPT), the backend of ChatGPT, and Google's Gemini, previously known as Bard, to establish benchmarks for assessing their visualization capabilities. While LLMs have shown promise in generating chart descriptions, captions, and design suggestions, their potential for evaluating visualizations remains under-explored. Collecting data from humans for evaluations has been a bottleneck for visualization research in terms of both time and money, and if LLMs were able to serve, even in some limited role, as evaluators, they could be a significant resource. To investigate the feasibility of using LLMs in the visualization evaluation process, we explore the extent to which LLMs possess visualization literacy -- a crucial factor for their effective utility in the field. We conducted a series of experiments using a modified 53-item Visualization Literacy Assessment Test (VLAT) for GPT-4 and Gemini. Our findings indicate that the LLMs we explored currently fail to achieve the same levels of visualization literacy when compared to data from the general public reported in VLAT, and LLMs heavily relied on their pre-existing knowledge to answer questions instead of utilizing the information provided by the visualization when answering questions.

1 INTRODUCTION

The paper proposes assessing whether LLMs can support visualization evaluation by measuring their visualization literacy against human benchmarks. Using modified VLAT-based studies, it examines performance, interpretation limits, and evaluation costs for GPT-4 and Gemini.

  • Motivation: Human-centered visualization evaluation is costly, motivating investigation of LLMs as possible evaluation assistants.The paper frames LLM support as potentially useful for visualization assessment and readability checks.
  • Study design: The study uses a modified visualization-literacy assessment to evaluate GPT-4 and Gemini across 12 visualizations and 8 visual tasks.Accuracy rate is the primary performance metric.
  • Study design: The experiments examine model performance across visualizations and tasks, reliance on knowledge versus visual information, contextual information, answer choices, and cost.Costs are measured in both time and money and compared with previously tested humans.
  • Findings: Compared with humans, GPT-4 and Gemini lack visualization literacy and rely heavily on pre-existing knowledge rather than information in the visualizations.Their performance varies across specific tasks and visualizations.
  • Contributions: The paper contributes an evaluation methodology, empirical studies, analysis of interpretation limitations, and discussion of future uses for LLM-based visualization evaluation.These contributions address both benchmarking and the practical potential of leveraging LLMs.

2 RELATED WORK

Prior work establishes visualization literacy and chart question answering as relevant foundations, while leaving unresolved whether off-the-shelf LLMs genuinely interpret visualizations rather than reproduce learned answers. This paper targets that gap with a comprehensive evaluation of LLM visualization literacy.

  • Visualization Literacy: Visualization literacy concerns reading and interpreting visual representations and supports understanding unfamiliar charting techniques.Its importance has grown with the widespread adoption of visualization.
  • Visualization Literacy: The VLAT measures visualization literacy with 53 multiple-choice questions designed for non-expert users.Prior work used quantitative evaluation methods, including item response theory.
  • Visualization Literacy: The paper addresses a gap in evidence about whether LLMs possess visualization literacy, defining it as the ability to read, understand, and interpret information from visualizations.The gap matters because LLM use in visualization-related tasks is increasing.
  • Chart Question Answering: Chart question answering research develops datasets and models for answering questions about graphical data, including bar charts and relational plot reasoning.These efforts underscore the need for models to answer visualization-related questions correctly.
  • LLMs and Visualization: Existing LLM visualization studies emphasize creation and enhancement, whereas evaluation of the quality of those outputs remains less developed.This work contrasts generation capabilities with evaluation needs.
  • Research Gap: Earlier LLM visualization-literacy studies reused VLAT materials without accounting for training-data knowledge, answer choices, or option order, potentially inflating performance.The present work instead aims to determine whether off-the-shelf LLMs possess visualization literacy.

3 PILOT/EXPLORATORY EXPERIMENTS

Pilot experiments tested whether LLMs could extract chart data and generate visualization code, revealing unreliable direct extraction and motivating evaluation with PNG visualizations stripped of value labels.

  • Exploratory setup: The exploratory work examined LLM capabilities from two perspectives: extracting data from visualizations and generating visualization code.These pilots preceded the systematic visualization-literacy investigation.
  • Data extraction: On simple bar charts without value annotations, models consistently made data-extraction errors across all five attempts, regardless of grid lines.They failed to extract precise data from chart geometry.
  • Data extraction: Models relied heavily on data labels even when labels conflicted with the plotted values.They interpreted SVG files more accurately than PNG files, but PNG was retained as the more common format.
  • Design decision: The pilot findings led the authors to test direct reading of PNG visualizations rather than text-transformed charts.Data value labels were removed to reduce reliance on those labels.

4 ASSESSING LLMS’ VISUALIZATION LITERACY

The study evaluated GPT-4 and Gemini with modified VLAT visualizations and found that both fell below human visualization-literacy levels. Their performance varied by visualization and task, while responses often reflected prior knowledge rather than information extracted from charts.

  • 4.1 Study Design: GPT-4 and Gemini were evaluated with modified VLAT charts, comparing their performance with general-public results.The study used multimodal models and reproduced VLAT charts with randomized values to reduce training-data exposure.
  • 4.1 Study Design: GPT-4’s prior answers matched VLAT responses, motivating randomized chart values and withheld answers in the modified assessment.The authors interpreted this pattern as potential exposure to VLAT questions and answers or related ground-truth data.
  • 4.3 Overall Literacy: GPT-4 and Gemini did not reach visualization-literacy levels comparable to humans.The comparison used general-public performance reported in the original VLAT as the human baseline.
  • 4.3 Overall Literacy: GPT-4 performed better than random on 25 of 53 questions, while Gemini exceeded random on 24.The authors attributed this pattern partly to reliance on prior knowledge that could conflict with the displayed visualization.
  • 4.3 Regression Analysis: Visualization presence produced a joint mean coefficient of approximately -0.5051, indicating poor performance when visualizations were present.The regression analysis used bootstrapping and examined visualization, task, model, and interaction effects.
  • 4.3 Regression Analysis: Only 18 of 49 visualization/task interactions were significant for GPT-4 and 20 for Gemini, supporting reliance on knowledge beyond chart information.Most visualization/task interactions were not significant for either model, so the authors did not support H4.

5 FURTHER ANALYSIS OF LLMS IN VISUALIZATION UNDER-

Further experiments examined answer-choice constraints, numerical responses, and contextualization. Answer choices generally improved accuracy, while decontextualized visualizations improved GPT-4’s performance and modestly changed Gemini’s.

  • 5 Further Analysis: Experiments 3–5 tested whether answer choices and visualization context affected LLM performance.The experiments reused the study’s visualizations and questions, with choice-free tests and a decontextualized-visualization follow-up.
  • 5.3 Numerical Responses: For single-value questions, GPT-4 improved with visualizations while Gemini worsened, suggesting greater reliance on prior knowledge for Gemini.The analysis covered 4,614 valid single-value responses and used relative error as the accuracy measure.
  • 5.3 Choice-Free Performance: Answer choices appeared to guide LLM responses toward expected answers rather than constrain their performance.The authors noted that choices may improve accuracy but require additional design costs.
  • 5.4 Contextualization: GPT-4’s average accuracy rose from 30.90% with contextualized visualizations to 41.96% with decontextualized visualizations.The authors cautioned that the limited number of task types prevented statistical conclusions for some comparisons.
  • 5.4 Contextualization: Gemini’s average accuracy changed from 38.98% with contextualized visualizations to about 43.13% with decontextualized visualizations, without significant differences across visualization types.The authors concluded that contextualized visualizations appeared to increase reliance on pre-existing knowledge, while calling for further research.

6 COSTS DIFFERENCES BETWEEN LLMS AND HUMANS

The paper compares the time and monetary costs of LLM- and human-based visualization assessment. LLMs were faster and potentially cost-effective, but their lower visualization literacy creates a trade-off for evaluation use.

  • Cost Comparison: GPT-4 cost approximately $0.01 per visualization question and $0.001 per question without visualization, while Gemini was free through its API at the time.The formal study cost approximately $160 excluding pilot and testing activities.
  • Cost Comparison: LLMs were significantly faster than humans, with Gemini notably faster than GPT-4.The timing comparison used separate sessions for each question, unlike the one-session structure required by the human VLAT procedure.
  • Implications: LLMs may be cost-effective and time-efficient for visualization interpretation, but limited visualization literacy creates a trade-off.The authors state that current LLMs cannot replace humans based on their results.

7 DISCUSSION

The evaluated LLMs are not yet ready to replace humans in visualization evaluation, but their cost efficiency and task-dependent strengths support cautious, model-specific experimentation. Their reliance on factual prior knowledge and safety-oriented responses can reduce the usefulness of visualization-based answers.

  • Implications: Both LLMs currently lack sufficient visualization literacy and should not replace humans in visualization evaluation.The paper recommends experimentation with diverse models rather than replacement of human evaluators.
  • Implications: GPT-4 performed relatively well on scatterplot correlation trends and comparisons, whereas both models were unreliable with pie charts and histograms.The authors recommend testing models on specific visualization types before adopting them for initial evaluation.
  • Interpretation: LLMs’ factual prior knowledge can override visualization evidence when questions concern real-world facts, especially without answer choices.This may cause models to answer from learned knowledge rather than demonstrate the visualization’s intended patterns.
  • Implications: LLMs may still help designers reflect on data sources by assessing data for potential fraud.This proposed use treats the models as reminders rather than independent visualization evaluators.
  • Implications: More data across visualization types and tasks is needed before the paper’s model-specific recommendations can be made more precise.The authors frame current recommendations as requiring further verification.
  • Interpretation: Overalignment can produce broad or conservative answers that preserve safety at the expense of specificity and usefulness.The paper illustrates this behavior with range questions where models may cover the full y-axis or extend beyond it.

8 LIMITATIONS AND FUTURE WORK

The authors identify methodological choices that may affect measured LLM visualization literacy and outline future tests of prompting, anonymization, autonomous reading, and evaluation strategies. They also present the methodology as a template for monitoring future models.

  • Limitations: Using separate sessions, unlike VLAT’s single 53-question session, may produce different LLM performance estimates.Future work could have LLMs answer all shuffled questions in one session.
  • Limitations: Restricting open-ended answers to one word may limit LLM performance and leaves alternative prompting methods for future study.The authors specifically mention smaller trial sets and chain-of-thought prompting as possible alternatives.
  • Future Work: Prompt order may affect results because the experiments placed questions before visualizations.The authors propose testing the reverse order.
  • Future Work: Different levels of decontextualization and anonymization may influence visualization literacy in distinct ways.Proposed manipulations include removing category terms and replacing data attributes with generic labels such as x and y.
  • Future Work: The paper assesses visualization literacy with a heuristic approach following VLAT, leaving autonomous visual reading and broader evaluation strategies for future work.The authors also seek to investigate why models succeed or fail under particular conditions.
  • Future Work: The methodology could serve as a template for evaluating and regularly monitoring future models’ visualization literacy.This proposed use does not imply that current models already meet the required standard.

SUPPLEMENTAL MATERIAL POINTERS

The supplemental materials document the logistic-regression design, hyperparameter selection, bootstrap analysis, and released implementation resources.

  • Released materials: The authors provide code, visualization assets, experimental results, and manuscript figures through GitHub and OSF repositories.The released materials include scripts for interacting with LLMs and analyzing experimental results.
  • Variables and interactions: The regression models 629 variables and interactions spanning visualization types, task types, LLMs, visualization presence, and their interactions.The model includes 24 one-way, 133 two-way, 276 three-way, and 196 four-way terms.
  • Hyperparameter tuning: Hyperparameter tuning tested penalties, solvers, and regularization values using repeated ten-fold cross-validation and accuracy scores.The tested combinations produced 11,700 recorded test-set accuracy scores.
  • Model selection: The final model was selected primarily by Average Precision because only approximately 36.1% of the 25,440 samples were correct.AUPRC was prioritized because it is robust to data imbalance.
  • Bootstrapping: Bootstrapping sampled the experiment data with replacement and refit the logistic model 1,000 times to analyze coefficient distributions and combination probabilities.Each bootstrap model stored 629 coefficients and supported hypothesis testing.

A.4.1 Coefficients

The coefficient analysis finds broad statistical significance across the fitted model and reports mixed GPT-4–Gemini differences across visualization/task interactions.

  • Coefficient results: All but six logistic-regression coefficients were statistically significant, while general coefficients involving LLM and visualization presence were significant.General coefficients are visualized as a ridge plot, with mean visualization and task coefficients reported separately.
  • Model comparison: Across 49 visualization/task interactions, GPT-4 outperformed Gemini in 17, Gemini performed better in 25, and 7 showed no difference.The comparison used two-sided tests of significant GPT-4–Gemini differences when the visualization was present.
  • Visualization presence: Visualization-presence effects were statistically significant in 18 of 49 interactions for GPT-4 and 20 for Gemini.The reported tests compared conditions with visualization presence and absence.

A.5 Time Analysis

The time analysis compares response durations across the two LLMs with and without answer choices. GPT-4 generally responded more slowly and less consistently than Gemini.

  • Response time: GPT-4 generally took longer than Gemini both with and without answer choices.The comparison covers the response-time conditions shown in the time-consumption figure.
  • Response-time stability: GPT-4 exhibited relatively unstable response times during continuous request sequences compared with Gemini.The reported instability was observed in both answer-choice conditions.

B.1 Numerical Range Analysis and Results

The numerical range analysis uses four overlap metrics to compare correct and predicted intervals, while noting that individual metrics can misrepresent usefulness when interval lengths differ substantially.

  • Metrics: Four metrics—Percentage Overlap, Jaccard Index, Sørensen-Dice Coefficient, and Overlap Coefficient—quantify overlap between correct and predicted ranges.The metrics are commutative, so exchanging the two intervals does not change their values.
  • Percentage Overlap: Percentage Overlap can report 100% when one interval contains the other, even if their sizes differ dramatically.For example, a predicted range of 0–1000 can receive the same percentage-overlap result as a much narrower contained range.
  • Jaccard Index: The Jaccard Index measures overall interval similarity from 0 for no overlap to 1 for identical intervals but may underrepresent proportionality when lengths vary widely.Its example compares intervals spanning 1–4 and 3–6.
  • Sørensen-Dice Coefficient: The Sørensen-Dice Coefficient expresses overlap relative to the combined size of both intervals, ranging from 0 for no overlap to 1 for identical intervals.The metric emphasizes the shared portion relative to both intervals’ total extent.
  • Overlap Coefficient: The Overlap Coefficient measures containment by dividing intersection size by the smaller interval’s size, ranging from 0 to 1.A value of 1 indicates that the smaller interval is completely contained within the larger one.

C DETAILS ABOUT EXPERIMENT 5

The follow-up experiment compared GPT-4 and Gemini on anonymized visualizations against Experiment 1, using accuracy-rate comparisons and multiple overlap metrics. GPT-4 tended to improve with decontextualized visualizations, whereas Gemini’s differences were not significant.

  • Experiment 5: Decontextualized visualizations were tested in a follow-up experiment and compared qualitatively with Experiment 1 results.The follow-up used anonymized visualizations shown in Figure 8.
  • Experiment 5: GPT-4 tended to improve when visualizations were decontextualized, while Gemini showed no significant differences.This comparison concerns the follow-up experiment relative to Experiment 1.
  • Experiment 5: The overlap analysis used Jaccard Index, Sorensen Dice Coefficient, Percentage Overlap, and Overlap Coefficient measures across LLM and visualization settings.These metrics were presented as distributions or comparisons across settings.
  • Experiment 5: Table 6 color-encodes follow-up reading accuracy as much better, close to, or much worse than Experiment 1 using a 0.05 threshold.Green denotes more than 0.05 higher performance, yellow denotes values within 0.05, and red denotes much worse performance.
Loading 2501.16277v1…