PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting and Understanding Ambiguity in Visualizations with Role-Playing Vision Language Model Interpretations

Garrick Chan, Yun-Hsin Kuo, Sam Yu-Te Lee, Yinuo Tang, Kwan-Liu Ma

arXiv:2610.33997v1cs.HC

TL;DR

Visualizations can support divergent chart-grounded takeaways, yet eliciting audience interpretations is costly and difficult to scale. The paper uses persona-conditioned VLMs to generate and compare candidate interpretations, finding that the resulting profiles can surface potential ambiguity for authors to inspect.

  • Problem

    Visualizations may support divergent chart-grounded takeaways, while evaluating audience interpretations can be costly and difficult to scale.

    Source: §1 · p. 1

  • Method

    The pipeline uses relevant personas and chart-type-routed VLMs to generate contextual takeaways with supporting Target–Pattern–Scope statements, then labels statement pairs and summarizes their distribution.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    Case studies show the pipeline can identify opposing views from synthetic personas with competing interests and areas that visualization authors and distributors should handle with more care.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Ambiguity profiles make interpretive variation an evaluation target and offer a way to make differences actionable during visualization authoring.

    Source: §7 · p. 8

    VLM chart-reading and statement-labeling errors remain, and applicability beyond evaluated chart types is limited by datasets weighted toward bar and line charts.

    Source: §7.6 · p. 9

Abstract

from arXiv · show

Communicative visualizations can support multiple chart-grounded takeaways, including divergent or conflicting ones; we refer to this variation as interpretive ambiguity. Such ambiguity can undermine a visualization's communicative goal when the supported takeaways diverge from its intended message. Evaluating visualizations with target audiences can surface this variation, but such evaluations can be costly and difficult to scale. In this work, we explore persona-conditioned Vision Language Models (VLMs) as a complementary evaluation tool for interpretive ambiguity. For a given visualization, our pipeline uses topic-relevant personas to elicit candidate contextual takeaways supported by interpretations of perceived visual patterns, such as trends and clusters. We derive a Target--Pattern--Scope framework for comparing these interpretations, use LLM judges to assign Compatible, Disjoint, or Opposing ambiguity labels to statement pairs, and summarize their distribution in an ambiguity profile. To improve chart grounding, we benchmark candidate VLMs on chart comprehension and select the highest-performing model for each chart type. We evaluate statement-pair labeling against human-authored annotations and apply the approach to real-world, public-facing visualizations curated from established datasets. Through case studies, we examine how perceived visual patterns relate to contextual takeaways and how a visualization redesign changes the sampled interpretations. Our findings show how persona-conditioned VLMs can complement human feedback by generating candidate contextual takeaways and surfacing potential ambiguity for further inspection.

1 INTRODUCTION

The introduction frames interpretive ambiguity as variation in chart-grounded takeaways that can diverge from a visualization’s intended message. It presents persona-conditioned VLMs as a scalable complement to audience evaluation, with a pipeline and Target–Pattern–Scope framework for surfacing candidate interpretations.

1 INTRODUCTION

  • Interpretive ambiguity is variation among chart-grounded takeaways, and divergence from an intended message may undermine a visualization’s communicative goal.Different viewers may attend to different data, describe a pattern at different scopes, or frame a comparison in opposing ways.Source: §1 · p. 1
  • Audience studies can reveal interpretive variation, but repeated evaluations are costly and may not span a broad range of relevant perspectives.The authors position persona-conditioned VLMs as complementary diagnostic probes, not substitutes for human participants or predictors of actual audience interpretation.Source: §1 · p. 1
  • The pipeline benchmarks candidate VLMs by chart type, elicits persona-conditioned contextual takeaways and supporting interpretations, then summarizes pairwise ambiguity labels for author inspection.It does not make a definitive binary judgment that a visualization is or is not ambiguous.Source: §1 · p. 1
  • The paper contributes a Target–Pattern–Scope framework, a persona-conditioned VLM pipeline for ambiguity profiles, and case studies on real-world visualizations.The framework compares the target described, the claimed pattern, and the scope over which the claim applies; pairs are labeled Compatible, Disjoint, or Opposing.Source: §1 · p. 1

2 RELATED WORK

The related work connects viewer-dependent interpretation, VLM chart comprehension, and persona-conditioned simulation. Together, these strands motivate using simulated personas to probe possible chart interpretations while distinguishing such probes from representative human responses.

2.1 Interpretive Ambiguity in Visualization

  • Interpretive ambiguity concerns variation in chart-grounded takeaways, distinct from unclear visual mappings or misleading design choices.Misleadingness concerns steering viewers toward an improper interpretation; ambiguity concerns whether a chart supports multiple takeaways.Source: §2.1 · pp. 1–2
  • Prior work establishes varied viewer-dependent interpretations and the challenge of eliciting and comparing them systematically at scale.Interpretations may depend on chart design and viewer characteristics, while ambiguity can reinforce existing beliefs.Source: §2.1 · p. 2

2.2 VLM Chart Comprehension

  • VLM chart-comprehension evidence is mixed, and structured reasoning prompts can improve accuracy on some chart-question tasks.Prior evaluations also report that VLMs can fall short of human performance on VLAT, while generated takeaways may be less diverse than human takeaways for bar charts.Source: §2.2 · p. 2

2.3 Persona-Conditioned Simulation for Design Evaluation

  • Persona-conditioned simulations have been used to generate candidate perspectives for design exploration and evaluation.Examples include feedback from reader personas, poster-design agents, and aggregated interaction traces for identifying usability issues.Source: §2.3 · p. 2
  • Persona-conditioned outputs are diagnostic probes, not representative human responses or estimates of population prevalence.Their usefulness may depend on whether the prompted persona variables explain human variation, and AI surrogates do not provide actual participation or lived experience.Source: §2.3 · p. 2

3 PROBLEM FORMULATION AND METHODOLOGY

The methodology treats ambiguity as variation in chart-grounded claims elicited for the same interpretive question, then compares statements across personas. It samples contextual takeaways and supporting pattern statements, decomposes them into Target, Pattern, and Scope, and summarizes pairwise labels in question-specific profiles.

3.1 Operationalizing Interpretive Ambiguity

  • Interpretive ambiguity is the potential for substantively different claims about visible patterns in response to the same interpretive question.The analysis focuses on L3 chart-grounded pattern statements; L4 takeaways may contain multiple supporting L3 statements, and pairs compare different personas answering the same chart question.Source: §3.1 · p. 2
  • Ambiguity profiles describe semantic relationships between statements, not factual correctness, prevalence, or a definitive judgment that a chart is ambiguous.Chart comprehension and grounding are prerequisites for interpreting the labels.Source: §3.1 · p. 2
  • The label vocabulary consolidates Equivalent and Entailment as Compatible, uses Disjoint for different jointly acceptable aspects, and Opposing for conflicting readings.The initial vocabulary adapted Natural Language Inference relations, renaming Neutral as Disjoint and broadening Contradiction to Opposing.Source: §3.1 · p. 2

3.3 Ambiguity Profiling Pipeline

  • The four-stage pipeline profiles the chart, selects personas, generates chart interpretations, and labels statement pairs to form question-specific ambiguity profiles.Chart profiling occurs once per chart; persona selection uses its topic, and labeling compares resulting L3 statements within each question.Source: §3 · p. 2, Figure 1 · p. 3, §3.3 · p. 3

3.2 Persona-based VLM Takeaway Sampling

  • Role-playing VLMs probe plausible interpretive variation because human elicitation is costly and difficult to scale; the sampled personas are not population estimates.The fixed PersonaHub pool contains 194,728 filtered descriptions combining roles with interests, concerns, goals, or attitudes.Source: §3.2 · p. 3

3.3.1 Chart Profiling

  • Chart profiling extracts a chart description and topic, selects tasks from visible structure, and generates shared open-ended questions that allow multiple chart-grounded responses.The same question is posed to each persona to control the subsequent comparison.Source: §3.3.1 · p. 3

3.3.2 Persona Selection

  • Persona selection filters for relevance, then clusters candidates and chooses cluster medoids to represent distinct regions of the relevant pool.The smaller of two estimates of stable groups governs selection; embedding-space diversity does not guarantee demographic or ideological coverage, so personas are not an audience sample.Source: §3.3.2 · p. 3

3.3.3 Chart Interpretation

  • For each persona and question, a VLM generates a contextual L4 takeaway and identifies visible observations and supporting L3 statements, decomposed into Target, Pattern, and Scope.The persona conditions which visible patterns are foregrounded, while the chart remains the required evidential basis.Source: §3.3.3 · p. 3

4 DATASET CURATION AND VLM SELECTION

The authors curate real-world charts and chart-question pairs, benchmark general-purpose VLMs for chart comprehension, and route each chart type to its strongest candidate. Results show that the overall leader is not the best model for every chart type.

4 DATASET CURATION AND VLM SELECTION

  • The study assembled 783 unique charts and 689 chart-question pairs to develop the pipeline and select VLMs.The chart corpus was intended to include real-world, public-facing charts spanning VLAT chart types.Source: §4 · p. 4, §4.1 · p. 4

4.1 Dataset Curation

  • The curated corpus draws primarily on ChartQAPro, with sparse pie and scatterplot categories supplemented from ChartQA and ChartMuseum.The authors did not verify every chart against design guidelines, so the corpus is not a certified set of well-designed visualizations.Source: §4.1 · p. 4

4.2 VLM Selection

  • Because chart-comprehension performance can vary by chart type, the pipeline routes each type to its highest-scoring benchmark model to reduce chart-reading errors as a source of takeaway variation.For chart types absent from the benchmark, it defaults to Gemini 3.6 Flash, the model with the highest overall accuracy.Source: §4.2 · p. 4, §4.2.3 · p. 4

4.2.1 Candidate Models

  • The benchmark evaluated nine proprietary GPT, Gemini, and Claude VLMs, selected to span provider families and fast, balanced, and performance tiers.Provider-described tiers organize the candidate set rather than define analytical categories.Source: §4.2.1 · p. 4

4.2.2 Benchmark Procedure

  • All nine models answered the same 689 chart-question pairs, with anonymized responses scored semantically against reference answers using specified numerical tolerances.Gemini 3.5 Flash-Lite judged responses; failed calls and unintelligible or malformed answers counted as incorrect.Source: §4.2.2 · p. 4

4.2.3 Results and Model Routing

  • 83.16% overall accuracy: Gemini 3.6 Flash led the benchmark and the three largest chart categories, while other models led on the remaining five types.The results motivate chart-type-specific routing rather than using the overall winner for every chart.Source: §4.2.3 · p. 4
  • Higher provider-described tiers did not consistently improve accuracy, and no single model performed best across all chart types.Gemini 3.6 Flash led line, bar, and stacked bar; other models led on the remaining five types.Source: §4.2.3 · p. 4

5 THE AMBIGUITY FRAMEWORK

The section refines an initial ambiguity taxonomy through human review, defines Target, Pattern, and Scope (TPS) for comparing chart-grounded statements, and operationalizes the resulting labels for ambiguity profiling.

5.1 Qualitative Framework-Derivation Procedure

  • Three authors independently labeled 115 sampled statement pairs, discussed every non-unanimous judgment, and reached agreement on labels and component assignments.The sample came from eight charts and was selected for roughly balanced coverage across chart types and initial labels.Source: §5.1 · p. 5

5.2 Target, Pattern, Scope

  • TPS represents a Pattern asserted of one or more Targets within a Scope, with components that can be implicit, compound, or expressed at different specificity levels.Across 115 reviewed pairs, the authors found no recurring labeling distinction that TPS could not capture, while limiting this claim to chart-grounded statements in their analysis.Source: §5.2 · p. 5

5.3 Ambiguity Taxonomy

  • The final taxonomy groups equivalent and entailing readings as Compatible, reserves Disjoint for distinct nonconflicting readings, and assigns Opposing to contradictions or conflicting impressions.Opposing takes precedence when statements contradict or create conflicting impressions.Source: §5.3 · p. 5, Table 3 · p. 6

5.3.1 Compatible

  • Compatible applies when statements share a reading or one entails the other without conflict, including equivalent claims and consistent one-way specificity.The illustrated age-range example is Compatible when one statement's scope includes the other's.Source: §5.3.1 · p. 6, Figure 2 · p. 6

5.3.2 Disjoint

  • Disjoint applies when TPS differences prevent entailment without creating conflict, such as describing a nonconflicting extremum and trend.The example shares a Target and overlapping Scopes but separates the statements by Pattern.Source: §5.3.2 · p. 6

5.3.3 Opposing

  • Opposing covers direct contradictions and factually sustainable statements whose framing or emphasis creates conflicting overall impressions.In the illustrated example, statements place a change in direction at incompatible points; another contrasts overall distributions with peak locations.Source: §5.3.3 · p. 6, Figure 2 · p. 6

5.4 Operationalizing and Evaluating the Taxonomy

  • The pipeline identifies TPS components for each statement, compares every within-question, between-persona pair, and uses a holistic council to assign top-level labels.The final council combines balanced-tier models from three provider families.Source: §5.4 · pp. 6–7
  • The holistic labeling strategy achieved higher exact agreement than the gated decision tree on both development and evaluation partitions, so the authors adopted it.Separately, the final council classified 101 of 115 human-labeled pairs correctly (87.8%); two pairs were Divided.Source: Table 4 · p. 7, §5.4 · p. 7

6 CASE STUDIES

Across 24 public-facing visualizations, the pipeline found that most statement pairs were Disjoint. Case studies show that L3 pattern agreement does not determine whether personas share contextual takeaways, and that redesign effects depend on the question and communicative goal being evaluated.

6 CASE STUDIES

  • Across 11,083 statement pairs, 69.1% were Disjoint, 26.6% Compatible, 3.7% Opposing, and 71 Divided.Among Opposing pairs, 178 (43.2%) were factual conflicts and 234 (56.8%) were framing conflicts.Source: §6 · p. 7

6.1 Case I. From Perceived Patterns to Takeaways

  • Compatible #BlackLivesMatter L3 pairs shared a decline but supported different takeaways, while Disjoint police-violence pairs still converged on concern about racial disparities.The cases show that L3 statement labels alone do not determine agreement among L4 takeaways.Source: §6.1 · p. 7

6.2 Case II. Redesigning a Communicative Visualization

  • The shift followed adding a question about the 8% promise; Opposing readings persisted, so the profile remained meaningful relative to the question and communicative goal.The findings frame interpretive variation as a design objective and ambiguity profiles as inspectable feedback.Source: §6.2 · pp. 7–8
  • With a benchmark-focused question, redesign raised Compatible labels from 27 of 109 (24.8%) to 37 of 84 (44.0%) and lowered Disjoint labels from 78 of 109 (71.6%) to 44 of 84 (52.4%).The redesign more often anchored L3 statements to the 8% benchmark; the original and redesigned charts used the same data, personas, and question.Source: §6.2 · pp. 7–8, Figure 3 · p. 8

7 DISCUSSION

The discussion treats interpretive variation as a design objective whose desired range depends on communicative intent, and presents ambiguity profiles as structured feedback for authoring and evaluation. It also identifies limits in model reliability, persona sampling, validation, and evaluation scope.

7 DISCUSSION

  • Interpretive variation is an evaluation target in its own right, and its profile can make differences actionable and help assess coverage of possible perspectives.The discussion also frames this as a way to incorporate communicative concerns into agentic visualization workflows.Source: §7 · p. 8

7.1 Interpretive Range as a Visualization Design Objective

  • Interpretive range may be concentrated around an intended message for explanatory charts or broadened for exploratory charts; Disjoint pairs reveal differing attention, while Opposing pairs reveal competing characterizations.Compatible L3 statements can still support divergent L4 takeaways, and Disjoint statements can converge on a shared contextual concern.Source: §7.1 · p. 8

7.2 Ambiguity Profiles Are Conditional on Evaluation Goals

  • An ambiguity profile depends on the interpretive question and communicative goal; comparing its takeaways with a stated objective can help identify which patterns or scopes drive divergence.A design tool could use sampled interpretations to show whether revisions narrow, broaden, or redirect the profile, complementing audience studies.Source: §7.2 · p. 9

7.3 Ambiguity Profiles as Inspectable Design Feedback

  • Compatible, Disjoint, and Opposing labels summarize the kind of variation, while Target–Pattern–Scope locates it and lets authors inspect representative statement pairs and takeaways.Disjoint Targets or Scopes may prompt changes to titles, annotations, or visual emphasis; Opposing Patterns may prompt inspection of intervals, baselines, scales, or framing.Source: §7.3 · p. 9

7.4 From Synthetic Users to Perspective Sampling

  • Persona-conditioned simulation expands the interpretation space rather than estimating real-viewer prevalence: topic-based retrieval and diversity sampling surface distinct readings, while persona attributes can affect coverage.Validation of representation and correspondence with human audiences remains an important direction.Source: §7.4 · p. 9

7.5 Ambiguity Evaluation in Agentic Workflows

  • The framework offers agentic workflows an evaluation representation that returns contextual takeaways, supporting statements, labels, and Target–Pattern–Scope explanations, enabling a possible revision-and-reevaluation feedback loop.Authors specify the intended message and decide which differences matter.Source: §7.5 · p. 9

7.6 Limitations

  • VLM chart-reading and statement-labeling errors remain, applicability beyond evaluated chart types is limited, and prompting or stochastic model behavior may not reflect human reasoning.Visualization-literacy benchmarks and chart datasets are heavily weighted toward bar and line charts.Source: §7.6 · p. 9
  • PersonaHub’s semantic coverage can omit people interested in a topic for non-semantic reasons and thereby bias profiles; the 115 human-labeled pairs are limited and reference-label confidence may vary.The scope also excludes animated or interactive visualizations, publication context, and charts requiring external knowledge; pair counts scale quadratically with personas and statements, making large-scale testing costly.Source: §7.6 · p. 9

8 CONCLUSION

The conclusion presents a model-agnostic pipeline that samples persona-conditioned takeaways and uses Target–Pattern–Scope to profile interpretive variation. Case studies show it can surface opposing synthetic-persona views and areas requiring care, while authors retain responsibility for consequential judgments and audience engagement.

8 CONCLUSION

  • The pipeline constructs ambiguity profiles by labeling pairs of persona-generated L3 statements supporting L4 takeaways as Compatible, Disjoint, or Opposing.Case studies involving controversial-topic charts show opposing views from synthetic personas with competing interests and identify areas for authors and distributors to handle with more care.Source: §8 · p. 9
  • In data-communication workflows, evaluation can inform iteration, while authors retain responsibility for consequential judgments and audience engagement.Source: §8 · p. 10

A AMBIGUITY-PROFILE PIPELINE IMPLEMENTATION DETAILS

The implementation supplement records pipeline choices and defines boundaries on what ambiguity profiles compare and establish. It also describes construction of the PersonaHub persona pool.

A AMBIGUITY-PROFILE PIPELINE IMPLEMENTATION DETAILS

  • The supplement documents implementation choices and parameter settings for the four-stage pipeline; chart profiling and interpretation use chart images, while persona selection and pair labeling use text.Model calls are stateless and independent.Source: A AMBIGUITY-PROFILE PIPELINE… · p. 13

A.1 Analytical Boundaries

  • Profiles compare visible-evidence L3 statements from the same chart and interpretive question; pair labels describe semantic relations, not factual correctness, and profiles diagnose potential variation rather than prevalence or definitive ambiguity.Source: §A.1 · p. 13

A.2 PersonaHub Pool Construction

  • PersonaHub’s 200,000 descriptions were filtered to remove 1,462 likely non-English entries and 3,810 non-human entities, leaving 194,728 personas.Source: §A.2 · p. 13

A.3 Chart Profiling

The pipeline profiles each chart, retrieves and diversifies relevant personas, and elicits chart-grounded interpretations for comparison. It then pairs statements responding to the same question and classifies their relationships using the Target, Pattern, and Scope taxonomy.

A.3 Chart Profiling

  • Chart profiling records chart metadata and topic, then selects visible-structure-supported tasks and generates one open-ended question per task for all personas.The six tasks cover distributions, clusters, correlations, large and small trends, and anomalies.Source: §A.3 · p. 13

A.4 Persona Retrieval and Threshold Selection

  • Personas are ranked by topic similarity, and a chart-specific LLM-judged threshold selects those at or above it.The binary search assumes relevance increases monotonically with similarity; that assumption is not separately tested.Source: §A.4 · p. 13

A.5 Persona Clustering

  • Persona clustering defines knatural as groups remaining after an average-linkage dendrogram cut at median pairwise cosine distance, and kmax as the largest reproducible k-means count, assessed using 20 subsamples of 80% of Pselected with a 0.6 ARI threshold.Source: §A.5 · pp. 13–14

A.6 Chart Interpretation

  • For each question-persona combination, the VLM produces a contextual takeaway first, then supporting chart observations and pattern statements.Pilot experiments found that this order yielded more question-responsive statements that were less likely to include information not visible in the chart.Source: §A.6 · p. 14

A.7 Statement-Pair Construction and Scaling

  • Statement pairs are constructed across personas only within the same question, reducing irrelevant pairings; call counts scale quadratically with personas and average statements per persona.Source: §A.7 · p. 14

A.8 Additional Ambiguity-Taxonomy Examples

  • Taxonomy examples distinguish equivalent statements, directional entailment, complementary distribution parts, and mixed Target–Pattern–Scope specificity.Directional entailment can make a pair Compatible even when one statement has narrower scope; complementary parts need not entail or conflict.Source: §A.8 · p. 14

B.1 Dataset Selection and Curation

The curated corpus contains real-world charts selected for conventional structures and chart-comprehension questions, with scope restricted to specified chart types. The benchmark also compares nine proprietary VLMs across provider-described tiers.

B.1 Dataset Selection and Curation

  • Chart selection prioritized public-facing, conventional chart structures and excluded synthetic-data datasets, infographics, dashboards, and other multi-chart visualizations.Charts were not all screened against established design guidelines, so the corpus is not a verified collection of well-designed visualizations.Source: §B.1 · p. 14
  • The final corpus contains 783 unique charts, including 730 retained ChartQAPro charts and 53 supplemental charts.The chart–question benchmark uses 689 pairs across eight chart types after excluding conversational, hypothetical, and unanswerable entries.Source: §B.1 · pp. 14–15

B.2 Candidate Model Coverage

  • Nine proprietary VLMs were purposively selected, with one model from each of three families at each of three provider-described tiers.The benchmark ran in August 2026, and reported costs use token counts and provider prices from the evaluation period.Source: §B.2 · p. 15

B.3 Benchmark Procedure

Each candidate VLM answered retained chart–question pairs using a chart-comprehension prompt, and a semantic LLM judge assessed correctness against reference answers. The pipeline routes interpretation to the best-performing model for each chart type, with specified tie and missing-type rules.

B.3 Benchmark Procedure

  • Candidate models received every retained chart–question pair with a chart-values extraction, sorting, verification, and answering prompt.Calls were retried up to four times; after five unsuccessful attempts, an empty answer was recorded.Source: §B.3 · p. 15
  • Gemini 3.5 Flash-Lite judged anonymized responses against ChartQAPro references, and overall accuracy was computed across all 689 pairs.Numerical answers allowed 5% tolerance, or one year for year questions; failed calls and unintelligible responses counted as incorrect.Source: §B.3 · p. 15
  • For each chart type, the pipeline selected the model with highest type-specific accuracy, breaking ties by overall accuracy; unbenchmarked types defaulted to Gemini 3.6 Flash.Routing applies during chart interpretation, while chart profiling uses a fixed model.Source: §B.3 · p. 15
Loading 2610.33997v1…