PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture

Cody Kommers, Mingrui Ye, Evelyn Gius, Daniela Mihai, Hoyt Long, Zheng Yuan, Drew Hemment

arXiv:2609.12575v1cs.CLcs.AIcs.HC

TL;DR

Ambiguity can be valuable in communication and culture, yet AI evaluation often treats it as something to resolve. This paper uses a Dixit-based task and coding rubric to compare human and model clues, finding that models are less calibrated and rarely use culturally situated references.

  • Problem

    AI research often evaluates ambiguity as something to remove, despite its role in communication and culture and the need to account for multiple legitimate interpretations.

    Source: §1 · pp. 1–2

  • Method

    The paper compares human and multimodal-model Dixit clues using a novel rubric for calibrated ambiguity, with models generating clues on shared boards.

    Source: §1 · p. 2, §1.1 · p. 3, §7 · p. 11

  • Results

    Models are less well calibrated than humans and almost never use culturally situated references; prompting helps to a degree but does not fully prevent ambiguity collapse.

    Source: §1.1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    Appropriate ambiguity depends on context, and social-cognition prompting alone does not fully address model miscalibration.

    Source: §1 · p. 2, §1.1 · p. 3

    The operationalization captures only a limited range of ambiguity’s textures, and the rubric’s dimensions map imperfectly onto different forms of ambiguity.

    Source: §8.1 · p. 13

Abstract

from arXiv · show

Ambiguity is often treated as a bug for AI systems to resolve---but in human communication and culture, ambiguity can also be a generative resource. From humour to politics to art, people express themselves in words and images that are open enough to invite different interpretations, yet constrained enough to be interpretable. We operationalise this notion of calibrated ambiguity with a task drawn from the parlour game Dixit. We compare differences in clues generated by human vs multimodal language models, based on a novel coding rubric for calibrated ambiguity, and find that models consistently exhibit ambiguity collapse (i.e., their outputs are over-specified, leaving no room for multiple legitimate interpretations). Unlike human clues, AI-generated clues also exhibit cultural flattening; they almost never make reference to culturally-situated knowledge, even when prompted to use allusion and figurative language.

1 Introduction

Although AI research often treats ambiguity as something to remove, ambiguity can be central to communication and culture, where multiple interpretations may be legitimate. The paper introduces a Dixit-based paradigm and rubric for studying calibrated ambiguity, finding that models and humans manage it differently.

1 Introduction

  • Ambiguity is not only semantic vagueness: it can be a vital aspect of human communication and culture, despite being difficult to quantify.The paper notes that cultural appropriateness depends on navigating tradeoffs among ambiguous concepts, not merely checking demographic categories.Source: §1 · pp. 1–2
  • Ambiguity can support narrative meaning, interpersonal tact, and deliberation, while AI systems’ single authoritative judgments risk collapsing multiple legitimate interpretations.The paper argues that appropriate ambiguity depends on the people, situation, and purpose involved.Source: §1 · pp. 1–2

1.1 Summary of Contributions

  • The paper presents a Dixit-based paradigm and a coding rubric to measure calibrated ambiguity in multimodal AI outputs.The rubric is intended to distinguish dimensions relevant to the task.Source: §1 · p. 2, §1.1 · p. 3
  • Models are less well calibrated than humans and almost never use the culturally situated references common in human clues; prompting helps but does not fully prevent ambiguity collapse.The paper also reports significant variation across models in how they calibrate ambiguity.Source: §1.1 · p. 3

2 Related Work

Prior work has examined ambiguity recognition and resolution, perspectivist annotation, and Dixit as a testbed for vision–language association. This paper’s context includes approaches that treat ambiguity as desirable and prior Dixit studies that vary in scale and focus.

2 Related Work

  • Prior work frames ambiguity collapse as a risk to plural interpretations and links it to the flattening of culturally situated outputs.This paper focuses specifically on output collapse: whether systems produce artefacts supporting multiple legitimate interpretations.Source: §2 · p. 3

2.1 Measuring ambiguity

  • Existing ambiguity evaluations test recognition, resolution, or uncertainty, while perspectivist and variability-based work treats disagreement as potentially desirable but not uniquely attributable to ambiguity.These approaches often use human judgments as a reference for the appropriate form of variability.Source: §2.1 · p. 3

2.2 Dixit-based tasks

  • Dixit has been used to study complex tasks without a single right answer, including creative captioning that integrates visual perception, language understanding, and social reasoning.Source: §2.2 · p. 3
  • Computational Dixit studies have addressed image–word association, voting, and card–hint matching; one small study found the AI had the lowest median hint relevance and greatest variability.Prior work ranges from controlled studies of 15 turns or 20 cards to archival analyses of about 116,000 rounds.Source: §2.2 · p. 3

3 Method

The study operationalizes calibrated ambiguity through Dixit, where a clue should identify the storyteller’s card for some but not all players. It compares human clues with model-generated clues on shared boards, using human-coded ratings and LLM coders for larger-scale comparison.

3.1 Dixit Paradigm

  • A Dixit clue is calibrated when some but not all other players select the storyteller’s target card.The storyteller wins only under that condition, so clues must be neither uninterpretable nor unmistakably specific.Source: §3.1 · p. 4

3.2 Existing dataset of human-generated clues

  • The human dataset contains 116,226 preprocessed English-language rounds; the study selected 350 five-player rounds after structural checks and duplicate-clue exclusion.The selected boards comprise 200 randomly sampled baseline rounds and 150 rounds with human clues coded for comparison.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 LLM-generated Dixit Clues and Judgments

  • Six open-weight vision–language models generated clues for the same boards, with a minimal prompt or a bundled social-cognition prompt that included audience modeling, figurative language, and a target of two guessers.The social-cognition prompt comparison tests the bundled instructions together, not their individual effects.Source: §3.3 · pp. 4–5
  • Models saw only the target card, and each model–prompt combination generated one clue per round under fixed sampling settings, with overlength outputs resampled rather than truncated.The study generated 4,200 LLM clues in total.Source: §3.3 · p. 5
  • Two LLM coders independently rated every clue using the same rubric protocol as human annotators, seeing the clue and five candidate images but not the answer key or clue source.Unless otherwise stated, analyses use the mean of their two ratings per clue.Source: §3.3 · p. 5
  • LLM coders’ ratings correlated with the human panel mean at r=0.60–0.79 across four dimensions, comparable to individual human annotators; behavioral voting tracked individual clues only weakly.The authors therefore used rubric coding as the primary assessment and LLM coders for model comparison.Source: §3.3 · p. 5

4 Ambiguity Coding Rubric

The rubric treats calibrated ambiguity as four separate ratings, operationalizing calibration, cultural situatedness, literalness, and figurativeness for Dixit clues. The section also describes how the rubric was refined and applied, and reports strong inter-rater reliability.

4 Ambiguity Coding Rubric

  • Each clue receives four 3-point ordinal ratings: calibration, situatedness, literalness, and figurativeness.The dimensions were selected to address the study’s hypotheses rather than to exhaustively classify ambiguity.Source: §4 · p. 5, Table 2 · p. 6
  • Calibration targets ambiguity collapse by distinguishing over-specific clues from calibrated or overly ambiguous ones.Raw target-card accuracy was insufficiently sensitive because chance guesses could make completely ambiguous clues appear successful.Source: §4 · p. 5
  • Situatedness measures how much cultural knowledge a clue requires, addressing observed differences in human and model use of cultural references.The term is drawn from humanities usage and captures whether specialized cultural knowledge is needed to interpret a clue.Source: §4 · pp. 5–6

4.1 Procedure

  • Five expert annotators reviewed examples and rated clues while blind to both the target answer and whether each clue was human- or model-generated.They first selected a target from the five-image set, then rated the clue on all four dimensions.Source: §4.1 · p. 6
  • A fifth rating recorded externally verifiable references so interpretation would not depend solely on annotators’ prior cultural knowledge.A rater used Gemini and Google search to check possible referents, including cultural references such as the example “Colors of the wind...”Source: §4.1 · p. 6

4 Ambiguity Coding Rubric

  • Literalness and figurativeness are separate dimensions because clues can combine concrete image features with figurative language.The earlier primary-mechanism and literal-versus-figurative schemes proved difficult to apply when clues had both aspects.Source: §4.2 · p. 6, §4 · p. 6

4.3 Reliability

  • Across 150 clues rated by five annotators, all four dimensions had Krippendorff’s alpha values above 0.80.Calibration α=0.866, situatedness α=0.927, literalness α=0.884, and figurativeness α=0.859.Source: §4.3 · p. 6

5 Results

The Dixit results compare human and model clues using the ambiguity rubric, finding differences in calibration, cultural situatedness, and language style. The size and nature of those differences vary between the two models examined.

5.1.1 LLMs exhibit ambiguity collapse.

  • LLM clues were more over-specified than human clues: mean calibration was 2.39 versus 1.93 on the 1–3 scale (p<0.001; Fig. 2a).The rubric operationalized ambiguity collapse as over-specification; the figure presents human-coded ratings and group means with 95% confidence intervals.Source: §5.1.1 · p. 7, Figure 2 · p. 7

5.1.2 LLMs tend to flatten cultural context.

  • LLM clues were less culturally situated than human clues (mean situatedness 1.15 vs. 1.82; p<0.001), and almost never used culturally specific references.The situatedness scale distinguishes universal, mainstream, and niche cultural knowledge.Source: §5.1.2 · p. 7, Figure 2 · p. 7

5.1.3 LLMs tend to be more literal and less figurative than humans.

  • LLM clues were more literal than human clues (2.21 vs. 1.55) and less figurative (1.94 vs. 2.37; both p<0.001).Some highly literal clues were over-specified, while other miscalibration involved vague visual descriptions rather than exact labels.Source: §5.1.3 · p. 7, Figure 2 · p. 7

5.1.4 Ambiguity calibration depends crucially on the model.

  • Qwen clues were more literal than Gemma clues (2.55 vs. 1.86; p<0.001), while Gemma clues were more figurative (2.25 vs. 1.64; p<0.001).Figure 3 separates model-generated clues and marks the human mean with a dashed line.Source: §5.1.4 · p. 7, Figure 3 · p. 8
  • Qwen drove the over-specification result (mean calibration 2.75 vs. human 1.93; p<0.001), while Gemma’s calibration was statistically indistinguishable from humans (2.01 vs. 1.93; p=0.36).Both models were similarly low in situatedness relative to humans (1.18 and 1.12 vs. 1.82; both p<0.001).Source: §5.1.4 · pp. 7–8, Figure 3 · p. 8

5.2 Can formal linguistic features capture the differences without the rubric?

The section tests whether formal linguistic features can distinguish human from model clues without the ambiguity rubric. Individual features provide limited separation, while their aggregate captures a stronger pattern of model-clue homogeneity.

5.2 Can formal linguistic features capture the differences without the rubric?

  • The analysis used twelve surface features on 4,200 LLM clues and a random sample of 2,000 human clues.The LLM clues came from six models, two prompts, and 350 rounds; the human sample came from a corpus of 62,688 English clues.Source: §5.2.1 · p. 8

5.2.1 Individually, formal features separate human and LLM clues

  • Despite statistically reliable average differences, individual features overlapped substantially and were insufficient on their own to distinguish human- and model-generated clues.For example, more than four words appeared in 93% of LLM clues and 46% of human clues.Source: §5.2.1 · p. 8
  • Word count was the strongest individual feature (AUC = 0.74, 95% CI [0.72, 0.75]), followed by concreteness (0.69).The word-count distinction was at least partly an artifact of prompts requesting 3–6-word clues; LLM clues were also more concrete on average.Source: §5.2.1 · p. 8

5.2.2 LLM clues tend to be more homogeneous than human clues.

  • Across the twelve-feature space, model clues were more homogeneous than human clues, clustering more tightly despite weak one-to-one signals from individual features.The figure shows individual-feature ROC AUCs, the held-out combined score, and a two-component projection.Source: §5.2.2 · p. 8, Figure 4 · p. 9
  • The twelve-feature Mahalanobis score separated held-out LLM and human clues with AUC = 0.92, while LLM clues occupied a tighter subregion of the human feature space.In that space, 99.8% of LLM clues fell inside the human 95% ellipsoid, compared with 23% of human clues inside the LLM ellipsoid.Source: §5.2.2 · pp. 8–9, Figure 4 · p. 9
  • Mean pairwise distance was 3.2 for LLM clues versus 4.7 for human clues; individual models were narrower than the human cloud, with distances of 2.2–3.4.Model centroids occupied different regions, so there was no single LLM style; relying on one model understates the range of human clues.Source: §5.2.2 · p. 9

6 Model Comparison

The expanded comparison tests six vision–language models with two prompts, using LLM coders checked against a human panel. Models differ in ambiguity collapse, prompting partly reduces it, and cultural flattening remains across models and prompts.

6.2 Model Comparison Results

  • The comparison expanded to six models spanning 9B–35B parameters and dense and mixture-of-experts architectures, with 350 clues per model-prompt condition.Across this sample, neither architecture nor parameter count predicted calibration ranking.Source: §6 · p. 9, §6.2 · p. 10, §6.2.1 · p. 10

6.1 Scaling annotation with LLM coders.

  • LLM coders’ ratings correlated with the human panel mean at r=0.60–0.79, comparable to individual human annotators’ leave-one-out agreement (r=0.58–0.80).They matched the rounded panel mean exactly on 0.55–0.80 of clues and identified the target at least as often as human annotators (0.62–0.65 vs. 0.57).Source: §6.1 · p. 9, Table 3 · p. 9

6.2.1 All six models collapse, regardless of architecture or size.

  • All six models’ minimal-prompt clues were more specified than human clues on the same boards (all p<10^-40; Cliff’s δ=0.54–0.97).GLM-4.6V-Flash was overspecified on 93% of rounds versus 2% for humans; Gemma-3-27B on 28%.Source: Figure 5 · p. 10, §6.2.1 · p. 10

6.2.2 Prompting mitigates collapse but does not remove it.

  • The social cognition prompt shifted calibration toward the human distribution, with the largest change for GLM (0.76 points, from 2.96 to 2.21).Prompt, model, and their interaction significantly affected calibration (all p<0.001).Source: §6.2.2 · p. 11
  • Only Gemma’s social-cognition clues reached human-consistent calibration; figurative language did not reliably translate into better calibration.Gemma was rated calibrated on 98% of rounds versus 97% for humans, while GLM’s more figurative clues remained overspecified on 12% of rounds.Source: §6.2.2 · p. 11

6.2.3 Cultural flattening persists across models and prompts.

  • All twelve model-prompt conditions were near the floor for situatedness (1.01–1.07), and the prompt did not significantly affect situatedness (p=0.07).Human clues were rated Mainstream or Niche 43% of the time, versus 3.9% of model clues rated above Universal.Source: §6.2.3 · p. 11

7 Discussion

The discussion interprets the findings as evidence of output ambiguity collapse and cultural flattening, while emphasizing that collapse varies by model and can be partly mitigated by prompting. The authors argue that prompting does not erase the gap between model and human clueing.

7 Discussion

  • The study’s central finding is that models are less calibrated than humans and almost never use the cultural references found in human clues.Surface-level features distinguish human and model clues only weakly, while models vary in how they calibrate ambiguity.Source: §7 · p. 11

7.1 Do LLMs exhibit ambiguity output collapse?

  • The demonstrated collapse concerns output plurality, and its degree varies: social cognition prompting brought Gemma to human-level calibration, but did not eliminate collapse overall.The authors report susceptibility across 9B–35B dense and mixture-of-experts models, with model-specific differences.Source: §7.1 · pp. 11–12

7.2 Do LLMs avoid cultural references when calibrating ambiguity?

  • Models avoided culturally situated knowledge even under a prompt inviting allusion and figurative language, unlike humans’ use of cultural references.The authors describe this avoidance as a defining difference in how humans and models managed ambiguity in this task.Source: §7.2 · p. 12

7.3 Do models tend to calibrate ambiguity in the same way?

  • Models use different strategies and respond differently to prompts, but avoidance of culturally situated knowledge was robust across models and conditions.The authors characterize model clues as homogeneous while noting each model uses an idiosyncratic strategy.Source: §7.3 · p. 12

7.4 Can ambiguity collapse be fixed with prompting?

  • Social cognition prompting improved calibration for some models but did not remove collapse, and encouraging figurative language did not automatically yield better-calibrated clues.Gemma performed best and sometimes reached human-level performance in the examined task.Source: §7.4 · p. 13

8 Limitations and Alternative Perspectives

The authors present the Dixit-based rubric as a useful baseline, while emphasizing that ambiguity calibration depends on context and that prompting and model-scale questions remain open.

8 Limitations and Alternative Perspectives

  • The rubric operationalises calibrated ambiguity as multiple legitimate interpretations, but captures only some aspects of ambiguity and maps imperfectly onto situatedness, literalness, and figurativeness.The authors note that culturally situated knowledge may help analyse moral ambiguity in Unforgiven, while the role of literal and figurative language is less clear.Source: §8.1 · p. 13
  • The desirable kind of ambiguity depends on use case, and future work could adapt the paradigm to specific contexts or populations.Examples include moral ambiguity in AI-generated narratives and labelling inappropriate images for content moderation.Source: §8.1 · p. 13
  • Prompting appears to affect overall calibration and collapse, but the findings on cultural flattening are less clear.The authors suggest investigating whether effective elicitation requires cues of participation in a cultural milieu, rather than simply asking for behaviour representing it.Source: §8.2 · p. 13
  • The study does not establish how ambiguity calibration scales with model size; Gemma 3 (27B) performed near human level in some cases under the studied conditions.The authors suggest testing larger models with adapted tasks, such as more distractors or background information about players’ shared cultural frames.Source: §8.3 · p. 13

Ethics and Privacy Statement

The study used publicly available human clues and author-coders, and the authors report minimal ethical or privacy risk for the Dixit images and work presented.

Ethics and Privacy Statement

  • Human clues were publicly available and expert coders were paper authors, while Dixit imagery posed minimal ethical or privacy risk; adaptive work may involve fraught contexts such as content moderation.Source: Ethics and Privacy Statement · p. 14

A Storyteller prompts

The two prompts shared the same text up to a marked point, after which the social cognition prompt added coaching.

A Storyteller prompts

  • The baseline prompt frames clue generation as a four-player Dixit round and asks for a three-to-six-word clue that hints at the target card.Source: A Storyteller prompts · p. 16
  • The social cognition prompt encourages clues using figurative or abstract language that could apply to multiple cards while favoring the target and being guessed by two of four players.Source: A Storyteller prompts · p. 16

B Behavioural robustness check: a vision–language judge panel

A vision–language judge panel broadly matched human audiences on population-level target-range rates, but its correspondence with individual human rounds was limited.

B Behavioural robustness check: a vision–language judge panel

  • The earlier check used 200 human-dataset rounds with Gemma-3-27B and Qwen3.5-35B-A3B under two social-cognition prompt wordings, yielding 800 clues clipped to seven words.Source: B Behavioural robustness check: a… · p. 16
  • Four open-weight vision–language judges voted for one card per clue, with a clue in range when some but not all valid judges selected the target.The same panel scored human- and model-generated clues; it returned valid votes in 76% of the 1,000 judged rounds.Source: B Behavioural robustness check: a… · p. 16
  • The panel placed 60.5% of human clues in the target range versus 60.0% for original human audiences, but per-round hit rates correlated modestly (Spearman ρ=.29, p<.001).Source: B Behavioural robustness check: a… · p. 16
  • The panel is not a stand-in for how a particular human audience reads a clue; it supports population-level producer comparisons on identical boards.Source: B Behavioural robustness check: a… · p. 16

B.1 Model Comparison Robustness

Three checks indicate that the reported results are robust to coder choice, board-subset selection, and clue length. In particular, word count does not explain the calibration gap.

B.1 Model Comparison Robustness

  • Word count did not explain calibration: the LLM-clue correlation was ρ=.11; Kimi-VL’s shortest clues (3.5 words) had δ=.61, while equal-length GLM banks (6.4 words) had δ=.97 and .27.Kimi-VL's minimal-prompt clues were substantially overspecified despite being the shortest bank.Source: §B.1 · p. 16
  • Calibration means differed by at most 0.10 between the two board subsets for every bank.The 350 boards comprised two subsets selected in different ways.Source: §B.1 · p. 16
  • Coder agreement ranged from r=.66 for Calibration to r=.84 for Figurativeness, with exact agreement of .76–.96; reported gaps held for either coder alone except the Gemma social-cognition bank.One coder rated that Gemma bank as equivalent to the human clues.Source: §B.1 · p. 16
Loading 2609.12575v1…