Source-linked AI summary
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
Nan Li, Albert Gatt, Massimo Poesio
TL;DR
Vision-language models must distinguish potential shared understanding from common ground established through dialogue grounding. The paper evaluates this ability under controlled dialogue-context and map-information conditions, finding that task-relevant map content makes models over-predict alignment, whether presented visually or textually.
Problem
The paper asks whether VLMs distinguish potentially shared interpretations from mutual understanding established through grounding in asymmetric collaborative dialogue.
Method
The study tests VLM interpretation matching on HCRC MapTask reference expressions while varying dialogue context and map-information access across two model families.
Results
Authentic maps improve overall performance but cause over-predicted alignment, a bias reproduced by textual map descriptions and absent from non-informative visual inputs.
Takeaways & Limitations
Models better assess potential referential overlap from map content than track grounding as an incremental interactional process, conflating possible overlap with established alignment.
Takeaways & Limitations
Detailed condition-grid analysis relies primarily on one model and one MapTask corpus, so generalization to other information-asymmetric settings remains unestablished.
Abstract
from arXiv · showhide
In collaborative dialogue, shared perception does not guarantee shared interpretation. Mutual understanding must be established through interaction. We investigate whether vision-language models (VLMs) can distinguish what could be shared from what has been shared between dialogue participants through grounding. We formulate this as an interpretation-matching task on 13,077 annotated reference expressions from HCRC MapTask dialogues, and evaluate VLMs under systematically controlled manipulations of dialogue context and map-information access. Our results show that providing authentic map images improves overall performance but shifts models toward over-predicting alignment. Textual descriptions of the same map content reproduce this bias, while non-informative images suppress alignment predictions entirely, indicating that the bias is driven by task-relevant map content, not the visual channel. This improvement comes at the cost of degraded accuracy on non-aligned cases. Calibration analysis and reference-chain tracking further suggest that models rely on static referential cues on the maps rather than tracking how grounding unfolds through dialogue history. We observe these patterns most clearly in Qwen3-VL-8B-Instruct and, to varying degrees, in four additional models from two architecture families. In models that exhibit the bias, map content, whether presented visually or textually, is treated as evidence of mutual understanding, conflating potential with established common ground.
1 Introduction
This study tests whether vision-language models distinguish potential from established common ground in information-asymmetric dialogue by matching participants’ interpretations of reference expressions. Across controlled dialogue-context and map-information manipulations, models often treat task-relevant map content as evidence of alignment rather than tracking grounding through dialogue history.
- Task and evaluation: The study frames interpretation matching as a binary judgment of whether two participants’ interpretations of a marked reference expression match.Models are evaluated while varying dialogue context and map information, using Qwen3-VL and Gemma3 models ranging from 2B to 12B parameters.
- Main findings: Authentic map images shift models toward over-predicting alignment by making landmark co-presence appear to signal mutual understanding.This conflates what could be shared with what has been shared.
- Main findings: Textual descriptions of the same map content reproduce the alignment bias, showing that task-relevant content rather than the visual channel drives it.Non-informative visual inputs instead make models more conservative about predicting alignment.
- Main findings: Calibration and reference-chain analysis indicate that models rely on static referential cues on maps instead of tracking grounding through dialogue history.This explanation is reported as converging across both analyses.
- Contributions: The model-dependent failure mode is observed most clearly in Qwen3-VL-8B, where possible referential overlap is conflated with communicative alignment.The work contributes an evaluation methodology for distinguishing potential from established common ground in information-asymmetric dialogue.
2 Related Work
Related work frames common ground as an interactional achievement rather than a consequence of shared perception, highlighting the limitations of dialogue systems that only observe static transcripts. Reference corpora likewise often assume speaker–addressee convergence through single gold referents, an assumption challenged under information asymmetry.
- Grounding and Overhearers: Common ground develops incrementally through negotiation, confirmation, and repair rather than direct shared perceptual access.
- Grounding and Overhearers: The overhearer illusion shows that participants who cannot contribute grounding acts form systematically weaker interpretations than addressees.Dialogue systems trained and evaluated on static transcripts are structurally overhearers, limiting what they can learn about grounding and clarification.
- Reference corpora test shared interpretation through repeated mention, partial information, and clarification but typically assign each expression a single gold referent.This annotation practice implicitly assumes speaker–addressee convergence, which breaks under information asymmetry and motivates perspectivist annotation.
3 Experimental Setup
The experiments evaluate VLM interpretation matching on perspectivist HCRC MapTask annotations, varying dialogue context and map-information access across two open-source model families. The setup includes informative, textual, and non-informative map conditions, with Qwen3-VL-8B-Instruct used for the full condition grid.
- Dataset: The dataset contains 13,077 annotated reference expressions from 128 HCRC MapTask dialogues with separate speaker intentions and addressee interpretations.The dialogues involve a giver and follower collaborating across slightly different maps, enabling landmark misalignment through naming, missingness, or quantity discrepancies.
- Task: Models make a binary YES/NO judgment of whether participants share the same grounded interpretation of a marked reference expression.YES means matching landmark IDs; NO includes both not-yet-grounded pending states and misunderstandings involving different landmarks.
- Experimental manipulations: Dialogue context varies across four windows: curL, curT, startL, and startT, testing the effects of prior grounding history and future repair or clarification.The windows range from the current transaction through the end of that transaction, or from the dialogue beginning through the target line or current transaction end.
- Experimental manipulations: Map access includes Text-only, Both maps, single-authentic-map conditions, textual map descriptions, Blank maps, and Shuffled maps.Blank and shuffled images test whether over-alignment is a generic multimodal artifact or depends on task-relevant map content.
- Models: The study evaluates five instruction-tuned VLMs from the Qwen3-VL and Gemma3 families, spanning 2B to 12B parameters, with Qwen3-VL-8B-Instruct used for full-grid analysis.The other four models are evaluated on the baseline condition grid to assess generality.
- Evaluation: Evaluation reports accuracy, macro-averaged F1, per-class recall, and yes-rate, with additional calibration, grounding-status, and reference-chain analyses.The yes-rate measures response bias, while the further analyses examine token-level confidence, grounding states, and repeated landmark mentions.
4 Results
Map access improves Qwen3-VL-8B-Instruct’s overall detection but shifts predictions toward alignment, reducing performance on not-aligned cases. Task-relevant map content reproduces this bias across visual and textual formats, while broader dialogue context provides additional but smaller gains under map access.
- Baseline map access: Adding both maps raises F1macro from .591 to .671, while recallpos increases from .590 to .822 and recallneg decreases from .677 to .518.The yes-rate rises from .515 to .727, exceeding the gold base rate of .721.
- Baseline map access: Either single-map condition triggers stronger over-alignment than both maps, with yes-rates of .791 and .794 and recallneg values of .436 and .408.Giver-only F1macro is .669 versus .671 with both maps; both-map access slightly moderates the bias through cross-map discrepancy evidence.
- Map-information modality: Content-free images suppress alignment predictions: blank and shuffled maps yield yes-rates of .184 and .193, below the text-only baseline of .515.This rules out image presence alone as the cause of over-alignment.
- Map-information modality: Textual map descriptions reproduce authentic-map behavior, with yes-rates of .675–.716, F1macro of .636–.668, elevated recallpos of .756–.810, and reduced recallneg of .528–.533.These values remain close to real maps and well above the text-only baseline, indicating that the bias follows task-relevant content rather than the visual channel.
5 Further Analysis and Discussion
Further analyses show that task-relevant map content drives over-alignment by making models confident in interpretations that have not been grounded, producing an asymmetric accuracy and calibration trade-off. This pattern intensifies across reference chains but varies substantially by model, rather than following architecture or size uniformly.
- Calibration analysis: Maps improve calibration on aligned instances but produce severe miscalibration on non-aligned ones, especially under follower-only access.Under both-maps, yes-rate is .727, with ECEyes = .094 and ECEno = .403; follower-only creates an ECE gap of .463.
- Status-level trade-off: Map access raises aligned accuracy while reducing pending accuracy and, under single-map conditions, misunderstood accuracy.Aligned accuracy increases by 23–29 points (.590 →.822 / .872 / .879), while pending accuracy falls by 17–27 points (.691 → .419 / .441 / .523).
- Status-level trade-off: Content-rich maps and textual descriptions over-predict alignment, whereas blank and shuffled maps elicit near-constant NO responses.Fake visual controls yield aligned accuracy of .220 / .222 and yes-rates of .184 / .193, indicating that task-relevant content, not modality, drives the trade-off.
- Mechanism: Models over-weight map evidence about what could be shared instead of dialogue history showing what has been established through interaction.This conflates potential common ground with established common ground and parallels the overhearer’s illusion.
- Reference-chain tracking: Over-prediction grows with repeated mention: both-maps yes-rates rise from .549 to .789 across chain lengths, while text-only accuracy declines from .719 to .599.Within chains, P(YES) also rises with reference position, reaching .708–.794 under map conditions at later mentions.
- Cross-model comparison: Models diverge in their responses to maps, and model size does not predict performance or calibration.Qwen3-VL-4B becomes more conservative (.368 →.225), while Qwen3-VL-2B outperforms 4B on both-maps (.566 vs. .411) and has ECE = .144.
6 Conclusion
Authentic maps improve VLM performance but bias models toward treating map content as evidence of established mutual understanding. This over-alignment reflects difficulty tracking grounding as an incremental interactional process and motivates evaluation in interactive settings and beyond MapTask.
- Main findings: Authentic maps improve overall performance but push models toward YES, treating landmark co-presence as evidence of mutual understanding.Textual descriptions of identical map content reproduce this bias, whereas non-informative images do not.
- Main findings: Under map conditions, models gain on aligned cases but lose accuracy on pending and misunderstood ones.They also become more confidently biased toward aligned judgments.
- Main findings: Models grow more likely to predict alignment across repeated mentions, suggesting reliance on potential referential overlap rather than interactional grounding.The conclusion characterizes current VLMs as better at assessing overlap from map content than tracking grounding incrementally.
- Limitations and future work: The experiments are limited to an overhearer setting in a single domain.Future work should test interactive settings in which models can request clarification, express uncertainty, or revise judgments over time.
- Limitations and future work: Extending evaluation beyond MapTask and using mechanistic interpretability may clarify whether over-alignment is general or tied to the task, setting, or model family.Suggested analyses include attention flow analysis and relating behavioural heterogeneity to model properties.
Limitations
The study’s limitations concern restricted generalization, corpus-specific effects, and an overhearer-based binary evaluation design. Textual map conditions also incompletely reconstruct spatial layout, complicating comparisons with visual conditions.
- Generalization: The detailed condition grid uses only Qwen3-VL-8B-Instruct, while additional models are tested only on baseline conditions.This limits how broadly the reported condition effects can be generalized across models.
- Generalization: The dataset is a single HCRC MapTask corpus with a 72.1%/27.9% aligned-case imbalance and only 239 misunderstood instances among 13,077.Its discrepancy types and class distribution may shape the observed effects, motivating replication with perspectivist annotations for both interlocutors’ interpretations.
- Task Design: The overhearer setup measures judgment failure rather than interaction failure.The model evaluates dialogue from an overhearer position rather than participating in the interaction.
- Task Design: Greedy decoding elicits a binary judgment that may not reflect the model’s full distributional beliefs about alignment.Multi-choice or free-form responses could reveal a more nuanced picture.
- Textual Map Reconstruction: Textual conditions provide landmark names and inter-map discrepancies but do not fully reconstruct spatial layout.Thus, behavioral differences between textual and visual conditions may partly reflect incomplete reconstruction rather than a genuine visual-channel effect.
A Dataset and Annotation
The study uses a perspectivist HCRC MapTask release that separately annotates giver and follower interpretations for 13,077 reference expressions, enabling interpretation matching. Its annotations encode landmark discrepancies, incremental resolution, and three post-hoc understanding states.
- Dataset: The dataset contains 13,077 reference expressions across 128 dialogues, paired with 16 distinct map pairs, with separate giver and follower interpretation annotations.These annotations define the YES/NO interpretation-matching task.
- Landmark discrepancies: Existence and multiplicity discrepancies are the structural source of most misalignment, whereas lexical variants are unified by name during annotation.The corpus distinguishes identical landmarks, lexical variants, existence discrepancies, and multiplicity discrepancies.
- Landmark identifiers: A unified landmark ID scheme disambiguates multiplicity landmarks and records map-side provenance using the format <map_id>_<concept>#<ordinal>@<side>.The map identifier ranges from m0–m15; ordinal 0 denotes the lower instance and 1 the upper instance, while @g and @f mark giver and follower maps.
- Annotation cascade: Each reference expression passes through a five-step annotation cascade that models incremental resolution, with downstream attributes set to null when the cascade terminates early.Each step is evaluated only when its preceding conditions are met, and step 5 is terminal.
- Understanding states: 72.1% of reference expressions are aligned, 26.0% are pending, and 1.8% are misunderstood under the released dataset’s three understanding states.These correspond to 9,435 aligned, 3,403 pending, and 239 misunderstood expressions.
B Experimental Setup and Reproducibility · C Prompt Template · D Textual Map-Information Examples
The paper uses tightly controlled, reproducible inference settings and a fixed prompt template across experimental conditions. Textual map access varies from landmark-name lists to landmark lists augmented with explicit map-discrepancy descriptions.
- B Experimental Setup and Reproducibility: All inference runs use vLLM with greedy decoding, temperature 0.0, random seed 42, a 16-token limit, and constrained YES/NO output.Experiments run on one NVIDIA A100 80 GB GPU, with Qwen3-VL built-in thinking disabled.
- B Experimental Setup and Reproducibility: Calibration analyses additionally record the top-20 logprobs at each generation step.These log probabilities support the calibration analyses reported in §5.1.
- C Prompt Template: Figure 3 presents the complete prompt template used across all experimental conditions.The template provides the structure into which condition-specific map information is inserted.
- D Textual Map-Information Examples: The textual map-information examples use dialogue q1ec1, map pair m12, and target reference expression “a caravan park.”Using one instance allows direct comparison between the two textual variants.
- D Textual Map-Information Examples: The text-landmark-names condition provides only the landmark-name list for each participant’s map.Its system prompt states that the lists appear below in the dialogue context.
- D Textual Map-Information Examples: The text-discrepancy-detail condition adds an explicit summary of differences between the two maps to their per-map landmark lists.The summary covers per-side exclusives, multiplicity landmarks, and shared landmarks.
- D Textual Map-Information Examples: The text-discrepancy-detail map-access block is inserted into the user prompt’s ${map_access} slot, as shown in Figure 5.Its system prompt explicitly describes both landmark names and how the maps differ.
E Map Reading Sanity Check
The sanity check finds that all five VLMs can read MapTask maps, but Qwen3-VL identifies landmarks and their positions more accurately than Gemma3.
- Evaluation setup: The check evaluated landmark names and spatial positions across all 32 hand-drawn MapTask maps using free-form greedy decoding.Landmark-name scores use exact string matching after lowercasing and whitespace normalization.
- Landmark identification: Qwen3-VL models achieve higher landmark-identification F1 (.876–.897) than Gemma3 models (.813–.820).Qwen3-VL-8B also has the highest precision (.956), with almost no spurious landmark names.
- Landmark identification: Qwen3-VL-8B avoids the character-level landmark naming errors introduced by both Gemma3 models.Examples include “picker fence” for “picket fence” and “pits of forest fire” for “site of forest fire.”
- Spatial description: Qwen3-VL models produce correct or near-correct positions on all 14 landmarks in the spatial-description check.Gemma-3-4B misplaces east lake, START, camera shop, and picket fence on map0g.
F Cross-Model Detailed Results
This section presents detailed cross-model comparisons, including macro F1 and yes-rate across map-access conditions and macro F1 by grounding status. The analyses cover Qwen3-VL and Gemma-3 model families at startT.
- Figures 7–9 provide detailed breakdowns of the cross-model comparison in §5.4.
- Figures 7 and 8 compare macro F1 and yes-rate across map-access conditions at startT.Figure 7 covers Qwen3-VL models (8B, 2B, 4B), while Figure 8 covers Gemma-3 models (4B, 12B).
- Figure 9 reports macro F1 by grounding status across all models and map-access conditions at startT.The grounding-status categories are aligned, pending, and misunderstood.