Source-linked AI summary
Beyond Local Surprise: Grounded Dialogue as Selective Belief Revision under Referential Uncertainty
Ziming Liu, Bhanu Chaitanya Jasti, Ziyang Xu, Hongyu Wu, Yi Wu, Jiqun Liu
TL;DR
The paper asks how listeners should preserve or revise grounded interpretations when referential evidence arrives incrementally under hidden visual context. It introduces a controlled framework comparing four revision policies under identical conditions, finding that selective revision guided by accumulated uncertainty supports coherent grounding better than reacting to local mismatch alone. The resulting preservation-by-default pattern is broadly consistent with conceptual pact theory.
Problem
Incremental grounding models often treat local mismatch as a cue for stronger updating, leaving unclear whether mismatch alone should determine revision of established interpretations.
Method
The paper models turn-by-turn grounding as an evolving image-region evidence state and compares four revision policies while holding dialogue, visual evidence, and architecture fixed.
Results
Successful grounding depends less on reacting to local mismatch than on selective revision under uncertainty; mismatch is associated with reduced revision, while uncertainty permits accumulated understanding to change.
Takeaways & Limitations
Conversational grounding is better understood as selective belief modulation, with interpretations preserved by default and revised when ambiguity becomes sufficiently high.
Takeaways & Limitations
The experiments use only PhotoChat, so whether the separation between local mismatch and successful revision generalizes to less stable communicative settings remains open.
Abstract
from arXiv · showhide
When a speaker refers to a scene that the listener cannot directly see, the listener must decide whether to preserve its current understanding or revise it as new utterances arrive. Many language systems treat local mismatch as a cue for updating: divergence from the current understanding encourages adjustment. Yet conversational understanding may be more conservative, interpreting mismatching evidence relative to prior understanding rather than immediately revising it. We introduce a controlled, data-driven framework for turn-by-turn preserve/revise decisions in dialogue, where competing revision policies are learned under otherwise identical conditions. We compare four theory-driven revision strategies, each reflecting a different assumption about when listeners should preserve or revise. Two findings stand out. First, a mismatch-driven policy that updates solely based on local divergence reacts strongly to mismatch but destabilizes grounding and degrades retrieval. Second, an uncertainty-sensitive policy extends mismatch-based updating with accumulated evidence, preserving coherent understanding while maintaining strong retrieval performance. Surprisingly, coherent understanding emerges from a counterintuitive pattern: local mismatch promotes preservation, whereas accumulated uncertainty promotes revision, suggesting that listeners maintain prior understanding despite local mismatch and revise only when uncertainty sufficiently accumulates. This pattern is consistent with conceptual pact theory.
1 Introduction
The paper frames grounded dialogue as incremental decisions to preserve or revise established interpretations. It asks whether revision should respond to local mismatch alone or to uncertainty accumulated across turns.
- Motivation: Listeners incrementally decide whether to preserve established interpretations or revise them as new conversational evidence arrives.Grounding is interpreted relative to what speakers have already established.
- Motivation: Incremental models commonly revise referent beliefs as each new word provides evidence, with larger mismatch implicitly warranting stronger updates.The paper identifies this as an unresolved assumption about how revision should be driven.
- Motivation: Conceptual pact theory instead suggests preserving shared interpretations rather than renegotiating meaning after every local mismatch.Repeated mismatch may need to accumulate into uncertainty before revision is warranted.
- Contribution: The framework holds the underlying architecture fixed while comparing PURE-CORR, FIXED, DCP-B, and MULTI-C revision policies.This controlled comparison targets competing assumptions about preserve-or-revise behavior.
- Contribution: Local mismatch destabilizes grounding when it drives strong revision, whereas uncertainty over accumulated evidence supports coherent selective revision.The resulting pattern is that surprise suppresses revision while accumulated uncertainty promotes it.
2 Related Work
Prior work models grounding as incremental interpretation, repair, or downstream task success. This paper distinguishes itself by making preserve-or-revise behavior itself the object of study.
- Grounding and repair: Conversational grounding research commonly emphasizes clarification and recovery when misunderstandings cause communication breakdown.Dialogue systems often treat revision as a discrete clarification decision under uncertainty.
- Incremental interpretation: Incremental reference models update interpretations as linguistic evidence accumulates, gradually narrowing or shifting toward likely referents.These approaches typically assume a revision mechanism rather than studying revision decisions directly.
- Research gap: A common computational intuition is that larger mismatch should lead to larger updates.The paper identifies preserve-or-revise behavior as rarely formulated explicitly in prior work.
- Grounded multimodal dialogue: Grounded multimodal dialogue evaluates alignment with a shared visual environment through downstream outcomes such as retrieval or collaborative performance.The paper uses grounded dialogue not only as a task but as a setting for analyzing revision dynamics.
3 Method
The framework represents grounding as an evolving evidence state over image regions and learns a scalar revision signal while keeping dialogue, visual evidence, and architecture controlled. Four policies encode distinct assumptions about revision.
- Evidence state: At each turn, an evidence state over image regions tracks what has been grounded from the dialogue and shared visual scene.The goal is to model grounding evolution over time rather than only dialogue–image matching.
- Revision mechanism: A scalar ρ_t balances prior grounding against new evidence: low ρ_t preserves accumulated grounding, whereas high ρ_t favors revision.Varying how ρ_t is determined isolates competing revision assumptions under fixed evidence.
- Evidence state: Turn-local evidence is computed as a softmax attention distribution over patch-level image features, representing where the current utterance points in the scene.The local signal reflects the current utterance without prior conversational context.
- Coherence: The model separately tracks coherence by combining the current grounding state with image features and updating a turn-level coherence trajectory.This coupling distinguishes local evidence from longer-term grounding coherence.
- Revision policies: PURE-CORR and FIXED use constant ρ = 0.1, while DCP-B makes revision increase with local divergence and MULTI-C learns from stability, uncertainty, and expected coherence change.MULTI-C uses five conversational signals, including local divergence, prior coherence, expected coherence change, and prior and incoming uncertainty.
- Learning objectives: Training applies preserve-or-revise constraints that encourage stability during low-revision turns and grounding change during high-revision turns, with perceptual evidence fixed.Contrastive alignment anchors utterances to their dialogue images while the preserve and revise objectives regulate state changes.
4 Experimental Setup
The experiments use PhotoChat to evaluate incremental grounding before image disclosure, with a fixed hidden-image target and deterministic train/test splitting. This setting requires grounding to develop from dialogue across turns.
- Dataset: PhotoChat contains dialogues in which one participant privately sees an image and reveals it only after conversation unfolds.Before disclosure, interlocutors must establish shared understanding without direct image access.
- Evaluation protocol: The evaluation retains only pre-disclosure turns and treats the hidden image as the grounding target, preventing trivial matching from explicit visual descriptions.This tests grounding through accumulated conversational evidence.
- Evaluation protocol: The dataset contains 21 training shards with a deterministic 19-train, 2-test split using random seed 42.The split is fixed across revision policies for comparability.
5 Evaluation Protocol
The evaluation measures both retrieval performance and turn-level grounding dynamics across revision policies, using intrinsic metrics and multimodal perturbations. Retrieval is treated as a behavioral test, while grounding dynamics reveal whether successful retrieval depends on coherent revision.
- Evaluation Protocol: Retrieval evaluates whether a dialogue trajectory retrieves its target image from a candidate gallery, serving as a behavioral test of grounding quality.The protocol distinguishes retrieval success from the coherence of the revision dynamics that produce it.
- Reference Conditions: The study compares four revision policies alongside Random and CLIP-only reference conditions.Random provides a chance-level lower bound, whereas CLIP-only supplies a static frozen-embedding reference without incremental grounding updates.
- Intrinsic Metrics: Grounding dynamics are characterized with preservation index, revision sensitivity, and mean revision rate over evaluation turns and random seeds.These metrics capture stability under low revision, correspondence between revision signals and evidence shifts, and selective versus continuous evolution.
- Reporting: Table 1 reports retrieval and intrinsic dynamics as mean ± std across three seeds.The table caption states that PURE-CORR and FIXED remain within one standard deviation of MULTI-C on all retrieval metrics, while DCP-B substantially underperforms.
- Perturbation Experiments: Inference-time perturbations replace text or image representations with zero, random Gaussian, or empirical mean vectors to test whether MULTI-C requires coordinated multimodal grounding.The perturbations are applied without retraining, and coordinated grounding predicts retrieval degradation toward chance when either modality is disrupted.
6 Results
The results distinguish reactive mismatch-driven revision from bounded, uncertainty-sensitive adaptation. DCP-B destabilizes grounding and retrieval, whereas MULTI-C preserves accumulated understanding while remaining responsive to new evidence and using both modalities.
- Retrieval Performance and Grounding Dynamics: DCP-B collapses on retrieval despite high revision sensitivity, with Rev. Sens. = 0.900 and Switch KL = 12.31 ± 0.88.Local mismatch triggers abrupt grounding shifts that repeatedly disrupt accumulated understanding.
- Retrieval Performance and Grounding Dynamics: MULTI-C matches PURE-CORR and FIXED in retrieval while maintaining structured revision, with Rev. Sens. = 0.598 and Mean ρ = 0.148.Its revision is selective and distributed across turns rather than strongly reacting to isolated mismatch.
- Signal Contributions: In MULTI-C, local divergence correlates negatively with revision (r = −0.935), whereas incoming uncertainty correlates positively (r = +0.562).The learned controller distributes revision across multiple conversational signals instead of relying primarily on divergence.
- Revision Policies: The policies learn distinct revision profiles: low deterministic revision for PURE-CORR, near-binary mismatch-driven switching for DCP-B, and smooth bounded revision for MULTI-C.These profiles correspond to stable accumulation, surprise-driven switching, and uncertainty-sensitive revision assumptions.
- Turn-Level Trajectories: Across two pre-reveal dialogues, DCP-B shows spike–freeze behavior, while MULTI-C temporarily loses coherence at turning points and recovers as evidence accumulates.MULTI-C remains active but bounded, preserving accumulated grounding while adapting to conversational change.
- Multimodal Coordination: Perturbing either modality reduces MULTI-C retrieval to near-random levels across metrics.This confirms that its grounding dynamics rely on both visual and linguistic information rather than either modality alone.
7 Discussion
The discussion argues that successful grounding is conservative by default: local mismatch often supports preservation, while accumulated uncertainty selectively motivates revision. Retrieval alone cannot reveal these underlying dynamics.
- Selective revision: MULTI-C learns selective revision without explicit supervision, with accumulation dominating and local divergence reducing revision.Its mean revision rate is ρt = 0.148, while uncertainty is more strongly associated with revision than local conflict alone.
- Selective revision: The observed preservation of partner-specific interpretations is broadly consistent with conceptual pact theory.The discussion frames revision as selective belief modulation rather than purely reactive repair.
- Selective revision: Detecting mismatch and deciding whether it warrants revision are distinct operations.Mismatch may reflect ambiguity, incomplete reference, or temporary repair that still benefits from preserving prior grounding.
- Observable dynamics: MULTI-C and PURE-CORR can achieve comparable retrieval through qualitatively different revision dynamics.Retrieval indicates success but not the mechanism, so coherence trajectories and revision behavior provide additional evidence.
8 Conclusion
The paper introduces a computational framework for studying conversational grounding as a preserve-or-revise problem. Its results indicate that selective revision under accumulated uncertainty better captures coherent grounding than purely reactive mismatch-based updating.
- 8 Conclusion: The framework studies conversational grounding as a turn-by-turn preserve-or-revise problem.It treats understanding as evolving through interaction rather than as purely reactive repair.
- 8 Conclusion: Successful grounding depends less on local mismatch than on selective revision under uncertainty.Mismatch is associated with reduced revision, whereas uncertainty permits accumulated understanding to change.
- 8 Conclusion: The findings are broadly consistent with conceptual pact theory.The paper characterizes grounding as selective belief modulation and provides a computational instrument for studying how understanding evolves through interaction.
9 Limitations
The study’s generalizability is constrained by its single cooperative dataset and by a deliberate emphasis on interpretability over architectural expressivity. Whether the findings extend to less stable settings or richer models remains open.
- Single dataset: All experiments use PhotoChat, limiting evidence about generalization beyond one dataset.The authors plan evaluation on I-CONECT as a stress test involving less stable grounding and greater conversational scaffolding.
- Single dataset: PhotoChat’s cooperative, goal-oriented interactions typically maintain coherent grounding and contain relatively infrequent strong repair events.This may favor persistence-based strategies and leaves generalization to less stable communicative settings unresolved.
- Interpretability versus expressivity: The design prioritizes interpretability over capacity, leaving richer architectures’ ability to retain transparency while capturing finer-grained grounding unresolved.This is an explicit scope boundary of the current framework.
A.1 Training Protocol
Training uses a shared frozen CLIP encoder, fixed data splitting, common optimization settings, and repeated seeds across regime variants. The evaluation pipeline aggregates per-seed results with a fixed script.
- Training Protocol: All grounding regimes share an identical frozen CLIP encoder and optimization protocol.The encoder is ViT-B/32, and the backbone remains frozen throughout training.
- Training Protocol: Training uses batch size 16 and random seeds {0, 1, 2}.Results are reported as mean ± std over the three seeds.
- Training Protocol: Dynamic regimes share target revision rate r = 0.15 and DCP sharpness initialization a = 2.0.These shared settings accompany the common loss-weight configuration.
- Training Protocol: PURE-CORR sets α = β = 0, disabling all grounding-dynamics losses.This distinguishes it from the dynamic regimes that retain those losses.
- Training Protocol: PhotoChat is split deterministically at the shard level into 19 training shards and 2 test shards.The split uses seed 42 and is shared across all training seeds and regime variants.
- Training Protocol: Six regime variants train in two parallel waves across three GPUs per wave, while evaluation runs sequentially per seed.Per-seed results are aggregated into mean ± std tables using a fixed script, and interrupted runs resume from checkpoints.
B Appendix B: Modality Ablation
Inference-time modality ablation replaces each modality independently without retraining; replacing either modality collapses MULTI-C retrieval to near-random.
- Each modality is independently replaced with zero vectors, random Gaussian vectors, or empirical mean embeddings computed over the test set.The perturbations are applied at inference time.
- Replacing either modality collapses MULTI-C retrieval to near-random, indicating that grounding depends jointly on both modalities.