PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Gap: Theory-of-Mind-Grounded Friction for Epistemic Alignment

Yifan Zhu, Kyeongmin Rim, James Pustejovsky

arXiv:2608.30719v1cs.CL

TL;DR

The paper addresses the gap between smooth conversational coordination and convergence of participants’ beliefs by embedding ToM-grounded friction in Frictive Policy Optimization. Its four-identifier representation makes referential divergence computable, and FAR and FTR improve intervention-related outcomes and training stability relative to DPO. The authors frame this signal as a basis for context-sensitive intervention while identifying limits around inferred second-order labels, task generalization, and dual-use risk.

  • Problem

    Standard preference-based and turn-level evaluations do not explicitly represent evolving epistemic states, allowing incompatible interpretations to appear coordinated.

  • Method

    The paper extracts a four-identifier ToM schema for each referring expression and converts belief divergence into friction that conditions FPO objectives.

  • Results

    FAR and FTR improve intervention behavior, warranted-context calibration, and training stability relative to DPO.

  • Takeaways & Limitations

    ToM-grounded friction provides a trainable signal for regulating belief relations when locally coherent interaction masks incompatible interpretations.

  • Takeaways & Limitations

    Second-order labels are inferred by GPT-5 without gold mental-state annotations, and the evidence cannot establish genuine mentalizing or rule out correlated cues.

Abstract

from arXiv · show

Productive dialogue alignment requires distinguishing \emph{surface coordination} (acknowledgments and smooth task progression) from \emph{epistemic alignment} (convergence of belief states); standard preference-based methods typically optimize response-level preferences without explicitly modeling the latter. We operationalize Theory-of-Mind (ToM) inference as a control signal within Frictive Policy Optimization by extracting, at each referring expression, a four-part belief structure: the speaker's intended referent, the addressee's interpretation, and each participant's model of the other's belief. This makes friction mechanically computable from epistemic-state comparisons, capturing \emph{silent divergence}, where both participants proceed confidently while grounding to different referents. We evaluate the signal at two levels. At the representation level, ablating the second-order channel reduces misunderstanding recall from $65\%$ to $26\%$. At the policy level, reward-shaping (FAR) and trust-region (FTR) variants improve intervention F1 and warranted-context calibration over DPO, with Brier scores independently supporting the calibration gains. Across three training runs, FAR and FTR remain substantially more stable, whereas DPO varies widely and can degrade intervention competence already present in the base policy. Thus, ToM-grounded friction provides a trainable signal for context-sensitive intervention under referential belief divergence.

1 Introduction

The paper distinguishes surface coordination from epistemic alignment and extends Frictive Policy Optimization with ToM-grounded friction to expose latent referential divergence.

  • Standard turn-level evaluation can mistake acknowledgments and smooth task progression for alignment despite incompatible referential interpretations.
  • The proposed point-in-time ToM representation tracks intended, interpreted, and higher-order beliefs for each referring expression.
  • Second-order belief structure improves recovery of misunderstandings, including silent intent fixing.
  • Figure 1 presents extraction, human auditing, deterministic friction computation, and conditioning of FPO objectives.
  • FAR and FTR improve intervention behavior, warranted-context calibration, and training stability relative to DPO.

2 Background

Prior work frames alignment through turn-level preferences, grounding, and ToM, but motivates explicit representations of evolving belief relations in collaborative interaction.

  • RLHF, DPO, and GRPO optimize turn-level preference signals without explicitly representing evolving epistemic states.
  • Collaborative participants may maintain divergent interpretations while behaving as though mutual understanding has been achieved.
  • Surface transcripts alone are insufficient for tracking shared beliefs in collaborative tasks.
  • ToM distinguishes first-order beliefs from beliefs about others’ beliefs, making silent divergence representable.
  • Existing interactive benchmarks evaluate belief attribution and reasoning, whereas this work focuses on converting epistemic divergence into a trainable signal.

3 Data

The study builds friction-aware FPO instances from perspectivist HCRC MapTask annotations, preserving explicit grounding statuses and controlled misunderstanding rates.

  • HCRC MapTask pairs mismatched giver and follower maps in a navigation task with fixed dialogue roles and varying speaker–addressee assignments.
  • Each referring expression yields dialogue history, participant beliefs, and a PENDING, MISUNDERSTOOD, or ALIGNED grounding status.
  • 99 training dialogues contain 2,705 referring expressions, including 147 misunderstood cases, while 24 test dialogues contain 761 expressions and 69 misunderstandings.
  • The split preserves misunderstanding rates of 5.5% for training and 9.1% for testing.
  • After policy-adaptation filtering, 663 training and 278 held-out evaluation instances remain.

4 Theory-of-Mind-Grounded Friction Construction

The paper converts point-in-time first- and second-order belief comparisons, auxiliary dialogue signals, and referential configurations into deterministic productive and unproductive friction signals.

  • 4.1 Point-in-Time ToM Extraction: Second-order identifiers are inferred from dialogue, map, and dialogue-act context, making silent divergence detectable through cross-party comparison.
  • 4.1 Point-in-Time ToM Extraction: The four-identifier schema records intended referent, interpreted referent, and each participant’s belief about the other’s interpretation.
  • 4.1 Point-in-Time ToM Extraction: UPTAKE_QUALITY captures addressee response confidence, with withholding mapped to the strongest unresolved-uncertainty signal.
  • 4.1 Point-in-Time ToM Extraction: Existence queries are excluded from training because discovering asymmetry through questioning is treated as successful information exchange.
  • 4.1 Point-in-Time ToM Extraction: The five canonical configurations include Aligned as non-frictive and silent intent fixing as divergence masked by mutual presumptions of agreement.
  • 4.2 Friction Computation: The friction module combines uncertainty, surfaced repair, propagation risk, and higher-order belief conflict rather than simple agreement polarity.
  • 4.2 Friction Computation: The friction weights are hand-set, emphasize propagation risk and higher-order conflict, and are not claimed to be optimal.

5 Friction Signal Validation

The four-identifier schema yields usable belief and misunderstanding signals, with second-order structure materially supporting recovery and friction separation across train and test partitions.

  • Schema and misunderstanding recovery: 98.5% train parse success (2665/2705) and 100% test parse success (761/761) establish reliable four-identifier schema extraction.Remaining train failures are primarily API truncations from long dialogues and are filtered before downstream training.
  • Belief accuracy: 35.9–43.5% speaker ToM and 41.4–42.0% addressee ToM consistency are soft signals because the corpus lacks direct second-order annotations.A stratified blind manual annotation on 60 samples independently assesses second-order prediction reliability, with results indicating substantial agreement with human judgments.
  • Schema and misunderstanding recovery: 67.6% train recall (98/145) and 65.2% test recall (45/69) show the schema recovers misunderstanding labels.These results improve by more than 60 points over the cascade-based alternatives discussed in Appendix B.
  • Friction signal: Mean F− increases from 0.185 to 0.375 on train (∆= 0.190) and from 0.193 to 0.390 on test (∆= 0.197) from pending to misunderstood contexts.The consistent ordering supports using F− as a monotonic risk signal for downstream trainer adaptation.
  • Detection channels and ToM contribution: Removing the ToM-dependent channel reduces recall from 67.6% to 23.4% on train and from 65.2% to 26.1% on test.Removing multiplicity fallback as well reduces recall to 1.4% and 0%, respectively, indicating non-redundant channels.

6 FPO Trainer Adaptation

The adaptation study applies DPO and three FPO variants to participant-agent MapTask policies, using observable belief-state inputs while keeping full cross-party ToM annotations offline.

  • Training setup: The participant-agent policy observes dialogue history and its own map-derived information, then chooses whether to answer or intervene.Full cross-party ToM annotations construct friction supervision offline but are never direct policy inputs.
  • Training setup: Each training instance pairs a referring expression with its immediate addressee response and includes a compact observable belief-state preamble.Repair is preferred over acknowledgment in divergent contexts, while unnecessary intervention is dispreferred in aligned contexts.
  • Trainer variants: DPO uses β = 0.1 with a frozen reference policy and receives no friction signal.The baseline is compared with FAR, FPP, and FTR FPO variants adapted to offline MapTask data.
  • Trainer variants: FAR uses friction-derived risk for reward shaping, encouraging intervention at high epistemic risk and penalizing unnecessary intervention cost.Its risk is instantiated from the friction-derived epistemic state.
  • Trainer variants: FPP converts friction into pairwise preference supervision by requiring y+ to have greater productive friction F+ than y−.It retains the same underlying policy backbone as the DPO-style objective.
  • Trainer variants: FTR uses epistemic risk to vary trust-region distance from the base model, permitting larger updates in high-friction contexts and stronger anchoring in low-friction contexts.The friction-dependent coefficient is defined as β(h) = 1 ϵ0 + κF−(h).

7 Evaluation

The evaluation tests whether ToM-grounded friction improves intervention, calibration, and stability, and separates gains from friction-aware supervision, objective conditioning, and second-order belief structure.

  • Evaluation design: Rule-based intervention labels and held-out warrant annotations make the principal results independent of LLM judging, which is treated as secondary.Pairwise judge differences become inconclusive after response-length control.
  • Main results: FAR and FTR improve intervention F1 over DPO while reducing warranted-context miscalibration.ECE falls from 0.432 to 0.227 for FAR and 0.178 for FTR; Brier score decreases from 0.411 to 0.281 and 0.293, respectively.
  • Training stability: Across three independent runs, DPO is substantially more variable than FAR and FTR.DPO’s F1 ranges from 0.152 to 0.398 with standard deviation 0.106, compared with 0.023 for FAR and 0.009 for FTR.
  • Main results: FPP achieves superficially high F1 by intervening on every test instance, producing aligned-context ECE of 1.0 and strongly negative judge preference.Its perfect warranted-context scores follow mechanically from perfect recall, while the pairwise objective lacks an absolute signal for unnecessary intervention.
  • Sources of the improvement: The staged controls raise intervention F1 from 0.282 for DPO-FB to 0.344 for DPO and 0.416–0.417 for FAR/FTR.Warranted-context Brier error decreases from 0.532 to 0.411 and then to 0.281–0.293, separating friction-informed supervision from direct per-instance conditioning.
  • Sources of the improvement: Full ToM-grounded friction improves intervention F1 beyond lower-order friction by 13.0% for FAR and 4.3% for FTR.The lower-order variants retain the same objectives, data, and surface and first-order channels; calibration effects are mixed rather than uniformly improved.
  • Sources of the improvement: The signal-level ablation reduces misunderstanding recall from approximately 65% to 26% when the second-order channel is removed.At the policy level, the second-order contribution is interpreted as improved intervention behavior rather than a general calibration advantage.
  • Inference controls: The observable belief preamble supports substantial base-policy intervention competence, while masking reduces F1 for the base model, FAR, and FTR.The untrained backbone reaches F1 0.410, ICL reaches 0.490–0.510, and masking reduces F1 by 0.060, 0.073, and 0.068, respectively.

8 Conclusion

The paper makes epistemic misalignment computationally accessible by representing participants’ first- and second-order beliefs and using their divergence to guide intervention. FAR and FTR improve intervention behavior, calibration, and training stability over DPO, while the approach remains bounded by controlled data, offline supervision, and unverified transfer.

  • Conclusion: The four-identifier ToM schema tracks what each participant intends, understands, and believes about the other’s interpretation, exposing latent misalignment.The representation converts epistemic asymmetries into a friction signal for collaborative dialogue.
  • Conclusion: FAR and FTR improve intervention behavior and warranted-context calibration relative to DPO while substantially increasing training stability.The gains depend in part on second-order information rather than generic friction conditioning alone.
  • Conclusion: Intervention is particularly valuable when locally coherent interaction masks incompatible interpretations that could otherwise persist.The paper frames alignment as regulation of evolving belief relations beyond preference optimization over utterances.
  • Limitations: The experiments use the controlled HCRC MapTask corpus, whose generalization to open-domain or multi-party dialogue remains untested.Its misunderstanding rate of 5.5–9.1% may vary across collaborative corpora.
  • Limitations: Second-order labels are inferred by GPT-5 rather than established as ground-truth mental states, and ablations cannot rule out reliance on correlated cues.A blind human pilot supports agreement with referential judgments but does not establish genuine mentalizing.
  • Limitations: Friction is computed offline, so fully online estimation, extended-interaction error propagation, and rollout-based variants remain unevaluated.The policy instead receives dialogue history and a compact agent-observable belief projection during training and standard inference.
  • Limitations: Transfer of the friction formulation beyond the studied objectives and intervention space has not been established.Other settings would require defining productive friction for their corresponding action spaces, and tested constant robustness does not establish optimality.

Potential Risks

The paper identifies dual-use and calibration risks in a ToM-derived friction signal. It emphasizes that inferred mental states and deterministic scalar mappings can affect downstream interventions, especially in high-stakes settings.

  • Dual-use risk: The friction signal could be used to detect and strategically exploit belief asymmetries rather than repair them.The framework is instead engineered for transparency, verification, and mutual repair.
  • Inference risk: GPT-5-based second-order inference may reflect demographic or stylistic biases, producing systematic ToM attribution errors.Such errors could miscalibrate friction, penalize appropriate progress, or incentivize redundant interventions.
  • Mitigation: High-stakes adaptations should maintain strict human-in-the-loop oversight to audit calibration during continuous interaction.
  • Design choices: The F − scalar mappings are deterministic surrogates normalized to [0, 1], not parameters fitted to evaluation data.
  • Design choices: VALCONF encodes second-order belief conflict, while UNC, CONTR, and HAZ capture uptake uncertainty, surfaced repair or contradiction, and map-level propagation risk.

A.2 Sensitivity to Design-Choice Constants

The sensitivity analysis tests whether friction rankings and downstream method comparisons depend on hand-specified constants or representation choices. The reported ordering remains stable under weight and mapping perturbations, while preliminary alternatives motivate the final four-identifier schema.

  • Constant perturbations: The analysis perturbs component weights and scalar mappings by ±20% and recomputes friction ordering and FTR coefficients over 278 held-out instances.The weights are hand-specified as w=(0.15,0.20,0.30,0.35).
  • Constant perturbations: The perturbations preserve structurally informative configuration ordering and the gap between silent_intent_fixing and aligned_deep.The FTR risk contrast remains stable, and F + perturbations do not reverse inter-method ordering.
  • Weight comparison: The proposed weighting increases cross-class separation by 11.9% relative to the uniform alternative, 0.463 versus 0.414.
  • Interpretation: The constants are treated as robust structural surrogates rather than optimal fitted values.Uniform weighting preserves relevant ordering, and tested perturbations leave qualitative downstream conclusions unchanged.
  • Preliminary alternatives: Local-context belief prediction performed below a constant-True baseline, so latent divergence often required later repair, contradiction, or task-failure evidence.
  • Preliminary alternatives: Retrospective grounding improved contextual access but remained insufficient because intra-party binary grounding cannot represent two participants confidently selecting different referents.
  • Final representation: The final representation attributes beliefs at reference time while using retrospective evidence offline and separates first- from second-order beliefs across participants.
  • Human annotation: The human pilot evaluated GPT-5 second-order predictions on 60 held-out referring expressions covering all eight epistemic configurations.Its purpose was alignment with human-interpretable referential judgments, not establishing schema-wide inter-annotator agreement.

D.3 FTR Implementation

FTR estimates sequence-level KL with a non-negative k3 estimator and directly sets risk from F−(h), using ε0 = 1.0 and κ = 3.0. The resulting weighting remains approximately [0.36, 1.0], while tighter regimes increased calibration error in pilot runs.

  • KL estimation: The single-completion offline setting uses the non-negative k3 estimator because exact sequence-level KL is unavailable.Signed and squared log-ratio estimators did not preserve the desired asymmetry for decreases in gold-completion probability.
  • KL estimation: The policy/reference likelihood ratio r defines the estimator’s comparison quantity.
  • Risk weighting: FTR sets Risk(h) directly to F−(h) rather than fitting a separate risk predictor, with ε0 = 1.0 and κ = 3.0.
  • Risk weighting: Approximately [0.36, 1.0] is the resulting range over the observed friction range.Tighter regimes over-anchored the policy in pilot runs and increased calibration error.

E.1 Per-Seed Stability

Across three independent QLoRA initializations, FAR and FTR consistently outperform DPO in same-seed comparisons and show greater training stability. DPO varies substantially across seeds, while FPP’s core metrics are fixed by construction.

  • Per-seed comparisons: All six same-seed FAR and FTR comparisons against DPO are significant under McNemar tests.The reported p-value is p ≤ 3 × 10−3.
  • Per-seed breakdown: FPP’s F1, ECE, and ECEwarr are identical across seeds by construction, but its win rate versus DPO varies as DPO’s completions change.
  • Per-seed comparisons: +14.4–+42.5 percentage points for FAR and +13.3–+36.7 for FTR are the net margins against DPO.
  • Training stability: DPO’s F1 ranges from 0.152 to 0.398, with standard deviation 0.106, while FAR and FTR have F1 standard deviations of 0.023 and 0.009.Warranted-context ECE reaches 0.646 in DPO’s weakest run.
  • Training stability: The cross-run evidence is interpreted primarily as greater training stability because only three seeds make aggregate seed-paired tests underpowered.
Loading 2608.30719v1…