PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Knows What - and What Does It Support? Perspective-Bounded Evidence Calibration for Multimodal Theory of Mind

Ziqi Liao, Jiaqi Shen, Yiming Na, Kedi Lyu, Shao-Yuan Lo, Hongxia Xie

arXiv:2610.36824v1cs.AI

TL;DR

Multimodal ToM asks a solver to infer an agent’s mental state from evidence available to that agent, while existing approaches leave the role of agent-admissible evidence in hypothesis evaluation under-specified. PIP-ToM constructs agent-specific evidence stores and structured mental-state claims to calibrate model-based likelihoods. With GPT-4o, it outperforms AutoToM on MuMA-ToM and reports 95.00% average accuracy on both MMToM-QA and Hi-ToM.

  • Problem

    Existing approaches leave under-specified which agent-accessible evidence may legitimately affect a mental-state hypothesis.

    Source: Introduction · p. 1

  • Method

    PIP-ToM builds agent-specific evidence stores, aligns accessible evidence with structured mental-state claims, and calibrates hypothesis likelihoods while preserving model-based inference.

    Source: Introduction · p. 2, Structured Mental-State Schema · pp. 3–4

  • Results

    With GPT-4o, PIP-ToM outperforms AutoToM on MuMA-ToM (92.56% vs. 81.44%) and achieves 95.00% average accuracy on MMToM-QA and 95.00% on Hi-ToM.

    Source: Performance across Tasks and Reasoning… · p. 5

  • Takeaways & Limitations

    Evidence accessibility and claim-specific relevance address complementary sources of error, while structured evidence matching alone remains insufficient for goal and recursive mental-state inference.

    Source: Introduction · p. 2, Conclusion · p. 7

    Structured evidence matching alone remains insufficient for goal and recursive mental-state inference.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Theory of Mind (ToM), the ability to infer others' beliefs, goals, and intentions, is a cornerstone of social intelligence. Multimodal ToM requires inferring an agent's mental state from evidence that is available to that agent, rather than from the complete episode observed by the solver. Existing LLM-based and Bayesian model-based approaches reconstruct global event traces, yet they do not explicitly distinguish objective events from agent-admissible evidence during hypothesis evaluation. Consequently, agent-inaccessible events or semantically misaligned actions and utterances may influence the inferred belief, goal, or social intention, leading to perspective leakage and source or route confusion. We formulate this limitation as an evidence-admissibility problem: evidence should contribute to a mental-state hypothesis only when it is accessible to the relevant agent, preserves its information source, and is diagnostic of the queried mental-state variable. To operationalize this principle, we propose PIP-ToM, an evidence-admissibility interface that constructs agent-specific evidence stores, aligns them with structured mental-state claims, and calibrates model-based hypothesis scores while preserving the underlying Bayesian inference process. PIP-ToM with GPT-4o outperforms existing ToM methods across MuMA-ToM, MMToM-QA, and Hi-ToM, achieving average accuracies of 92.56%, 95.00%, and 95.00%, respectively.

Introduction

Multimodal ToM requires inferring an agent’s mental state from evidence available to that agent, not simply reconstructing the objective episode. The paper frames evidence admissibility as a constraint on hypothesis evaluation and introduces PIP-ToM to operationalize it.

Introduction

  • To infer beliefs, goals, and communicative motives, a solver must determine what evidence was available to the queried agent and what mental-state hypotheses it can support.Relevant evidence may span visual observations, object interactions, actions, communication, and memory.Source: Introduction · p. 1
  • Jessica’s utterance must be interpreted alongside her belief: she knows the magazine is on the table but tells Kevin it is in the bedroom cabinet.Inferring her social goal requires recovering her belief when speaking, preserving the utterance as a communicative act, and considering its intended effect on Kevin’s search.Source: Introduction · p. 1
  • The interface between agent-relative evidence reconstruction and hypothesis scoring remains under-specified, including whether evidence is accessible and relevant to the queried mental-state variable.The paper calls for evidence to be accessible to the perspective holder, source-preserving, and diagnostic of the queried variable.Source: Introduction · pp. 1–2
  • PIP-ToM builds agent-specific perspective stores, parses candidates into structured mental-state claims, and routes accessible evidence to calibrate hypothesis likelihoods.The paper’s contributions also report consistent improvements across three benchmarks and multiple backbones.Source: Introduction · p. 2

Related Work

Prior work reconstructs global state, derives agent-relative perspectives, and infers candidate mental states, but leaves under-specified which accessible evidence may affect a given hypothesis.

Explicit State Tracking

  • Explicit state-tracking methods represent narratives as event-driven transitions, entity-state and spatial knowledge, or predefined action domains with access predicates.These structured global trajectories provide a basis for agent-specific knowledge inference.Source: Explicit State Tracking · p. 2

Perspective Reconstruction

  • Perspective-reconstruction methods extract agent-observable information or track evolving beliefs and higher-order mental states over time.Examples include separating perception extraction from belief inference and representing multiple characters’ beliefs in graphs.Source: Perspective Reconstruction · p. 2

Bayesian and Model-Based Mental-State Inference

  • Bayesian and model-based approaches infer latent mental states from agent-relative representations, but the interface remains unclear: which accessible evidence may affect each hypothesis likelihood?The paper positions evidence admissibility as an explicit modeling problem between perspective reconstruction and inference.Source: Bayesian and Model-Based Mental-State… · p. 2

Methodology

PIP-ToM separates evidence ownership from evidential role, then calibrates agent-accessible evidence against structured mental-state claims while retaining the underlying model’s inference process.

Overview of PIP-ToM

  • PIP-ToM builds agent-specific perspective stores, parses candidate answers, and applies route-specific evidence calibration to local scores without redesigning the latent-state model.Its interface separates which evidence an agent can access from how that evidence may support a mental-state hypothesis.Source: Overview of PIP-ToM · p. 3

Agent-Specific Perspective Storage

  • Perspective stores retain evidence accessible to each agent, including observations, actions, utterances, and source-aware memory across timesteps.They organize evidence rather than assert oracle mental states, reducing reliance on objective world-state facts when forming agent-centered scores.Source: Agent-Specific Perspective Storage · p. 3

Structured Mental-State Schema

  • PIP-ToM parses a shared query frame and candidate schemas, then selects complementary evidence routes according to each claim’s semantics.The schema distinguishes the mental-state type, perspective holder, target when applicable, queried entity, relation or polarity, candidate value, and optional behavioral or social effect.Source: Structured Mental-State Schema · pp. 3–4

Evidence-Calibrated Bayesian Inverse Planning

Benchmarks

Evaluation covers MuMA-ToM, MMToM-QA, and Hi-ToM, with task settings, dataset sizes, and model configurations described in the supplement.

Baselines

The comparison includes general-purpose MLLMs, ToM-specific prompting, state-tracking, and model-based inference methods, using matched fused video-text event traces for PIP-ToM and AutoToM.

  • Baselines span general-purpose MLLMs, ToM-specific prompting methods, state-tracking methods, and model-based inference systems.The listed systems include GPT-4o, GPT-5.4, Qwen3-VL-8B, InternVL3.5-8B, SimToM, ThoughtTracing, DWM, PDDL-Mind, BIP-ALM, LIMP, and AutoToM.Source: Baselines · p. 4
  • PIP-ToM uses the same fused video-text event traces as AutoToM for a fair comparison (Zhang et al. 2025b).Source: Baselines · p. 4
  • Table 1 reports accuracy across MuMA-ToM, MMToM-QA, and Hi-ToM, marking the best and second-best non-human results.BoG denotes Belief of Goal; a dash means a result is unavailable, unreported, or not applicable.Source: Table 1 · p. 5
  • Figure 3 presents fine-grained accuracy across MMToM-QA belief and goal subtasks and Hi-ToM reasoning orders 0–4.Source: Figure 3 · p. 5

Overall Performance

With GPT-4o, PIP-ToM exceeds AutoToM on all three reported benchmarks; the comparison also reports gains across open-source backbones and a favorable accuracy-cost balance.

Overall Performance

Accuracy-Compute Trade-off

  • PIP-ToM generally achieves higher accuracy at comparable or lower token costs under matched backbones across the three benchmarks.Figure 4 compares average accuracy and inference-token consumption; task structure and reasoning depth may partly explain differences.Source: Accuracy-Compute Trade-off · p. 5, Figure 4 · p. 6

Analysis by Research Questions

The analyses test how perspective boundaries, claim-aware evidence calibration, and direct matching contribute to ToM inference. They find that the first two address distinct evidence errors, while direct matching is less reliable for latent mental states.

RQ1: Perspective-Bounded Evidence

  • Removing Perspective Storage lowers average accuracy most on perspective-divergent tasks; 62.8% of 121 Full-only audited cases involved reliance on inaccessible or global evidence.Performance changed little on true-belief and order-0 tasks, where agent-relative and global information largely align.Source: RQ1: Perspective-Bounded Evidence · p. 6

RQ2: Claim-Aware Evidence Calibration

  • Removing claim-aware calibration produces the largest overall ablation decline, including MuMA-ToM Social Goal accuracy from 84.33% to 67.67% and MMToM-QA Goal from 90.33% to 70.50%.The analysis concludes that evidence accessibility alone is insufficient for goal and social reasoning.Source: RQ2: Claim-Aware Evidence Calibration · pp. 6–7
  • With GPT-5.4, rescaling active factors across α = 0.75, 1.0, and 1.25 yields mean absolute deviations of 1.06 percentage points on MuMA-ToM and 1.09 on MMToM-QA.The reported deviations compare weak and strong settings with the reference configuration.Source: RQ2: Claim-Aware Evidence Calibration · p. 7

RQ3: Matching–Inference Complementarity

  • Direct matching reaches 100.00% accuracy on covered subsets of both belief tasks, showing that explicit-belief questions can often match candidates to accessible observations or memories.The takeaway distinguishes these covered explicit-belief cases from latent and nested mental states, which require model-based inference.Source: RQ3: Matching–Inference Complementarity · p. 7
  • For latent goals, direct accuracy is 65.95% on MMToM-QA Goal versus 84.35% for Bayesian fallback, and 76.00% on MuMA-ToM Belief-of-Goal versus 96.67% for full PIP-ToM.The direct-posterior variant normalizes structured matching scores when available and otherwise falls back to uncalibrated Bayesian inference.Source: RQ3: Matching–Inference Complementarity · p. 7

Qualitative Case Studies

The cases distinguish two complementary constraints: perspective boundaries determine what evidence an agent can access, while claim-aware routing determines which mental-state variable that evidence can support.

Qualitative Case Studies

  • Perspective boundaries exclude inaccessible global facts, whereas claim-aware routing prevents accessible evidence from being used for an unsupported mental-state claim.The two cases illustrate complementary rather than interchangeable constraints on inference.Source: Figure 6 · p. 7, Qualitative Case Studies · p. 7
  • The resulting posterior updates use agent-accessible evidence relevant to the queried mental state, rather than relying on additional context.The paper attributes gains to an interface between global episode reconstruction and model-based latent-state inference.Source: Qualitative Case Studies · p. 7

Conclusion

PIP-ToM calibrates model-based multimodal ToM inference using evidence available to the queried agent and relevant to the mental-state hypothesis. Across three benchmarks, the paper reports consistent improvements, while its analyses find that evidence matching alone remains insufficient for goal and recursive mental-state inference.

Conclusion

  • PIP-ToM determines which evidence was accessible to the queried agent, how it supports a mental-state hypothesis, and calibrates likelihoods while preserving model-based inference.The framework is presented as perspective-bounded evidence calibration for multimodal ToM.Source: Conclusion · p. 7
  • Experiments on MuMA-ToM, MMToM-QA, and Hi-ToM show consistent improvements across closed- and open-source backbones.Source: Conclusion · p. 7
  • The analyses find that accessibility and claim-specific relevance address complementary errors, while structured evidence matching alone remains insufficient for goal and recursive mental-state inference.The paper concludes that ToM reasoning requires tracking agents' access to information and how it should influence inferred mental states.Source: Conclusion · p. 7
Loading 2610.36824v1…