PaperlayerView exact revision on arXiv

Source-linked AI summary

MMToM-QA: Multimodal Theory of Mind Question Answering

Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum, Tianmin Shu

arXiv:2401.08743v2cs.AIcs.CLcs.CVcs.LG

TL;DR

Existing ToM benchmarks use either video or text, motivating a benchmark that tests mental-state inference across both modalities. The paper introduces MMToM-QA and BIP-ALM, which combines symbolic representations with Bayesian inverse planning and language models. Existing models fail to match human performance, while BIP-ALM significantly outperforms them; the benchmark remains limited to household object-search videos.

  • Problem

    Existing ToM benchmarks use video or text separately, leaving multimodal inference about people's mental states unevaluated.

    Source: §1 · p. 1

  • Method

    MMToM-QA tests goals and beliefs from video and text, while BIP-ALM uses unified symbolic representations and language models to support Bayesian inverse planning.

    Source: §1 · p. 2

  • Results

    Existing models make large systematic errors and fail to match human performance, while BIP-ALM significantly outperforms them.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The comparison indicates that large language and multimodal models still lack robust ToM capacity, while BIP-ALM shows promising results.

    Source: §1 · p. 2, §6 · p. 9

    MMToM-QA includes only videos of people looking for objects in household environments.

    Source: §6 · p. 9

Abstract

from arXiv · show

Theory of Mind (ToM), the ability to understand people's mental states, is an essential ingredient for developing machines with human-level social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. However, existing ToM benchmarks use unimodal datasets - either video or text. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person's mind based on conceptual representations (e.g., goals, beliefs, plans) extracted from any available data. To address this, we introduce a multimodal Theory of Mind question answering (MMToM-QA) benchmark. MMToM-QA comprehensively evaluates machine ToM both on multimodal data and on different kinds of unimodal data about a person's activity in a household environment. To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models). BIP-ALM extracts unified representations from multimodal data and utilizes language models for scalable Bayesian inverse planning. We conducted a systematic comparison of human performance, BIP-ALM, and state-of-the-art models, including GPT-4. The experiments demonstrate that large language models and large multimodal models still lack robust ToM capacity. BIP-ALM, on the other hand, shows promising results, by leveraging the power of both model-based mental inference and language models.

1 Introduction

The paper introduces MMToM-QA to test Theory of Mind (ToM) across video, text, and fused inputs, and proposes BIP-ALM for multimodal mental-state inference. Existing models make systematic errors and fail to match human performance, while BIP-ALM significantly outperforms them.

1 Introduction

  • Earlier ToM benchmarks use video or text separately, while ToM reasoning can infer mental states from either modality or fuse them into one coherent scene.The paper frames this multimodal reasoning as a stronger test for current machine-learning models, particularly LLMs.Source: §1 · pp. 1–2
  • MMToM-QA evaluates inference about goals and beliefs from video and text, with some questions requiring information fusion across both modalities.The benchmark includes questions at different points in videos, each with two options, one more likely given the observations.Source: §1 · p. 2, Figure 1 · p. 2
  • BIP-ALM extracts symbolic scene and action representations from video and text, then uses language models to evaluate hypotheses within Bayesian inverse planning.The method combines Bayesian inverse planning's robustness with language models' scalability and open-endedness.Source: §1 · p. 2
  • Existing models, including GPT-4(V), make large systematic errors and fail to match human performance, while BIP-ALM significantly outperforms them.The benchmark also provides a human baseline.Source: §1 · p. 2
  • The paper contributes a multimodal ToM benchmark, a method combining Bayesian inverse planning and language models, and a systematic comparison with human performance.The comparison includes state-of-the-art text and multimodal QA models.Source: §1 · pp. 2–3

2 Related Work

Prior ToM evaluations largely use either visual or textual inputs, and machine-ToM methods have mostly focused on unimodal data and simple domains. MMToM-QA and BIP-ALM extend this work to multimodal inference in complex scenarios.

2 Related Work

  • Visual ToM benchmarks use animations, while text-based benchmarks adapt false-belief tests; MMToM-QA instead tests goal and belief inference from multimodal data and long everyday activities.ToM questions require mental-state inference, unlike visual question answering that does not.Source: §2 · p. 3
  • Existing multimodal QA benchmarks test retrieval of relevant information, but the cited work identifies no multimodal QA benchmark for ToM.ToM differs from multimodal information retrieval because it concerns people's mental states.Source: §2 · p. 3
  • Machine-ToM approaches include end-to-end neural networks and model-based Bayesian inverse planning, but both have mainly addressed unimodal data and simple domains.Recent LLM studies report apparent ToM abilities, while systematic evaluations find these abilities less robust than humans and vulnerable to test variants [Ullman, 2023].Source: §2 · p. 3
  • BIP-ALM builds on end-to-end and model-based approaches by extending Bayesian inverse planning to multimodal fusion and complex scenarios with language models.The related-work discussion positions the method as combining strengths of these approaches.Source: §2 · p. 3

3 MMToM-QA Benchmark

MMToM-QA comprises household-activity videos paired with text and questions about goals and beliefs. Its question types probe joint mental-state inference, and its evaluation compares video, text, and multimodal input conditions in a zero-shot setting.

3.1 Overview

  • The benchmark contains 134 household videos and 600 two-choice questions about goals and beliefs, paired with RGB-D clips and text descriptions; its seven types split evenly between belief and goal questions.Videos average 1,462 frames and 36 actions; the training set includes 1,000 procedurally synthesized videos with annotations.Source: §3.1 · p. 3

3.2 Question Types

  • Seven question types test belief and goal inference, including belief tracking, goals conditioned on belief, belief updating, and reasoning about possible future actions.Questions may require joint inference of goal and belief rather than treating each independently.Source: §3.2 · pp. 4–5

3.3 Procedural Generation

  • The benchmark is generated from procedurally synthesized household videos, with question types and opposing hypotheses sampled from each type's definition and text questions created from ground-truth states, actions, and hypotheses.Video generation uses the VirtualHome-Social simulator; GPT-4 generates text descriptions and questions.Source: §3.3 · p. 5

3.4 Evaluation Protocol

  • Evaluation uses multimodal, text-only, and video-only conditions, with all models tested zero-shot and without example QAs during training.The training data is intended to support generalization about mental-state updates and behavior in physical environments.Source: §3.4 · p. 5

4 The BIP-ALM Model

BIP-ALM extends Bayesian inverse planning with unified symbolic representations of multimodal scenes and actions, then uses a language model to support inverse symbolic planning. The method compares goal-belief hypotheses using observed behavior.

4 The BIP-ALM Model

  • BIP-ALM extends Bayesian inverse planning with unified multimodal representations and a finetuned language model for efficient inverse symbolic planning.It builds on BIP, a cognitively grounded ToM method originally designed for visual data.Source: §4 · p. 5

4.1 Unified Symbolic Representations

  • Video and text inputs are translated into symbolic scene and action representations, aligned and fused into a unified event representation for mental-state inference.The planner uses the unified representation to compare the question’s two goal-belief hypotheses.Source: §4 · p. 5, §4.1 · p. 6

4.2 Inverse Symbolic Planner

  • The planner models behavior with a POMDP and uses observed states and actions to infer an agent’s goal and belief, assuming deterministic state transitions.It estimates past beliefs from observable state predicates and assumes a uniform initial belief.Source: §4.2 · pp. 6–7
  • A language model amortizes the computationally costly policy likelihood by scoring observed actions from symbolic states, goals, and estimated beliefs.The resulting likelihoods incorporate the current action, belief transition, and prior actions when comparing hypotheses.Source: §4.2 · p. 7

5 Experiments

BIP-ALM outperforms the baselines by a large margin, while human accuracy is highest with multimodal input and falls with unimodal input. Baseline models remain unreliable on harder questions, and BIP-ALM generalizes to unseen environments and human behavior.

5.2 Results

  • 93% accuracy across question types was achieved by humans in the multimodal condition, while accuracy fell with unimodal input and video-only questions were harder than text-only.Text-based true-belief/false-belief performance remained unchanged when input was unimodal.Source: §5.2 · p. 8
  • Most baselines were close to random guessing, while GPT-4(V) and SymbolicToM performed well on some types but still struggled with false-belief and harder questions.GPT-4(V) confused belief with the true world state and struggled with goal inference.Source: §5.2 · p. 8, Table 1 · p. 8
  • BIP-ALM models outperform all baselines by a large margin, and even without finetuning small pretrained LMs beat larger pretrained LMs used alone.The method supports ToM reasoning from unimodal or multimodal data through unified representations.Source: §5.2 · p. 8
  • Few-shot and chain-of-thought prompting did not consistently improve baselines, and finetuned Video-LLaMA 2 remained no better than chance overall.Finetuning moderately improved a few simpler question types.Source: §5.2 · p. 8
  • On a test set with 40 videos and 120 questions from unseen apartments and new human participants, BIP-ALM generalized to real human behavior and unseen physical environments.Three participants controlled the avatar to reach assigned goals, providing human belief updates and actions.Source: §5.2 · p. 9

6 Discussion & Conclusion

The discussion presents MMToM-QA as the first multimodal machine ToM benchmark and reports that current language and multimodal models struggle with mental-state reasoning. BIP-ALM performs better through model-based inference, while the authors identify limitations in its symbolic planning and in the benchmark’s scope.

6 Discussion & Conclusion

  • MMToM-QA is presented as the first multimodal benchmark for machine ToM, alongside a systematic evaluation of humans, state-of-the-art methods, and BIP-ALM.The authors propose extending its object-search household videos to more diverse scenarios and adding desires, emotions, and constraints.Source: §6 · p. 9
  • Video supplies dynamic changes and movement cues, while text can add world-state information obscured by limited camera views or occlusion.The combined information supports reasoning about expected action plans, observations, and false beliefs.Source: §6 · p. 9
  • GPT-4 and GPT-4V retrieve true-world information well but struggle with mental-state reasoning, especially goal judgments and changes over time.Goal inference was not evaluated in existing text-based benchmarks.Source: §6 · p. 9
  • BIP-ALM’s model-based inference performs better on the main test set and generalizes to real human behavior in unseen environments.The method imagines possible actions under hypothetical mental states and state contexts using language models.Source: §6 · p. 9
  • BIP-ALM’s symbolic planning cannot represent continuous spatial information needed to imagine a person’s future path in Type 2.4.The authors also note that language models sometimes estimate action likelihoods inaccurately when their planning fails.Source: §6 · p. 9

Ethics Statement

The authors connect ToM evaluation to the development of human-centered AI and describe steps taken to address benchmark diversity and validation.

Ethics Statement

  • The authors frame understanding human mental states as a foundation for AI systems that interact safely and cooperatively with people.They connect benchmarking machine ToM to assessing whether current models adequately understand humans.Source: Ethics Statement · p. 9
  • The authors report making efforts to increase the diversity of avatars used to generate benchmark videos.They also report validating the benchmark in a human experiment.Source: Ethics Statement · p. 9
  • The authors invite community feedback to further improve the benchmark.Source: Ethics Statement · p. 9

A Comparison of Theory of Mind Benchmarks

The comparison positions MMToM-QA as distinct from prior ToM benchmarks because it offers multimodal inputs and asks about both goals and beliefs.

A Comparison of Theory of Mind Benchmarks

B.1 More Quantitative Results

The additional results report gains from finetuning language models within BIP-ALM, but find that other tested baselines generally remain at or below chance. The study also evaluates generalization to real human behavior in unseen environments.

B.1 More Quantitative Results

  • Finetuning GPT-J and LLaMA 2 significantly boosts BIP-ALM, while pretrained versions already outperform larger models including GPT-4.The authors attribute this comparison to the advantage of inverse symbolic planning for multimodal ToM reasoning.Source: §B.1 · p. 12
  • Open-source baselines perform no better than chance across model sizes, and few-shot or chain-of-thought prompting yields no meaningful improvement for almost all baselines.Finetuned Video-LLaMA 2 improves on a few simpler types but remains no better than chance overall.Source: §B.1 · p. 12
  • The authors evaluate model generalization on a human test set featuring real behaviors in unseen environments.Source: §B.1 · p. 12

B.2 Qualitative Results

The examples illustrate how BIP-ALM uses language-model action likelihoods to infer mental states from observed behavior. Combining video and text can provide a fuller state for predicting a person’s next action.

B.2 Qualitative Results

  • Video supplies proximity while text supplies the cupcake’s location; their fused state predicts opening the microwave, the ground-truth next action.The video-only and text-only states each lack one of these cues.Source: §B.2 · pp. 13–14
  • The ablation table distinguishes pretrained-language-model variants (“w/o FT”) and the multimodal condition (“MM”).Source: Table 3 · p. 14
  • BIP-ALM uses action likelihoods to infer Elizabeth’s false belief and revise Karen’s goal hypotheses as new actions are observed.The examples infer that Elizabeth believes a glass is in an empty microwave and use Karen closing the oven to disfavor the plate goal.Source: §B.2 · pp. 12–13, Figure 5 · p. 14

B.3 Discussion on SimToM and SymbolicToM

The discussion contrasts text-based ToM methods with MMToM-QA’s more complex activity setting and describes the benchmark’s human study, data, statistics, and construction. It also outlines how GPT-4 was used to improve the wording of descriptions.

B.3 Discussion on SimToM and SymbolicToM

  • SimToM filters context by what a character knows and marginally improves GPT-4, but text-only methods cannot answer video-only or multimodal questions.Source: §B.3 · pp. 14, 15, 17
  • SymbolicToM’s GPT-4 accuracy is 63% on MMToM-QA versus 100% on ToMi, and remains below BIP-ALM.The discussion contrasts ToMi’s short sequences and simple scenes with MMToM-QA’s long activities, complex environments, and varying inference uncertainty.Source: §B.3 · pp. 15–16

B.4 More Details About the Human Experiment

  • Each question was answered by five participants; 180 people were recruited, and the study was IRB-approved with anonymized data.Participants had a mean age of 29.7; 65 were female, and pay was $12 per hour.Source: §B.4 · p. 17

B.5 Available Data

  • The benchmark provides RGB-D images, instance segmentation maps, human 3D poses, ground-truth scene graphs and actions, and camera data.Source: §B.5 · p. 17

B.6 Benchmark Statistics

  • MMToM-QA averages 1,595 text tokens and 1,902 video frames per question, with longer contexts than existing ToM benchmarks.The paper notes that these lengths demand information retrieval and fusion capabilities.Source: §B.6 · p. 17

B.7 Details of the Procedural Generation

  • The benchmark samples apartments, goals, and initial states, plans actions in a POMDP, and renders videos with ground-truth annotations.An ideal observer filters goal-belief hypotheses against observations and simulated actions; questions pair a surviving hypothesis with one ruled out by the model.Source: §B.7 · p. 18
  • Question generation selects at most one question per video moment and samples remaining questions to balance question types.Source: §B.7 · p. 18

B.8 Utilizing GPT-4 for Enhanced Text Generation

  • GPT-4 refines template-generated state and action descriptions to improve phrasing and diversify expression without changing their meaning.Source: §B.8 · p. 19

C BIP-ALM Implementation Details

Appendix C details how BIP-ALM converts visual and textual inputs into symbolic representations, combines them, and uses belief-conditioned action likelihoods for inference. It also describes the perception, parsing, and training choices used in the implementation.

C.1 Visual Perception

  • Visual perception builds voxel maps and 3D object boxes from images, then uses the boxes and human poses to construct scene graphs.Current experiments use ground-truth instance segmentation maps and 3D human poses; graph edges represent containment and proximity.Source: §C.1 · pp. 19–20

C.2 Text Parsing

  • GPT-4 parses questions into inference type, candidate objects, and relevant conditions, while separate prompts parse action descriptions into sequential actions.The question parser distinguishes belief inference from goal inference.Source: §C.2 · pp. 20–21
  • State parsing converts apartment descriptions into statements about room contents and object locations.The prompt specifies a sentence pattern and preserves original line breaks.Source: §C.2 · p. 20

C.3 Representation Fusion

  • Video-only and text-only predicates are fused into a unified symbolic representation and full state sequence, with text retained when modalities conflict.The stated rationale is that visual predicates are more likely to be erroneous from noisy perception.Source: §C.3 · p. 21

C.4 Belief Representation and Update

  • Beliefs are represented per object as possible locations with nonzero probabilities, and are updated at each step using the estimated observation.A negated predicate is represented by removing that location from the possible-location list.Source: §C.4 · pp. 21–22

C.5 Prompt for Language Models in Inverse Symbolic Planner

  • GPT-J or LLaMA 2 estimates the likelihood of the person’s last action given a hypothetical belief and goal, and competing belief hypotheses are compared through those likelihoods.For belief inference, the hypothesis whose excluded location yields the lowest action likelihood among alternatives indicates the more likely belief.Source: §C.5 · p. 22

C.6 Training Details

  • GPT-J and LLaMA 2 are finetuned on 20,000 state-belief-goal-to-action samples drawn from 1,000 training videos, without example question-answer pairs.Training uses LoRA for five epochs and takes about 20 GPU hours on one A100 GPU.Source: §C.6 · pp. 22–23

D Implementation Details of Baselines

The appendix describes baseline model settings and gives household examples of the benchmark’s belief- and goal-inference questions. The examples pair apartment contents and activity descriptions with questions about what a person believes or is trying to obtain.

D Implementation Details of Baselines

  • The multimodal baselines use sampled video frames, and Video-LLaMA 2 is finetuned on video descriptions generated from training annotations.Frame counts vary by model; the passage also specifies the baseline language-model versions.Source: D Implementation Details of Baselines · p. 23

E.1 Type 1.1 Example

  • Elizabeth’s example asks whether she believes a bottle of wine is in the fridge after searching several cabinets and approaching it.The apartment description places wine in both a cabinet and the fridge.Source: §E.1 · p. 23

E.2 Type 1.2 Example

  • Jennifer’s example asks whether she believes a cupcake is in the cabinet she is about to open.The apartment description places cupcakes in the microwave and not the cabinet in question.Source: §E.2 · p. 24

E.3 Type 1.3 Example

  • Charles’s example asks whether he believes a salmon is in the fridge after checking two cabinets and approaching the dishwasher.The apartment description places salmon in the oven, not the fridge.Source: §E.3 · p. 24

E.4 Type 2.1 Example

  • James’s example asks which object he has been trying to get—wine or an apple—after opening the stove, fridge, and microwave.The apartment description places apples in the fridge and wine in the living-room cabinet.Source: §E.4 · p. 25

E.5 Type 2.2 Example

  • Mark’s example asks whether he is trying to get a water glass or cupcake, given his stated belief that the seventh cabinet lacks a water glass.His activity description has him approach that cabinet.Source: §E.5 · p. 25

E.6 Type 2.3 Example

  • Mary’s example asks whether she is seeking chips or a condiment bottle after checking several cabinets and the dishwasher.The apartment description places chips in the first cabinet and a condiment bottle in the microwave.Source: §E.6 · p. 26

E.7 Type 2.4 Example

  • William’s example asks whether he is trying to get a wine glass or dish bowl after opening the first cabinet and approaching the fifth.The apartment description places a wine glass in the first cabinet and a dish bowl in the dishwasher.Source: §E.7 · p. 26
Loading 2401.08743v2…