Source-linked AI summary
Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, Yulia Tsvetkov
TL;DR
Large language models still lack basic ToM capabilities, and the implicit nature of ToM motivates approaches beyond explicit supervision. SymbolicToM uses nested symbolic belief graphs at inference time; on ToMi it improves GPT-3-Davinci accuracy by 38 absolute points to 92% and is reported more robust than supervised methods on perturbed and out-of-domain inputs.
Problem
Large language models still lack reliable theory-of-mind skills, while the phenomenon is largely implicit in their training data and difficult to learn.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The results show promise for augmenting neural language models with symbolic knowledge to improve social reasoning.
Source: §7 · p. 9The method assumes stories are chronological and may propagate errors from off-the-shelf graph-construction models.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Theory of Mind (ToM)$\unicode{x2014}$the ability to reason about the mental states of other people$\unicode{x2014}$is a key element of our social intelligence. Yet, despite their ever more impressive performance, large-scale neural language models still lack basic theory of mind capabilities out-of-the-box. We posit that simply scaling up models will not imbue them with theory of mind due to the inherently symbolic and implicit nature of the phenomenon, and instead investigate an alternative: can we design a decoding-time algorithm that enhances theory of mind of off-the-shelf neural language models without explicit supervision? We present SymbolicToM, a plug-and-play approach to reason about the belief states of multiple characters in reading comprehension tasks via explicit symbolic representation. More concretely, our approach tracks each entity's beliefs, their estimation of other entities' beliefs, and higher-order levels of reasoning, all through graphical representations, allowing for more precise and interpretable reasoning than previous approaches. Empirical results on the well-known ToMi benchmark (Le et al., 2019) demonstrate that SymbolicToM dramatically enhances off-the-shelf neural networks' theory of mind in a zero-shot setting while showing robust out-of-distribution performance compared to supervised baselines. Our work also reveals spurious patterns in existing theory of mind benchmarks, emphasizing the importance of out-of-distribution evaluation and methods that do not overfit a particular dataset.
1 Introduction
Theory of Mind (ToM)—reasoning about others’ mental states—is important for social intelligence and understanding stories, yet large language models still lack reliable ToM skills. SymbolicToM adds explicit, nested belief graphs at inference time and improves ToM performance and robustness.
1 Introduction
- ToM supports social intelligence and communication, and readers use first- and second-order reasoning to model characters’ mental states in stories.First-order reasoning models Bob’s mental state; second-order reasoning models Bob’s estimate of Alice’s mental state.Source: §1 · p. 1
- Large language models lack reliable ToM skills, while existing ToM reading-comprehension datasets are simplistic, lack diversity, and leave supervised models brittle under slight out-of-distribution perturbations.The authors argue that implicit ToM knowledge is difficult for models to learn from training data alone.Source: §1 · pp. 1–2
- SymbolicToM augments language models with explicit symbolic graphs of each character’s beliefs, using inference-time subtasks and off-the-shelf models rather than training.Its graph set represents nested estimates of what one character thinks another believes, and supports reasoning at user-specified depths.Source: §1 · p. 2
- 38 absolute points: GPT-3-Davinci’s ToMi accuracy rises to 92% averaged across question types with SymbolicToM.The authors also report greater robustness than supervised approaches on diverse story structures and sentence paraphrases.Source: §1 · p. 2
2 Motivation and Background
Language models can answer false-belief questions incorrectly, and current ToM reading-comprehension datasets are limited or prone to spurious patterns. The paper motivates an inference-time, unsupervised alternative and outlines SymbolicToM’s graph-based pipeline.
2 Motivation and Background
- In the illustrated false-belief story, ChatGPT and GPT3 answer that Bob expects Alice to search in the celery’s new location.The paper notes that these exact responses may vary because the models are nondeterministic and continuously changing.Source: §2 · p. 2
- Templated narratives cover limited interactions and simple discourse, human-generated data faces annotation-cost barriers, and text-based games often model mental states for intent or character style.The authors instead focus on differences in knowledge and beliefs among multiple characters.Source: §2 · p. 2
- ToM-bAbI and ToMi are the only available reading-comprehension datasets identified by the authors, and their templated nature makes supervised training prone to overfitting spurious artifacts.ToMi adds flexible sentence ordering and distractors, but the authors say it still has these pitfalls.Source: §2 · p. 2
- SymbolicToM constructs character belief graphs from a passage, retrieves relevant sentences, and queries an off-the-shelf language model zero-shot without training.The authors argue for inference-time or unsupervised approaches that combine neural models and symbolic algorithms because ToM is implicit and naturally available data is limited.Source: Figure 2 · p. 3, §2 · p. 3
3 Methods
SymbolicToM builds nested belief-state graphs from stories, updates them as characters witness events, and retrieves the relevant graph and sentences to answer questions. Its memory requirements grow exponentially with the maximum reasoning depth, which the experiments set to two.
3.1 SYMBOLICTOM: Algorithm Overview
- Belief graphs encode entities and their story relations, with a separate graph for each nested perspective and a global graph for the true world state.The method limits represented beliefs to the current world state, assumes repeated self-belief does not change the state, and permits any belief state to be represented at the selected maximum depth.Source: §3.1 · p. 3
- For question answering, SymbolicToM heuristically identifies the relevant characters, rephrases the question about the world state, retrieves their graph, and queries an LLM zero-shot using the represented story sentences.The method optionally filters retrieved sentences based on entities in the question, though an ablation found this step can often be skipped.Source: §3.1 · p. 4
- Stories are processed sequentially: the method updates the global context and identifies witnesses, then updates only the local belief contexts of characters who witnessed the described actions.Unchanged local contexts remain untouched when a character did not witness the actions.Source: §3.2 · p. 4
3.2.1 Detecting Witnesses, Updating Graphs, and Propagating Knowledge
- Global graph updates detect contradictions, add nodes and edges extracted from the resulting state, identify witnesses in connected components, and remove invalid edges.The method removes contradicted edges after finding witnesses so that witness detection can use the pre-removal graph.Source: §3.2.1 · pp. 4–5
- For each witness, local contexts gain the edges in that character’s connected component of the global graph and lose outdated edges absent from it.This propagation represents implicit knowledge gain, such as learning about visible people and objects upon entering a room.Source: §3.2.1 · p. 5
3.3 Notes on Memory Efficiency
- Memory requirements grow exponentially with maximum ToM order; all experiments use m = 2, the maximum order evaluated by current datasets.Processing questions first could reduce memory use by computing only the graphs required for target queries.Source: §3.3 · p. 5
4 Fundamental Issues in Existing ToM Datasets
The paper identifies template-related ambiguities and label issues in ToMi that can make questions depend on physical commonsense or require extra reasoning beyond ToM. The authors modify stories to disambiguate container locations and correct some mislabeled second-order questions.
4 Fundamental Issues in Existing ToM Datasets
- ToMi follows the Sally-Anne setup, with questions testing first- or second-order true or false beliefs as well as reality and memory.Its six fixed sentence primitives are combined into a finite set of story orderings.Source: §4 · p. 5
- Physical commonsense can shift human answers despite ToMi’s fixed label: the caption reports 80% accuracy for hat and box, but 20% for apple and pantry.In both cases, ToMi’s correct label is that Ethan thinks the object is in the basket.Source: Table 1 · p. 6, §4 · p. 6
- Ambiguous object locations can require three-hop reasoning and assumptions about characters’ movement, complicating the task beyond theory of mind.The authors give the example of inferring that a basket is in the front yard so that Ethan saw the object there.Source: §4 · p. 6
- The authors add a sentence locating each container after relevant primitives to disambiguate stories, and correct some mislabeled second-order questions.The passage attributes the mislabeled-question issue to prior reports by Arodi and Cheung (2021) and Sap et al. (2022).Source: §4 · p. 6
5 Experiments
The experiments compare off-the-shelf language models with and without SYMBOLICTOM on ToMi and evaluate robustness to changes in story structure and wording. SYMBOLICTOM improves ToM-question performance and is more robust than supervised models on the tested out-of-distribution variations.
5 Experiments
- The evaluation compares several base LMs out of the box and with SYMBOLICTOM, alongside supervised TTT and GPT3-Curie baselines.GPT3-Curie, fine-tuned on 6,000 ToMi examples for one epoch, reaches 98.5% average accuracy; TTT reaches 92.3%.Source: §5 · p. 6
5.1 In-Domain Evaluation
- SYMBOLICTOM yields gains on ToMi’s theory-of-mind questions, including +62 points for Flan-T5-XL first-order false-belief questions and +78 points for GPT3.5 second-order false-belief questions.All models maintain near-perfect performance on memory questions with or without SYMBOLICTOM; supervised models show high accuracy across question types.Source: §5.1 · pp. 6–7, Figure 4 · p. 7
5.2 Story Structure Robustness Test Sets
- Supervised models overfit ToMi’s original story structures, whereas SYMBOLICTOM-equipped models have high accuracy on modified test sets, especially larger models.The modified sets include concatenated false-belief stories, three-character stories, and multiple object movements across four containers.Source: §5.2 · pp. 7–8, Table 2 · p. 7
- On the third-order ToM setting, SYMBOLICTOM consistently enhances off-the-shelf LLMs and outperforms supervised methods.The authors describe this as an initial test of third-order reasoning in LLMs; a comprehensive benchmark is outside the paper’s scope.Source: §5.2 · p. 8
5.3 Paraphrasing Robustness Evaluation
- TTT’s average accuracy drops 54 points on ParaphrasedToMi, while SYMBOLICTOM-equipped models outperform TTT across all theory-of-mind questions.Finetuned GPT3 also loses 40 average-accuracy points on false-belief questions, while remaining robust on other question types.Source: §5.3 · p. 8, Figure 5 · p. 8
6 Related Work
Related work covers decomposed reasoning, structured representations of changing worlds, and theory of mind beyond NLP. The paper distinguishes its approach by tracking multiple characters’ beliefs and updating them as stories progress.
Existing Approaches
- SYMBOLICTOM decomposes higher-level reasoning into graph-reasoning subtasks, while also updating facts and beliefs as a story progresses.The authors note that story facts change over time, are not universally accessible to characters, and require explicit commonsense assumptions such as object permanence.Source: §6 · p. 9
- Structured world-state approaches could update a world graph, but the authors say they do not address multiple-belief representation or propagation to witnesses’ graphs.Some approaches are explicitly described as unable to model second-order ToM.Source: §6 · p. 9
ToM beyond NLP
- The work focuses on cognitive ToM but says recursive belief representation could also apply to affective ToM and queries of any reasoning order.The authors note that, to their knowledge, no dataset requires beyond second-order ToM.Source: §6 · p. 9
7 Conclusions
SymbolicToM uses nested symbolic belief states without training to improve language models’ theory-of-mind reading comprehension. The conclusion highlights robustness to story perturbations and out-of-domain inputs, alongside the need for broader future research.
7 Conclusions
- Theory-of-mind reasoning supports meaningful, contextualized reading-comprehension answers and is relevant to tutoring, dialogue, personalization, and negotiation.Stories often use theory-of-mind reasoning to create conflict.Source: §7 · p. 9
- SymbolicToM improves base language models’ reading-comprehension accuracy, especially on false-belief scenarios, without training or fine-tuning.It represents nested belief states explicitly.Source: §7 · p. 9
- SymbolicToM is highly robust to story perturbations and out-of-domain inputs, where supervised methods suffer significant degradations.The authors leave extensions to other forms of social intelligence and more diverse interaction datasets for future work.Source: §7 · p. 9
Limitations
The approach assumes chronological stories and relies on off-the-shelf models to build and update belief graphs, which may propagate errors. Available datasets also constrain evaluation to Sally-Anne-style scenarios and a simplified assumption about who witnesses events.
Limitations
- SYMBOLICTOM assumes chronological stories, and errors from WANLI and OpenIE graph construction or updates may propagate.The authors suggest more sophisticated models as a possible mitigation but did not test them because of budget constraints.Source: Limitations · pp. 9–10
- The authors created a more challenging version of available ToM datasets for out-of-domain testing, alongside a corrected ToMi version..Source: Limitations · p. 9
- Available theory-of-mind datasets describe Sally-Anne tests and assume anyone located at a place witnesses its events, an assumption that can fail in realistic settings.The authors propose improving witness detection and obtaining datasets with more interaction types and physical commonsense.Source: Limitations · p. 10
Ethics Statement
The ethics discussion focuses on potential harms from errors, misuse, and biased predictions, while noting that experiments use prototypical characters and physical-object reasoning. The appendix clarifies that the global graph represents observable true-world state, with verbal information stored locally when heard.
Ethics Statement
- Errors about characters’ mental states, misuse on private situations, and predictions reinforcing social biases are ethical concerns; the experiments use prototypical characters, not actual people.The paper also notes that physical-object location reasoning is less prone to ethical concerns, but can still produce biased decisions.Source: Ethics Statement · p. 10
- The authors argue that risks merit consideration even though current machine theory-of-mind research remains far from human capabilities..Source: Ethics Statement · p. 10
A.1 Detailed Description of Information Contained in Global Context G
- Global context G stores observable true-world locations, while verbally conveyed information belongs in listeners’ local contexts; this distinction is not relevant to ToMi because available datasets do not test verbal interactions..Source: §A.1 · p. 13
A.2 Prompts for Resulting State Extraction
The appendix describes prompts and model choices for extracting resulting states, and reports how language-model size and paraphrase robustness affect this subtask. It also notes GPT-3’s one-shot generalization from first- to second-order questions.
A.2 Prompts for Resulting State Extraction
- GPT-3-Curie receives two examples, then a sentence, and is prompted to continue with the resulting state after the action.The examples show job quitting and contract signing as state changes.Source: §A.2 · p. 13
- GPT-3-Davinci, Flan-T5-XL, GPT-3.5, and GPT-4 can answer the resulting-state subtask zero-shot, while smaller models benefit from few-shot prompts.Macaw was not queried; GPT-3-Curie was used instead as a comparable-size model.Source: §A.2 · p. 13
- GPT-3 is significantly more robust to paraphrases than Flan-T5, whose resulting-state detection falters on florid paraphrases.Flan-T5 handles the original phrasings straightforwardly.Source: §A.2 · p. 13
- GPT-3.5 and GPT-4 perform the task well zero-shot, while LLaMA models need fewer demonstrations than Flan-T5; the main experiments use GPT-3..Source: §A.2 · p. 13
A.3 Solving PROCESSQUESTION using GPT3
- The authors’ explorations suggest GPT-3 can extract entities and rephrase the question for the PROCESSQUESTION subtask..Source: §A.3 · p. 13
- Figure 6 presents GPT-3’s one-shot generalization from first-order to second-order questions..Source: Figure 6 · p. 13
B Details on Out-Of-Domain Evaluation
The appendix documents linguistic paraphrases and alternative story templates used for out-of-domain evaluation. It also records when the expanded GPT-3.5 and GPT-4 experiments were added.
B.1.1 All Paraphrases of PersonX entered the RoomY.
- The appendix lists alternative phrasings for entering and exiting rooms, alongside a table counting paraphrases per original sentence template.The GPT-3.5 and GPT-4 experiments were added in May 2023.Source: §B.1.1 · pp. 13–14, Table 3 · p. 13, §B.1.2 · p. 14
B.1.3 All Paraphrases of The Object1 is in the Container1.
- Object-state paraphrases vary both how container contents are described and how a character's movement of an object is worded.The object-state list includes container-centered formulations; the movement list includes verbs such as “stowed,” “hid,” and “transported.”Source: §B.1.3 · p. 14, §B.1.4 · pp. 14–15
B.1.5 All Paraphrases of PersonX is in the RoomY.
- Room-presence statements include alternatives such as “situated in” and “shelters.”The list recasts the same person-in-room relation in both person-centered and room-centered forms.Source: §B.1.5 · p. 15
B.1.6 All Paraphrases of positive distractor sentences
- Positive distractor paraphrases express attraction to an object through varied descriptions of enthusiasm, fascination, or affection.The examples include both adjective-based descriptions and statements of strong attachment.Source: §B.1.6 · p. 15
B.1.7 All Paraphrases of positive negative sentences (PersonX hates ObjectY)
- Negative distractor paraphrases express dislike through varied reactions, including aversion, distress, and disgust.Some formulations make the object the subject of the negative reaction.Source: §B.1.7 · pp. 15–16
B.2.1 Double Room False-Belief Episode
- The double-room false-belief template moves two characters and objects across two rooms, with object locations changing as characters leave.Source: §B.2.1 · p. 16
B.2.2 Three Active Characters Story
- The three-active-character template has one character move an object twice, with a different character making the second move.Source: §B.2.2 · p. 16
B.2.3 True-Belief Interaction, Falsified by Unwitnessed Third-Person Story
- Other templates pair a true-belief interaction with an unwitnessed third-person move, and a story involving four containers and multiple object movements.Source: §B.2.3 · p. 16, §B.2.4 · p. 16
C.1 Ablating FILTERBASEDONQUESTION from SYMBOLICTOM
The appendix tests the question-based story filter, evaluates second-order representations on third-order questions, and varies which language model implements resulting-state reasoning. It also supplies detailed result tables and figure conventions.
C.1 Ablating FILTERBASEDONQUESTION from SYMBOLICTOM
- SYMBOLICTOM improves theory-of-mind question performance across models with or without the final filter, while the filter is needed to beat the base model on those questions only for Flan-T5-XXL.Without the filter, GPT4+SYMBOLICTOM improves on all four ToM question types and maintains Reality and Memory performance; Flan-T5-XL improves on all four ToM types, with smaller Reality and Memory declines than under the full algorithm.Source: §C.1 · p. 17
C.2 Third-Order Theory of Mind Evaluation
- Using k = 2 representations significantly outperforms supervised baselines and dramatically improves over off-the-shelf models on the third-order evaluation.The authors hypothesize that second-order representations help models avoid attending to story events inaccessible to relevant characters.Source: Table 4 · p. 17, §C.2 · p. 17
C.3 Alternative RESULTINGSTATE Implementations
- With careful prompting, Flan-T5-XL and XXL use nine and eleven shots, while LLaMA-7B and 13B use six and seven; LLaMA accuracy exceeds GPT3 in these explorations.The figure compares ParaphrasedToMi results with out-of-the-box models and supervised-model performance lines.Source: Figure 8 · p. 18, §C.3 · p. 18
C.4 Detailed Result Tables
- Tables 5–7 report performance per model and question, with and without the final filter and the differences between them; Figures 5–7 show accuracy ratios and the corresponding filter comparison.Source: §C.4 · p. 18, Table 5 · p. 19, Table 6 · p. 19
C.1 Ablating FILTERBASEDONQUESTION from SYMBOLICTOM
- +7, +3.5, +12.8, and +15 points in average accuracy gain for Macaw-3B, GPT3, Flan-T5-XXL, and LLaMA-7B; filter effects were negative for Flan-T5-XL and GPT4.The respective negative effects were -5.3 and -4 points of accuracy on average; the filter keeps story edges with an endpoint mentioned in the question.Source: §C.1 · p. 16, Table 7 · p. 19