Source-linked AI summary
Let's Put Ourselves in Sally's Shoes: Shoes-of-Others Prefilling Improves Theory of Mind in Large Language Models
Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Yoshihiro Yamazaki, Keita Suzuki, Hiroaki Sugiyama, Kuniko Saito
TL;DR
LLMs still lag behind humans on Theory of Mind benchmarks, while fine-tuning on ToM datasets can degrade out-of-distribution generalization. This paper introduces Shoes-of-Others prefilling, a character-specific output prefix, and reports improved ToM across five mental-state categories in conversational and narrative contexts without world-state changes.
Problem
LLMs’ ToM performance remains below humans on recent benchmarks, and fine-tuning on ToM datasets can overfit and degrade out-of-distribution generalization.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The analysis suggests SoO mitigates unfaithful thoughts, while identifying other crucial factors for ToM remains future work.
Source: §5 · p. 5, §4 · p. 5The method was not evaluated with proprietary LLMs and cannot be applied to OpenAI models such as GPT-4 without support for specifying an output prefix.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Recent studies have shown that Theory of Mind (ToM) in large language models (LLMs) has not reached human-level performance yet. Since fine-tuning LLMs on ToM datasets often degrades their generalization, several inference-time methods have been proposed to enhance ToM in LLMs. However, existing inference-time methods for ToM are specialized for inferring beliefs from contexts involving changes in the world state. In this study, we present a new inference-time method for ToM, Shoes-of-Others (SoO) prefilling, which makes fewer assumptions about contexts and is applicable to broader scenarios. SoO prefilling simply specifies the beginning of LLM outputs with ``Let's put ourselves in A's shoes.'', where A denotes the target character's name. We evaluate SoO prefilling on two benchmarks that assess ToM in conversational and narrative contexts without changes in the world state and find that it consistently improves ToM across five categories of mental states. Our analysis suggests that SoO prefilling elicits faithful thoughts, thereby improving the ToM performance.
1 Introduction
Theory of Mind (ToM)—inferring others’ mental states—is important for LLM use in social situations, but LLMs still lag behind humans on ToM benchmarks. The paper introduces Shoes-of-Others prefilling as a broader-scope inference-time approach and reports improved performance across five mental-state categories.
1 Introduction
- LLMs’ ToM performance remains below human performance in recently proposed benchmarks.ToM is relevant to uses such as email autocomplete, empathic conversation, and persuasion.Source: §1 · p. 1
- Fine-tuning on ToM datasets can improve in-distribution generalization but induce overfitting and degrade out-of-distribution generalization.Source: §1 · p. 1
- Existing inference-time methods target beliefs in contexts with world-state changes, limiting their applicability to broader scenarios.The paper argues that methods focused on beliefs in specific scenarios are insufficient for practical, human-like ToM.Source: §1 · pp. 1–2
- SoO prefilling specifies the beginning of LLM outputs to initiate reasoning and makes fewer assumptions about context.The paper presents it as applicable to a broader range of settings.Source: §1 · p. 2
- SoO prefilling improves ToM across five mental-state categories in conversations and narratives without world-state changes, outperforming other methods.The categories are beliefs, intentions, desires, emotions, and knowledge; the analysis also suggests it encourages more faithful thoughts than “Let’s think step-by-step.”Source: §1 · p. 2
2 Shoes-of-Others Prefilling
Shoes-of-Others prefilling starts an LLM output with a character-specific perspective cue, then lets the model continue. The method is designed to guide thought across contexts without relying on assumptions that restrict it to changing-world scenarios.
2 Shoes-of-Others Prefilling
- SoO prefilling begins the output with “Let’s put ourselves in {name}’s shoes,” filling the name with the character whose mental state is queried.The model then generates the continuation; names are extracted from questions using a rule-based method detailed in Appendix B.2.Source: §2 · p. 2
- SoO guides the LLM’s thought process through explicit perspective-taking, while making fewer context assumptions than belief-filtering methods.The paper contrasts it with filtering information a character does not know, which is applicable only to contexts involving world-state changes.Source: §2 · p. 2
- Unlike traditional prompt instructions, prefilling supplies the initial portion of the output and lets the model generate its continuation.The proposed strategy uses this format to prompt the model explicitly to adopt another’s perspective.Source: §2 · p. 2
3 Experiments
Across conversational and narrative benchmarks, SoO prefilling improved first- and second-order ToM across mental-state categories, while prompting methods were not always effective. The experiments used multiple models and compared five inference-time methods.
3.1 Experimental Setups
- The experiments evaluated accuracy on ToMATO and ToMBench, using five methods: Vanilla, CoT prompting, SoO prompting, CoT prefilling, and SoO prefilling.The benchmarks cover conversational and narrative contexts, respectively, without changes to the world state; proprietary models were evaluated only with prompting methods because they do not support output-prefix specification.Source: §3.1 · p. 2
- Table 1 reports first-order ToM accuracies on both benchmarks, averaged over three runs, with results organized by mental state and model.Best scores among the five inference-time methods are bolded, and scores degraded versus Vanilla are red.Source: Table 1 · p. 3
3.2 Results
- Accuracy and faithfulness are positively correlated on both benchmarks for Llama-3-8B-Instruct.Figure 2 presents this correlation analysis.Source: Figure 2 · p. 4
- SoO prefilling consistently improved first-order ToM across five mental states on ToMATO and ToMBench, with some exceptions.Prompting methods were not always effective and tended to score below Vanilla.Source: Table 1 · p. 3, §3.2 · p. 4
- SoO prefilling outperformed other methods on average for Llama3 70B in second-order ToM, consistently improved scores, and was more effective on false-belief tasks than true-belief subsets.Prompting methods often degraded scores compared with Vanilla for proprietary and open-weight models.Source: Table 2 · p. 3, §3.2 · p. 4
- Including the target character’s name in SoO prefilling consistently helped on both benchmarks in the Llama-3-8B-Instruct ablation.The authors state that explicitly specifying names is necessary to guide ToM reasoning properly.Source: Table 3 · p. 3, §3.2 · p. 4
4 Analysis
The analysis relates SoO’s accuracy gains to more faithful thoughts, while finding that gains do not necessarily track longer thoughts. The authors interpret these findings as evidence that SoO mitigates thought unfaithfulness without relying solely on increased test-time compute.
Faithfulness and accuracy
- Faithfulness win rates were positively correlated with accuracy increases from CoT to SoO prefilling on both benchmarks, with a similar trend in other models.The analysis used GPT-4o mini as an LLM judge and compared examples where Llama3 8B was correct under both methods.Source: §4 · pp. 4–5
Thought length and accuracy
- Thought-length increases from CoT to SoO prefilling were not necessarily positively correlated with accuracy increases, suggesting gains do not rely solely on longer thoughts.The authors frame longer thoughts as scaling test-time compute.Source: §4 · p. 5
5 Discussion & Conclusion
The paper concludes that SoO prefilling improves first- and second-order ToM across five mental states in most cases and may mitigate unfaithful thoughts. It also relates explicit perspective-taking prompts for LLMs to findings from ASD research.
Discussion & Conclusion
- Across two benchmarks, SoO prefilling improved first- and second-order ToM across five mental states in most cases, and the analysis suggests it mitigated thought unfaithfulness.Identifying other crucial factors for ToM is left for future work.Source: §5 · p. 5
- The authors relate LLM gains from explicit perspective-taking to ASD research reporting comparable social-cognition performance when adolescents with ASD are explicitly encouraged to take perspectives.They also note a parallel between explicit reasoning outperforming vanilla prompting and findings on explicit versus implicit ToM in ASD research.Source: §5 · p. 5
6 Limitations
The method’s evaluation and performance gains have limits tied to model access, model and data choices, and the analysis judge.
Limitations
- SoO prefilling was not evaluated with proprietary LLMs that lack output-prefix specification, and its gains depend on models and data.The authors also caution that LLM-as-a-judge may have position and verbosity biases that undermine the reliability of analysis insights.Source: §6 · p. 5
7 Ethical Considerations
The paper flags potential harms from AI systems that predict or act on people’s mental states. The appendices also specify prompts for benchmark answering and faithfulness judging.
7 Ethical Considerations
- AI ToM could enable deceptive persuasion, intrusive profiling, or disadvantageous decisions when systems misinterpret mental states or users over-rely on them.The discussion calls for care to avoid these harms.Source: §7 · p. 5
A.1 Prompts for Evaluation
- The ToMATO and ToMBench evaluation prompts ask for the most probable answer to a question from options and require an A–D response.ToMATO prompts use a transcript; ToMBench prompts use a context.Source: §A.1 · p. 7
A.2 Prompts for LLM-as-a-judge
- The LLM-as-a-judge setup randomly assigns the two assistants’ positions to mitigate position bias.The cited bias source is Zheng et al. (2023).Source: §A.2 · p. 7
B.1 Experimental Setups
The experiments use open-weight and proprietary models, with quantization for the open-weight models and English-only ToMBench questions. Target-character names are extracted differently across the two benchmarks, and failed extractions are excluded.
B.1 Experimental Setups
- The experiments use three open-weight LLMs quantized to 4-bit to reduce computational costs, alongside proprietary gpt-3.5-turbo-0125 and gpt-4o-mini-2024-07-18.For ToMBench, they filter out two-option questions and use only its English subset.Source: §B.1 · p. 8
B.2 Name Extraction
- ToMATO target-character names are determined from its rule-based questions, while ToMBench names are extracted from manually written questions using spaCy parsing.Questions are excluded when this rule-based extraction fails; LLM-based extraction is left for future work.Source: §B.2 · p. 8
B.3 Robustness to Personality Traits
SoO prefilling improves first- and second-order ToM performance across character personality traits in ToMATO, without overfitting to specific personalities.
B.3 Robustness to Personality Traits
- SoO prefilling consistently improves first- and second-order ToM performance for each Big Five personality factor, without overfitting to specific personalities.Tables 5 and 6 report results for high and low subsets of each trait, averaged over three runs.Source: §B.3 · p. 8, Table 5 · p. 9, Table 6 · p. 9
C Analysis
The analyses examine thought length, word choice, faithfulness, and pairwise faithfulness judgments. They suggest that SoO’s accuracy gains are associated with faithful thoughts, not simply longer thoughts, while acknowledging exceptions and other possible contributing factors.
C.1 Thought Length
- SoO thoughts were generally longer than CoT thoughts on ToMATO but shorter on ToMBench, even as SoO improved performance on both benchmarks.The token counts use the Llama-3-8B-Instruct tokenizer.Source: §C.1 · p. 10, Table 7 · p. 10, Figure 4 · p. 10
C.2 Word-level Correlation Analysis
- The word-level analysis found many words significantly more or less frequent under SoO than chance rates, indicating different word sets in SoO and CoT thoughts.The analysis uses word frequencies, p(y|xi), and a z-statistics procedure.Source: §C.2 · p. 10, Figure 5 · p. 11
C.3.1 Accuracy vs. Faithfulness
- Faithfulness win rates were positively associated with accuracy increases from CoT to SoO, except for some outliers such as ToMATO’s Intention (2nd, FB) subset.The pattern was reported for Llama-3-8B-Instruct and also observed in other models.Source: §C.3.1 · p. 10, Figure 6 · p. 11, Figure 7 · p. 11
C.3.2 Accuracy vs. Thought Length
- Accuracy increases from CoT to SoO were not necessarily positively correlated with thought-length increases, supporting that gains do not rely solely on longer thoughts or scaled test-time compute.Figures 8 and 9 show the relationship for Mistral-7B-Instruct-v0.3 and Llama-3-70B-Instruct.Source: §C.3.2 · p. 10, Figure 8 · p. 11, Figure 9 · p. 12
C.3.3 Faithfulness vs. Thought Length
- Thought lengths were not positively correlated with faithfulness win rates in most cases, suggesting the judge did not simply favor longer outputs.The authors also infer that shorter thoughts are more faithful in ToM tasks.Source: §C.3.3 · p. 10, Figure 10 · p. 12, Figure 11 · p. 12
C.4 Pairwise Comparison w.r.t. Faithfulness
- SoO prefilling’s faithfulness win rate exceeded its loss rate in most cases, but losses were larger in some cases, leaving other factors as possible contributors.Exceptions included Llama3 70B on ToMBench and Llama3 8B on ToMATO’s Intention (2nd, TB) subset.Source: §C.4 · p. 13, Table 8 · p. 13, Table 9 · p. 13
D Examples
- The examples include a CoT case whose thought claimed Ethan thought Liam was concerned despite predicting the wrong answer about Liam feeling empathetic.The example illustrates the paper’s description of unfaithful thoughts.Source: D Examples · p. 13, Table 11 · p. 14