Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States

• Existing LLM ToM evaluations predominantly test static snapshots in isolated scenarios, overlooking how mental states evolve across connected interactions.Source: §1 · p. 1
• Tracking changing user mental states is presented as relevant to helping users in support conversations.Source: §1 · p. 2
• DYNTOM evaluates temporal mental-state tracking through a four-step process: constructing social contexts, designing trajectories, generating scenarios, and formulating questions, with human validation for realism and quality.Source: §1 · p. 2
• 44.7%: across ten representative LLMs, average performance lagged humans, with the gap widening when models tracked mental-state changes across scenarios.Source: §1 · p. 2
• The benchmark includes 1,100 social contexts, 5,500 scenarios, and 78,100 questions, and its contributions include analysis of challenges in tracking state changes.Source: §1 · p. 2

2.1 ToM Benchmarks

• False-belief tests found LMs performed below humans, while other work identified shortcuts and spurious correlations in LLM ToM responses.Source: §2.1 · p. 2
• Earlier ToM evaluations cover false belief, emotion, intention, perception, and multimodal settings, but most omit continuous evolution across connected scenarios.Source: §2.1 · p. 2

2.2 Human Behavior Simulation

• DYNTOM uses LLM-generated dialogue reflecting predetermined character states and human evaluation of social-context authenticity and question validity.Source: §2.2 · p. 2

3.1 DYNTOM Construction Framework

• Social contexts combine a location, two character profiles, and their relationship; annotators retained 92% of generated profiles and relationships as realistic.Source: §3.1 · p. 3
• DYNTOM models beliefs, emotions, and intentions and their resulting actions across five scenarios, using relationships in which beliefs influence emotions, states shape intentions, and these drive actions.Source: §3.1 · p. 3
• LLMs generate dialogue and background descriptions to manifest prescribed mental states, with scenarios designed to align character utterances and behavior to their trajectories.Source: §3.1 · p. 3

3.2 Question Genres

• Understanding questions assess belief, emotion, intention, or action at a specific time; transformation questions test changes between scenarios and the mechanisms behind them.Source: §3.2 · p. 4, Figure 2 · p. 4
• Transformation-3 questions test tracking state evolution across all scenarios, while the question progression helps distinguish basic state-comprehension failures from temporal-reasoning limitations.Source: §3.2 · p. 4
• Distractors draw on other states or scenario-specific values in the designed trajectories, grounding both correct answers and alternatives in the benchmark’s explicit state records.Source: §3.2 · p. 4

3.3 Statistics

• 1,100 social stages with five connected scenarios and 71 questions each yield 78,100 questions; DYNTOM captures temporal evolution where prior benchmarks mainly assess static snapshots.Source: Table 1 · p. 5, §3.3 · p. 5, Table 2 · p. 5

4.1 Experimental Setup

• The study evaluates ten representative language models, including Qwen2 72B, with vanilla and zero-shot CoT prompting, and uses ten graduate students as a human performance baseline.Students evaluated a randomly sampled 30% of the dataset: 330 social stages and 23,430 questions.Source: §4.1 · p. 5

4.2 Main Results

• 77.7% human average accuracy exceeds LLM performance by 44.7%; GPT-4o reaches 64.0%, 13.7% below humans.The comparison covers performance across tasks, with Table 3 reporting accuracy and human annotator standard deviations.Source: §4.2 · p. 5, Table 3 · p. 6
• 28.9 percentage points separate average emotion accuracy on understanding questions (54.7%) and transformation questions (25.8%).Models perform worse on transformation questions across mental states, which require tracking state evolution across scenarios.Source: §4.2 · p. 6, Table 3 · p. 6
• CoT effects vary by model: it improves several smaller models but degrades GPT-4o by 2.9%.The authors report that stepwise decomposition can help basic reasoning yet may treat scenarios independently and miss temporal dependencies.Source: §4.2 · pp. 6–7
• Emotion understanding averages 54.7%, versus 41.7% for belief understanding, a 13-point gap.The authors hypothesize that beliefs are more implicit and often require multi-step inference from indirect evidence.Source: §4.2 · p. 7

4.3 In-Depth Analysis

• Across all state types, fully correct responses—identifying both mental states and their changes—were rare (13-17%), while full errors dominated (50-58%); belief states had the highest full-error rate (58%).Local errors occurred in 13-18%, and restoration errors in 8-16%.Source: §4.3 · p. 7
• In seven-scenario sequences, middle-span accuracy rises from 26% to 47% when later scenarios are removed; six-scenario span 2–3 accuracy is 50%.Performance follows a U-shaped pattern across time spans, with models doing better at early and late spans than in the middle.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5 Conclusion

• DynToM moves beyond static ToM assessments by evaluating mental-state evolution across interconnected scenarios.The benchmark is presented as a framework for assessing and improving LLMs’ ability to understand dynamic human mental states.Source: §5 · p. 8
• LLMs trail human performance by 11% and particularly struggle to track mental-state changes across extended interactions.The authors state that human evaluation validates the benchmark’s alignment with real-world social dynamics.Source: §5 · pp. 8–9

Limitations

• The evaluation covers ten representative models, potentially overlooking insights from other emerging open-source and commercial models such as Claude.The authors identify computing resources and budget as constraints on the model-selection scope.Source: Limitations · p. 9
• The evaluation uses vanilla and chain-of-thought prompting, while other prompting methods remain unexplored.The authors mention think-twice, belief tracker, and self-consistency as possible methods for future exploration.Source: Limitations · p. 9
• The framework models belief, emotion, intention, and action in dialogue, while knowledge and multimodal cues remain outside its current evaluation scope.The authors propose exploring knowledge and visual or auditory cues in future work.Source: Limitations · p. 9

Ethics Statement

• Annotators were advised to stop if content caused discomfort and were paid at 2–3 times their local hourly minimum wage.The authors state that they adhered to the ACL Code of Ethics and prioritized annotator well-being.Source: Ethics Statement · p. 9
• The authors instructed annotators to validate data without bias and keep it free from unsafe, toxic, biased, offensive, and harmful content.They state that they made efforts to adhere to ACL ethical norms.Source: Ethics Statement · p. 9
• The authors clarify that evaluating LLMs’ capacity to interpret human mental states is intended to support social interaction, not anthropomorphize LLMs.The paper notes that ToM is a distinctive social cognitive capability intrinsic to humans.Source: Ethics Statement · p. 9

A.1 the candidate pool of social location

• The candidate pool contains 13 types of social locations, totaling 261 locations.The paper describes social location as the environment where individuals live, work, and learn.Source: §A.1 · p. 13

A.2 the candidate pool of profile

• Character profiles cover seven aspects: surname, name, gender, occupation, education, race, and personality traits.The paper identifies U.S. Census Bureau, Social Security Administration, and Bureau of Labor Statistics sources for surname, name, and occupation statistics.Source: §A.2 · p. 13

A.3 The prompt used to generate the mental state trajectory

• A prompt uses supplied information to generate a mental-state trajectory, with an example shown in Figure 12.Figure 11 illustrates the trajectory-generation prompt.Source: §A.3 · p. 13

A.4 The prompt used to generate the social scenarios

• A prompt uses supplied information to generate social scenarios, with an example shown in Figure 14.Figure 13 illustrates the social-scenario prompt.Source: §A.4 · p. 13

A.5 The templates for the four types of questions and question examples

• Four question templates assess scenario-specific reasoning and whether, why, and how a ToM reasoning item changes across scenarios; each context receives 71 questions.The templates are labeled Understanding-1 and Transformation-1, Transformation-2, and Transformation-3.Source: §A.5 · p. 13

A.6 Human validation of the Quality of DYNTOM

• Argilla is used as the annotation platform for data validation, and Figure 16 shows its annotation interface.Source: §A.6 · p. 13

B.1 Model detail

• The evaluation includes ten models: GPT-4o, GPT-4-Turbo, Llama-3.1-8B, Llama-3.1-70B, Mistral-7B, Mixtral-8x7B, Qwen2-7B, Qwen2-72B, DeepSeek-V2, and GLM-4.Source: §B.1 · p. 13
• The authors accessed the models through official APIs or model weights, following their terms.Source: §B.1 · p. 13
• Table 6 lists model versions, parameter sizes, context-window sizes, and prompts for the two evaluation methods.Source: §B.1 · p. 13

B.2 Prompting methods

• Vanilla prompting directly asks LLMs to answer, whereas CoT prompting elicits step-by-step reasoning before answering.Figure 17 shows the prompts used for both methods.Source: §B.2 · p. 13, Figure 17 · p. 21

B.3 Case Study for CoT Prompting

• CoT prompting does not always improve ToM reasoning across the item and question-type results; the appendix shows a GPT-4o failure case.The results are reported in Table 3, and the failure case appears in Figure 18.Source: §B.3 · p. 13, Figure 18 · p. 22
• Figures 5 and 6 show the candidate social locations and an example social context, respectively.Source: Figure 5 · p. 14, Figure 6 · p. 15
• Figures 7–9 pair races with popular surnames and genders with popular names and occupations.Source: Figure 7 · p. 15, Figure 8 · p. 16, Figure 9 · p. 16
• Figure 10 presents personality traits and educations.Source: Figure 10 · p. 16
• Figure 11 presents a prompt for generating character relationships and mental-state trajectories, while Figure 12 gives a trajectory example.Source: Figure 11 · p. 17, Figure 12 · p. 18
• Figure 13 shows a scenario-generation prompt, and Figure 14 provides an example of the resulting social scenarios.Source: Figure 13 · p. 19, Figure 14 · p. 19
• Figure 15 presents question types and generation templates, while Figure 16 shows the story-quality annotation platform.Source: Figure 15 · p. 20, Figure 16 · p. 20