PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoning

Soyoung Oh, Nursulu Sagimbayeva, Nicholas Jennings, Paulina Hussein, Mehwish Alam, Vera Demberg

arXiv:2610.19796v1cs.AIcs.CL

TL;DR

LLMs may infer agents’ mental states yet fail to use those inferences to predict their actions, a ToM compositionality gap. This paper trains models with concise if–then perception-belief-action rules using GRPO; rule-generating GRPO achieves the strongest accuracy, and larger-model rules improve GPT-4o-mini’s downstream accuracy.

  • Problem

    LLMs can answer explicit questions about agents’ mental states yet struggle to predict or judge their future actions despite relevant perception and belief knowledge.

    Source: §1 · p. 1

  • Method

    The paper trains models with GRPO to generate concise if–then rules connecting perception, belief, and action before answering.

    Source: §1 · pp. 1–2

  • Results

    Across training approaches, GRPOr achieves 74.2% accuracy, above GRPOt (67.0%), SFTa (66.7%), SFTr (61.4%), Baser (52.4%), and Base (45.5%).

    Source: §3 · p. 3

  • Takeaways & Limitations

    Structured rule generation is effective when jointly optimized with answer correctness, and rules from larger models improve GPT-4o-mini’s downstream accuracy.

    Source: §1 · p. 2, §4 · p. 5

    The evaluation covers four controlled ToM benchmarks, so the results may not generalize to more naturalistic settings; the reward does not verify rule semantics or causal use.

    Source: Limitations · p. 5

Abstract

from arXiv · show

In Theory-of-Mind (ToM) reasoning, large language models may correctly infer the agent's mental state but fail to use that inference when predicting or judging the agent's subsequent actions. In this work, we investigate whether explicitly connecting perception, belief, and action with rules improves ToM reasoning. We generate concise \textit{if--then} rules that express relations between perception to action and use Group Relative Policy Optimization (GRPO) to train Qwen2.5 and Llama3 models across four ToM benchmarks. GRPO with rule generation outperforms prompting, supervised fine-tuning, and think reasoning. Moreover, rules generated by larger models ($\geq$3B) provide transferable knowledge that improves downstream performance of GPT-4o-mini. These findings suggest that explicit rules help models bridge the reasoning gap in ToM and support knowledge transfer across models.

1 Introduction

The authors argue that LLMs can infer agents’ mental states yet fail to use those inferences when predicting actions, framing this as a compositionality gap. They test whether concise perception-belief-action rules trained with GRPO improve ToM and report accuracy and transfer gains.

  • LLMs may answer explicit mental-state questions correctly yet struggle to predict or judge agents’ future actions despite relevant perception and belief knowledge.Source: §1 · p. 1
  • The approach trains models to generate concise if–then rules connecting inferences, such as perception to belief and belief to action, before answering.The authors hypothesize that these rules constrain the reasoning space.Source: §1 · pp. 1–2
  • Rule-generating GRPO improves average accuracy by 7.2% over free-form GRPO reasoning while using fewer thinking tokens.Source: §1 · p. 2
  • Rules generated by models of at least 3B parameters improve GPT-4o-mini’s downstream accuracy.Source: §1 · p. 2

2 Methods

The methods evaluate several model families and training conditions across four ToM benchmarks, using generated rules and GRPO to train models to produce a rule before answering.

2.1 Data

  • The experiments use BigToM, HiToM, ToMI, and OpenToM, reserving 100 examples per benchmark for SFT warm-up and stratifying the remaining splits by benchmark-specific factors.Source: §2.1 · p. 2

2.2 Rule generation

  • Ground-truth rules are generated with gpt-5-mini without revealing the gold answer, then treated as valid when that model answers correctly using the rule.Source: §2.2 · p. 2

2.3 Models

  • The models include Qwen2.5-Instruct at 0.5B, 1.5B, 3B, 7B, and 14B, plus Llama-Instruct at 1B, 3B, and 8B.Source: §2.3 · p. 2

2.4 Experimental setup

  • The setup compares Base and rule-generating Baser prompt baselines with answer-only SFT, rule-answer SFT, free-form-reasoning GRPO, and rule-answer GRPO.Source: §2.4 · p. 2

2.5 Training

  • GRPO combines format and answer rewards, normalizes them across groups of eight completions, and uses format-validity SFT warm-up and development-accuracy checkpoint selection.The training uses the same examples for two epochs.Source: §2.5 · pp. 2–3

3 Results

Across training approaches, GRPOr has the strongest reported overall accuracy; the section also examines rule prompting, trace length, and transfer of generated rules.

3 Results

  • GRPOr achieves 74.2% overall accuracy, outperforming GRPOt (67.0%), SFTa (66.7%), SFTr (61.4%), Baser (52.4%), and Base (45.5%).Source: §3 · p. 3
  • Rule prompting increases average accuracy by 6.9% and improves results in most available comparisons across temperatures.Source: §3 · p. 3, Table 2 · p. 3
  • SFT rule generation decreases average accuracy relative to answer-only fine-tuning, unlike rule prompting.Source: §3 · p. 3, Table 3 · p. 4
  • GRPOr improves mean accuracy by 7.2% over GRPOt across seeds while using 25.2 versus 52.1 tokens; its no-warmup variant also outperforms GRPOt.Source: §3 · pp. 3–4

3.1 Rule utility beyond trace length

  • On HiToM, GRPOr outperforms both long- and short-thinking GRPOt, although shorter reasoning modestly improves performance.The compared GRPOr and GRPOt traces average 29.5 and 71.4 tokens, respectively.Source: §3.1 · p. 4, Table 5 · p. 4
  • Rules from smaller models (≤1.5B) reduce GPT-4o-mini downstream accuracy, while rules from larger models improve it, except for the 3B model on HiToM.Source: §3.1 · p. 4, Figure 2 · p. 4

4 Conclusion

The conclusion reports that perception-belief-action rules improve ToM reasoning, but supervised rule imitation does not reliably improve answer prediction. GRPO with rule generation yields the strongest accuracy, shorter traces than free-form GRPO, and downstream gains from larger-model rules.

4 Conclusion

  • Supervised rule imitation does not reliably improve answer prediction, unlike instructing models to generate perception-belief-action rules.The conclusion describes rule generation as improving ToM reasoning.Source: §4 · p. 4
  • GRPO with rule generation achieves the strongest accuracy, uses shorter traces than free-form GRPO, and improves GPT-4o-mini downstream accuracy with rules from larger models.The findings suggest structured rule generation is effective when jointly optimized with answer correctness.Source: §4 · pp. 4–5
  • The conclusion frames joint optimization of structured rule generation and answer correctness as effective for ToM reasoning.Rules connect perception, belief, and action.Source: §4 · pp. 4–5

Limitations

The evaluation spans four controlled ToM benchmarks, and its reward does not verify whether generated rules are semantically correct or causally used. The authors also identify alternative explanations for the gains and caution that results may not generalize to naturalistic settings.

Limitations

  • Four controlled ToM benchmarks leave open whether the results generalize to benchmarks assessing ToM in more naturalistic settings.The paper explicitly identifies this as a scope limitation.Source: Limitations · p. 5
  • The reward checks output format and answer correctness, but not rule semantic correctness or causal use in answer prediction.Consequently, uninformative rules could still receive high reward.Source: Limitations · p. 5
  • The gains may reflect concise intermediate traces or structured reasoning templates rather than the generated rules themselves.The authors cite prior work on concise traces and templates as possible explanations.Source: Limitations · p. 5

Ethical Considerations

The ethical considerations identify potential overreliance on model predictions of human beliefs and intentions, and possible use of improved social reasoning for manipulation. They also warn that credible-looking rules and benchmark gains should not be treated as proof of reliable real-world mental-state understanding.

Ethical Considerations

  • Improved social reasoning could be used for manipulation, while overreliance on predictions of human beliefs and intentions is another potential risk.Both risks are presented as possibilities, not established outcomes.Source: Ethical Considerations · p. 5
  • Generated rules may appear credible without being semantically correct or faithfully explaining model predictions.The paper cautions against treating benchmark gains as evidence of reliable real-world mental-state understanding.Source: Ethical Considerations · p. 5
  • Benchmark improvements should not be interpreted as evidence of reliable mental-state understanding in real-world settings.This is the paper’s stated boundary on interpreting the results.Source: Ethical Considerations · p. 5

A Benchmark details

The appendix describes benchmark variation, the constraints used to elicit rules, and prompt formats for rule-based and think-based answers. It also reports the reasoning settings and split-level counts used in validating generated rules.

A Benchmark details

  • The five ToM benchmarks vary in task factors and answer formats; BigToM, HiToM, and OpenToM use discrete choices, while ToMi uses open-ended predictions.The listed factors include task condition, belief or action type, question order, and character observations or personality.Source: A Benchmark details · p. 6, Table 6 · p. 6

B.1 Rule generation

  • Rule-generation instructions require a concise, single-sentence, story-specific IF-THEN rule with causally related conditions and conclusion, capturing the decision logic rather than naming the answer.The prompt asks for necessary and sufficient conditions and excludes redundant ones.Source: §B.1 · pp. 6–7

B.2 Prompts for training and evaluation

  • Rule prompts request a rule and final answer, whereas think prompts request step-by-step reasoning and an answer; both specify constrained JSON outputs.The rule condition leads to a conclusion, and the answer field contains only the final answer.Source: §B.2 · pp. 7–8

C Validation of the generated rules

  • Rule generation used medium-to-high reasoning and question answering low-to-medium reasoning; Table 8 reports validated-rule counts for train, development, and test splits.The table caption specifies the split-level counts.Source: C Validation of the generated rules · p. 8, Table 8 · p. 8

D Effect of SFT warm-up on GRPO performance

The comparison examines whether SFT warm-up adds benefit to GRPO rule training relative to GRPO think and rule training without warm-up.

D Effect of SFT warm-up on GRPO performance

  • GRPO rules without warm-up improve mean accuracy by 0.84 points across models and benchmarks compared with GRPO think.The reported pattern is that GRPO without warm-up broadly matches or exceeds GRPO think.Source: D Effect of SFT warm-up on GRPO… · p. 8, Table 9 · p. 8
  • GRPO rules with SFT warm-up exceed GRPO rules without warm-up by 5.95 points on average.The comparison controls for the effect of SFT warm-up on GRPO rule training.Source: D Effect of SFT warm-up on GRPO… · p. 8
  • Table 9 reports accuracy for GRPO think and GRPO rules without warm-up, with changes comparing the same model.The table caption specifies the scope of the comparison.Source: Table 9 · p. 8

E Reasoning trace length analysis

The analysis finds that rule traces are shorter than think traces, including when think reasoning is constrained to one sentence on HiToM.

E Reasoning trace length analysis

  • Think traces average 1.9 times the length of rule traces, rising to 2.41 times for Qwen 3B.These are average trace lengths across models and datasets, with the Qwen 3B comparison reported as the largest difference.Source: E Reasoning trace length analysis · p. 8
  • On HiToM, think_short traces average 25.5 tokens, versus 29.45 for rule traces and 68.61 for think traces.Think_short instructs models to produce one-sentence think traces.Source: E Reasoning trace length analysis · p. 8
  • The HiToM comparison measures average reasoning length across think_short, rule, and think conditions.Figure 4 presents the average token counts for the three conditions.Source: E Reasoning trace length analysis · p. 8

F Qualitative analysis for generated rules from small models

Small models can produce if–then-shaped rules without capturing coherent or relevant relationships, and may omit the queried agent’s perspective from reasoning.

F Qualitative analysis for generated rules from small models

  • Small models’ rules often link unrelated story elements or assert nonsensical relationships, despite following an if–then format.The analysis describes these rules as lacking semantic coherence or relevance to the question.Source: F Qualitative analysis for generated… · p. 8
  • Small models often fail to incorporate the queried agent’s perspective, even when their final answers are correct.Source: F Qualitative analysis for generated… · p. 8
  • The section compares average token use for rule and think reasoning traces.Figure 3 is described as comparing rule and think reasoning; Figure 4 adds think_short on HiToM.Source: Figure 3 · p. 9, Figure 4 · p. 9

G Generalization to Llama family

The experiments extend rule-based GRPO training to Llama models and report that it outperforms other conditions in most tested models and benchmarks. Examples also illustrate how rules and reasoning traces differ in representing beliefs about what others know.

G Generalization to Llama family

  • The Llama experiments use Llama-3.2-1B, Llama-3.2-3B, and Llama-3.1-8B with the Qwen-family training hyperparameters.Source: G Generalization to Llama family · p. 9
  • GRPOr outperforms other conditions for most Llama models and benchmarks, except Llama-3.2-1B on OpenToM.Table 10 reports answer accuracy and identifies the best-performing training regime for each model and benchmark.Source: Table 10 · p. 10, G Generalization to Llama family · p. 10
  • In the Jackson example, generated rules vary from a simple last-seen-location rule to an explicit nested-belief rule about what Jackson thinks Benjamin believes.The example asks where Jackson thinks Benjamin thinks the corn is; the stated answer is blue_treasure_chest.Source: G Generalization to Llama family · p. 11
  • The Qwen3B reasoning trace explains Noah’s false belief by noting that he left before the sweater was moved, while the 14B trace adds that he did not witness the move.Both traces conclude that Noah would look for the sweater where he last saw it.Source: G Generalization to Llama family · p. 11

Related papers

Loading 2610.19796v1…