Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?

Motivation and task

• The work argues that HRI-specific ToM evaluation matters because real-world robots operate with irreversible consequences and higher stakes.Source: §1 · p. 2
• The study examines LLM-based human proxies for judging whether robot behavior would be perceived as explicable, legible, predictable, or obfuscatory in HRI.The task targets internal critique during behavior synthesis, where the robot queries an LLM about a human observer’s likely perception.Source: §1 · p. 2

Study scope

• The evaluation covers five HRI domains and four behavior types, producing 20 curated situations for mental-model queries.Source: §1 · pp. 2–3

Human validation

• A user study validates that lay users can correctly answer the curated Theory of Mind queries.Source: §1 · p. 3

Perturbation tests

• Vanilla prompts can inflate expectations of LLM ToM, while Inconsistent Belief, Uninformative Context, and Conviction tests are designed to break that illusion.These tests probe whether responses remain reliable under perturbations rather than merely succeeding on familiar prompt patterns.Source: §1 · pp. 2–3

Theory of Mind

• Theory of Mind supports communication and human-agent interaction, while HRI behavior synthesis requires modeling and reasoning over the human mental model.Source: §2 · p. 3

LLMs and emergent abilities

• LLMs are studied as possible emergent reasoners with reported abilities in reasoning, deception, and Theory of Mind, but newer work emphasizes brittleness and prompt dependence.Source: §2 · p. 3

Positioning

• The paper differentiates itself through literature-grounded HRI scenarios, a large-scale user study, and a Fetch-robot case study.Source: §2 · p. 3

Behavior types

• Explicability measures how closely a plan matches an observer’s expectations given a known goal.Source: §3 · p. 3
• Legibility reduces ambiguity about the goal being achieved, whereas predictability reduces ambiguity about possible plans given a goal.Source: §3 · p. 3
• Obfuscatory behavior makes a plan consistent with at least k goals, including the true goal and k−1 decoy goals intended to deceive the observer.Source: §3 · p. 3

Mental-model requirements

• Synthesizing these behaviors requires selecting or balancing behavior types, modeling the human, and reasoning over that mental model.The behavior types can be competing rather than mutually exclusive, and cross-type synthesis may involve multiple objectives.Source: §3 · p. 3

4 METHODOLOGY

• The study asks how humans perform, how LLM responses align with oracle and human judgments, and how robust LLM mental-model reasoning is.Source: §4 · p. 3

4.1 Prompt Construction

• Each situation specifies a domain, goals, behavior type, constraints, optional perturbation, and two questions about perceived behavior and its explanation.The prompts are evaluated first with human subjects and then with GPT-4 and GPT-3.5-turbo as potential human proxies.Source: §4 · p. 4, §4.1 · p. 4

4.2 Human Subject Study on ToM-PROBE task

• The human study evaluates explicable, legible, predictable, and obfuscatory behavior across five domains using a pilot-refined, IRB-approved protocol.Source: §4.2 · p. 4

4.2.1 Question Design.

• Participants answer binary and multiple-choice reasoning questions, evaluate GPT-4’s response, and rate its reasoning on a five-point Likert scale.The mixed design varies domains between subjects and response ordering within subjects.Source: §4.2.1 · p. 4

4.2.2 Setup & Procedure.

• The final study includes 100 participants, with 20 assigned to each evaluation domain, following a 25-participant pilot.Participants were recruited through Prolific and completed the study in approximately six minutes.Source: §4.2.3 · p. 4, §4.2.2 · p. 4

4.3 Testing LLMs for Robustness

• Robustness tests add uninformative context, inconsistent beliefs, and repeated-query conviction checks whose correct responses should remain invariant to irrelevant perturbations.Inconsistent-belief prompts hide the robot from the observer and add “Can’t Say” as the correct response option; conviction tests vary response stochasticity through temperature.Source: §4.3 · pp. 4–5

• Five domains—Fetch, Passage Gridworld, Environment Design, Urban Search and Rescue, and Package Delivery—provide the evaluation settings.Source: §5 · p. 5
• Fetch evaluates legibility by asking whether an initial leftward move identifies the robot’s intended goal.Source: §5 · p. 5
• Passage Gridworld evaluates explicability when the robot takes a longer route that is justified by blockades unknown to the observer.Source: §5 · p. 5
• Environment Design tests whether gridworld design choices support obfuscatory behavior by keeping alternative goals difficult to distinguish.Source: §5 · pp. 5–6
• Package Delivery evaluates obfuscation when a robot holds a package between the port and ship without reducing the observer’s possible goals.Source: §5 · p. 6
• Urban Search and Rescue evaluates predictability by asking whether a robot carrying a medkit and moving left has a uniquely identifiable completion.Source: §5 · p. 6

6 RESULTS

• The results section uses human performance to validate the prompts and establish a baseline before analyzing LLM performance and robustness.Source: §6 · p. 6

6.1 RQ1 : Human Subjects on ToM-PROBE task

• Users achieved at least 69% average accuracy on both questions in each domain except lower Legibility and Predictability accuracy for Package Delivery.Question 2 often identified the correct reasoning even when users had answered Question 1 incorrectly.Source: §6.1 · p. 6, Figure 4 · p. 6

6.2 RQ2: LLMs Performance Alignment

• GPT-4 performs better than GPT-3.5-turbo across behavior types except predictability, where their accuracies are equal.Source: §6.2.1 · p. 6, Figure 5 · p. 7

6.2.2 Kolmogorov-Smirnov (KS) Test :

• The KS test yields an observed critical value of 0.04 between GPT-4 and human responses, supporting alignment at the tested significance levels.GPT-4 versus ground truth produces 0.3, accepted at 0.05 but not 0.01.Source: §6.2.2 · pp. 6–7

6.2.3 Subjective Response Evaluation:

• Human ratings generally agree with GPT-4’s vanilla descriptive responses, with Likert averages of 3.47, 4.23, 4.0, and 4.08 across behavior types.Source: §6.2.3 · p. 7, Figure 6 · p. 7
• Automated comparison of GPT-4 explanations yields average ratings of 3.4, 4, 3.8, and 3.4 for explicability, legibility, predictability, and obfuscation.Source: §6.2.3 · p. 7, Figure 6 · p. 7

6.3 RQ3: Failure modes of LLMs in ToM-PROBE

• GPT-4’s positive vanilla-prompt results are overturned by robustness testing, which the authors use to argue that ToM abilities are absent.Source: §6.3 · p. 7, §6.3.3 · p. 8
• The first perturbation causes performance to drop across all four robot behavior types.Source: §6.3.1 · p. 7

6.3.1 Uninformative Context.

• Under the second perturbation, GPT-4 gives zero correct responses in 19 of 20 cases.Source: §6.3.2 · p. 7

6.3.2 Inconsistent Belief.

• When sampled 10 times at τ=2, GPT-4 switches between Yes and No even when answers are incorrect, while GPT-3.5-turbo produces worse, irrelevant responses.Source: §6.3.3 · p. 7

6.3.3 Conviction Test.

• Such inconsistent and irrelevant outputs could make an automated HRI system unreliable and affect human-robot interaction.Source: §6.3.3 · p. 8

7.1 Study design & Procedure

• The study uses Fetch videos, text prompts, and two perturbations, recruiting 20 new Prolific participants to answer behavior and explanation questions.Source: §7.1 · p. 8, §7 · p. 8

7.2 Results

• The perturbation operationalization was validated because 100% could not read the robot’s label and 95% could not see the robot acting.Source: §7.2 · p. 8
• Users correctly identified legible behavior at 85% and selected the correct Q2 explanation at 95% under the reported cases.Source: §7.2 · p. 8
• For Inconsistent Belief perturbations, 75% of users correctly opted for the “Can’t Say” choice, while 25% chose the “No” option.Source: §7.2 · p. 8
• Human responses were not affected by the perturbations, whereas GPT4-Vision failed the Inconsistent Belief and Conviction tests despite correctly handling the vanilla case.Source: §7.2 · p. 8

• LLMs perform better than chance on vanilla perceived-behavior tasks but fail Inconsistent Belief, Uninformative Context, and Conviction tests.These failures indicate that their responses are not robust to prompt perturbations or repeated querying.Source: §8 · p. 8
• The study examines explicable, legible, predictable, and obfuscatory behavior across five HRI domains using human comparisons and LLM evaluations.The task asks an LLM to predict how a human observer would perceive a robot’s behavior.Source: §8 · p. 8
• The work provides an initial HRI-focused analysis of LLM ToM abilities and identifies further opportunities for studying ToM facets and failure modes.The authors emphasize both potential benefits and cautions when using LLMs in HRI.Source: §8 · p. 8

• GPT-3.5 was prompted to compare expert-written and GPT-4-generated explanations as a semantic text-similarity tool.The comparison produces Likert-scale values for the two explanations.Source: §A.1 · p. 11
• The appendix labels this procedure as Prompt 6: Likert Score Prompt.Source: §A.1 · p. 11
• The similarity prompt defines 5 as very similar and 1 as very different for the two supplied sentences.The input consists of an expert explanation and GPT-4’s response.Source: §A.1 · p. 11

• Table 2 reports average correct responses and standard deviations over 20 participants across five domains for Q1 and Q2.Q1 is the binary response question and Q2 is the multiple-choice reasoning question.Source: §A.2 · p. 11, Table 2 · p. 12
• Figure 9 organizes correct Q1 answers by explicability, legibility, predictability, and obfuscation across five domains.Each bar represents the number of correct answers for a behavior type in the five domains.Source: Figure 9 · p. 12
• Figures 10 and 11 show GPT-4 and GPT-3.5-turbo performance distributions across the four behavior types after ten repeated prompts at τ=1 and τ=0.Source: Figure 10 · p. 13, Figure 11 · p. 13
• Figure 12 presents human evaluations of GPT-4 reasoning responses using average ratings on a 1–5 scale.Source: Figure 12 · p. 14

• Table 3 compares GPT-4v results for vanilla, Uninformative Context, and Inconsistent Belief prompts, marking incorrect answers in bold.Source: Table 3 · p. 16
• GPT-4v can answer the Uninformative Context perturbation correctly, yet its answers vary when the prompt is repeated in the Conviction test.Source: Table 3 · p. 16

Prompt structure

• Across domains, prompts ask observers to classify a robot behavior using explicit definitions and answer with constrained binary or numbered-choice responses.The prompt suite operationalizes perceived behavior recognition through scenario descriptions, plans or partial plans, definitions, and restricted response formats.Source: §B.1 · p. 17, §B.2 · p. 17, §B.3 · p. 18

B.1 Fetch Robot Domain

• Fetch scenarios test explicability, legibility, predictability, and obfuscation using object transport, alternative destinations or paths, and behaviors that preserve multiple possible goals.Examples include unnecessary arm-tucking, movement toward one of two locations, path selection toward locX, and motion that leaves loc2 and loc3 equally plausible.Source: §B.1 · p. 17

B.2 Passage Gridworld Domain

• Passage Gridworld prompts vary whether the human knows hidden blockades and ask how routes affect explicability, legibility, predictability, and obfuscation.The scenarios use shortest-path expectations, alternative goal cells, known or unknown barriers, and partial movements that narrow or preserve possible completions.Source: §B.2 · pp. 17–18

B.3 Environment Design Domain

• Environment Design contrasts an unobstructed setup with a uniquely constrained setup to test whether fewer valid plans improve explicability, legibility, and predictability.Setup A permits multiple plans, whereas Setup B has one valid plan under the stated obstacles; the prompts ask which setup exhibits each property.Source: §B.3 · p. 18
• A separate Environment Design prompt tests obfuscation by asking whether multiple equally poor plan completions can hide which of two goals the agent pursues.The setup contains no obstacles and allows alternative plans while the agent may seek either G1 or G2.Source: §B.3 · p. 19

B.4 USAR Domain

• USAR scenarios apply the four behavior definitions to med-kit selection, patient-versus-commander destinations, alternative routes, and equal distances between possible goals.The observer lacks med-kit properties in the explicability case, while later prompts use turning direction, route choice, and equal-distance positioning to distinguish interpretations.Source: §B.4 · p. 19

B.5 Package Delivery Domain

• The Package Delivery domain introduces partial observability, asking whether acquiring an unspecified crate before loading one onto the ship is explicable when speed is required.The observer knows the initial inventory and learns only that new cargo arrived, not its identity.Source: §B.5 · p. 20
• The explicability prompt asks why a plan is not explicable, offering explanations based on action order, observer knowledge, or the impossibility of explicable plans.The options distinguish a plan-level failure from the observer’s ability to identify the loaded crate.Source: §B.5 · p. 20
• The subsection labels the legibility, predictability, and obfuscatory cases as separate package-delivery prompts.They are identified as Prompt 24, Prompt 25, and Prompt 26, respectively.Source: §B.5 · p. 20
• The legibility prompt defines legibility by whether the observer can identify the robot’s intended goal from a partial plan.The example plan picks up a package and brings it closer to the ship, followed by a Yes/No judgment and a forced-choice explanation.Source: §B.5 · p. 20
• The legibility explanation choices attribute the judgment to loading the package, acquiring it at the port, or the observer knowing its identity.These alternatives test whether the inferred goal depends on the robot’s next action or on information available to the observer.Source: §B.5 · p. 20
• All cases constrain the model to answer the primary judgment with only Yes or No and the follow-up rationale with one numbered option.This format is explicit in the legibility, predictability, and obfuscatory scenarios and their paired questions.Source: §B.5 · p. 20
• The package-delivery domain models a port robot whose actions may acquire packages or load them onto a ship under an observer’s partial information.The observer knows the initial inventory but may learn only that new cargo arrived, or may identify a crate when it is loaded; the robot’s goal varies by prompt.Source: §B.5 · p. 20
• The predictability prompt defines a partial plan as predictable when the observer can identify whether it has one possible completion.The example involves picking up a package and moving toward the port, with explanations tied to acquisition, loading, or directly showing the package.Source: §B.5 · p. 20
• The obfuscatory prompt defines behavior as obfuscatory when it does not reduce the observer’s set of possible goals.The example holds a package between the port and ship, and the explanation options distinguish between retaining both possible actions versus committing to one.Source: §B.5 · p. 20

C.1 Fetch Robot Domain

• Fetch prompts test whether observers identify unnecessary actions, infer goals from partial plans, determine unique completions, and recognize unchanged goal sets as obfuscatory.Scenarios cover object transport, alternative locations or paths, and behavior labels that observers cannot read.Source: §C.1 · pp. 20–21

C.2 Passage Gridworld Domain

• Passage Gridworld prompts test behavior interpretation when the robot knows blockades that the observer may not, including route expectation, goal inference, completion uniqueness, and goal-set reduction.The scenarios contrast observer knowledge and robot knowledge across explicability, legibility, predictability, and obfuscation judgments.Source: §C.2 · pp. 22–23

C.3 Environment Design Domain

• Environment Design prompts compare obstacle-free and obstacle-constrained 3x3 grids to test whether plan multiplicity affects explicability, legibility, and predictability.Setup A permits multiple plans, whereas Setup B has a single valid plan; the prompts also test whether plans preserve ambiguity between two goals.Source: §C.3 · pp. 23–24
• The Environment Design obfuscation prompt defines an obfuscatory setup as one where equally poor completions preserve ambiguity about which of two goals the agent seeks.It presents an obstacle-free setup with multiple candidate plans and asks whether the environment supports prolonged goal hiding.Source: §C.3 · p. 25

C.4 USAR Domain

• USAR prompts instantiate the four behavior judgments with med-kit delivery, contrasting observer expectations, alternative destinations, alternative routes, and equal-distance goals.The scenarios involve a commander, patient room, med-kit properties unknown to the observer, and forced-choice explanations for each judgment.Source: §C.4 · pp. 25–27

C.5 Package Delivery Domain

• The Package Delivery subsection begins an explicability prompt, but this supplied part contains no further scenario description or response options.Only the prompt heading is provided here.Source: §C.5 · p. 27
• The explicability prompt asks whether acquiring a crate before loading one is expected when the goal is to load a crate quickly.The observer knows the initial inventory but receives only unspecified information about newly acquired cargo, and cannot read the robot’s “Explicable Agent” label.Source: §C.5 · p. 27
• Its rationale question distinguishes unnecessary ordering from information about the loaded crate and the claim that no plan is explicable.Respondents must choose among three explanations: loading first to achieve the goal, knowing the loaded crate’s identity, or rejecting all plans as explicable.Source: §C.5 · p. 27
• The legibility prompt asks whether picking up a package and moving it toward the ship identifies the robot’s intended goal.Legibility is defined by reducing the number of possible goal locations, under partial observations and an unreadable “Not a Legible Agent” label.Source: §C.5 · p. 28
• Its rationale question contrasts an intended ship-loading goal, an acquisition goal, and the observer’s knowledge of package identity.The response format restricts participants to one of three numbered explanations.Source: §C.5 · p. 28
• The predictability prompt asks whether picking up a package and moving toward the port leaves only one possible completion.Predictability is defined by the number of possible plan completions, while the observer must ignore an unreadable “Not a Predictable Agent” label.Source: §C.5 · p. 28
• Its rationale question separates acquiring the package, loading it onto the ship, and directly showing it to the human as possible next steps.The prompt frames the goal as revealing the package’s identity to the human.Source: §C.5 · p. 28
• The obfuscatory prompt asks whether holding a package between the port and ship fails to narrow the robot’s possible goals.A behavior is obfuscatory when the observer still considers both acquisition and loading possible, despite the unreadable “Not an Obfuscatory Agent” label.Source: §C.5 · p. 29
• Its rationale question identifies continued ambiguity between acquiring the package and loading it on the ship as the basis for obfuscation.Participants choose whether both actions remain possible or whether only one action is available.Source: §C.5 · p. 29

D.1 Fetch Robot Domain

• Fetch prompts test explicability, legibility, predictability, and obfuscation when the human observer cannot see actions and may lack environmental understanding.Examples include unnecessary tucking before fetching, a leftward step indicating one of two goals, two possible routes, and movement that preserves equal goal distances.Source: §D.1 · pp. 29–30

Cross-domain prompt structure

• Across prompts, follow-up questions require selecting whether inexplicability, legibility, predictability, or obfuscation stems from limited visibility, dislike of the plan, or inability to understand the environment.These forced-choice explanations are paired with Yes, No, or Can’t Say judgments.Source: §D.1 · p. 29, §D.2 · pp. 30–31, §D.3 · p. 32

D.2 Passage Gridworld Domain

• Passage Gridworld prompts introduce blockades unknown to the observer or known to both parties, testing whether indirect routes appear explicable, legible, predictable, or obfuscatory.The scenarios contrast shortest-path expectations with routes around hidden blockades, ambiguous goal-directed movement, constrained completions, and movement that leaves multiple goals possible.Source: §D.2 · pp. 30–31

D.3 Environment Design Domain

• Environment Design prompts compare obstacle-free Setup A with obstacle-constrained Setup B, where the number of valid plans determines explicability, legibility, and predictability.The obfuscation prompt instead asks whether an obstacle-free environment permits equally poor completions that conceal which of two goals the agent seeks.Source: §D.3 · pp. 32–33

D.4 USAR Domain

• USAR prompts test observer judgments when med-kit properties are unknown, goals are ambiguous, routes have alternatives, or the robot remains equally distant from competing goals.The four prompts cover selecting a farther heavy med-kit, turning toward a patient, choosing between two paths, and preserving equal distance to the patient and commander.Source: §D.4 · pp. 34–35

D.5 Package Delivery Domain

• The Package Delivery explicability prompt tests whether acquiring an unspecified crate before loading one appears expected when the observer receives only partial cargo information.The observer knows the initial inventory and learns that new cargo arrived, but not its identity, while the goal is rapid loading.Source: §D.5 · p. 36
• The explicability prompt asks why the robot’s plan is not explicable, offering lack of visibility, dislike of the plan, or inability to understand the environment.Responses must contain only option 1, 2, or 3.Source: §D.5 · p. 36
• The prompt set repeatedly restricts outputs to the requested categorical labels or numbered explanations, supporting standardized assessment across the four perceived-behavior properties.The supplied package-delivery materials explicitly provide these response constraints for explicability, legibility, predictability, and obfuscation.Source: §D.5 · pp. 36–37
• The package-delivery prompts place a robot at a shipping port where the human supervisor receives only partial activity information and cannot see the robot acting.The observer may learn crate identities in some situations, while newly acquired cargo is reported without its number or identity.Source: §D.5 · pp. 36–37
• The legibility prompt defines a partial plan as legible when the observer can identify the robot’s goal, with greater legibility corresponding to fewer possible goal locations.The tested plan picks up a package and brings it closer to the ship; the observer answers Yes, No, or Can’t Say, then selects an explanation option.Source: §D.5 · p. 36
• The predictability prompt defines a partial plan as predictable when the observer can identify one possible completion, with greater predictability corresponding to fewer possible completions.The tested plan picks up a package and moves toward the port, followed by a Yes, No, or Can’t Say judgment and a constrained explanation response.Source: §D.5 · p. 36
• The obfuscatory prompt defines behavior as obfuscatory when it fails to reduce the observer’s set of possible goals.The example plan has the robot pick up a package and hold it between the port and ship before the observer gives a categorical judgment.Source: §D.5 · p. 37
• The obfuscatory explanation question offers the same three reasons: the human cannot see the robot acting, dislikes its plan, or cannot understand the environment.The response is restricted to option 1, 2, or 3.Source: §D.5 · p. 37