PaperlayerView exact revision on arXiv

Source-linked AI summary

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

Kawai Chung, Chunkit Chan, Yuxuan Liu, Yauwai Yim, Xi Yang, Wenbin Hu, Huihao Jing, Chiu Yeung Tang, Changxuan Fan, Ki Sen Hung, Tsun On Kwok, Haoran Li

arXiv:2610.18610v1cs.CLcs.AI

TL;DR

Dialogue and collaborative assistants need to judge what partners know, believe, or intend, but the competitiveness and reliability of typed models for bounded ToM selection require evaluation. Jev-ToM compares typed models with generative baselines across benchmarks and analyzes confidence, reasons, and output validity. Jev ranks first or second across the six evaluated capabilities, while confidence errors and reason effects vary by benchmark and attachment.

  • Problem

    The paper asks whether Jev is competitive and reliable as a bounded ToM decision component, since aggregate accuracy can conceal differences across mental-state tasks and selection reliability matters to assistants.

    Source: §1 · p. 1

  • Method

    Jev-ToM compares five typed decision models and seven generative baselines on 11,117 questions from nine benchmarks across six capability groups, also examining confidence, reasons, and output validity.

    Source: §1 · p. 2

  • Results

    Jev ranks first or second across all six capabilities, including 84.8% in emotions versus GPT-3.5’s 79.7%, while confidence errors and reason effects remain benchmark-dependent.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    The findings support testing Jev as a bounded selection component, with confidence calibration, candidate sensitivity, and output validity prioritized for integration.

    Source: §1 · p. 2

    The evaluation covers English text questions with supplied candidates and does not test integration into a complete interactive assistant.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.

1 Introduction

Jev-ToM evaluates whether typed decision models can provide accurate, reliable bounded Theory of Mind selections for dialogue and collaborative assistants. It finds competitive accuracy alongside benchmark-dependent confidence errors and reason effects.

1 Introduction

  • Interactive responses depend on a partner’s perspective, while mental-state judgments can serve as bounded intermediate selections in dialogue and collaboration.The surrounding system can retain candidate generation and response composition.Source: §1 · p. 1
  • Fast structured output alone does not establish suitability: aggregate accuracy can conceal differences across mental states, and assistants need reliable selections and valid outputs.The paper asks how competitive and reliable Jev is as a bounded ToM decision component.Source: §1 · p. 2
  • Jev-ToM compares five typed decision models and seven generative baselines on 11,117 questions from nine ToM benchmarks across six capability groups.The evaluation also examines confidence-based selection, supplied reasons, output validity, response time, and reference-price cost.Source: §1 · p. 2
  • Competitive accuracy coexists with benchmark-dependent confidence errors, while supplied reasons can improve or disrupt selections depending on attachment and evaluated materials.The evaluation identifies confidence calibration, candidate sensitivity, and output validity as integration priorities.Source: §1 · p. 2

2 Related Work

Prior work motivates ToM judgments as intermediate inputs to planning, dialogue, response selection, and action prediction. Jev-ToM connects these uses and Jev interface research through cross-benchmark profiles and reliability analyses.

2 Related Work

  • Mental-state judgments support cooperative planning (Li et al., 2023), dialogue perspective tracking (Qiu et al., 2024, Yang et al., 2025), response selection (Hwang et al., 2026), and action prediction (Mu et al., 2026).Benchmarks vary in whether they test belief attribution, information asymmetry, psychological states, or socially situated judgment, making scope and aggregation important to interpreting scores.Source: §2 · p. 2
  • Typed Jev interfaces expose choices and probabilities, while prior work examines Jev task performance, calibration, resource use, annotation, and sensitivity to request content (Almeida, 2026, TypeSafe AI, 2026; Deußer et al., 2026; Ibrahim and Zaki, 2026, Rao and Callison-Burch, 2026; Hu et al., 2026, Sun et al., 2026).Mnemon uses Jev for retrieved-record evaluation while an LLM plans searches and composes responses (Wang, 2026).Source: §2 · p. 2
  • Jev-ToM extends this work by linking cross-benchmark ToM profiles with confidence-based selection, bidirectional changes under supplied reasons, and output validity.This connects the evaluation to contextual additions studied in JevOut on ToMBench (Xu et al., 2026).Source: §2 · p. 3

3 Experiments

The experiments evaluate fixed-choice ToM judgments across benchmark subsets, capability groups, models, and reliability analyses. Their design standardizes records and scoring while retaining source-specific tasks and benchmark-balanced aggregation.

3 Experiments

  • The evaluation compares fixed answer choices and examines confidence, supplied reasons, and resource use in separate subsets of models and questions.It uses strict scoring requiring valid outputs as well as correct selections.Source: §3 · p. 3, §3.3 · p. 3

3.1 Theory of Mind Ability Categories

  • Six capability groups cover knowledge access, beliefs, goals/desires/intentions, emotions, social pragmatics, and mental-state application.Application concerns using a perspective to predict, select, or judge behavior and can overlap state categories; coverage varies by source, from eight benchmarks for beliefs to two for social pragmatics.Source: §3.1 · p. 3

3.2 Dataset Selection and Harmonization

  • The dataset uses fixed development or previously used subsets from nine English ToM benchmarks, with source-specific adapters standardizing records while preserving contexts and candidate meanings.Candidate counts range from two to fifteen.Source: §3.2 · p. 3
  • After excluding 50 BigToM items whose stories omit a required subsequent event, the frozen evaluation contains 11,117 questions with references withheld from models.Native labels map to capability groups; ToMBench target mapping received agent-assisted review, not independent human validation.Source: §3.2 · p. 3

3.3 Evaluation Metrics

  • Exact match counts invalid, unsupported, or failed responses as incorrect; overall scores include controls and unmapped questions, whereas capability scores include assigned questions only.Equal benchmark weighting prevents larger datasets from dominating, and overall exact match is not the mean of the six capability scores.Source: §3.3 · pp. 3–4
  • Figure 2 reports accuracy across the six capabilities, averages equally across contributing benchmarks, and excludes controls and unmapped questions.Its panels sort models by accuracy and distinguish typed models from generative baselines by color.Source: Figure 2 · p. 4

3.4 Evaluated Models

  • The evaluation includes five typed models and seven generative baselines, which receive the same contexts, questions, and ordered candidates through model-appropriate interfaces.Typed models return a choice and candidate probabilities; generative models return an option label, and outputs are mapped back to original candidates for strict scoring.Source: §3.4 · p. 4
  • Excluding native controls, Jev scores 69.1% and GPT-3.5 scores 70.9% on exact-match accuracy.Table 2 covers 11,117 questions, weights benchmarks equally, and counts invalid outputs as errors.Source: Table 2 · p. 5

4 Results and Analysis

Jev ranks first or second across all six capabilities, though benchmark-dependent confidence, reason attachment, and candidate expansion complicate selection. Related-question cues improve action prediction, and separate deployment comparisons show cost–time advantages over nano and mini.

4.1 Where Are Typed Models Competitive?

  • Jev ranks first or second across all six capabilities, including 84.8% in emotions against GPT-3.5’s 79.7%, but benchmark rankings can reverse.Without native controls, Jev scores 69.1% against GPT-3.5’s 70.9%; on ToMi, Open-Jev-2B reaches 72.0% against Jev’s 69.7%.Source: §4.1 · p. 5

4.2 Can Confidence Identify Reliable Answers?

  • At the 0.90 confidence cutoff, retained error is 1.8% for ToMATO and 34.9% for SimpleToM despite similar coverage of about 47%.Hi-ToM’s zero observed errors cover only 33 retained answers; the cutoff needs task-specific validation before use as an acceptance policy.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3 Intervention 1: How Do Supplied Reasons Affect Selection?

  • On earlier-correct cases, aligned reasons raise fresh accuracy from 96.4% to 99.0%, while swapped reasons lower it to 25.3%.Neutral text preserves 96.8%, indicating that extra text alone does not explain the loss; on earlier errors, aligned and shared keyed reasons reach 71.0% and 69.4% against a 10.0% baseline.Source: Figure 4 · p. 6, §4.3 · p. 6
  • On new materials, Jev’s aligned gain falls from 51.4 to 3.7 points, while crossing keeps repair near seven points but raises damage from 3.3 to 7.1.Despite a net −0.4-point change, 13.8% of weighted outcomes change strict correctness.Source: Figure 5 · p. 6, §4.3 · pp. 6–7
  • Across crossed pairs, Jev and Luna have 78.2% and 77.6% both-correct outcomes, yet their invalid or failed shares are 7.5% and less than 0.1%.Every model selects only the keyed reason more often than only the keyed answer, so pair selection requires checking semantic choices and output validity.Source: Figure 6 · p. 7, §4.3 · p. 7
  • Candidate expansion lowers Jev’s strict accuracy from 81.0% to 76.4%; post-hoc saved-choice scoring widens the strict-to-saved gap from 3.2 to 6.0 points.Expansion also reduces GPT-3.5 accuracy by 13.9 points despite valid outputs, distinguishing selection loss from additional validation failures.Source: Figure 7 · p. 7, §4.3 · p. 7

4.4 Intervention 2: Does Mental-State Feedback Improve Action Prediction?

  • Question cues account for 10.75 of Jev’s 14.5-point action-prediction gain, with 3.75 further points from adding the predicted answer.The additional gain concentrates on SimpleToM; predicted-state feedback has task-dependent effects and can reduce accuracy for some typed models.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 What Are the Cost–Time Tradeoffs?

  • At US$0.0295 per 1,000 questions, Jev costs 32.7% less than nano and 56.3% less than mini, with accuracy advantages of 11.7 and 6.7 points.In a separate timing run, Jev records 73.0% accuracy and a 282 ms mean response time; observed timings vary with hardware, concurrency, and retry settings.Source: §4.5 · p. 8, Figure 8 · p. 8

5 Conclusion

Jev-ToM finds Jev competitive for bounded ToM selection, while identifying reliability and integration checks needed before testing it in complete assistants.

5 Conclusion

  • Jev combines competitive bounded ToM selection with favorable reference-cost and observed-time comparisons in the evaluated deployments.Benchmark-dependent confidence errors and sensitivity to supplied reasons show why accuracy alone is insufficient for adoption.Source: §5 · p. 8
  • Benchmark-dependent confidence errors and sensitivity to supplied reasons qualify Jev’s competitive accuracy.Source: §5 · p. 8
  • The results prioritize task-specific confidence calibration, candidate design, and output validation before testing Jev within complete assistants.Source: §5 · p. 9

Limitations

The evaluation is limited to English text questions with supplied candidates and does not establish broader ToM generalization or performance in complete assistants.

Limitations

  • The nine benchmarks cover several ToM judgments, but do not establish generalization to other languages, cultural settings, or multimodal social situations.The six capability groups are not an exhaustive measure of ToM.Source: Limitations · p. 9
  • The evaluation uses English text-based questions with supplied candidates, and possible exposure to the public benchmarks cannot be excluded.Source: Limitations · p. 9
  • Jev is evaluated as a bounded decision component, not within a complete interactive assistant.Candidate generation, partner-state updates across turns, and interactive task success remain outside the empirical claims; confidence and reason analyses are exploratory.Source: Limitations · p. 9

Ethical Considerations

The study relies on existing benchmark situations and experienced student annotations, and cautions against treating inferred mental states as facts or using results to justify consequential autonomous decisions.

Ethical Considerations

  • The study uses existing benchmark situations, model outputs, and annotations by three postgraduate students at English-medium universities with prior ToM-benchmark experience.Source: Ethical Considerations · p. 9
  • Predictions about beliefs or intentions can harm users when treated as facts, so applications should distinguish observed information from inferred states and validate error consequences.Source: Ethical Considerations · p. 9
  • The results do not justify autonomous consequential decisions about individuals, and redistribution remains subject to source terms.The authors plan a reproducibility package whose release is governed by the original licenses.Source: Ethical Considerations · p. 9

A.1 ToM benchmarks and application

The related work spans benchmark designs that vary in perspective, task, and evidence, Jev reliability and decision-interface studies, multilingual and multimodal evaluations, and mental-state reasoning in agents.

A.1 ToM benchmarks and application

  • ToM benchmarks vary in perspective depth and task structure, while broader datasets add psychological diversity and conversational information asymmetry.SimpleToM separates mental-state inference from behavior prediction and judgment; SimToM filters stories to information available to the queried character.Source: §A.1 · pp. 11–12

A.2 Jev evaluation and decision reliability

  • Prior Jev work examines task performance, calibration, and resource use, while studies of confidence routing, request cues, and candidate coverage motivate separating selection from output validity.These studies report task-dependent confidence behavior and cases where well-formed outputs may still reflect unintended request cues.Source: §A.2 · p. 12

A.3 Multilingual and multimodal reasoning

  • Multilingual and cultural audits, along with visual-narrative motivation tasks, extend evaluation beyond fixed English text questions.The cited work studies language and social framing, visual grounding, multi-label motivations, and evolving narrative evidence.Source: §A.3 · pp. 12–13

A.4 Mental-state reasoning in agents

  • Agent systems use explicit or tracked mental states for planning, dialogue alignment, goal achievement, and response composition, motivating component evaluation alongside surrounding interaction.Source: §A.4 · p. 13

B Evaluation Protocol

The protocol harmonizes benchmark questions and model outputs, then evaluates capability scores, confidence-based selection, and reason interventions under separate comparison designs. Results show that aggregate rankings depend on native controls and confidence separates errors differently across benchmarks.

B.2 Datasets and harmonization

  • The common primary set contains 11,117 questions from OpenToM, FANToM, ToMBench, and ToMi, after excluding 50 BigToM items missing a required subsequent event.Contexts are not truncated, and candidate order is frozen across models.Source: §B.2 · p. 13

B.1 Capability mapping

  • Capability scores average contributing benchmarks and exclude controls and unassigned items, while primary overall accuracy equally averages nine benchmark scores including them.The six capability assignments use native task, ability, and mental-state labels; application covers judgments of behavior or communicative appropriateness.Source: §B.3 · p. 13, §B.1 · p. 13, §C.1 · p. 14

B.4 Models and settings

  • The twelve-model evaluation uses one source question per request without task-specific fine-tuning, and paired uncertainty intervals resample source-defined families.The protocol uses 5,000 draws for primary comparisons, 2,000 for the six-condition study, and 10,000 for matched candidates; intervals do not measure deployment variability.Source: §B.4 · p. 13, §B.5 · pp. 13–14

C.2 Sensitivity to native controls

  • Jev falls from 73.2% to 69.1% without native controls, while GPT-3.5 changes from 71.6% to 70.9%, reversing their order.The paired difference is −1.79 points [−3.48, −0.07], and the aggregate ranking depends on whether native controls are included.Source: §C.2 · p. 14

D.2 Risk–coverage curves

  • At about half coverage, OpenToM retains 23.3% error at 48.8% coverage, versus SimpleToM’s 35.3% error at 48.9% coverage.Their full-coverage errors are 40.0% and 40.9%, respectively.Source: §D.2 · p. 14

E.2 Six-condition answer selection

  • The six-condition intervention compares no reasons, neutral filler, aligned and swapped reasons, and repeated keyed or foil reasons across original and reversed answer orders.For the eight-model pair study, reasons were prepared for 4,761 existing questions and reviewed for story support, perspective, attachment, and reference leakage.Source: §E.2 · pp. 14–15, §E.1 · p. 14

E.3 Human annotation

The reason studies examine annotation-supported attachment effects, repair and damage, matched candidate changes, and output validity. Their results show that reason attachment can sharply alter answers, while candidate expansion and validity checks reveal distinct issues.

E.3 Human annotation

  • Three experienced postgraduate coauthors annotated reasons and answers, first recording story-based answers before reasons were shown and saving later revisions separately.References, model predictions, machine review, and material origin were hidden from annotators.Source: §E.3 · p. 15
  • In the initial shared-reason annotations, story-support agreement was 85.56% (κ = 0.768) and attachment agreement was 95.19% (κ = 0.833).Agreement and endorsement do not establish correctness or complete adjudication.Source: §E.3 · p. 15
  • On 219 questions with human-majority answer references, aligned reasons raise Jev accuracy from 98.2% to 100.0%, while swapped reasons lower it to 24.9%.The paired changes are +1.8 and −73.3 points, with intervals [0.5, 3.6] and [−77.6, −68.9].Source: §E.3 · p. 15

E.4 Material selection, repair, and damage

  • On new reason materials, Jev’s aligned change is +3.69 points [1.93, 5.51], while its crossed change is −0.40 points [−3.03, 2.17].Repair and signed damage use all weighted outcomes, so the net change is not conditioned on earlier errors or successes.Source: §E.4 · p. 15

E.5 Matched candidates

  • Jev’s reattachment change is +2.31 points [−4.17, 8.80], while its expansion change is −4.63 [−10.19, 0.93] on matched candidates.GPT-3.5’s expansion change is −13.89 [−19.91, −7.87]; saved-choice scoring reads selections despite invalid probabilities and is post-hoc.Source: §E.5 · p. 16

E.6 Semantic outcomes and output validity

  • Across 9,522 crossed-pair responses, Jev has 448 probability-sum failures and three selected-maximum failures, while DeepSeek has 843 final HTTP 429 outcomes and four parsing failures.Jev’s sum failures are approximately 0.01 from one; strict scoring retains the 10−4 sum and 10−6 selected-maximum tolerances.Source: §E.6 · p. 16

F.1 Sample and conditions

The feedback study tests related-question cues and predicted states on BigToM and SimpleToM samples, while the resource comparison records deployment-specific serving conditions. The discussion proposes component-level integration tests rather than claiming complete-assistant performance.

F.1 Sample and conditions

  • The feedback sample pairs 200 BigToM belief/action cases with 200 SimpleToM awareness/behavior cases, using question-only, predicted-state, and flipped-state conditions.Conditions preserve the story prefix, downstream question, candidates, and order while changing the suffix.Source: §F.1 · p. 16

F.2 Fixed-No content control

  • On 200 SimpleToM behavior targets, Jev gets 177 correct under predicted feedback, and a fixed-No replay reproduces every selected option.The predicted-minus-question-only change over both sources is +3.75 points [1.75, 6.00], but this does not establish benefit from instance-specific predictions.Source: §F.2 · p. 16

G Cost and Response Time

  • Resource estimates cover 11,117 questions per deployment, with reference prices based on public tariffs and timing measured under different serving setups.Timing excludes queuing and model initialization; hosted hardware and provider rate limits were not recorded, so the comparison reflects observed deployments rather than isolated architectural latency.Source: G Cost and Response Time · pp. 16–17

H Potential of Jev Application

  • The discussion proposes testing Jev as a dialogue-agent selector, with candidates that distinguish partner perspective from agent information and are revised as evidence changes.It recommends separately checking candidate coverage, task-specific confidence calibration, and fallback handling for invalid outputs, then evaluating dialogue quality and collaborative task success in a complete system.Source: H Potential of Jev Application · p. 17

I Data Access and Reproduction

The appendix documents data access, reproduction materials, supplementary comparisons, and the exact prompts used in the primary, reason, and feedback studies. It also distinguishes prepared aggregate records from supplementary per-question materials whose release remains planned.

I Data Access and Reproduction

  • The versioned reproduction materials identify source questions, families, subsets, candidate orders, references, predictions, and available probability vectors.The prepared anonymous review archive contains materials and aggregate reason results, while per-question reason outputs and matched-candidate records are archived separately and supplementary release remains planned.Source: I Data Access and Reproduction · p. 17

Supplementary Tables and Figures

  • Tables 4–9 document comparison designs, capability assignments, evaluated subsets, and model versions and settings.The tables specify that model comparisons use one question per request without task-specific fine-tuning and that capability assignments include overlapping application questions.Source: Table 4 · p. 18, Table 5 · p. 18, Table 6 · p. 18
  • Tables 10–14 and Figure 9 report benchmark-equal control-free exact match on 10,197 of 11,117 questions after removing 920 native controls, Jev selection and risk–coverage on 11,117 questions, reason interventions, and matched-candidate accuracy on 108 questions.Table 12 reports pooled accuracy and paired percentage-point differences; Table 13 averages accuracies equally across benchmarks. Figure 9 defines coverage over all questions and error among retained answers.Source: Table 10 · p. 20, Table 11 · p. 20, Figure 9 · p. 21

J.1 Primary evaluation and output example

  • The primary evaluation gives generative models shared context, question, and ordered options, and instructs them to return only an option label.The example contrasts this with typed output containing a choice and candidate probabilities.Source: §J.1 · p. 25

J.2 Reason-intervention instructions

  • The reason-intervention instructions distinguish selecting an answer itself from selecting a complete answer–reason pair.For answer selection, a suggested reason is an untrusted explanation to evaluate rather than an additional story fact.Source: §J.2 · p. 25

J.3 Reason generation and review

  • Reason-generation instructions require one explanation per option, supporting the keyed answer or affirmatively supporting a plausible mistaken distractor without revealing the key.The review rubric checks support, attachment, mapping, leakage, and unresolved material uncertainty, using gold-informed machine review rather than human annotation.Source: §J.3 · pp. 25–26

J.4 Mental-state feedback suffixes

  • Feedback suffixes append the related upstream question and, for predicted or flipped conditions, the selected answer text without references or probabilities.The predicted and flipped suffixes differ only in which upstream answer text is supplied.Source: §J.4 · p. 26

Related papers

Loading 2610.18610v1…