PaperlayerView exact revision on arXiv

Source-linked AI summary

MindZero: Learning Online Mental Reasoning With Zero Annotations

Shunchi Zhang, Jin Lu, Chuanyang Jin, Yichao Zhou, Zhining Zhang, Tianmin Shu

arXiv:2606.00240v1cs.AIcs.MA

TL;DR

Real-world assistance needs Theory of Mind that continuously tracks uncertain mental-state hypotheses, but existing approaches can be costly or annotation-dependent. MindZero trains multimodal language models through self-supervised reinforcement learning and significantly outperforms baselines across mental-reasoning and assistance tasks while reducing computational cost.

  • Problem

    Existing ToM work largely relies on question answering, whereas real-world assistance requires continuous uncertainty tracking over competing hypotheses and computationally feasible updates.

  • Method

    MindZero trains multimodal language models with self-supervised reinforcement learning to generate mental-state hypotheses rewarded for explaining observed actions, enabling annotation-free single-pass inference.

  • Results

    MindZero significantly outperformed baselines across question-answering and proactive-assistance tasks; in GridWorld QA, it delivered a 2.1–2.5× accuracy gain over base models.

  • Takeaways & Limitations

    Mental reasoning can be learned as a self-supervised skill while retaining robustness and uncertainty tracking comparable to model-based methods at substantially lower computational cost.

  • Takeaways & Limitations

    MindZero does not currently model recursive reasoning between multiple agents, and longer input sequences require proportionally more input tokens.

Abstract

from arXiv · show

Effective real-world assistance requires AI agents with robust Theory of Mind (ToM): inferring human mental states from their behavior. Despite recent advances, several key challenges remain, including (1) online inference with robust uncertainty updates over multiple hypotheses; (2) efficient reasoning suitable for real-time assistance; and (3) the lack of ground-truth mental state annotations in real-world domains. We address these challenges by introducing MindZero, a self-supervised reinforcement learning framework that trains multimodal large language models (MLLMs) for efficient and robust online mental reasoning. During training, the model is rewarded for generating mental state hypotheses that maximize the likelihood of observed actions estimated by a planner, similar to model-based ToM reasoning. This method thus eliminates the need for explicit mental state annotations. After training, MindZero internalizes model-based reasoning into fast single-pass inference. We evaluate MindZero against baselines across challenging mental reasoning and AI assistance tasks in gridworld and household domains. We found that LLMs alone are insufficient; model-based methods improve accuracy but are slow, costly, and limited by backbone MLLM capacity. In contrast, MindZero enhances MLLMs' intrinsic ToM ability and significantly outperforms model-based methods in both accuracy and efficiency, showing that mental reasoning can be effectively learned as a self-supervised skill.

1. Introduction

MindZero addresses the difficulty of training robust online Theory-of-Mind systems when mental states are latent, dynamic, and costly to annotate. It uses self-supervised reinforcement learning to train multimodal language models for efficient, annotation-free mental reasoning and reports improvements over baselines across gridworld and household tasks.

  • Motivation: Proactive real-world assistance requires Theory of Mind to infer users’ desires, beliefs, and goals from their behavior.These mental-state inferences support anticipating users’ needs.
  • Challenges: Online mental reasoning is difficult because mental states are latent, ambiguous, dynamically changing, and costly to annotate reliably in real-world domains.These challenges have limited prior learning-based Theory-of-Mind methods to controlled settings.
  • Prior approaches: Inference-time reasoning combined with model-based Theory of Mind, such as Bayesian inverse planning, avoids training annotations but has limitations addressed by MindZero.The passage motivates moving beyond inference-time approaches while retaining model-based grounding.
  • MindZero: MindZero trains multimodal language models with self-supervised reinforcement learning to generate mental-state hypotheses rewarded for explaining observed behavior, enabling single-pass inference without annotations.Its objective remains grounded in model-based reasoning to preserve robustness and interpretability.
  • Evaluation: Small multimodal language models trained with MindZero significantly outperformed baselines across question-answering and proactive-assistance tasks in gridworld and household environments.They matched the robustness of model-based methods while significantly reducing computational cost, with further validation in an IRB-approved human study.

2. Related Work

Related work spans prompting-based, model-based, and other approaches to Theory of Mind reasoning, while ToM-guided assistance requires continuous mental-state inference and uncertainty updates beyond question answering. Prompting methods improve base LLMs but retain systematic reasoning errors, whereas assistance settings demand ongoing inference during interaction.

  • Theory of Mind Methods: ToM reasoning methods comprise prompting-based, model-based, and other approaches, with prompting improving base LLMs but retaining systematic errors in long-context, complex, and recursive reasoning.The passage identifies three main categories and specifically characterizes the limitations of prompting-based methods.
  • ToM-Guided Assistance: ToM research has mainly addressed question answering from stories or videos, whereas ToM-guided assistance requires continuously inferring and updating mental states under uncertainty.Assistance therefore differs from answering questions about already-presented scenarios.

3. Problem Formulation

The problem formulation unifies online inference of users’ latent beliefs and goals from sequential behavior with uncertainty-aware proactive assistance. It emphasizes jointly modeling history, updating multiple hypotheses efficiently, and replanning in real time despite computational bottlenecks.

  • 3. Problem Formulation: MindZero formulates online mental state inference and proactive assistance within a unified probabilistic framework for dynamic environments.The framework connects latent-state reasoning from sequential observations to assistive decision making.
  • 3.1. Online Mental Reasoning: Given observed states s1:t and actions a1:t, a ToM model infers the user’s latest mental state mt, including beliefs bt and goals gt.The formulation is inspired by Bayesian inverse planning (BIP).
  • 3.1. Online Mental Reasoning: Online inference estimates the posterior P(mt | s1:t, a1:t) over mental states from the user’s behavioral history.The posterior is defined through Bayesian inference using the observed state-action sequence.
  • 3.1. Online Mental Reasoning: Unlike typical Markovian BIP formulations, MindZero models all past behavior jointly, but inference can be intractable because of infinite hypotheses and costly action-likelihood estimation.Action likelihoods require forward planning conditioned on hypothetical mental states.
  • 3.2. Proactive Assistance Guided by Online Mental Reasoning: At every step, online mental reasoning updates multiple hypotheses {mt} and estimates their probabilities {qt} from behavior history before assistive planning.The assistive agent plans actions using the top mental-state hypotheses.
  • 3.2. Proactive Assistance Guided by Online Mental Reasoning: Proactive assistance must account for uncertainty across hypotheses, frequently update plans from recent behavior, and support fast real-time replanning.MindZero targets these requirements by training a small multimodal language model with low computational cost and latency.

4. MindZero

MindZero is a self-supervised reinforcement-learning framework that trains multimodal language models for efficient, robust online mental reasoning without ground-truth mental-state annotations. It rewards explicit mental-state hypotheses that explain observed actions, while entropy preserves diverse posterior reasoning and enables real-time planning.

  • MindZero: MindZero trains multimodal language models for efficient and robust online mental reasoning directly from behavioral data using self-supervised signals.This addresses the scarcity and collection difficulty of ground-truth mental-state annotations in real-world domains.
  • MindZero: An action-likelihood evaluator, either a model-based planner or an LLM, estimates observed-action likelihoods under each proposed mental-state hypothesis.Planners are used in GridWorld, whereas LLMs are used in the Household domain.
  • MindZero: MindZero internalizes Bayesian inverse planning and enables real-time planning for proactive assistance after reinforcement-learning training.Mental-state priors may be uniform or incorporate prior knowledge from symbolic rules or LLMs to constrain the hypothesis space.
  • MindZero: The model is rewarded for generating mental-state hypotheses that maximize the likelihood of observed user actions, treating actions as evidence rather than prediction targets.This reframes mental reasoning as inverse modeling centered on explanatory consistency.
  • MindZero: MindZero approximates the posterior of mental states through variational inference and optimizes an Evidence Lower Bound because traversing the full hypothesis space is intractable.The model generates a finite candidate set with normalized posterior probabilities, treating those candidates as the effective variational-posterior support.
  • MindZero: The entropy term encourages exploration over mental-state hypotheses, preventing premature collapse to one mode and promoting robust, diverse posterior approximations.High rewards also require hypotheses to be explicit and meaningful to both action-likelihood and mental-state-prior estimators.

5. Experimental Setup

MindZero is evaluated in four settings spanning GridWorld and household question answering and proactive assistance. These tasks test visual or multimodal mental-state inference, continuous online goal updating, and efficient help under ambiguity.

  • Experimental settings: The evaluation covers GridWorld Question Answering, GridWorld Proactive Assistance, Household Question Answering, and Household Proactive Assistance.Question answering asks directly about mental states, whereas assistance requires fast, online reasoning.
  • GridWorld: In GridWorld, models infer which colored blocks a human intends to assemble from partial trajectories while grounding language and actions in the visual scene.The Construction environment contains obstacles, colored objects, and human navigation actions.
  • GridWorld Proactive Assistance: GridWorld proactive assistance requires timestep-by-timestep hypothesis generation and early-but-reliable online goal inference to accelerate task completion.Helping performance is measured by speedup, the amount by which assistance accelerates completion.
  • Household Question Answering: Household question answering uses MMToM-QA to jointly infer a person’s beliefs and goals from visual and textual inputs while searching for objects.All methods receive the same fused textual representations produced using established information-fusion methods.
  • Household Proactive Assistance: Household proactive assistance uses O-WAH, requiring online goal updates in realistic environments with larger spaces, partial observability, and longer episodes.An uncertainty-aware helping planner generates assistance from the latest observations at each step.
  • Baselines: Baselines include open-weight base models, zero-shot large models, and test-time scaling methods ThoughtTracing and AutoToM, with scaling methods excluded from proactive assistance for slow inference.MindZero uses the same open-source base models for fair comparison; proactive assistance reports non-thinking large-model results because decisions must be real time.

6. Experimental Results

MindZero consistently improves accuracy and proactive-assistance speed across GridWorld and Household tasks while maintaining low inference cost. Its online goal inference steadily improves as evidence accumulates, and ablations plus a human study identify robust uncertainty modeling and practical real-world effectiveness.

  • Question Answering: MindZero consistently outperforms pretrained and test-time scaling baselines in GridWorld and Household question answering while maintaining low inference cost.
  • Question Answering: 2.1–2.5× accuracy gain over Qwen3-VL-4B and Qwen3-VL-8B base models makes MindZero the most accurate method in GridWorld QA.
  • Proactive Assistance: MindZero achieves the best proactive-assistance performance, with 23.0% and 24.5% speedup for Qwen3-VL-4B and Qwen3-VL-8B in GridWorld.GPT-5.2 and Gemini-3-Flash yield no speedup because constantly changing goal predictions destabilize actions.
  • Proactive Assistance: 19.1% speedup is achieved by MindZero with Qwen3-4B in Household Proactive Assistance, while Llama-3.2-3B shows no significant gain because it cannot produce correctly formatted goal hypotheses.The Llama-3.2-3B model is first fine-tuned on generations sampled from pretrained Llama-3.1-8B before RL training.
  • Online Goal Inference: MindZero’s online goal-prediction accuracy steadily increases with task progress, substantially outperforming base models and matching much larger pretrained models.Stable online goal inference is identified as a key reason for effective proactive assistance.
  • Ablation Studies: 13.9% speedup drop without entropy bonus shows that retaining diverse hypotheses is important, while removing explicit prior modeling causes a 2.1% speedup drop.Multiple hypotheses prevent premature commitment under ambiguous early observations, and entropy regularization mitigates overconfident mode collapse.
  • Human Study: 19.7% speedup is achieved by MindZero in a human study, compared with 2.6% for pretrained Qwen3-4B; its difference from Gemini-3-Flash’s 23.4% is not statistically significant.The study involved 12 Johns Hopkins University participants completing four household tasks.

7. Conclusion

MindZero trains multimodal language models through self-supervised reinforcement learning for robust, efficient online Theory of Mind reasoning without mental-state annotations. Future work targets recursive multi-agent reasoning and more efficient handling of long input sequences.

  • Contributions: MindZero rewards hypotheses that best explain observed behavior, helping models internalize model-based Theory of Mind reasoning while retaining single-pass inference speed.The framework is designed for robust and efficient online reasoning without mental state annotations.
  • Contributions: MindZero was evaluated across question answering and proactive assistance tasks.
  • Limitations and Future Work: MindZero does not currently model recursive reasoning between multiple agents.The authors plan to incorporate multi-agent recursive mental reasoning into training.
  • Limitations and Future Work: As input sequence length increases, the model’s required input token length increases accordingly.Future work includes developing a more efficient model structure for long input sequences.

Impact Statement · A. MindZero Implementation Details

MindZero aims to make online Theory of Mind reasoning in assistive AI more robust and efficient without explicit annotations. The paper also highlights risks including manipulation, surveillance, profiling, inappropriate assistance, reduced autonomy, and bias without safeguards.

  • Impact Statement: MindZero targets robust and efficient online Theory of Mind reasoning for assistive AI systems.The stated aim is to advance machine learning through improved online mental reasoning.
  • Impact Statement: The approach infers human intentions and uncertainty from behavior without relying on explicit annotations.This annotation-free capability is presented as a potential benefit for real-world AI agents.
  • Impact Statement: Potential applications include household assistance, digital services, and human–computer interaction.These domains are identified as examples of real-world deployment.
  • Impact Statement: More accurate modeling of human intentions and beliefs could be misused for manipulation, surveillance, or unwanted behavioral profiling.These risks may arise if systems are deployed without appropriate safeguards.
  • Impact Statement: Errors in inferred mental states could produce inappropriate assistance or reduce user autonomy.The paper identifies these as possible consequences of inaccurate mental-state inference.
  • Impact Statement: Behavioral-data biases could be reinforced by systems that infer human mental states.Responsible deployment therefore requires safeguards against these risks.

A.1. Model Training … B.1. Environment Setup

MindZero training uses GRPO with specified hardware, rollout, optimization, and precision settings, while evaluation defines collaboration speedup and FLOP-based inference cost. Prompts and GridWorld environments are standardized through controlled contexts, procedural generation, multimodal observations, and reproducible episode records.

  • A.1. Model Training: MindZero models are trained with standard GRPO in VeRL using 4×H100 GPUs, with 32 rollout samples per prompt and 20 training epochs in bf16.Training uses a rollout batch size of 32, global batch size of 8, AdamW, learning rate 1 × 10−6, weight decay 1 × 10−2, and max grad norm 1.0.
  • A.2. Evaluation Metrics: Proactive assistance is evaluated using a speedup metric that measures collaborative efficiency relative to remaining stationary.Thuman denotes stationary-helper time, while Tcollab denotes time with active assistance.
  • A.2. Evaluation Metrics: Inference cost is reported in floating point operations using FLOPs approximated from active parameters and processed tokens.The supplied approximation is FLOPs = 2 × Pactive × Ntokens, with Pactive as active parameter count and Ntokens as total processed tokens.
  • A.3. Prompt Examples: Each task uses the same instruction with different context inputs, and Household reward evaluation adopts prompts similar to AutoToM.The paper provides task-context examples in Figures 6–9.
  • A.3. Prompt Examples: Prompts and datasets are open-sourced through the MindZero GitHub repository and Hugging Face dataset page.The supplied passage gives both public resource links.
  • B. GridWorld Experiments: The GridWorld experiments include Question Answering and Proactive Assistance, with experimental details provided for Sections 5.1 and 5.2.This passage identifies the two experiment types covered in the GridWorld experiments.
  • B.1. Environment Setup: Episodes use a 10 × 10 grid with U(0, 20) obstacles and 8 uniquely colored and shaped objects, filtered for trajectory length and goal ambiguity.The dataset contains rendered visual observations and detailed textual descriptions of environment rules.
  • B.1. Environment Setup: GridWorld environments and agents accept explicit seeds, while configurations, initial states, and full action histories are stored for reproducibility.Stored records support reproducing any episode or visualization.

B.2. Data Generation · B.3. Agent Policies

The paper generates binary-choice QA data from GridWorld episodes, sampling questions before and after object pickup to probe human intentions and placement targets. Its helping and simulated-human agents use probabilistic, goal-directed policies with collaboration heuristics and stochastic evaluation across multiple seeds.

  • B.2. Data Generation: B.2. Data Generation uses binary-choice questions with grounded natural-language descriptions to test distinct aspects of social reasoning.Each episode generates three query types.
  • B.2. Data Generation: Pre-pick questions infer the intended object or overall goal configuration before the human picks up an object.These queries are sampled at timesteps before pickup and are conditioned on the placement goal.
  • B.2. Data Generation: Post-pick questions infer the intended placement target given the object currently held by the human.These queries are sampled after the human is holding an object.
  • B.2. Data Generation: The QA dataset contains 800 training episodes with 2,400 questions and 100 evaluation episodes with 300 questions.The questions are used for training and evaluation of the QA task.
  • B.2. Data Generation: A separate stepwise inference model is trained on 1000 unlabeled episodes unrolled into timesteps and evaluated on 20 randomly sampled episodes for online assistance.This setup assesses online assistance performance without episode-level labels.
  • B.3. Agent Policies: B.3. Agent Policies defines a helping planner that maintains a goal distribution B = {(g_i, p_i)} and selects actions using a Boltzmann policy based on probability-weighted expected return.The helper predicts the human’s likely target, typically the closer one, and pursues the other while yielding, avoiding blocked paths, and preventing deadlocks.
  • B.3. Agent Policies: The simulated human uses a shortest-path, sequential goal planner with a Boltzmann policy at temperature τ = 0.01 and logical interaction constraints.The planner acquires the proximal target and transports it adjacent to the distal target; proactive assistance also models alternating move-then-pause behavior.
  • B.3. Agent Policies: GridWorld proactive assistance is evaluated across random seeds 10, 20, and 30, with averaged results accounting for planner and human stochasticity.The evaluation averages outcomes across the three specified seeds.

C. Household Experiments · C.1. Environment Setup · C.2. Data Generation

The household experiments use VirtualHome with goal-conditioned planning and sequential Monte Carlo for online goal inference. Data generation covers question answering and proactive assistance, including filtered binary hypotheses and apartment-based household tasks.

  • C. Household Experiments: The experiments evaluate MindZero on Household Question Answering and Proactive Assistance tasks.These experiments are described in Sections 5.3 and 5.4.
  • C.1. Environment Setup: VirtualHome v2.2.4 serves as the household simulator, with agent policies implemented by a goal-conditioned MCTS planner.The simulator and planner define the household environment used in the experiments.
  • C.1. Environment Setup: Online goal inference maintains goal hypotheses over time using Sequential Monte Carlo, following AutoToM.This provides the temporal hypothesis-tracking mechanism for household agents.
  • C.2. Data Generation: MMToM-QA training data is used to construct MindZero’s Question Answering training data.The dataset supplies the source material for training in the household question-answering setting.
  • C.2. Data Generation: Because test questions use binary choices, hypothesis filtering constructs binary options rather than sampling from the full hypothesis space.For goal-related questions, choices pair a randomly sampled observed object with an unobserved one.
  • C.2. Data Generation: Proactive Assistance data uses Apartments #0, #1, #2, #3, and #5 for training generation, and Apartments #3 and #6 for testing generation.The apartment split follows the standard VirtualHome setting.
  • C.2. Data Generation: The proactive-assistance dataset contains 20 training episodes with 968 timesteps and 16 testing episodes.Training and testing data are generated for four evenly distributed task types.
  • C.2. Data Generation: The four proactive-assistance task types are setting up a table, loading the fridge, preparing food, and loading the dishwasher.These tasks define the household activities represented in the generated episodes.

D. Human Experiment … F. Full Results of Question Answering

The paper evaluates MindZero in a human household-assistance experiment and reports methodological details for comparing question-answering baselines across GridWorld and Household domains. It also provides per-task human-assistance speedups and full question-answering results, with textual transcripts used when baselines lack multimodal support.

  • D. Human Experiment: The human experiment recruited 12 Johns Hopkins University students, obtained IRB approval and informed consent, and allowed voluntary withdrawal.Participants included undergraduates, master’s students, and Ph.D. students; five were male and seven female.
  • D. Human Experiment: Each approximately 60-minute session involved household tasks in a simulated apartment, while the system recorded task-related interaction logs.Participants operated the environment through a computer interface.
  • D. Human Experiment: Across all four tasks, MindZero trained from Qwen3-4B achieved positive speedup, whereas pretrained Qwen3-4B slowed humans on Tasks 5 and 13.The results report average task-completion steps and speedup over the Single Human setting, using MindZero w/ Qwen3-4B as the base model.
  • D. Human Experiment: MindZero’s per-task speedup gap relative to Gemini-3-Flash was small and varied in sign, consistent with no statistically significant difference.The passage describes this comparison across the four household-assistance tasks.
  • E.1. ThoughtTracing: ThoughtTracing was evaluated using its original unmodified implementation and prompts on all 600 Household Question Answering test instances.The complete test set was used for fair comparison with other baselines and the main experiments.
  • E.2. AutoToM: AutoToM was evaluated with its original implementation and prompts across multiple backend models, using a uniform inference distribution after parsing errors.Parsing errors may occur for smaller models with limited instruction-following capabilities, such as Llama-3.2-3B.
  • E.3. Textual Transcripts: For GridWorld Question Answering, textual transcripts replaced multimodal inputs because ThoughtTracing and AutoToM do not support multimodal evaluation.The paper illustrates this protocol with an example transcript and marks text-only methods with an asterisk.
  • E. Test-Time Scaling Methods: The question-answering evaluation compared ThoughtTracing and AutoToM with MindZero and other baselines across GridWorld and Household domains.Full results are presented in Table 4, with best overall and open-weight results highlighted.
Loading 2606.00240v1…