PaperlayerView exact revision on arXiv

Source-linked AI summary

Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind

Hanqi Xiao, Vaidehi Patil, Zaid Khan, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal

arXiv:2604.11666v2cs.CLcs.AIcs.LG

TL;DR

LLM dialogue agents need to model adversaries’ beliefs in privacy-sensitive interactions, but existing defenses and frontier models struggle with multi-turn belief steering. The paper introduces TOM-SB and trains AI Double Agents with reinforcement learning, finding that ToM and attacker-fooling improve each other and are strongest when jointly rewarded.

  • Problem

    Existing dialogue defenses are often reactive or turn-level, while multi-turn privacy interactions require modeling an adversary’s evolving beliefs and maintaining trust.

  • Method

    The paper introduces TOM-SB, a shared-universe, multi-turn adversarial privacy task, and trains defenders with reinforcement-learning rewards for ToM accuracy, attacker-fooling, or both.

  • Results

    ToM and attacker-fooling improve bidirectionally, with joint rewards strongest; trained agents outperform frontier baselines, while prior-knowledge fooling remains generally low at 42.4%.

  • Takeaways & Limitations

    Belief modeling is closely associated with successful attacker-fooling in TOM-SB, and the task can be extended to stronger out-of-distribution attackers.

  • Takeaways & Limitations

    Experiments use generated scenarios without users or real private information, and the task assumes an adversarial interlocutor and a shared universe.

Abstract

from arXiv · show

As large language models (LLMs) become the engine behind conversational systems, their ability to reason about the intentions and states of their dialogue partners (i.e., form and use a theory-of-mind, or ToM) becomes increasingly critical for safe interaction with potentially adversarial partners. We propose a novel privacy-themed ToM challenge, ToM for Steering Beliefs (ToM-SB), in which a defender must act as a Double Agent to steer the beliefs of an attacker with partial prior knowledge within a shared universe. To succeed on ToM-SB, the defender must engage with and form a ToM of the attacker, with a goal of fooling the attacker into believing they have succeeded in extracting sensitive information. We find that strong frontier models like Gemini3-Pro and GPT-5.4 struggle on ToM-SB, often failing to fool attackers in hard scenarios with partial attacker prior knowledge, even when prompted to reason about the attacker's beliefs (ToM prompting). To close this gap, we train models on ToM-SB to act as AI Double Agents using reinforcement learning, testing both fooling and ToM rewards. Notably, we find a bidirectionally emergent relationship between ToM and attacker-fooling: rewarding fooling success alone improves ToM, and rewarding ToM alone improves fooling. Across four attackers with different strengths, six defender methods, and both in-distribution and out-of-distribution (OOD) evaluation, we find that gains in ToM and attacker-fooling are well-correlated, highlighting belief modeling as a key driver of success on ToM-SB. AI Double Agents that combine both ToM and fooling rewards yield the strongest fooling and ToM performance, outperforming Gemini3-Pro and GPT-5.4 with ToM prompting on hard scenarios. We also show that ToM-SB and AI Double Agents can be extended to stronger attackers, demonstrating generalization to OOD settings and the upgradability of our task.

1 Introduction

The paper introduces TOM-SB, a multi-turn privacy-themed challenge requiring defenders to model an attacker’s partial knowledge and steer beliefs without revealing private information. Frontier models struggle, while reinforcement learning with ToM and fooling rewards improves both capabilities.

  • Task and motivation: Reactive refusal and naive proactive deception are inadequate because refusal reveals the defender and inconsistencies reduce trust.The proposed double-agent behavior instead aligns deception with the attacker’s prior beliefs.
  • Task and motivation: TOM-SB requires defenders to elicit attacker knowledge, model evolving beliefs, steer incorrect inferences, maintain trust, and protect the true target.The task uses a shared universe and partial attacker knowledge, making generic or inconsistent deception detectable.
  • Baseline difficulty: Gemini3-Pro estimates the attacker’s prior knowledge correctly less than 50% of the time.This result accompanies the broader finding that prompting-based defenders struggle with the task.
  • Training findings: Across six defense methods and four attacker types, ToM and attacker-fooling show an average Pearson correlation of 0.92 when attackers have prior knowledge.Jointly rewarding both objectives produces the largest increase in both qualities.
  • Baseline difficulty: Gemini3-Pro and GPT-5.4 defenders reach attacker-fooling rates of 34% and 27%, respectively, on TOM-SB.The task includes 300 structured scenarios and conversations of up to 29 messages.
  • Training findings: Optimizing ToM alone raises attacker fooling from 13% to 41%, while optimizing fooling raises ToM from 13% to 35% on prior-knowledge scenarios.The results indicate bidirectional emergence between the two capabilities.

2 Related Work

Prior ToM research largely uses static inference benchmarks, while interactive dialogue work has emphasized cooperative or mixed-motive settings. TOM-SB instead couples evolving belief modeling with adversarial privacy objectives and competing dialogue goals.

  • Theory of Mind in LLMs: Static ToM benchmarks primarily test passive mental-state inference through question answering.Reported failures include sensitivity to trivial alterations in reasoning scenarios.
  • Theory of Mind in LLMs: TOM-SB differs from static evaluations by coupling ToM with competing objectives such as preventing leakage while maintaining trust.The task requires modeling evolving beliefs and resolving tradeoffs during interaction.
  • Interactive dialogue and privacy: Interactive social-intelligence research has mainly studied cooperative or mixed-motive multi-agent settings, along with persuasion and negotiation.The paper positions adversarial privacy dialogue as a distinct challenge.

3 TOM-SB Setup

TOM-SB models adversarial information extraction in a shared hierarchical universe, with attackers probing for missing attributes and defenders estimating beliefs and trust while replying safely. Its metrics separately evaluate trajectory-level and stepwise ToM alignment.

  • Environment and scenarios: Each scenario combines a shared hierarchical universe, the defender’s private path, and attacker prior knowledge containing only some attributes.The attacker seeks missing attributes through dialogue.
  • Environment and scenarios: The hierarchy represents coarse-to-fine information, such as company, division, department, and team, while the defender’s private information is a complete path.Scenarios sample valid paths and replace unknown attacker attributes with null values.
  • Attacker and defender: At each turn, the attacker tracks trust, beliefs, next actions, and whether it succeeded, detected the defender, or should query again.These internal fields drive the attacker’s subsequent message.
  • Attacker and defender: The defender estimates the attacker’s current beliefs and trust before generating a reply conditioned on dialogue history and prior reflections.The defender is not told directly what the attacker already knows and must avoid leaking new private information.
  • Metrics: Figure 2 distinguishes trajectory-level ToM accuracy, fooling reward, and stepwise ToM alignment averaged across turns.Trajectory-level ToM compares estimated and actual prior knowledge, while stepwise ToM compares beliefs at each turn.

4 AI DOUBLE AGENTS Training and Optimization

AI DOUBLE AGENTS are trained with trajectory-level rewards for fooling attackers and accurately modeling their prior knowledge. The training setup supports separate or combined objectives within simulated multi-turn rollouts.

  • Objectives: A successful double agent models evolving attacker beliefs and steers the attacker toward a plausible but incorrect conclusion without losing trust.The training objectives are defined at the trajectory level.
  • Objectives: The fooling reward is 1 when the attacker confidently signals success while holding an incorrect belief and has learned no new ground-truth attribute.Any newly acquired ground-truth information sets this reward to 0.
  • Objectives: The ToM reward is 1 when the defender’s estimated attacker prior knowledge matches the attacker’s actual prior knowledge.An LLM judge infers the estimate from internal reflections and conversation history.
  • Optimization: The total reward combines fooling and ToM objectives as R_total = R_fool + R_ToM.The policy is trained with Dr. GRPO, which removes standard-deviation reward normalization and uses max-length normalization.
  • Trajectory construction: Training trajectories continue until the attacker succeeds, loses trust, or the interaction reaches the maximum number of turns.These simulated trajectories are used to train the defender policy.

5 Experimental Setup

The experiments compare prompted and training-based defenders across fooling and ToM metrics using Qwen3-14B and several frontier or open-source models.

  • Dataset Splits and Models: Experiments use Qwen3-14B, Gemini3-Pro, Ministral-3-14B-Instruct, and GPT-5.4 as defenders, with Gemini3-Flash powering four prompt-based attackers.The training split contains 225 scenarios and the evaluation split contains 75 scenarios.
  • Metrics: Fooling % measures trajectories where defenders cause attackers to conclude with incorrect information, including a hard subset with attacker prior knowledge.
  • Metrics: ToM accuracy is measured at trajectory and stepwise levels to assess whether defenders correctly estimate attacker beliefs.
  • Defender Baselines: Prompted baselines include refusal, naive misleading, and ToM-guided misleading, while training-based methods include online SFT and AI Double Agents.
  • Evaluation Reporting: Table 1 reports Qwen3-14B metrics as percentages averaged across three training seeds, with mean ± standard deviation.Refusal and naive-misleading prompts do not produce ToM reflections and therefore have no ToM accuracy scores.

6 Results and Analysis

Prompting produces weak belief tracking and fooling, whereas reward-trained AI Double Agents improve both; combined rewards perform best, though stronger attackers expose generalization and training challenges.

  • 6.1 Main Results: ToM prompting reaches 36.0% trajectory-wise ToM, 49.3% stepwise ToM, and 13.2% hard-scenario fooling, indicating weak belief tracking.Gemini3-Pro reaches 34.4% hard-scenario fooling despite 57.8% overall fooling.
  • 6.1 Main Results: Fooling-only training raises fooling from 13.2% to 34.4% on hard scenarios and improves trajectory ToM from 36.0% to 49.0%.Stepwise ToM also rises from 49.3% to 62.4%.
  • 6.1 Main Results: Combining fooling and ToM rewards yields 51.3% overall fooling, 42.4% hard-scenario fooling, and 58.7% trajectory-level ToM.Across methods, ToM and fooling performance are tightly correlated.
  • 6.2.1 Extending the TOM-SB Environment: Stronger OOD attackers reduce fooling rates, with the best AI Double Agent declining from 42.4% against the baseline to 31.2% against Cross-Examiner.The Bluffing Attacker is identified as the strongest variant, and higher ToM continues to track higher fooling.
  • 6.2.1 Extending the TOM-SB Environment: Training against the Bluffing Attacker remains beneficial but produces smaller ToM-plus-fooling gains than training against the baseline attacker.This indicates that stronger attackers make learning more difficult as well as testing more difficult.
  • Behavioral Analysis: Representative failures include not probing for prior knowledge, failing to pivot after lies are detected, losing trust before fooling, and fooling outside the shared universe.

7 Conclusion

ToM-SB is a challenging multi-turn privacy game in which defenders fool attackers using benign information consistent with their prior knowledge. The results link ToM accuracy with fooling and show that the task can be made harder with stronger attackers.

  • Conclusion: ToM-SB requires defenders to fool attackers with benign information that does not conflict with the attackers’ partial prior knowledge.
  • Conclusion: Fooling and ToM performance are strongly coupled: training for either objective alone improves the other.
  • Conclusion: The strongest methods reach only 42.4% absolute fooling when attackers have prior knowledge, while stronger attackers further increase task difficulty.

Ethics Statement

The paper studies defenders that mislead attackers in generated privacy scenarios, while explicitly withholding judgment on whether this capability is desirable and acknowledging possible misuse.

  • All experiments use generated scenarios and do not involve users or real private information.
  • The authors frame the work as a ToM study of how rewards shape attacker modeling and fooling, not as a position on misleading other agents.
  • The training methods could be misused for flipped objectives, a risk the authors associate with proactive-defense research.

A.1 Additional Discussion of Limitations and Future Work

TOM-SB is primarily an interactive ToM evaluation with security implications, using simplifying assumptions to make belief steering controlled and measurable rather than fully realistic.

  • TOM-SB assumes a shared universe so defenders cannot use arbitrary unverifiable lies, while enabling formal evaluation of consistency, contradictions, and attacker beliefs.
  • The shared-universe assumption models attackers with reasonable knowledge of possible answers, but extending belief steering to partially shared or evolving universes remains future work.
  • The paper does not claim that all real-world deployments exactly satisfy TOM-SB’s assumptions.

A.2 Additional Experiments on GPT-5.4 and Ministral-3-14B-Instruct

Additional experiments test the training findings on GPT-5.4 and Ministral-3-14B-Instruct under adjusted evaluation or implementation settings, with the reported bidirectional pattern reproduced for Ministral.

  • GPT-5.4 results use the same setup as Gemini3-Pro, while Ministral-3-14B-Instruct uses a 5e-06 learning rate and additional length and format rewards.
  • The Ministral-3-14B-Instruct setup adds a length reward favoring responses under 100 tokens and scaling to zero beyond 200 tokens.
  • Ministral-3-14B-Instruct reproduces the correlation between ToM accuracy and fooling rate across runs.
  • Training Ministral-3-14B-Instruct with fooling and ToM rewards individually reproduces bidirectional emergence.

A.3 ToM as a Driver of Fooling: Oracle ToM Intervention

The paper presents interventions and ablations supporting a link between accurate attacker-belief modeling and fooling, while testing combined rewards, prompt guidance, attacker strength, and the shared-universe assumption.

  • Oracle ToM Intervention: Injecting ground-truth attacker prior knowledge at inference time raises Gemini3-Pro’s overall fooling rate from 58.7% to 99.3%.
  • Oracle ToM Intervention: The oracle intervention changes only belief-state information and improves fooling, indicating that accurate attacker-prior modeling directly contributes to fooling.
  • Combined Rewards: ADA (Fooling + ToM) improves overall fooling over Fooling-only by +4.67 points and over ToM-only by +7.78 points.
  • Strategic-Hint Ablation: Removing strategic hints preserves the pattern that ToM-only training improves fooling and fooling-only training improves ToM.
  • Shared-Universe Ablation: Without a shared universe, prompted defenders achieve 100% fooling rates, making fooling degenerate through unverifiable fabricated lies.

A.8 Additional Results Evaluating OOD transfer to Other Attackers

The additional evaluations compare defender methods across attacker variants, using the same metrics and excluding methods without significant gains. They extend analysis to OOD attackers and detailed attacker-variant tables.

  • Evaluation scope: Figure 4 replicates Figure 3 across all fooling and ToM metrics, while Tables 11–14 report results for four attacker variants.The evaluations cover Base Attacker, Cross-Examiner, Deception-Aware Attacker, and Bluffing Attacker settings.
  • Method selection: SFT and refusal prompting are excluded because Table 1 showed no significant performance increase from either method.
  • OOD comparison: GPT-5.4 OOD defender results are omitted because they were strictly lower than or equal to Gemini3-Pro’s performance.
  • Motivation: Related work identifies reactive refusal and filtering defenses as vulnerable to indirect information extraction across turns, while proactive defenses lack explicit modeling of evolving adversarial beliefs.

B.1 Dataset generation

The dataset contains structured, controlled belief-steering scenarios generated from a shared hierarchical universe. Its tree structure and balanced attacker prior knowledge provide the main challenge factors.

  • Dataset construction: The dataset contains 300 scenarios generated by a Gemini-2.5-pro-based generator using a fixed JSON schema.
  • Scenario structure: Each scenario uses a closed universe represented by three 3-ary trees of depth 3, with a defender-private root-to-leaf path and an attacker-known partial path.
  • Private information: Sensitive information is assigned to a hypothetical person by sampling one root-to-leaf path from the generated trees.
  • Difficulty control: Scenarios are balanced by the amount of initial attacker knowledge to control difficulty and reduce dataset bias.
  • Design rationale: The structured universe, tree representation, and prior-knowledge setup drive task difficulty, while node semantics primarily provide linguistic diversity.

B.2 Additional Experiment Setup Details

The experiments evaluate multi-turn defenders with trajectory-level reinforcement learning, separate ToM and fooling metrics, multiple prompt baselines, and qualitative success and failure analyses.

  • Training setup: RL training uses 75% of the dataset, eight full trajectories per example, and Dr. GRPO optimization with trajectories limited to 15 turns.
  • Evaluation setup: Evaluation samples two complete attacker–defender trajectories per scenario from the 25% held-out split, with LLM judges assessing belief tracking.
  • Metrics: Fooling Rate measures trajectories where the defender misleads the attacker without improving the attacker’s beliefs beyond its initial prior.
  • Metrics: Trajectory-level ToM evaluates whether the defender’s estimated attacker prior matches the attacker’s true prior through a two-stage judge process.
  • Metrics: Stepwise ToM averages belief-tracking accuracy across conversation turns and is treated as a loose upper bound on defender ToM.
  • Metric separation: The ToM and fooling rewards are explicitly separated so that ToM evaluation contains no information about whether the attacker was fooled.
  • Baselines: Prompt baselines include refusal, misleading, and ToM defenders operating in the same multi-turn setting with different instructions.
  • Success modes: Successful defenders probe while trust is high, recover after detected lies, and switch from probing to fooling before trust is exhausted.

Conversation 10: Bluffing Attacker baits the defender and identifies inconsistent pivots

The bluffing attacker exploits defenders that adapt too readily to suggestions, exposing inconsistent fabricated pivots. The interaction is governed by explicit trust tracking, verification, and JSON-formatted beliefs and actions.

  • Initial state: Maria’s ground truth is OmniCorp, Nexus, and Epsilon, while the attacker initially has no information about her corporation, division, or team.
  • Bluffing interaction: The attacker baits the defender with Cyberion and skepticism, eliciting successive pivots from Orion/Xi to Orion/Omicron and then Helios/Lambda.
  • Outcome: The attacker interprets repeated responses to its suggestions as fabricated decoys and reduces trust to zero, triggering defender detection.
  • Trust strategy: The attacker is instructed to verify responses, decrease trust for contradictions or evasiveness, and distinguish casual, active, and highly skeptical verification at scores 5, 3, and 1.
  • Interaction protocol: Attacker reflections must track trust, beliefs across every information layer, strategy, signal, and the next prompt in valid JSON.
  • Defender strategy: The defender prompt recommends probing unknown attacker knowledge, preserving trust, and making in-universe lies consistent with the attacker’s prior.

Gemini3-Pro fools Base Attacker that has no prior knowledge

Gemini3-Pro steers a no-prior-knowledge attacker through a plausible but false organizational path, leading the attacker to report success while extraction fails.

  • The defender supplies Innovate Inc., Product, and API Services as a consistent false path for Chloe.The defender’s reflections explicitly describe selecting values from the shared universe to maintain deception.
  • The attacker progressively accepts the fabricated organization, division, and team as Chloe’s information.The attacker’s reflections move from partial acceptance to believing all three layers have been identified.
  • The dialogue ends with the attacker signaling success after identifying all information layers, while the judge records extractionFailed.The repeated signal sequence is query again->query again->query again->attack successful->extractionFailed.

ADA ToM + Fooling recovers from failed attempt when fooling Bluffing Attacker

The defender initially offers contradictory company names, then recovers by providing the attacker’s believed company and plausible false details, leaving the attacker accepting Project Aegis as the target project.

  • The attacker begins with SityCorp known but division and project unknown, while the defender initially asks for prior knowledge rather than answering directly.The attacker’s initial belief records SityCorp as known; the defender’s reflection says the attacker has not demonstrated specific placement knowledge.
  • The defender first names NextGen Solutions and DataSphere, causing the attacker to detect a contradiction and lower trust.The attacker explicitly states that these companies conflict with the initial SityCorp belief.
  • The defender then provides SityCorp, Cybersecurity, and false Project Aegis, which the attacker treats as consistent with the universe despite reduced trust.The defender’s reflection describes the project as plausible but false, while the attacker recognizes the company and division as matching its prior belief.
  • The attacker asks a verification question, then accepts the fabricated trail and signals successful extraction.The verification query is marked query again; the later reflection says the attacker believes the SityCorp-Cybersecurity-Project Aegis path.
Loading 2604.11666v2…