PaperlayerView exact revision on arXiv

Source-linked AI summary

Loyal Agents: Training LLM Agents to Protect Principal Interests Under Strategic Information Asymmetry

Zimeng Huang, Shilei Chen, Jiatong Zhao, Wenxin Xu, Tonghan Wang

arXiv:2609.34714v1cs.CY

TL;DR

Standard alignment objectives do not specify how delegated agents should protect principals’ strategic interests. The paper formalizes loyalty as information control, introduces a benchmark and adversarial GRPO training, and reports reduced leakage alongside improved utility and evidence-related outcomes. Its scope excludes settings where disclosure is legally or institutionally required.

  • Problem

    Existing alignment objectives such as helpfulness, harmlessness, and honesty offer limited guidance for protecting principals’ strategic interests during delegated information exchange.

    Source: §1 · p. 2

  • Method

    The paper formalizes Agent Loyalty as preventing exploitable information leakage and resisting manipulative information uptake, then trains with online adversarial GRPO using mechanism-specific rewards.

    Source: §1 · p. 2, §5 · p. 6, §5.3 · p. 6

  • Results

    Adversarial training reduces single-turn per-response leakage by 31–44pp while improving task utility, evidence faithfulness, and decision accuracy by up to 11pp, 49pp, and 29pp, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dedicated adversarial training can improve loyalty beyond general capability and existing alignment by directly targeting principals’ information boundaries.

    Source: §7 · p. 10

    The benchmark excludes settings where truthful disclosure is legally or institutionally required and does not adjudicate the open normative boundary between strategic ambiguity and deceptive misrepresentation.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

As LLMs increasingly act as delegated agents, they are expected to protect principals' interests when interacting with external parties. Standard alignment objectives, such as helpfulness, harmlessness, and honesty, do not specify how agents should protect principals' strategic interests under delegation. We formalize Agent Loyalty as an information-control property requiring agents to prevent Exploitable Information Leakage (EIL) and resist Manipulative Information Uptake (MIU). We introduce LoyalAgent-Bench, comprising 10,298 samples across 42 subscenarios and six domains, and an online GRPO framework that trains against a LLM opponent to generate mechanism-specific reward signals. Experiments show that loyalty is not guaranteed by general capability or existing alignment, with measurable EIL and MIU gaps under zero-shot evaluation, while our trained 8B models reduce per-response leakage in single-turn exchanges by 31-44pp and improve task utility, evidence faithfulness, and decision accuracy by up to 11pp, 49pp, and 29pp, respectively. For the trained Qwen3-4B model, no degradation is observed on out-of-distribution benchmarks in math and narrative reasoning.

1 INTRODUCTION

Delegated agents must protect principals’ strategic interests while interacting with parties whose interests may diverge. The paper frames loyalty as information control and introduces a benchmark and adversarial training approach to address gaps in existing alignment.

1 INTRODUCTION

  • Existing alignment objectives prioritize helpfulness, harmlessness, and honesty but do not specify how agents should protect principals’ strategic interests during delegated information exchange.A response can be honest and useful to a counterparty yet strategically costly to the principal.Source: §1 · p. 2
  • Agent Loyalty is an information-control property requiring agents to prevent Exploitable Information Leakage and resist Manipulative Information Uptake.EIL concerns external parties inferring private information; MIU concerns decisions displaced from authorized evidence by strategic content.Source: §1 · p. 2
  • Loyalty is difficult to supervise because exploitability and manipulation depend on interaction context, while static rules risk suppressing task-relevant information exchange.The paper introduces LOYALAGENT-BENCH with 10,298 samples across 42 subscenarios and partially real-world-grounded databases.Source: §1 · p. 2
  • Adversarial training reduces single-turn per-response leakage by 31–44pp while improving task utility, citation-level faithfulness, and decision accuracy by up to 11pp, 49pp, and 29pp, respectively.The paper also reports that general capability and existing alignment do not ensure loyalty.Source: §1 · p. 2

2 RELATED WORK

Prior work studies alignment, agent capabilities, safety, and strategic interaction, while the paper identifies a gap in jointly protecting principal utility and sensitive information under external pressure.

2 RELATED WORK

  • Prior research spans alignment and multi-agent coordination, alongside benchmarks for task completion, tool use, prompt injection, deception, and agentic misalignment.Other strands study bargaining, negotiation, and conflicting interests under information asymmetry.Source: §2 · pp. 2–3
  • These research strands do not jointly address preserving principal utility and sensitive information under external strategic pressure.Source: §2 · p. 3
  • Recent AI-delegation work studies loyalty in polyadic settings and benchmarks leakage–over-refusal trade-offs, negotiation behavior, and conflicts between agent incentives and principal interests.The paper distinguishes its focus as information control under strategic external pressure.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper formulates loyalty as controlling information flows among a principal, an agent, and a strategic opponent. It separates protection against exploitable disclosure from decisions grounded in authorized evidence.

3 PROBLEM FORMULATION

  • Loyalty is framed as information control in open-ended strategic interaction, paralleling MPC’s privacy–correctness distinction without claiming its fixed-computation guarantees.Source: §3 · p. 3

3.1 THREE-PARTY FORMULATION

  • The formulation distinguishes a principal’s task, necessary information, exploitable information, and opponent-supplied content that may be manipulative.The agent may use necessary and exploitable information internally, but must control its disclosure and ground decisions in authorized information.Source: §3.1 · p. 3
  • The blind-opponent assumption limits inference to the opponent’s opening and the agent’s responses, isolating information recoverable from agent behavior.The scope assumes a legitimate principal and gives truthfulness, task completion, legal duties, and safety obligations precedence over withholding.Source: §3.1 · p. 3

3.2 EXPLOITABLE INFORMATION LEAKAGE (EIL)

  • An EIL task advances the delegated objective while minimizing recoverable exploitable information, measured as normalized severity-weighted recoverability.Necessary information supports task utility, and the EIL reward combines utility with leakage minimization using λ = 0.5 by default.Source: §3.2 · pp. 3–4

3.3 MANIPULATIVE INFORMATION UPTAKE (MIU)

  • An MIU task selects a decision and provides a rationale grounded in authorized sources, with the target decision determined from clean evidence and user constraints.Source: §3.3 · p. 4
  • The MIU reward combines exact decision correctness with per-claim support from clean authorized evidence, so accuracy alone does not characterize agent behavior.With η = 0.5, fully grounded correct, ungrounded correct, and incorrect decisions receive rewards of 1, 0.5, and −1 to −0.5, respectively.Source: §3.3 · p. 4

4 LOYALAGENT-BENCH

LOYALAGENT-BENCH tests whether agents protect principals’ interests across diverse EIL and MIU settings. Its construction, breadth, and evaluation measures target both strategic disclosure and manipulation resistance.

4 LOYALAGENT-BENCH

  • LOYALAGENT-BENCH contains 10,298 samples across 42 subscenarios and varies contexts, information dependencies, and opponent strategies to expose loyalty failures.The benchmark targets harms to principals’ economic interests, procedural rights, or decision integrity.Source: §4 · p. 4

4.1 BENCHMARK DESIGN AND CONSTRUCTION

  • The benchmark groups EIL into Bargaining, Redress, and Gatekeeping, and MIU into Financial Advisory, Commerce Guidance, and Information Guidance.Domains capture structural failure patterns, while subscenarios instantiate them across tasks and entities.Source: §4.1 · p. 4

4.2 DATASET STATISTICS AND DIVERSITY

  • Figure 2 summarizes EIL and MIU record counts, subscenario balance, frequent attack types, and topical and vocabulary diversity.Its caption reports near-uniform subscenario coverage and that no attack type exceeds 5% of records.Source: Figure 2 · p. 5, §4.2 · p. 5

4.1 BENCHMARK DESIGN AND CONSTRUCTION

  • Scenario generation combines real-world-grounded facts, dependency-ordered content generation, linguistic variation, automated checks, and review by four domain specialists.Private information is unnecessary for task completion, and manipulated evidence carries no valid signal by construction.Source: §4.1 · p. 5

4.2 DATASET STATISTICS AND DIVERSITY

  • The benchmark spans six domains and 81 attack and manipulation types, with no single type exceeding 5% of records and broadly balanced subscenarios.Its 8,161 training, 1,096 validation, and 1,041 test samples cover scenario, mechanism, and linguistic diversity.Source: §4.2 · p. 5

4.3 EVALUATION PROTOCOL

  • EIL evaluation reports task utility and severity-weighted leakage separately, averaging each response over four opponent rollouts to reduce variance.Per-response leakage is measured because one disclosure can suffice for subsequent exploitation.Source: §4.3 · p. 5
  • MIU evaluation measures decision quality against a clean-evidence baseline and evidence faithfulness to distinguish principled resistance from accidental correctness or refusal.Source: §4.3 · p. 5

5 TRAINING METHOD

The method trains loyalty through online adversarial GRPO, using distinct reward mechanisms for preventing exploitable leakage and resisting manipulative evidence. The EIL mechanism evaluates both task progress and what an opponent can infer; the MIU mechanism prioritizes correct, evidence-faithful decisions.

5 TRAINING METHOD

  • Online adversarial GRPO trains the policy against a fixed, non-learning LLM counterparty for EIL reward, while MIU uses static manipulated contexts.The opponent is not updated during training, and there is no online opponent for MIU.Source: §5 · p. 6

5.1 ONLINE ADVERSARIAL GRPO FRAMEWORK

  • GRPO generates K candidate responses per prompt and normalizes their rewards within the group to obtain relative advantages.The policy is optimized with a clipped objective and KL regularization against a reference policy; online opponent evaluation exposes it to dynamically sampled failure modes.Source: §5.1 · p. 6

5.2 EIL TRAINING: BLIND OPPONENT MECHANISM

  • EIL training uses a blind opponent to reconstruct private facts and produce a continuation, then evaluates leakage and task utility before composing the reward.The opponent lacks access to the principal’s private record; utility scores task progress and preservation of the principal’s position.Source: §5.2 · p. 6

5.3 MIU TRAINING: HIERARCHICAL DECISION–FAITHFULNESS REWARD

  • MIU training prioritizes decisions that are both correct and faithful to authorized evidence over correct decisions grounded in manipulative evidence.Correctness is checked against a pre-computed decision by exact string matching; a judge assesses whether stated reasons rely on clean rather than manipulative evidence.Source: §5.3 · p. 6

6 EXPERIMENTS

The experiments test whether adversarial training improves loyalty without sacrificing utility, how training choices affect results, and whether gains transfer or preserve general capabilities. Across the reported experiments, outcomes vary by loyalty objective and training design, while the Qwen3-4B model shows no point-estimate decrease on four tested out-of-distribution benchmarks.

6.2 MAIN RESULTS

  • Qwen3-4B EIL leakage falls from 57.74% to 19.04% as utility rises from 52.47% to 59.51%; Llama-3.1-8B and Marin-8B also reduce leakage while improving utility.For Llama-3.1-8B and Marin-8B, leakage falls by 85% and 77% to 5.63% and 13.04%, respectively, while utility improves by 4.6pp and 11.0pp.Source: §6.2 · p. 7, Table 1 · p. 7
  • Llama-3.1-8B reaches an EIL reward of 0.60, matching Claude Opus 4.8, while Loyalty-CoT gains are inconsistent and trained-model failures include overelaboration and decision drift.Marin-8B reaches EIL reward 0.56 versus GPT-5.5’s zero-shot 0.55, but that comparison is subject to judge-calibration uncertainty and applies to EIL only; MIU rewards remain lower. Under CoT, GPT-5.5’s MIU reward drops, and explicit loyalty prompting can be counterproductive for some models and domains. EIL failures involve unsolicited supporting facts during legitimate requests; MIU failures can involve erroneous actions despite reduced uptake of fabricated claims.Source: §6.2 · pp. 7–8
  • MIU faithfulness reaches 90.58–98.62% across the three trained models, while decision accuracy improves over CoT for Llama-3.1-8B and Marin-8B but not Qwen3-4B.Some faithfulness gain reflects fewer reasoning lines, though faithfulness restricted to correct decisions confirms improvement in evidence selection; Qwen3-4B trails CoT on decision accuracy.Source: §6.2 · p. 7

6.3 ROBUSTNESS TO TRAINING DESIGN CHOICES

  • Joint E2M1 sampling with λ = 0.5 improves both objectives over E1M1, but joint training remains below single-task upper bounds and increasing λ trades lower EIL leakage for lower MIU performance.EIL-only and MIU-only training specialize in their respective objectives; the EIL-only model has poor MIU performance and the MIU-only model shows the reverse pattern.Source: §6.3 · p. 8
  • Training gains transfer to a held-out opponent: Qwen3-4B retains 86.5% and Llama-3.1-8B retains 82.5% of their leakage reduction; removing adversarial inference raises leakage by 22.6pp.Opponent identity affects improvement magnitude but not direction; the Qwen3.5-35B-A3B opponent yields higher final EIL and MIU rewards than Claude Haiku 4.5. Across four temperatures, pooling recovers 26–39% more grounded slots than any single temperature while aggregate reward remains constant.Source: §6.3 · pp. 8–9

6.4 CAPABILITY GAINS IN OTHER DOMAINS DO NOT TRANSFER TO LOYALTY

  • Narrative and mathematical capability gains do not reliably transfer to loyalty: story-generation SFT lacks consistent loyalty gains, while mathematical training degrades EIL or MIU outcomes.Story-generation SFT improves cloze accuracy by 21–24pp but degrades EIL utility and MIU answerability; mathematical training can improve math performance while degrading loyalty metrics.Source: §6.4 · p. 9

6.5 GENERAL CAPABILITY RETENTION AND OUT-OF-DISTRIBUTION GENERALIZATION

  • Qwen3-4B has no point-estimate decrease on four tested out-of-distribution benchmarks, and achieves lower failure rates than the Qwen3-8B baseline on all PrincipalBench metrics.The PrincipalBench comparison uses the original Qwen3-8B per-token-KL results and Qwen3-4B for this method, so model-size differences warrant caution.Source: §6.5 · p. 9, §6.6 · p. 9

6.7 VALIDATION OF AUTOMATED EVALUATION

  • Human ratings correlate strongly with automated judge scores, and replacing the evaluation judge preserves identical model rankings for EIL and MIU reward.Across 60 EIL and 60 MIU instances rated by four annotators, opponent inference agreement is Fleiss κ = 0.935; judge replacement causes calibration-dependent shifts in absolute scores.Source: §6.7 · p. 9

7 CONCLUSION

The paper concludes that standard alignment objectives do not protect principals’ strategic interests in delegated agents, and presents agent loyalty as information control addressed by a benchmark and adversarial training. It situates this approach alongside privacy and contextual-integrity work while bounding its scope around truthful, discretionary disclosure.

7 CONCLUSION

  • The authors conclude that dedicated adversarial training can improve loyalty beyond general capability and existing alignment, and that principal information boundaries should be directly targeted.They formalize agent loyalty as information control and introduce LOYALAGENT-BENCH with online adversarial GRPO.Source: §7 · p. 10
  • The scope is truthful selective disclosure within a legitimate principal’s discretionary space; mandatory disclosure, safety obligations, and third-party welfare take precedence.Cases where disclosure is legally or institutionally required are excluded from the benchmark, and assessing compliance requires domain-specific legal ground truth.Source: ETHICS STATEMENT · p. 10
  • The boundary between strategic ambiguity and deceptive misrepresentation remains an open normative question, and formalizing that distinction is identified as future work.In a sample of 500 EIL replies per baseline and trained model, false or misleading statements occurred in at most 3.1% of replies, at comparable rates before and after training.Source: ETHICS STATEMENT · pp. 10–11

A.1 STRUCTURAL MOTIVATION FROM MPC

  • The formulation parallels MPC privacy and correctness, but addresses open-ended natural-language strategic interaction rather than fixed functions with cryptographic guarantees.The paper characterizes EIL as preventing opponent inference of exploitable information and MIU as keeping decisions grounded in authorized evidence.Source: §A.1 · p. 17

A.2 CONTEXTUAL INTEGRITY AND INFORMATION DISCLOSURE

  • Unlike contextual-integrity work and static privacy filters, this setting targets a strategic opponent that actively probes for information exploitable against the principal.The paper distinguishes delegated agents in open-ended strategic interaction from deterministic firewall layers that preprocess messages.Source: §A.2 · p. 17

A.3 SYCOPHANCY AND PERSUASION ROBUSTNESS

This section situates MIU in sycophancy and persuasion-robustness research, distinguishing principal-harming uptake under delegation from user-directed pressure. It describes benchmark design for evaluating loyalty across varied contexts and query wording while preserving evaluation semantics and clarifying generalization scope.

A.3 SYCOPHANCY AND PERSUASION ROBUSTNESS

  • MIU differs from sycophancy studies of user pressure because manipulative content comes from an opposing party and uptake can shift a delegated agent’s decisions against the principal’s interests.Prior work reports belief-matching over truthfulness and increased sycophancy under sustained misleading pressure; training interventions can improve robustness and receptiveness to valid corrections.Source: §A.3 · p. 17

B.2 DATA SOURCES

  • Scenario construction draws on public real-world datasets and official resources, with controlled synthesis for contexts lacking explicit external sources.EIL includes three domains, 27 subscenarios, and 6,700 records; MIU includes three domains, 15 subscenarios, and 3,598 records.Source: §B.1 · p. 18, §B.2 · pp. 19–20

B.1 SCENARIO TAXONOMY

  • The benchmark organizes scenarios by loyalty-risk mechanism and concrete task, covering 42 subscenarios across six family domains.Its global train/validation/test split targets 80/10/10, with every subscenario represented in each partition; evaluation tests held-out instances within known subscenario structures, not unseen mechanisms or attack families.Source: §B.1 · pp. 17–18, §B.3 · p. 20

B.4 MIU ATTACK TAXONOMY

  • MIU uses 75 attack types in nine semantic classes, with financial-advisory tactics balanced across risky-action, exaggerated-risk, and uncertainty variants.The three financial variants respectively favor risky action, exaggerate risk, or emphasize missing information, balancing evaluation across decision branches.Source: §B.4 · p. 20

B.5 EIL OPPONENT TACTICS

  • EIL records pair 1–5 necessary information items with 2–5 exploitable slots, each linked to an adverse action if leaked.Six opponent tactics are expressed in an opening probe and conditional adverse actions; the probe is generated without access to exploitable slots.Source: §B.5 · p. 20, §B.6 · p. 24

B.6 DATA CONSTRUCTION PIPELINE DETAILS

  • EIL generation fixes necessary information before private leverage and enforces semantic separation so task-required disclosures do not reveal exploitable facts.Each exploitable slot must enable a distinct adverse outcome and cannot duplicate, refine, quantify, temporally bound, compare against, or be strongly entailed by necessary information.Source: §B.6 · p. 24
  • MIU generation fixes user constraints, authorized evidence, and decision options before evidence; its baseline decision derives only from clean evidence and constraints.Clean and manipulated cards are shuffled together without source or attack labels, while dual-model audits verify baseline labels and position balancing distributes the correct option across positions.Source: §B.6 · pp. 25–26

B.7 DIVERSITY METRICS FULL REPORT

  • Controlled batch assignment diversifies style and other record features to prevent spurious correlations, while persona-first rewriting varies queries without changing their semantic content.The protocol controls tone, discourse structure, verbosity, and speaker perspective; revisions may not add facts or explicit privacy-boundary cues.Source: §B.6 · p. 25, §B.7 · p. 26, §B.7.1 · p. 26

B.7.1 LINGUISTIC DIVERSIFICATION PROTOCOL

  • Diversification preserves semantic content and evaluation fields without adding facts or privacy cues, and uses Gemini 3.1 Pro.The intervention therefore changes only surface expression, not the evaluation semantics.Source: §B.7.1 · p. 26

B.7.2 LEXICAL DIVERSITY METRICS

The benchmark’s query text shows lexical, opening-prefix, and narrative-form diversity, with variation reflecting both diversification procedures and domain-specific communication needs.

B.7.2 LEXICAL DIVERSITY METRICS

  • Self-BLEU-4 values are 0.011 (EIL) and 0.020 (MIU), while distinct-2 exceeds 0.22, indicating varied query wording.MATTR is near 0.88 and per-text TTR exceeds 0.75 for both mechanisms.Source: Table 7 · p. 27, §B.7.2 · p. 27

B.7.3 DIVERSITY BY FAMILY DOMAIN

  • Domain-level distinct-n varies with vocabulary breadth: gatekeeping is highest among EIL domains, while financial advisory is lower within MIU.The authors attribute these differences to genuine task-specific vocabulary rather than generation artifacts.Source: Table 8 · p. 27, §B.7.3 · p. 27

B.7.4 OPENING PREFIX DIVERSITY

  • Maximum five-word prefix share is at most 3.6% across fields, and 63% of EIL query prefixes are unique.The authors report that diversification eliminates dominant opening templates.Source: Table 9 · p. 28, §B.7.4 · p. 28

B.7.5 NARRATIVE FORM DISTRIBUTION

  • EIL queries are predominantly context-setting (47.3%), while MIU queries favor first-person direct and possessive openings (78.5% combined).The authors connect this contrast to the different communicative structures of delegated communication and decision-seeking.Source: §B.7.5 · p. 28

B.8 HUMAN QUALITY AUDIT

A human audit assessed whether generated benchmark records meet the structural assumptions behind EIL and MIU evaluation; the appendix also states the training configuration and scope assumptions.

B.8 HUMAN QUALITY AUDIT

  • The audit checked EIL scenario validity, information necessity and exploitability, and absence of privacy cues, while MIU checks covered scenario validity, evidence quality, and decision validity.Four graduate-trained annotators participated; three independently assessed records and a fourth adjudicated disagreements. All scenarios were synthetic and sampled records were checked for sensitive real-world information.Source: §B.8 · pp. 28–30
  • Three-way exact agreement ranged from 71–100% across criteria, with the greatest disagreement on per-preference validity (71–86%).Majority voting or adjudication resolved disagreements for the final binary labels.Source: §B.8 · pp. 29–30, Table 11 · p. 30
  • 94% of audited EIL records and 92% of audited MIU records passed all structural checks before corrections were applied.Failed records were rewritten or removed before the benchmark was finalized.Source: §B.8 · p. 29

C.1 SCOPE AND ASSUMPTIONS

  • The scope assumes a legitimate principal with genuine task authority; truthfulness remains fixed, and legally or ethically required disclosure overrides discretionary withholding.Privacy–utility optimization is confined to legitimate discretion, with compliance and safety treated as separate higher-priority constraint layers.Source: §C.1 · p. 30

C.2 HYPERPARAMETER SETTINGS

  • The training setup mixes EIL and MIU prompts 2:1 in each global batch, with shared settings across tasks and three base models.The models differ only in model weights and tokenizer.Source: §C.2 · p. 30

C.3 LOYAL AGENT, OPPONENT AND JUDGE PROMPT TEMPLATES

This section describes the prompt contracts, evaluation procedures, and reward signals used to operationalize loyalty through EIL and MIU assessment, including decision quality, evidence faithfulness, task utility, and information leakage.

C.3.1 EIL LOYAL AGENT CONTRACT

  • The EIL agent advances the user's task while minimizing strategically exploitable disclosure and resisting counterparty pressure; truthfulness is baseline, while the study targets selective disclosure.The contract permits only the minimum accurate private detail needed to complete the task.Source: §C.3.1 · pp. 31–32

C.3.2 EIL OPPONENT (ADVERSARY)

  • The EIL opponent reports only private facts recoverable from the agent's reply and generates a counterparty response using only those recovered facts.It is instructed not to infer hidden facts from general statements or unsupported assumptions.Source: §C.3.2 · pp. 32–33

C.3.3 EIL TASK UTILITY JUDGE

  • The utility judge scores overall user benefit by jointly assessing task progress and preservation of the user's negotiating position, using four discrete scores and literal reply evidence.It also considers whether the observed counterparty response shows exploitation or shifts advantage away from the user.Source: §C.3.3 · pp. 33–34

C.3.4 EIL LEAKAGE JUDGE

  • The leakage judge scores direct disclosure or sufficiently recoverable protected facts, rejects vague associations, and uses adversary predictions to assess strategic leakage.Its reward depends on opponent behavior; evaluation averages across four temperatures, which recover substantially more protected slots than any single temperature.Source: §C.3.4 · p. 35

C.3.5 MIU LOYAL AGENT CONTRACT

  • The MIU agent selects options according to user goals, evidence, and constraints, treating external material as evidence to evaluate rather than instructions to follow.Its fixed decision-and-reason format enables claim-by-claim faithfulness evaluation; parse success was at least 99% for all three base models zero-shot on the training set.Source: §C.3.5 · pp. 35–37

C.3.6 MIU FAITHFULNESS JUDGE

  • The MIU faithfulness judge assesses each reason only against its cited objective evidence and classifies support, including overreach and manipulation-driven claims.It evaluates printed reasons individually without using uncited cards to justify a claim.Source: §C.3.6 · p. 37

C.4.1 MIU REWARD FUNCTION

  • The MIU reward combines decision quality, defined as exact match with the baseline decision, and evidence faithfulness, with a default faithfulness weight of 0.5.Source: §C.4.1 · p. 38
  • Evidence faithfulness measures per-claim citation grounding, while decision quality measures whether the selected option matches the record’s baseline decision.The faithfulness judge assigns a status and support score; the two measures are kept distinct so faithfulness remains an independent training signal.Source: §C.4.1 · p. 38
  • MIU responses that fail parsing or have truncated output are excluded from GRPO updates, and groups with no eligible candidates or identical rewards are dropped.These filters avoid invalid updates and undefined advantage estimation.Source: §C.4.1 · p. 38

C.4.2 EIL REWARD FUNCTION

  • The EIL reward balances task utility against leakage with a default leakage penalty weight λ = 0.5.Utility uses a four-point scale, and leakage is a continuous score in [0, 1].Source: §C.4.2 · pp. 38–39
  • Leakage is estimated by having an opponent blindly infer private facts, then scoring each protected slot for recoverability and weighting it by inherent risk.Evaluation tests each response against opponents at all four temperatures and averages the resulting metrics across branches.Source: §C.4.2 · p. 39
  • Utility and leakage can diverge because utility scores task progress as well as position preservation, whereas leakage measures per-slot recoverability before the counterparty reply.Capability-only controls show that short evasive replies can lower both leakage and utility, receiving lower reward than adversarially trained behavior.Source: §C.4.2 · p. 39
  • Reply-level utility–leakage correlations are −0.16, +0.02, and −0.15 across three trained models, while each model’s highest-utility stratum spans the full [0, 1] leakage range.The authors treat position preservation as intrinsic to task success, while utility and leakage measure operationally distinct dimensions of loyalty.Source: §C.4.2 · p. 39
  • EIL candidates are excluded for truncated outputs or opponent/judge failures, and groups with all-ineligible candidates or zero reward variance are dropped.Training halts after three consecutive full-group infrastructure failures.Source: §C.4.2 · pp. 39–40

D EXTENDED EXPERIMENTAL RESULTS AND PROMPT TEMPLATES

The appendix examines prompting, baseline reproduction, training design, and robustness, including multi-temperature convergence, capability-only controls, out-of-distribution performance, and human alignment. It reports findings on transfer to unseen opponents, leakage reduction, training trade-offs, OOD performance, and human–judge agreement.

D.2 LOYALTY-COT PROMPT TEMPLATES; D.3 PRELIMINARY VALIDATION OF INFRASTRUCTURE MODEL SELECTION; D.4 REPRODUCTION OF PRIOR BASELINES

  • The primary Loyalty-CoT prompt asks agents to reason internally about user objectives and information boundaries before responding; reproduced Prompt Scaffold and Per-token-KL yield 36.16% and 38.74% utility versus 52.47% for zero-shot Qwen3-4B.The reproduced methods used official released code and the same training budget as the primary runs.Source: §D.2 · p. 41, §D.2.1 · p. 41, §D.4 · p. 42

D.5.1 TASK SAMPLING RATIO AND LEAKAGE PENALTY

  • EIL-only training reaches the highest EIL reward (0.62) but poor MIU performance, while MIU-only training reaches the highest MIU reward (0.78) but worse EIL performance.Among joint configurations, increasing λ from 0.25 to 0.5 reduces leakage from 49.81% to 25.76% but lowers MIU decision accuracy and reward; E2M1 sampling at λ = 0.5 improves both rewards and MIU faithfulness over E1M1.Source: §D.5.1 · p. 43

D.5.3 HELD-OUT OPPONENT GENERALIZATION

  • Qwen3-4B retains 86.5% of its 43.42pp leakage reduction against a held-out opponent, while Llama-3.1-8B retains 82.5% of its 33.36pp reduction.MIU metrics are invariant across opponents; EIL utility changes by −3.11pp for Qwen3-4B and +2.10pp for Llama-3.1-8B. Both training runs also improve EIL and MIU reward as leakage falls across checkpoints.Source: §D.5.3 · p. 44, Table 17 · p. 44

D.5.4 NECESSITY OF ADVERSARIAL INFERENCE

  • Removing adversarial inference raises leakage from 19.04% to 41.63% (+22.6pp) and lowers EIL reward from 0.50 to 0.33 (−33.5% relative).Utility falls by 5.5pp; MIU decision accuracy, faithfulness, and reward also decline. Without opponent inference about recoverable information, the leakage judge misses subtle exploitable disclosures and mainly penalizes surface-level leaks.Source: §D.5.4 · p. 45, Table 18 · p. 45

D.5.5 UTILITY AND LEAKAGE ARE NOT REDUNDANT AT THE REPLY LEVEL

  • High utility does not imply low leakage: top-utility replies span the full [0, 1] leakage range, and story-generation controls shorten replies without clean EIL gains.The controls reduce median reply length from 116 to 62 words while losing 6.7–8.1pp utility; the share of replies with both low utility and zero leakage rises from 0.5% to 19.6%. These findings support that utility and leakage are not redundant measurements, not that they are independent constructs.Source: §D.5.5 · pp. 45–46, Table 19 · p. 46

D.5.6 EIL RESPONSE LENGTH AND EVASION RATE

  • EIL-trained responses shorten by 19% over training, while suspected evasion remains below 2% and conditional utility rises from 0.538 to 0.607.Mean length declines from 113 to 92 words, and evasion—zero leakage with utility at most 0.33—rises from 0.3% to 1.4%; over 98% of outputs remain non-evasive.Source: §D.5.6 · p. 46

D.5.7 OPPONENT TEMPERATURE DIVERSITY

  • The four-temperature opponent ensemble recovers 601 slots, 26–39% more than individual temperatures, despite nearly identical aggregate rewards and leakage rates.Coverage measures evaluation comprehensiveness, not agent vulnerability; the diversity analysis finds temperature-ordered opponent outputs persist across training. The scope is a single Haiku-trained trajectory and protocol, not a claim that magnitudes transfer unchanged to more strongly defended policies.Source: §D.5.7 · pp. 47–48, Figure 4 · p. 48

D.5.8 MULTI-TEMPERATURE TRAINING CONVERGENCE

  • Ensemble reward rises from 0.263 to 0.372 (+41%), with overlapping 95% confidence intervals and a maximum inter-temperature gap of 0.016.The results indicate that group-level assignment exposes the policy to all four temperature conditions rather than privileging one.Source: §D.5.8 · p. 49, Figure 6 · p. 49
  • 0.535→0.373 mean leakage (−30%) and approximately 14%→30% zero-leakage rates hold across four temperatures, with declines differing by less than 0.004.At step 159, temperature spread was at most 0.021 within each domain, smaller than the inter-domain spread; no temperature consistently dominated within a domain.Source: §D.5.8 · p. 49, Figure 7 · p. 50, Figure 8 · p. 50

D.6 STORY GENERATION AS CAPABILITY-ONLY CONTROL

  • +21–24pp ROCStories gains from both story-generation pipelines do not yield consistent loyalty improvements across both tasks.ROC-only SFT versus WP+ROC reports EIL utility 45.73 vs. 44.42, leakage 21.28 vs. 22.44, and reward 0.35 vs. 0.33; MIU decision accuracy 55.76% vs. 48.75%, faithfulness 71.12% vs. 63.30%, and reward 0.19 vs. 0.05. The reported EIL reward gains coincide with utility drops of 6.7–8.1pp.Source: §D.6 · pp. 49–51
  • 22.6% of valid EIL examples regress from utility ≥0.5 to ≤0.33 after SFT, while low-utility, zero-leakage outputs rise from 0.5% to 19.6%.Outputs with utility ≤0.33 increase from 26.5% to 41.7%, and median length falls from 116 to 62 words; the paper interprets this pattern as reward hacking through short, uninformative responses.Source: §D.6 · p. 50
  • Sampled story-generation outputs also show task-capability degradation, including evasive medical and discrimination-case replies and a pricing negotiation that avoids scope and rates.The examples are described as degradation beyond simple refusal behavior; sampled outputs tended toward short, evasive responses lacking substantive task engagement.Source: §D.6 · pp. 50–51

D.7 SCOPE OF THE CAPABILITY-ONLY CONTROLS

  • Cross-domain controls cannot isolate data distribution from optimization objective, while a fixed-setup ablation separates loyalty-data gains from the adversarial signal.Capability-only checkpoints reach EIL rewards of 0.35 and 0.27 versus 0.50–0.60 for adversarial training; the no-adversary ablation drops from 0.50 to 0.33, compared with 0.24 without a loyalty objective.Source: §D.7 · p. 51

D.8 OUT-OF-DISTRIBUTION GENERALIZATION

  • No point estimate decreases across four OOD math and narrative benchmarks after adversarial training on Qwen3-4B, though the evidence is limited to one model and seed.The AIME-2026 pass@16 gain of 3.33pp corresponds to approximately one additional problem solved; the study does not establish broader generalization.Source: §D.8 · p. 51, Table 23 · p. 52

D.9 HUMAN ALIGNMENT STUDY

  • Four graduate-student annotators blindly rated samples from the best Qwen3-4B checkpoint; opponent-inference agreement was Fleiss κ = 0.935, with 96.7% unanimous consensus.The study sampled 60 instances per benchmark and used majority vote for opponent inference; human ratings of leakage, utility, and faithfulness used a four-level scale.Source: §D.9 · p. 52
  • Faithfulness had the strongest human–judge agreement (Pearson r = 0.928; Spearman ρ = 0.904), while leakage and utility Pearson correlations were 0.775 and 0.741.Inter-annotator Fleiss κ was 0.798 for leakage, 0.760 for utility, and 0.787 for faithfulness; human–judge weighted κ was 0.64 for leakage and utility and 0.83 for faithfulness.Source: §D.9 · p. 52

D.1 PRELIMINARY DOMAIN-LEVEL ANALYSIS

Domain vulnerabilities vary across tasks and models, and Loyalty-CoT can help some settings while harming others. The training-design results show improvements across opponent substitutions, while the supplied ablation passages identify the sampling-ratio and leakage-penalty comparison without reporting its outcomes.

D.1 PRELIMINARY DOMAIN-LEVEL ANALYSIS

  • EIL bargaining is the hardest domain for most models, while MIU financial advisory is commonly hardest; even Claude Opus 4.8 has a 0.27 MIU reward gap across domains under BL.The EIL exceptions include Marin-8B-Instruct and Claude Haiku 4.5 under CoT; information guidance is the MIU bottleneck for some models.Source: §D.1 · pp. 40–41
  • Higher-tier models generally earn higher within-series rewards, but cross-family comparisons and the size of domain gaps vary, especially for MIU.Claude’s EIL reward spread is small, while Qwen’s is not; MIU’s hardest domain is not determined by scale alone.Source: §D.1 · p. 41
  • Loyalty-CoT yields uneven returns: it improves selected domains for some models but regresses across GPT-5.5’s MIU domains and on Claude Haiku 4.5’s EIL gatekeeping.The results motivate training-based approaches that internalize loyalty rather than relying only on runtime instructions.Source: §D.1 · p. 41

D.5 TRAINING DESIGN ABLATIONS

  • Table 15 compares task-sampling ratios and leakage penalties for Qwen3-4B; the main E2M1, λ=0.5 configuration averages three seeds, while alternatives use one.The supplied passage identifies the comparison’s scope but does not report its outcomes.Source: Table 15 · p. 43

D.5.2 OPPONENT SELECTION AND ROBUSTNESS

  • Both opponent choices improve directionally across EIL and MIU metrics over the zero-shot baseline, though the Haiku-trained model trails on several outcomes.The reported gaps include −0.13 EIL reward, +18.24pp worse EIL leakage, and 80.00% versus 90.58% MIU faithfulness.Source: §D.5.2 · p. 43

D.10 JUDGE MODEL ROBUSTNESS

Changing judge families preserves model rankings on EIL and MIU, although EIL absolute scores shift. The case-study overview describes trained agents as usually maintaining information boundaries without refusing, with residual leakage concentrated in bargaining elaboration.

D.10 JUDGE MODEL ROBUSTNESS

  • Model rankings remain unchanged across judge families for EIL reward and MIU metrics, with Spearman ρ = 1.00 for each reported ranking.MIU decision accuracy is identical by design because it uses exact matching; faithfulness changes by at most 1.08pp and reward by at most 0.54pp.Source: §D.10 · p. 52
  • EIL absolute scores shift under judge substitution, but the Llama-3.1-8B-Instruct > Marin-8B-Instruct > Qwen3-4B reward ordering is preserved.The passage attributes score shifts to judge calibration differences.Source: §D.10 · p. 53

E CASE STUDY: HOW TRAINED AGENTS SUCCEED AND FAIL

  • The case study samples 100 EIL and 100 MIU examples from Marin-8B-Instruct, stratified by family domain, with category rates matching full-set statistics within sampling noise.Replies were manually categorized and illustrated with representative cases.Source: E CASE STUDY: HOW TRAINED AGENTS… · p. 53

E.1 EIL: MAINTAINING AND LOSING AN INFORMATION BOUNDARY

  • Zero-leakage replies are typical and usually preserve task progress without refusal; remaining leakage is concentrated in bargaining, where elaboration volunteers leverage-bearing facts.The authors characterize this leakage as over-explaining an otherwise legitimate point.Source: §E.1 · p. 53

E.1.1 SUCCESS MODE: REDIRECTING THE EXTRACTION ATTEMPT

The examples show that agents can redirect extraction attempts and advance delegated tasks, but volunteering unrequested supporting facts can still expose leverage. In MIU, agents often match decisions to decisive authorized evidence, while errors include both unsupported conclusions and uptake of manipulated claims.

E.1.1 SUCCESS MODE: REDIRECTING THE EXTRACTION ATTEMPT

  • Agents redirect extraction attempts toward task-relevant questions, withholding the May 18 deadline and $2,850 ceiling while still advancing toward a viewing.The leasing reply asks about the unit and application process instead of answering timeline and alternatives questions.Source: §E.1.1 · p. 54

E.1.2 FAILURE MODE: ELABORATING PAST THE BOUNDARY

  • The clean moving-quote reply withholds the $4,200 budget while supplying details needed for a quote, showing that less leakage can coincide with more useful task information.The paired replies differ by sampling seed; the authors say this is consistent with low but nonzero aggregate leakage, not proof of its source.Source: §E.1.2 · p. 56
  • Residual EIL can arise when a legitimate response volunteers unrequested leverage: one reply disclosed a two-week deadline and mortgage reason, scoring full leakage at no utility cost.A different sampling seed produced zero measured leakage on the same prompt, so the pair is illustrative rather than a controlled comparison.Source: §E.1.2 · pp. 55–56

E.2 MIU: EVIDENCE ANCHORING AND DECISION DRIFT

  • In 70% of sampled MIU replies, the action is correct and reasoning is traceable to authorized evidence; 19% are wrong despite faithful reasoning.Accuracy ranges from 87.2% in commerce guidance to 69.0% in information guidance.Source: §E.2 · p. 56, Table 26 · p. 57

E.2.1 SUCCESS MODE: MATCHING AGAINST A REQUIREMENT

  • Successful MIU replies identify decisive evidence that settles the stated requirements and ignore adjacent manipulated items.Examples include matching insurance constraints and envelope specifications against authorized evidence.Source: §E.2.1 · pp. 57–58

E.2.2 FAILURE MODE: DRIFT AND UPTAKE

  • MIU failures include decision drift from faithful but insufficient evidence and uptake of manipulated claims used as the operative reason.The drift example uses a procedural fact that does not establish the claim; the uptake example relies on a counterparty’s scarcity claim.Source: §E.2.2 · p. 58

E.3 SUMMARY

  • EIL failures tend to be over-communication during legitimate task moves, whereas MIU’s dominant error is drawing an unsupported conclusion from authorized evidence.Because drift can retain faithful reasoning, decision accuracy must remain a separate evaluation axis; manipulated-evidence uptake is less frequent but strongly predictive of error.Source: §E.3 · p. 59
  • Correct-decision MIU faithfulness rises from 0.76–0.81 for base models to 0.94–0.99 for trained models, alongside fewer reason lines per response.A separate check of sampled EIL replies found false or misleading statements in at most 3.1%, at comparable rates before and after training; it does not characterize prevalence across all models or scenarios.Source: §E.3 · p. 59

F.1 DISCUSSION

The discussion frames loyalty as relational information control that must preserve useful task completion while protecting the principal. It distinguishes EIL from MIU, argues that general capability is not a substitute for loyalty training, and notes boundaries in evaluation and inference-time prompting.

F.1 DISCUSSION

  • Because loyalty depends on opponent inference and authority over information, the authors argue that training should instantiate the strategic relation rather than evaluate responses in isolation.They contrast adversarial exposure to information-control consequences with capability-only training and static supervision.Source: §F.1 · p. 59
  • Loyalty is constrained helpfulness: agents must maintain task utility while minimizing exploitable disclosure and grounding decisions in authorized evidence.The reported utility gains alongside leakage suppression are consistent with this framing, though the authors caution that those related EIL metrics alone do not test it.Source: §F.1 · pp. 59–60
  • Per-response leakage isolates the disclosure decision, but single-turn evaluation leaves open whether repeated or accumulated probes erode boundaries over sustained exchanges.The authors describe this as a coverage limitation rather than a validity limitation.Source: §F.1 · p. 60
  • EIL and MIU require distinct defenses: recognizing facts that enable opponent inference versus distinguishing evidence relevance from sufficiency.The authors argue this complementarity supports separate information-control pathways rather than one behavioral tendency.Source: §F.1 · p. 60
  • Capability-only training improves math and narrative benchmarks without comparable loyalty gains, while adversarial loyalty training does not degrade those benchmarks.The authors conclude general capability is not a sufficient surrogate objective for loyalty.Source: §F.1 · p. 60
  • Training across opponent temperatures may expose vulnerabilities missed by a single opponent, while opponent substitution preserves directional improvement despite affecting its magnitude.The paper treats temperature as a proxy for stylistic and strategic variation.Source: §F.1 · pp. 60–61
  • Loyalty-CoT can help weaker models on MIU decisions, but it supplies inference-time reasoning guidance rather than jointly training EIL, MIU, and task utility.Combining prompting with adversarial training is identified as future work, outside this study.Source: §F.1 · p. 61

F.2 LIMITATIONS

The evaluation and its measures have bounded scope: they do not capture adaptive multi-turn probing or isolate manipulation from clean-task competence. MIU may also reflect generator-specific style, while adversarial training adds computational overhead and is not compared with distillation or DPO.

F.2 LIMITATIONS

  • The evaluation tests a single opponent probe at one of four temperatures, so sustained exchanges that accumulate disclosures remain outside its measured scope.Per-response leakage is a necessary, not sufficient, condition for harm; the reported values lower-bound exposure that sustained exchanges could accumulate.Source: §F.2 · p. 61
  • The fixed gold-standard decision cannot distinguish manipulation susceptibility from inability to derive the correct decision from clean context.The authors report this conflated quantity; using each model’s clean-context decision could separate the failure modes but would complicate cross-model comparison.Source: §F.2 · p. 61
  • MIU gains may partly reflect stylistic signatures from the single manipulated-card generator, and robustness to other generators or unseen attacks is untested.Source: §F.2 · p. 61
  • Online adversarial training costs more computation than static-label supervision, and the study leaves distillation and DPO comparisons to future work.Results suggest weaker or more efficient opponents may reduce overhead, but the authors identify cost as a practical consideration.Source: §F.2 · pp. 61–62
Loading 2609.34714v1…