PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints

Yutao Xie, Jingbo Yang, Guanyu Yao, Xiaoyu Yang, Duo Ding, Shiyu Chang

arXiv:2610.16155v1cs.AI

TL;DR

Collaborative-agent evaluations often use cooperative or predefined user behaviors, limiting coverage of diverse interactions and agent-specific weaknesses. TIPER evolves user personas under task-integrity constraints; it lowers held-out avg@3 across all reported GPT-5.4 settings and across seven agents and four domains.

  • Problem

    Collaborative-agent evaluations often rely on cooperative simulators or predefined behaviors, limiting coverage of realistic user diversity and agent-specific weaknesses.

    Source: §1 · pp. 1–2

  • Method

    TIPER evolves persistent user personas through reflection-guided behavioral mutations, task-wise Pareto selection, and task-integrity checks.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Held-out avg@3 is lower in all 12 GPT-5.4 settings and all 28 agent–domain pairs, with a 22.9-point median reduction across pairs.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    TIPER provides targeted robustness evaluation and a behavior-to-failure taxonomy for diagnostic analysis of interactive-agent failures.

    Source: §7 · p. 10

    Persona fidelity is not measured directly, so observed differences may reflect how simulator backbones realize the same persona instructions.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlooking failures that arise under diverse but realistic user interactions. This can lead to overly optimistic estimates of agent robustness and provide limited failure signals for training agents that generalize across users. Existing approaches largely rely on predefined user behaviors, which may miss agent- or domain-specific weaknesses, while failure-oriented search can reduce task success by invalidating the underlying task rather than exposing genuine robustness failures. To fill this gap, we introduce TIPER, a task-integrity-preserving persona evolution method that automatically discovers agent-specific user policies exposing robustness weaknesses while keeping the underlying task valid and solvable. TIPER evolves persistent user personas through reflection-guided behavioral mutations and task-wise Pareto selection under a task-integrity constraint. Across four interactive-agent benchmarks, TIPER yields lower held-out avg@3 scores in all 12 GPT-5.4 settings, with observed reductions of 5.8 to 33.7 percentage points, and in all 28 agent-domain pairs spanning seven agents, with a median observed reduction of 22.9 points. Further analysis yields a behavior-to-failure taxonomy that connects discovered user behaviors to recurring agent failure modes. Together, these results show that TIPER enables more diagnostic robustness evaluation of interactive agents and provides informative failure analysis for future agent robustness improvement. Code repository: https://github.com/UCSB-NLP-Chang/TIPER.

1 INTRODUCTION

Collaborative-agent evaluations often use cooperative user simulators, which may miss diverse and challenging interactions. TIPER searches for agent-specific weaknesses while preserving task validity, and exposes lower held-out task success across the reported benchmarks and agents.

1 INTRODUCTION

  • Cooperative LLM user simulators may underrepresent realistic interaction diversity and overestimate agent performance.The introduction also notes that agents trained against narrow simulator behavior may generalize poorly to diverse users.Source: §1 · p. 1
  • Hand-crafted behavior classes and traits may miss agent- or domain-specific weaknesses, while failure-oriented search can invalidate the task.An audit of 872 τ-Trait conversations found task-critical simulator errors in 244 cases (28.0%).Source: §1 · p. 2
  • TIPER evolves persistent user personas to expose agent-specific robustness weaknesses while preserving valid interactions.It reflects on observed failures to propose behavior-level mutations, then uses task-wise Pareto selection to retain complementary failure patterns.Source: §1 · p. 2
  • 5.8–33.7 percentage points: held-out GPT-5.4 task success is lower under TIPER personas in all 12 settings.Across 28 agent–domain pairs covering seven agents, task success is lower in every pair, with a median reduction of 22.9 points.Source: §1 · p. 2

2 PROBLEM FORMULATION

The formulation treats persona search as maximizing performance reduction relative to a benchmark-default user while imposing an aggregate task-integrity constraint. Persona performance is assessed on integrity-preserving trajectories, requiring rollouts for evaluation.

2 PROBLEM FORMULATION

  • Each task includes a goal, environment, verifier, and integrity judge, and each persona induces a user policy through the simulator.The persona is an instance-agnostic behavioral instruction.Source: §2 · p. 3
  • Persona validity requires an aggregate integrity rate of at least 1−ε across the evaluation tasks.The integrity-conditioned task score is restricted to trajectories judged to preserve task integrity.Source: §2 · p. 3
  • A persona seeks to maximize average task-performance reduction relative to the benchmark-default persona, subject to aggregate integrity.Positive reduction means lower agent performance; negative reduction means the persona improves it.Source: §2 · p. 3
  • Evaluating a persona requires rolling out the agent against its induced user policy because integrity and conditional performance are not available in closed form.Personas are compared under a fixed evaluation budget.Source: §2 · p. 3

3 TASK-INTEGRITY-PRESERVING PERSONA EVOLUTION

TIPER searches over transferable user personas through reflection-guided mutations, text checks, an empirical integrity gate, and task-wise Pareto selection. Search, rank, and test tasks are separated so the final persona is fixed before test evaluation.

3 TASK-INTEGRITY-PRESERVING PERSONA EVOLUTION

  • Search, rank, and test tasks are separated, with the test set held out during persona evolution and selection.The rank set supports integrity gating, Pareto selection, and final persona selection.Source: §3 · p. 3

3.1 PERSONA REPRESENTATION AND REFLECTIVE PROPOSAL

  • Personas are domain-aware but instance-agnostic instructions, and reflection on parent trajectories guides behavior-level mutations toward observed agent weaknesses.Text-level checks reject unclear, instance-specific, inconsistent, or near-duplicate proposals before rollout.Source: §3.1 · pp. 3–4

3.2 SEARCH-TIME INTEGRITY GATING

  • A child advances to rank-set evaluation only if it lowers the search-minibatch score and meets the allowed integrity-violation threshold among failed rollouts.The empirical search-time gate filters failures caused by simulated-user behavior but does not certify the formal validity constraint.Source: §3.2 · p. 4

3.3 TASK-WISE PARETO SELECTION

  • Task-wise Pareto selection retains nondominated personas to preserve complementary weaknesses, and the selected persona is fixed before test evaluation.If no proposal passes the checks and gate, the active set remains unchanged.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments test TIPER across four interactive-agent benchmarks and compare its held-out failure exposure with default and alternative user policies. They also assess integrity and alignment with human-user references.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses four benchmarks, compares TIPER with default, fixed-program, and objective-matched PPol policies, and selects personas on rank tasks before three-rollout test evaluation.Reported valid avg@3 averages task scores over valid test rollouts; passk is reported for tasks with three valid rollouts.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • On held-out tasks, rank-selected TIPER personas reduce avg@3 versus benchmark-default user simulators in all 12 GPT-5.4 test settings, with a median reduction of 16.1 percentage points.The largest avg@3 reductions were 33.7 points in Math, 25.3 in Telecom, and 25.0 in Delivery.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 INTEGRITY AND BEHAVIORAL ALIGNMENT

  • Among failed held-out rollouts, task-critical simulator-error rates are at most 2.5% for TIPER versus up to 21.4% for alternative policies.A blinded audit found 148 of 150 automatically accepted TIPER failure trajectories task-valid, with 143 showing an identifiable agent error.Source: §4.3 · p. 6
  • On Airline and Retail, TIPER’s aggregate USI scores exceed the benchmark-default simulator’s but remain below human-reference scores.USI aggregates five behavioral-alignment metrics; the reported comparison also describes alignment across communication, information disclosure, clarification, and reactions to agent errors.Source: §4.3 · p. 6

4.2 MAIN RESULTS

  • Across 28 agent-domain pairs, avg@3 decreases in every setting and pass3 in 25, with median reductions of 22.9 and 20.4 percentage points, respectively.Telecom avg@3 reductions span 13.0–39.5 points across all seven agents, while Math reductions vary from 0.8 to 33.7 points.Source: §4.2 · p. 6, Table 2 · p. 7
  • Across eight agent-domain settings, TIPER yields the largest avg@3 reduction, exceeding the strongest alternative by a median of 7.5 percentage points.The alternatives include fixed behavior programs and objective-matched PPol.Source: §4.2 · p. 6, Table 3 · p. 7

5 ANALYSIS

The analysis finds that integrity checks strengthen held-out failure exposure, while mutation-model and simulator-backbone effects vary by domain and metric. The behavior analysis connects integrity-valid user behaviors with recurring agent failures, but treats these links as diagnostic associations rather than causal effects.

5.1 ABLATION AND COMPONENT SENSITIVITY

  • Removing either validity check weakens held-out failure exposure, and the full method performs strongest for both agents and metrics.The checks may discourage task-invalidating or instance-specific shortcuts that do not transfer to held-out evaluation.Source: §5.1 · p. 7, Table 4 · p. 8
  • All three mutation models reduce Telecom avg@3 by 23.5–29.6 points and pass3 by 29.6–35.2 points, but Delivery sensitivity varies by model.On Delivery, GLM-5.2 yields a 7.5-point reduction on both metrics, while GPT-5.5 and DS-V4-Flash remain strong; no model is best in every case.Source: §5.1 · p. 8
  • Changing the simulator backbone preserves the direction of agent degradation but not its magnitude, with Telecom reductions nearly vanishing and Delivery metrics diverging.Persona fidelity is not measured directly, so backbone differences may reflect how each model realizes the same persona instructions.Source: §5.1 · p. 8

5.2 FROM USER BEHAVIORS TO AGENT FAILURE MODES

  • The diagnostic analysis uses 139 same-task contrastive pairs across eight domains in two benchmarks, retaining integrity-valid TIPER failures and judge-agreed behavior labels.The pairs contrast a successful default rollout with an integrity-valid failed TIPER rollout for GPT-5.4.Source: §5.2 · p. 8
  • In τ 3-bench, sparse information, incremental requests, and single-step requests account for 28.1%, 21.9%, and 20.8% of displayed annotations; VitaBench is led by incremental requests at 54.9%.Bundled actions account for 24.6% in VitaBench; the authors relate the benchmark contrast to information maintenance versus accumulating requirements and executing multi-part workflows.Source: §5.2 · p. 8, Figure 3 · p. 9
  • Incremental requests associate with incomplete option enumeration and assumed tool unavailability, while sparse information and uncorrected assumptions associate with trusting unverified reports.Single-step requests, vague references, and bundled actions spread across several failure modes.Source: §5.2 · p. 8, Figure 4 · p. 9
  • The observed pathways point to incomplete tracking, insufficient verification, and premature commitment, but represent diagnostic associations rather than causal effects.Each behavior can map to multiple failures, and each failure can arise from multiple behaviors.Source: §5.2 · pp. 8–9

6 RELATED WORK

Prior work studies interactive evaluation, persona-driven behavioral variation, and reflective prompt evolution. TIPER adapts these ideas to agent-conditioned failure search while checking that resulting trajectories preserve task integrity.

Interactive evaluation and user simulation

  • Interactive-agent evaluations use multi-turn simulated users, but simulators can be overly cooperative or behaviorally uniform, and unconstrained simulation can compromise validity.Interactive performance also depends on the interaction process and user experience.Source: §6 · p. 9

Persona-driven behavioral stress testing

  • Persona-driven work diversifies simulated behavior through non-collaborative users, human traits, and evolving intent, while persona-conditioned simulators can exhibit drift, role confusion, and behavioral echoing.PPol evolves persona generators for human-likeness and behavioral coverage while preserving task goals; TIPER instead searches for agent-specific failure-revealing personas and checks trajectory integrity.Source: §6 · p. 9

Reflective prompt evolution and adversarial search

  • Prompt optimization and red-teaming offer textual gradients, LLM-guided evolution, reflection with Pareto selection, and quality-diversity search; TIPER adapts these to interactive user policies.It retains complementary failures through task-wise Pareto selection and filters invalid candidates through text-level checks and trajectory-level integrity gating.Source: §6 · pp. 9–10

7 CONCLUSION

TIPER evolves instance-agnostic personas to expose failures overlooked by default user simulation while preserving task integrity. It produces lower observed task scores across evaluated settings and agent–domain pairs, supporting targeted robustness evaluation.

7 CONCLUSION

  • TIPER evolves instance-agnostic personas to expose failures overlooked by benchmark-default simulation while preserving task integrity.The conclusion describes the method as a practical framework for targeted, behaviorally grounded robustness evaluation.Source: §7 · p. 10
  • Across 12 GPT-5.4 settings and 28 agent–domain pairs, TIPER yields lower observed avg@3 scores than default policy, with a 22.9-point median reduction across pairs.Its observed reduction is largest among compared policies in all eight direct comparisons, where task-critical simulator-error rates are at most 2.5%.Source: §7 · p. 10
  • Ablations support the integrity gate and text checks, while trajectory analysis links exposed failures to execution, grounding, and replanning errors.The conclusion frames the results as supporting behaviorally grounded robustness evaluation.Source: §7 · p. 10

A ADDITIONAL USER-SIM INDEX RESULTS

This appendix reports additional user-simulator index comparisons and experimental configurations, covering search and evaluation settings, policy controls, cross-agent and component protocols, and bootstrap intervals.

A ADDITIONAL USER-SIM INDEX RESULTS

  • On Airline and Retail, TIPER has higher aggregate USI than the benchmark-default simulator in both domains, but remains below human-reference scores.Table 5 reports the aggregate USI values; its reference is based on Sim2Real human-reference trajectories.Source: Table 5 · p. 16

B EXPERIMENT DETAILS

  • Reported experiment details prioritize settings affecting search and evaluation, leaving provider-specific routing, retries, and caching unspecified.Source: B EXPERIMENT DETAILS · p. 16

B.1 SHARED CONFIGURATION

  • Search, rank, and test splits are task-disjoint; the selected persona is fixed using rank-set avg@2 before held-out evaluation on a fresh three-trial batch.Test outcomes do not affect persona selection.Source: §B.1 · pp. 16, 18
  • The serial loop proposes one child from a Pareto-active parent, screens it, and advances only when minibatch raw difficulty strictly increases; integrity remains a separate hard gate.The child then undergoes full rank evaluation and task-wise Pareto selection.Source: §B.1 · pp. 16–17

B.2 BASELINES AND COMPONENT PROTOCOLS

  • Controls comprise the no-persona default and four fixed NCUser programs; PPol shares the agent-difficulty objective but generates one persona per held-out task, so portfolio protocols differ.Source: §B.2 · p. 18
  • Cross-agent runs re-evolve personas separately for seven agents across Telecom, Math, Coding, and Delivery, while component tests vary the mutation model, simulator backbone, or search checks.Source: §B.2 · p. 18

B.3 TASK-LEVEL BOOTSTRAP INTERVALS

  • Tables 8–9 give percentile 95% task-level bootstrap CIs from 10,000 replicates, pairing aligned task identities and resampling unaligned task-score distributions independently.No interval is estimated when the raw task outcomes needed for resampling are unavailable.Source: §B.3 · p. 18, Table 8 · p. 18, Table 9 · p. 19

B.4 COMMON-VALID-TASK SENSITIVITY

This appendix examines Airline comparisons under a common-valid-task restriction, reports available run-scale and cost data, and presents prompts for evolving and auditing realistic, task-valid user behavior. The prompts distinguish valid but difficult interactions from task-invalid or unfair conduct, using conservative, transcript-grounded checks that the user coherently pursued the original goal.

B.4 COMMON-VALID-TASK SENSITIVITY

  • The direction of all four GPT-5.4 Airline fixed-policy comparisons is unchanged on common-valid tasks, with the largest shift at 0.5 percentage points.The sensitivity check retains only tasks with three integrity-valid rollouts in both conditions, using archived runs with aligned task identities and complete trial records.Source: Table 10 · p. 19, §B.4 · p. 19

B.5 RUN SCALE AND COST

  • VitaBench logs include 11,127 agent calls, 6,140 user-simulator calls, and USD 37.13 in logged costs; the cost and saved-span wall-clock values are lower bounds.The USD 37.13 comprises USD 17.15 for the agent and USD 19.98 for the user simulator; model-call counts exclude initial task messages.Source: §B.5 · p. 19, Table 11 · p. 20
  • The reported API-cost range excludes unpriced USI and taxonomy calls, failed pilot runs, human annotation, local compute, and storage.The estimates are planning estimates rather than reconstructed bills.Source: §B.5 · p. 20, Table 12 · p. 21

B.6 PROMPT TEMPLATES

  • ColBench persona prompts target systematic misdirection in requirement gathering, specification, algorithm guidance, or confirmation while forbidding fabricated facts and requiring known details to remain recoverable.They distinguish genuine wrong-code failures from timeout-producing friction and ask mutations to target agent strategies observed in conversation traces.Source: §B.6 · pp. 22–23
  • Serial support-agent prompts likewise seek wrong actions rather than timeouts, while requiring personas to remain grounded in the scenario and avoid impossible blockers.They use successful and failed conversation traces to guide targeted mutations, and explicitly prohibit fabrication, abandonment, and dead-end interactions.Source: §B.6 · p. 24
  • Persona mutation first analyzes success and failure traces, proposes three distinct weakness-targeting hypotheses, then selects and implements the most promising direction.The chosen hypothesis is expected to make the agent take the wrong action.Source: §B.6 · p. 24
  • The persona linter accepts only profiles that are natural, valid, and reusable across tasks, rejecting incoherent instructions, impossible or illegal behavior, and task-specific details.It permits realistic misdirection, terse answers, and drip-feeding information, while disallowing instructions to withhold known information when directly asked.Source: §B.6 · p. 25
  • The auditor flags customer behavior only when no reasonable cooperative customer would act that way and it plausibly changes or misdirects the task outcome.It prioritizes precision over recall, treating false alarms as more damaging than misses and judging uncertain cases as acceptable.Source: §B.6 · p. 26
  • The conversation auditor treats structural violations, cutting off promised in-flight actions, material ungrounded claims, and active contradictions of task constraints as critical violations.It distinguishes active contradictions from passive omissions and requires transcript evidence for disputed entity choices.Source: §B.6 · pp. 26–27
  • The integrity audit checks whether the simulated user coherently pursues the original goal and flags critical violations using quoted customer turns as evidence.The prompt distinguishes protocol or factual violations from expected outcomes, and instructs auditors to judge behavior rather than task success.Source: §B.6 · p. 28
  • For ColBench, the audit treats realistic friction as genuine failure and flags severe failure only when unreasonable user behavior plausibly causes wrong code.The standard is high-precision: when uncertain, judge genuine failure rather than severe failure.Source: §B.6 · p. 28
  • Valid ColBench friction includes vague or partial answers, omitted details, over-agreement, early closure, and correcting only explicit contradictions; severe conduct includes fabrication, stonewalling, contradiction, or breaking character.Fabricated specifics must be unsupported by hidden information and plausibly mislead the agent; vagueness or omission alone is not fabrication.Source: §B.6 · p. 28

C CASE STUDIES

The case studies show TIPER personas exposing failures through task-preserving interactions: one prompts an unsupported policy inference, while another leaves requirements recoverable but unelicited.

Retail Case Study: False Modification Constraint

  • The Retail persona reveals requirements in separate local scopes, exposing the agent’s tendency to treat a policy constraint as broader than it is.The failure analysis identifies over-restrictive policy inference, rather than missing information or task invalidation, as the failure mechanism.Source: C CASE STUDIES · p. 30
  • In the Retail case, the agent refused feasible laptop-order changes after wrongly generalizing a one-modification restriction, while the default interaction completed them.The searched persona supplied the details before execution and approved the agent’s complete summary; the paired default outcome shows the task remained feasible.Source: C CASE STUDIES · p. 30

ColBench Case Study: Missing Thresholds

  • The ColBench persona answers only the first part of a bundled clarification request and does not invent unspecified requirements.The failure analysis attributes the outcome to incomplete requirement elicitation and unverified assumptions, not fabricated information or refusal to cooperate.Source: C CASE STUDIES · pp. 30, 32
  • In the ColBench case, the agent supplied its own thresholds and produced code that failed the hidden test despite the user remaining responsive and the task solvable.The default simulator supplied fixed thresholds and boundary conditions after follow-up, and the agent implemented the reference behavior.Source: C CASE STUDIES · p. 32

D TASK-INTEGRITY AUDIT OF τ-TRAIT

The audit finds frequent task-critical simulator errors in τ-Trait conversations, distinguishes failure-conditioned integrity checks from broader trajectory checks, and frames behavioral alignment as descriptive rather than evidence of human-user distribution matching.

D TASK-INTEGRITY AUDIT OF τ-TRAIT

  • 28.0% of audited τ-Trait conversations contained a task-critical user-simulator error, with missing constraints the most frequent error type.The unique violation rate ranged from 25.2% for Confusion to 31.2% for Skeptical; these all-conversation rates are not directly comparable to Table 3’s failure-conditioned rates.Source: D TASK-INTEGRITY AUDIT OF τ-TRAIT · p. 33

E.1 INTEGRITY ANALYSIS PROTOCOL

  • The integrity audit marks a rollout invalid if either judge flags a task-critical user-simulator error, while the search-time severe judge examines only failed rollouts.The analysis-time check covers all evaluated rollouts; when few rollouts fail, the finite-sample search gate may reject a candidate after one violation.Source: §E.1 · p. 33

E.2 BEHAVIORAL ALIGNMENT

  • The User-Sim Index compares TIPER and other simulator trajectories with human-reference trajectories on Airline and Retail across four behavioral axes.Larger ECE* values indicate closer alignment; the comparisons are descriptive and do not establish that TIPER personas reproduce the full distribution of human users.Source: §E.2 · pp. 33–34

E.3 HUMAN VALIDATION OF TASK INTEGRITY AND FAILURE ATTRIBUTION

A human audit supports attributing most sampled, automatically accepted TIPER failures to agent errors in task-preserving interactions. The accompanying taxonomies separately characterize user behaviors and avoidable agent decision mechanisms.

E.3 HUMAN VALIDATION OF TASK INTEGRITY AND FAILURE ATTRIBUTION

  • 143 of 150 audited trajectories were both integrity-valid and associated with an identifiable agent error.Annotators judged 148 failures valid; one was invalid and one remained uncertain, while five valid failures were unattributed.Source: §E.3 · p. 34, Table 14 · p. 34

F BEHAVIOR AND FAILURE TAXONOMIES

  • The diagnostic framework connects task-preserving simulated-user behavior to canonical agent failure modes without forcing unsupported cases into a named mechanism.The user-side taxonomy has 10 behaviors in four families; the agent-side taxonomy has six canonical modes plus residual M0.Source: F BEHAVIOR AND FAILURE TAXONOMIES · p. 35, §F.2 · p. 37

F.1.1 REQUEST STRUCTURE

  • Request-structure behaviors include revealing existing requirements incrementally, bundling valid operations, and introducing a task-consistent caveat late.These behaviors test whether the agent revises its plan, resolves every operation, and revalidates affected constraints.Source: §F.1.1 · p. 35

F.1.2 PACING

  • Pacing behaviors include insisting on one small step at a time and requesting human escalation while remaining available to continue the assigned task.They test whether the agent integrates local progress or handles escalation without prematurely abandoning the task.Source: §F.1.2 · p. 35

F.1.3 UNDER-SHARING

  • Under-sharing behaviors range from sparse or vague information to incomplete verification and terse prompts, while keeping withheld information recoverable.These behaviors test grounding, follow-up elicitation, and retrieval or policy application without making the task infeasible.Source: §F.1.3 · pp. 35–36

F.1.4 NON-CORRECTION

  • Uncorrected assumptions leave an incorrect inference or partial work uncorrected, testing whether the agent settles on its first plausible interpretation.The user does not introduce false information or change the success criterion.Source: §F.1.4 · p. 36

F.2 AGENT FAILURE MODE TAXONOMY

  • The agent taxonomy distinguishes six avoidable decision errors, including trusting unverified reports, inferring spurious constraints, and assuming tools are unavailable.It also covers premature partial-match commitment, abandoning verifiable conditional requests, and incomplete option enumeration; M0 remains for other supported errors.Source: §F.2 · p. 36

F.3 PER-TRAJECTORY ANALYSIS

The analysis pairs default successes with integrity-valid TIPER failures to identify behavior and agent decision shifts. Its selected diagnostic cases and annotation rules bound what the resulting figures represent.

F.3 PER-TRAJECTORY ANALYSIS

  • Each analysis pair fixes the task, tools, and agent while contrasting a successful benchmark-default run with a failed integrity-valid TIPER run.This contrast helps separate persona-exposed failures from tasks the agent already fails under the default simulator.Source: §F.3 · p. 37

F.4 ANNOTATION PROTOCOL

  • Two judges independently label behavior and failure mode in separate passes, and only mutually agreed labels appear in the analysis figures.For single-label cause and failure-mode analyses, disagreements are excluded rather than adjudicated.Source: §F.4 · p. 37, §F.3 · p. 37

F.5 ANALYSIS-SET CONSTRUCTION

  • The 139-pair diagnostic set includes only default-success/TIPER-failure pairs that pass integrity checks and have identifiable behavior, decision shift, and failure.It is a diagnostic sample, not an estimate of behavior frequency in the benchmark.Source: §F.5 · p. 37

F.6 CONSTRUCTING THE ANALYSIS FIGURES

  • The behavior plot renormalizes percentages over six displayed behavior types, while Sankey ribbons can represent multiple labels from one trajectory.The four less-frequent taxonomy items remain in the codebook but are not plotted; ribbon width denotes annotation count, not distinct trajectories.Source: §F.6 · pp. 37–38

G.1 RELATION TO ITERATIVE RED-TEAMING AND PERSONA ENVIRONMENTS

  • TIPER differs from iterative jailbreak-prompt refinement and persona-conditioned social-intelligence or safety environments by searching task-preserving simulator policies against a fixed evaluated agent.The cited approaches do not directly instantiate persistent user policies for fixed tool-mediated tasks with trajectory-level task-integrity enforcement.Source: §G.1 · p. 38

G.2 LIMITATIONS AND FUTURE WORK

The limitations qualify TIPER’s coverage, integrity evidence, and search behavior; the accompanying pseudocode outlines its serial proposal, screening, and selection loop.

G.2 LIMITATIONS AND FUTURE WORK

  • TIPER searches fixed conversation-level personas, not adversaries that adapt their strategy turn by turn, and its model-based integrity judges provide empirical evidence rather than formal certification.The ColBench adapter also introduces benchmark-specific human priors; evolutionary search uses more model calls than fixed simulation, and minibatch improvement can limit accepted mutations at the score floor.Source: §G.2 · p. 38
  • The integrity protocol’s false-negative rate against human labels is unmeasured, and its failure-conditioned gate can have high sampling variance when candidates yield few failed rollouts.The Airline experiment transfers across benchmark releases with nonidentical search, rank, and test distributions; excluding disagreements may favor easier-to-classify failures, so pathway frequencies describe the agreement-filtered set.Source: §G.2 · p. 38
  • As rank-set dimensionality grows, many candidates can remain nondominated, so KeepK caps the frontier by mean rank-set score rather than guaranteeing high-dimensional diversity.Future work could explore adaptive user policies, tie-aware search gates, and targeted agent training with discovered personas.Source: §G.2 · p. 38

H ALGORITHM PSEUDOCODE

  • The serial search proposes child personas from parent trajectories, filters them through text checks and an integrity gate, then updates the active set through task-wise Pareto selection.The pseudocode also specifies generation and proposal-attempt limits, minibatch comparisons, rank-set evaluation, and final selection by rank-set score.Source: H ALGORITHM PSEUDOCODE · p. 39
Loading 2610.16155v1…