AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval

Gunja Agarwal; Arup Kumar Das; Arun Menon; Jitesh Chandra Mishra; Vignesh Divakaran

Source-linked AI summary

AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval

Gunja Agarwal, Arup Kumar Das, Arun Menon, Jitesh Chandra Mishra, Vignesh Divakaran

arXiv:2608.24076v1Aug 25, 2026cs.AI

TL;DR

Agentic information-retrieval evaluation is limited by scripted interactions that do not represent diverse users or adversarial brittleness. AgentWorld combines OCEAN user populations, stateful tool-use environments, consistency and fault measurement, training-data export, and adversarial risk analysis. Its experiments show personality-specific failures and quantify trajectory-level brittleness beyond pass^k.

Problem
Agentic information-retrieval evaluation remains limited by scripted interactions that fail to capture diverse users and adversarial brittleness.
Method
AgentWorld combines OCEAN-driven user simulation, stateful tool-use environments, pass^k consistency, structured diagnosis, handoff verification, score-gated export, and adversarial trajectory analysis.
Results
Experiments show personality variation exposes failures uniform testing misses, while the Risk Analyser quantifies trajectory-level brittleness beyond pass^k.
Takeaways & Limitations
Personality-aware evaluation provides controlled stress profiles for exposing agent failures across user variation and multi-step adversarial conditions.
Takeaways & Limitations
Persona-to-behaviour mappings lack validated calibration against human studies, and LLM judges introduce potential bias.

Abstract

from arXiv · show

Evaluation of agentic information retrieval remains limited to scripted interactions with uniform users, missing both natural personality diversity and adversarial brittleness. We present AgentWorld, a simulation framework combining (i)Big Five (OCEAN) personality-driven user populations with stateful tool-use environments; (ii)the pass$^k$ consistency metric with structured fault classification, partial-credit scoring, and dual-control handoff verification; (iii)score-thresholded training-data export in six fine-tuning formats; and (iv)an adversarial Risk Analyser that snapshots required-intermediate-state spines, branches Monte-Carlo rollouts under four task-aware perturbation types, and quantifies risk via $ΔP / ΔT$ scoring, Dempster--Shafer evidence fusion, and Shapley attack-category attribution. Three experiments demonstrate the framework: a conversational analytics agent across 10 OCEAN personas (240 evaluator judgments); a customer-support agent across 5 tasks $\times$ 4 persona variants; and adversarial stress-testing of 5 tasks revealing pre-existing trajectory brittleness ($V_{\min}=0.375$ without perturbation) and tool/infrastructure-layer attack dominance (Shapley: 46% system, 38% action). Personality variation surfaces failure modes uniform testing cannot expose---cross-domain leakage, contextual drift, a 0.27-point quality gap, and 50% vs. 100% pass-rate across personas on the same task---while the Risk Analyser quantifies trajectory-level brittleness that pass$^k$ alone cannot measure.

1 Introduction

AgentWorld addresses five shortcomings in agent evaluation by combining personality-diverse simulation, consistency and fault measurement, closed-loop training export, and adversarial trajectory analysis.

Motivation: Scripted tests often omit personality diversity, consistency measurement, detailed diagnosis, connected improvement, and adversarial brittleness measurement.These gaps leave uncertainty about failures across real-user variation and multi-turn attack surfaces.
Contributions: AgentWorld combines OCEAN-driven user simulation with stateful tool-use environments and pass^k consistency measurement.The framework evaluates multi-turn interactions rather than only scripted conversational responses.
Contributions: Structured fault classification, partial-credit scoring, and handoff verification diagnose who failed and what went wrong instead of using binary pass/fail alone.The handoff design includes a four-step cycle and personality-stratified coordination analysis.
Contributions: Behavioural score thresholds gate conversation export into six fine-tuning formats, connecting evaluation failures to training data.A model-comparison runner also tracks performance across OCEAN quadrants and checkpoints.
Contributions: The Risk Analyser branches from required intermediate states under four task-aware perturbation types and quantifies trajectory risk with ΔP/ΔT, Dempster–Shafer fusion, and Shapley attribution.This extends evaluation from cooperative testing to layer-specific adversarial brittleness.

2 Related Work

Prior frameworks cover complementary aspects of agent evaluation, user simulation, stateful environments, and adversarial testing, but surveyed systems do not combine AgentWorld’s full capability set.

Benchmarks and user simulation: τ-bench introduced pass^k consistency, τ2-bench added dual-control coordination, and AppWorld provided stateful apps with 457 APIs.Other benchmarks target browsers, operating systems, function calling, knowledge-grounded users, or automated error analysis.
Benchmarks and user simulation: Personality-like user simulation spans memory-driven characters, Big Five steering, social-intelligence evaluation, and multi-agent persona frameworks.These efforts establish components for simulating varied users but address different evaluation settings.
Coverage gap: Surveyed frameworks do not combine OCEAN personality, stateful apps, pass^k, structured faults, training export, and adversarial risk in one pipeline.The comparison is based on documented feature support as of submission.
Adversarial robustness and risk attribution: Existing adversarial systems attack prompt or tool surfaces and report binary pass/fail, whereas AgentWorld measures trajectory-level impact on multi-step task success.AgentWorld additionally fuses heterogeneous evidence and attributes aggregate risk across four attack categories.

3 AgentWorld Framework

AgentWorld simulates personality-driven users in stateful, tool-rich, optionally multi-agent environments, then evaluates and stress-tests complete task trajectories in a closed loop.

Architecture: AgentWorld runs personality-driven conversations through a four-phase pipeline that simulates users, evaluates interactions, exports training data, and reruns improved agents.The pipeline injects user messages into the agent under test and feeds evaluation back into improvement.
Personality-Driven User Simulation: OCEAN personas use five trait scores, optional custom traits, five-level natural-language mappings, and configurable population sampling.Trait combinations generate behaviours such as confrontational or emotionally reactive messages that conventional suites may omit.
Multi-Agent Topologies: Multi-agent simulations support full-mesh, hub-spoke, hierarchical, small-world, and scale-free topologies for multi-stage retrieval pipelines.Topology is configurable per simulation.
Stateful Tool-Use Environments: Stateful applications expose typed actions, persistent state, observations, and immutable audit trails so retrieval and action chains can be evaluated together.AgentWorld supports reproducible transactional execution and multiple built-in application categories.
Adversarial Risk Analyser: Task-aware perturbations target distinct simulation layers, while risk aggregation uses uncertainty-aware evidence fusion and attribution across attack categories.Perturbations are grounded in task state sequences and can be reviewed, edited, and persisted for reproducibility.
Adversarial Risk Analyser: The Risk Analyser captures required intermediate states, branches independent Monte-Carlo rollouts, and compares unperturbed V(s_t) with perturbed V′(s_t, p).Its four-phase architecture is spine, adversarial rollouts, scoring, and aggregation.

4 Evaluation Framework

AgentWorld evaluates agent reliability through repeated trials, diagnostic scoring, coordination overhead, and adversarial trajectory analysis. Its Risk Analyser combines counterfactual rollouts with uncertainty-aware aggregation and attack-category attribution.

Evaluation framework: AgentWorld evaluates completed simulations at both task level and individual-message behavioural level.Task-level scoring checks goal accomplishment, while behavioural scoring assesses message quality.
Consistency measurement: pass^k estimates the probability that an agent succeeds on every one of k trials, exposing reliability gaps hidden by one-off success.An agent succeeding in 8/10 trials has pass8 = 2.2%, whereas pass@8 reports 100%.
Diagnostic scoring: Structured fault classification identifies both responsibility and error type, turning aggregate failure labels into actionable diagnoses.Categories include Agent, Environment, or Task definition responsibility and errors such as wrong parameters, missing actions, and policy violations.
Diagnostic scoring: Partial-credit scoring combines completed-step and correct-field fractions, so a task with 9/10 steps and 4/5 fields scores 0.85.This distinguishes near-misses from immediate failures.
Coordination cost: A large Δcoord indicates degradation from solo to dual-control operation and points toward conversational-design fixes rather than a stronger base model.Dual-control tasks split backend-tool and device-app access between the service agent and user.
Adversarial risk: The Risk Analyser branches Monte-Carlo rollouts from required intermediate states to compare baseline V(s_t) with perturbed V′(s_t, p), deriving failure-probability and delay impacts.It scores checkpoint sensitivity, causal necessity, and importance before producing per-perturbation risk scores.
Adversarial risk: Dempster–Shafer fusion preserves ignorance and reports source conflict, while exact Shapley values attribute aggregate risk across four attack categories.The design targets dilution of strong evidence and silent masking of disagreement associated with flat averaging.

5 Experiments

Across three experiments, AgentWorld evaluates agents with persona-stratified behavioural and task-level measures, repeated consistency testing, and adversarial trajectory analysis. The experiments expose persona-dependent quality and completion gaps, step-budget bottlenecks, and brittleness concentrated in tool and infrastructure layers.

Experiment 1: The analytics-agent study evaluated 10 OCEAN personas across 60 messages and 240 evaluator judgments, with all 30 agent calls succeeding without fallbacks.The panel jointly varied persona, role, and query domain.
Experiment 1: A 0.27 relevance spread ran from 0.88 for Campaign Analyst to 0.61 for Executive Synthesis, while aggregate relevance was 0.75.Because role and domain co-varied with personality, the spread reflects combined difficulty rather than personality alone.
Experiment 1: Structured evaluation identified response truncation, contextual drift, and cross-domain data leakage that aggregate scoring concealed.Truncation affected 5/30 responses, contextual drift occurred in 3 instances, and leakage occurred once.
Experiment 2: In customer support, 17 of 19 single-run simulations passed under LLM-judge criteria, but completeness averaged 0.508 while other criteria exceeded 0.81.Four OCEAN persona variants were evaluated across five tasks.
Experiment 2: Within-task judge scores differed by 0.145 points for Complete Payment and 0.122 points for Cancel Payments across personas.The largest reported gap was 0.906 versus 0.761 for Complete Payment.
Experiment 2: Step-budget exhaustion drove low completeness: 14 of 19 simulations hit the maximum step budget, including Anxious Complete Payment at 0.420 completeness.Complete Payment reached 0.950 completeness for the Impatient persona in three steps but used all 15 steps for the Anxious persona.
Experiment 2: pass8 exposed reliability gaps hidden by single judge runs, with some cells falling to pass8=0.23–0.50 despite passing the original run.Complete Payment and Withdraw to Bank remained at passk=1.00, while other tasks eroded as k increased.
Experiment 3: Risk analysis found 100% behavioural pass coexisting with V_min=0.375, where only 3/8 unperturbed rollouts succeeded at a required intermediate state.State mutation produced the peak ΔP=0.458, and Dempster–Shafer fusion yielded Bel=0.727.

6 Discussion and Limitations

AgentWorld combines personality-aware simulation, reliability measurement, training-data generation, and adversarial risk analysis to expose brittleness in agentic information retrieval. Its limitations include unvalidated persona realism, LLM-judge dependence, equivalent sensitivity and causal measures under default perturbations, and degraded Dempster–Shafer fusion under high conflict.

Limitations: Persona-to-behaviour mappings are controlled, reproducible stress profiles rather than validated proxies for real users with matching OCEAN traits.Validation against human Big Five instruments and elicited agent behaviour remains future work.
Limitations: The conclusions depend on LLM judges, whose biases require cross-judge agreement analysis to bound their influence.The paper also describes DPO auto-construction as using score differentials as a proxy for human preference.
Discussion: V_min=0.375 without perturbation shows pre-existing trajectory brittleness in the Check Refund Eligibility example.At checkpoint t=1, 3 of 8 baseline rollouts succeeded.
Limitations: Under default single-dimension perturbations, sensitivity g(s_t) and causal c(s_t) are numerically equivalent, while Dempster’s rule degrades when conflict k>0.7.Multi-dimensional perturbations separate sensitivity and causal measures; attack-graph kill-chain analysis is deferred because single-attack dominance makes multi-step chains uninformative.
Discussion: AgentWorld unifies personality-driven simulation, reliability measurement, training-data generation, and adversarial risk analysis in one closed-loop pipeline.The stated goal is to expose IR failure modes that uniform testing misses and make personality-aware reliability evaluation practical for agentic IR.