PaperlayerView exact revision on arXiv

Source-linked AI summary

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Ashish Jain, Armaan Sandhu

arXiv:2609.38043v1cs.AI

TL;DR

Interactive benchmarks use LLMs as executable users, but their role adherence is not directly measured even though they control information timing and actions. UserProxyBench evaluates that contract fidelity with task-grounded rubrics and UFS; varying only the user proxy changes reward and can leave successful episodes with user-specification violations.

  • Problem

    Benchmarks score the agent but do not directly measure whether an LLM user proxy follows its assigned role, despite the proxy controlling information timing and sometimes user-side actions.

    Source: §1 · p. 1

  • Method

    UserProxyBench scores contract adherence independently of agent success using task-grounded criteria drawn from private blueprints, simulator guidelines, and user tools.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Mean Rτ ranges from .644 to .796 with GPT-5.5 fixed, and 24.4% of agent-successful episodes fail UFS.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    UFS can serve as a reliability constraint for selecting the least expensive simulator that meets a pipeline’s required fidelity.

    Source: §3.1 · p. 2, §1 · p. 2

    UFS is LLM-judged by the rubric generator, and the manual audit is single-rater; the study also uses one rollout per task and does not measure downstream trained-policy effects.

    Source: §4 · p. 4

Abstract

from arXiv · show

Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated user controls what information the agent receives and when, yet current benchmarks score only the agent and do not directly measure whether the user correctly executed its assigned role. We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success. Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks changes mean task reward by 15.2 points, while 24.4% of successful episodes contain a user-specification violation. The dominant failure is premature disclosure: users provide information before it is requested. This behavior has little effect on task reward, yet among successful episodes it causes the agent to make 1.06 fewer tool calls on average, changing the interaction being evaluated while preserving the reward. Finally, across seven proxies we identify an empirical cost-fidelity frontier, enabling practitioners to select the least expensive simulator that satisfies a required fidelity level.

1 Introduction

Interactive benchmarks and training increasingly use LLMs as executable users, making the proxy part of the evaluated interaction. UserProxyBench addresses the resulting gap by measuring whether proxies follow their assigned user contract, independently of agent reward.

1 Introduction

  • LLM user proxies control information timing, observations, and sometimes user-side actions, so their behavior can change the interaction an agent is meant to solve.Premature disclosure can remove a task’s intended information-gathering step even when reward is unchanged.Source: §1 · p. 1
  • Prior work examines human-likeness and simulator effects on assistant performance; this paper instead asks whether a proxy correctly executes the benchmark’s stated user specification.The paper calls this functional fidelity: contract adherence, not human realism.Source: §1 · p. 1
  • Nearly one quarter of successful trajectories contain a user-contract violation, and the most common failure is almost invisible to agent reward.UserProxyBench scores the user independently of the agent and treats proxy choice as a fidelity-versus-cost trade-off.Source: §1 · pp. 1–2
  • The paper positions user-proxy choice as an operational trade-off: use UFS as a reliability constraint and minimize user-side inference cost.Source: §1 · p. 2

2 UserProxyBench

UserProxyBench defines fidelity through task-specific user-contract criteria and evaluates proxies across four enterprise domains with the agent held fixed. Its rubrics are grounded in private task instructions and simulator guidelines, while UFS counts episodes satisfying every applicable criterion.

2 UserProxyBench

  • UFS is the fraction of episodes in which a proxy satisfies every applicable criterion in its own contract.A judge-free user-action probe is reported for Telecom but excluded because other domains lack the same signal.Source: §2 · p. 2
  • Each task’s rubric contains 4–8 atomic, falsifiable user-only criteria grounded in its private blueprint, simulator guidelines, user tools, and a sample interaction.The blueprint and guidelines remain the correctness authority; an independent verifier reviews rubric grounding, achievability, redundancy, and scope.Source: §2 · p. 2
  • The 2,141 criteria span groundedness, premature disclosure, goal deviation, and missed information across Telecom, Airline, Retail, and Banking.The paper scores compliance with explicit instructions such as waiting for the agent to request information, rather than an implicit style preference.Source: §2 · p. 2
  • The evaluation covers 375 tasks across four enterprise domains, with GPT-5.5 frozen as the agent and only the user proxy changing.Each task has one rollout per proxy; the reported set includes three open-weight and four hosted models.Source: §2 · p. 2

3 Results

With GPT-5.5 fixed, changing only the user proxy shifts task reward, and some successful episodes violate user specifications. The results show how to choose a proxy under a fidelity requirement and why reward alone can miss changes to the interaction.

3.1 Fidelity-constrained proxy selection

  • At q = .84, Gemma-31B is the lowest-cost qualifying proxy; at q = .86, Gemini-3.5-Flash becomes the operating point.Gemma-31B has UFS .845 at about $0.0175 per simulation; Gemini-3.5-Flash gains 1.9 UFS points but costs 3.2× more.Source: §3.1 · p. 2, Figure 1 · p. 3

3.2 The user proxy changes the benchmark result

  • With GPT-5.5 fixed, proxy choice shifts mean Rτ from .644 to .796, and 24.4% of agent-successful episodes fail UFS.Figure 2 shows that only the user simulator changes across proxy conditions.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 Why task reward cannot screen user proxies

  • Premature disclosure is the most common failure (480) but changes agent pass rate by only −.043 on average, making it hard for Rτ to detect.Its median effect is positive (+.033), with inconsistent signs across proxy–domain cells; goal deviation is less common (134) and reduces pass rate by .515 on average.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4 Implications for multi-turn RL

  • Among successful episodes, over-disclosing users lead the frozen agent to make 1.06 fewer tool calls and ask 0.26 fewer questions despite identical task reward.Across 374 tasks completed by all seven proxies, mean agent turns correlate more strongly with UFS (r = .955) than with Rτ (r = .594).Source: §3.4 · p. 4

4 Discussion

UserProxyBench measures contract fidelity rather than human realism, and the authors identify limits in its evaluation design and scope.

4 Discussion

  • UFS measures contract fidelity, not human realism; the two are complementary.The evaluation also uses one rollout per task, time-dependent serving prices, a pinned Banking version, and no measurement of downstream trained-policy effects.Source: §4 · p. 4
  • The manual audit covers ten tasks and 50 criteria or judgments, but uses a single rater.The authors also report a judge-free Telecom probe: 54 of 658 trajectories miss gold writes, with 23 passing all applicable criteria and 20 passing a directly relevant criterion.Source: §4 · p. 4

A Full proxy results

Table 1 reports aggregate proxy results, sorted by user-side cost, for the frozen GPT-5.5 agent.

A Full proxy results

  • Table 1 sorts aggregate proxy results by user-side cost and reports unweighted four-domain means for agent Rτ and UFS.The dagger marks proxies strictly dominated on the cost–UFS plane.Source: Table 1 · p. 5

B Per-domain Rτ and UFS

Table 2 presents per-domain results and reports the trajectory counts and scoring exceptions.

B Per-domain Rτ and UFS

  • Table 2 gives per-domain results for Telecom, Airline, Retail, and Banking, with stated sample-count exceptions and 2,618 of 2,625 trajectories scored.Banking has n = 96 in two infrastructure-error cells; Retail has n = 113 for five proxies where UFS is undefined because no criteria applied.Source: Table 2 · p. 6

C Failure taxonomy and examples

The section organizes user failures into four families and illustrates them with trajectories; it then describes rubric construction and a judge-free Telecom diagnostic.

C Failure taxonomy and examples

  • The taxonomy defines 19 failure modes across four families, with 18 exercised by the 2,141 retained criteria.Premature disclosure includes volunteering unasked information, preempting diagnosis, and dumping a blueprint; other families cover groundedness, goal deviation, and missed information.Source: C Failure taxonomy and examples · p. 6
  • One premature-disclosure example gives the user's name and number in the opening turn, before the agent requests identifying information.Other examples include claiming to turn off Airplane Mode without a tool call while the device remains in that state, and claiming an MMS succeeded without a successful tool result.Source: C Failure taxonomy and examples · p. 6

D Rubric construction and judge-free probe

  • Rubrics are generated from the private blueprint, simulator guidelines, user tool surface, and one sample conversation, then batch-reviewed against nine checks.The sample is explicitly not treated as an oracle, and the trajectory grader scores criteria one at a time without proxy identity.Source: D Rubric construction and judge-free… · p. 6
  • Among 658 scorable Telecom trajectories, 54 miss a gold user write; adding this judge-free signal changes UFS by .009–.070 per proxy but leaves rankings unchanged.The signal is reported as an independent diagnostic rather than incorporated into the cross-domain UFS definition.Source: D Rubric construction and judge-free… · pp. 6–7
  • Figure 4 plots mean agent turns across 374 commonly completed tasks against UFS and agent Rτ, with each proxy labeled.The supplied figure description identifies the plotted measures and scope but does not state a comparison outcome.Source: Figure 4 · p. 7

E Agent effort versus proxy metrics

Across 374 tasks completed by all seven proxies, mean agent turns align more strongly with UFS than with agent task reward, making UFS a stronger predictor of induced interaction length.

F Limitations and next validation

The current manual audit is limited in scale and independence; the authors identify blinded human validation and additional experiments as next steps.

  • The manual audit covers ten end-to-end tasks and 50 additional final criteria or judgments reviewed by one author.Source: F Limitations and next validation · p. 7
  • The audit is not blinded, stratified, or multi-rater, motivating a blinded human study of criterion validity and judge decisions with agreement statistics.Source: F Limitations and next validation · p. 7
  • Planned checks include Telecom rollouts, a second frozen agent, and an RL comparison of policies trained with high- versus low-UFS proxies.These would separately test stochasticity, interaction-partner dependence, and downstream training effects.Source: F Limitations and next validation · p. 7

G Experimental configuration and assets

The paper reports proxy configuration and benchmark implementation details, and cautions that contract fidelity does not establish human realism.

G Experimental configuration and assets

  • All seven proxies use temperature 1.0, while the frozen GPT-5.5 agent uses high reasoning effort; proxy thinking settings vary by model.Sonnet-4-6 ran without extended thinking because the trajectory runner did not forward its provider-specific parameter.Source: G Experimental configuration and assets · p. 7
  • Rubric generation and grading both use Claude Opus 4.8, while GPT-5.5 remains the frozen agent and is excluded from the proxy set because its outputs participate in rubric construction.The benchmark uses tau2-bench v1.0.0 with two local engineering patches; tables and figures are regenerated deterministically from graded trajectories.Source: G Experimental configuration and assets · p. 7

H Broader impact

  • High UFS may still underrepresent real users, dialects, accessibility needs, or adversarial behavior, so it should complement rather than replace real-user and population-sensitive evaluations.Cost-frontier results are time-dependent and are not long-lived commercial recommendations.Source: H Broader impact · p. 8

Related papers

Loading 2609.38043v1…