PaperlayerView exact revision on arXiv

Source-linked AI summary

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

Anjali Kantharuban, Jonas Mueller

arXiv:2610.02460v1cs.CL

TL;DR

User simulators can sound human without reproducing the outcomes seen with real users, limiting what fidelity alone establishes about benchmark validity. CUE encodes observed sessions or samples continuous user representations and decodes them into prompts; evaluations report improved outcome calibration over persona-based methods alongside competitive fidelity.

  • Problem

    Human-like dialogue does not ensure that simulators reproduce the agent outcomes observed with human participants, so outcome calibration requires separate evaluation.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    CUE encodes observed sessions or samples continuous representations, then decodes them into persona commands for replay or sampled user simulation.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    CUE improves outcome calibration over other persona-based simulators while retaining competitive fidelity, although a no-persona baseline beats several methods in some τ2-Bench settings.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    Benchmarking should evaluate outcome calibration directly rather than treating naturalness, mimicry, or coverage as sufficient evidence of validity.

    Source: §8 · p. 11

    The evaluation does not establish that improved outcome reproduction preserves comparative rankings across multiple agents or that retrospective replay predicts future user behavior.

    Source: Limitations · p. 11, §8 · p. 11

Abstract

from arXiv · show

Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking also requires alignment in when and how agents fail across simulated and real user populations. We find that existing simulators lack outcome calibration: agreement with observed success rates and failure patterns when real users interact with the same agent. We introduce Calibrated User Embeddings (CUE), a framework that both encodes observed sessions and samples continuous representations, then decodes them into persona commands to steer LLMs to act as user simulators without training. Through this, we evaluate user-conditioned replay of past sessions and aggregate metric agreement when sampling novel personas for the same tasks. On $\tau^2$-Bench, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs than other persona-based simulation methods. These gains coexist with competitive user fidelity as measured using metrics established in prior work. After being fit to mostly customer support interactions, the same CUEd simulators generalize to document creation, math tutoring, and casual conversation, and remain effective across different simulator LLMs without CUE retraining.

1. Introduction

Interactive benchmarks need simulators that reproduce real-user outcomes, not just human-like dialogue. The paper introduces CUE and evaluates fidelity, outcome calibration, and transfer across settings.

1. Introduction

  • Outcome calibration measures whether simulated interactions reproduce agent outcomes observed with human participants, a property that fluent, human-like dialogue alone does not ensure.The paper examines role adherence, failure reproduction, and aggregate and user-specific success alignment.Source: §1 · p. 1
  • CUE combines session encoding, persona-command decoding, training-example retrieval, and latent diffusion to support observed-session replay and sampling new user representations.It is trained on behavior and style without benchmark rewards or failure labels in the training objective.Source: §1 · p. 2
  • The paper evaluates fidelity alongside outcome agreement, presents reusable replay and sampled simulation, and tests transfer to held-out domains.The introduction reports improved outcome calibration over other persona-based simulators and competitive fidelity.Source: §1 · p. 2

2. Related Work

Prior work addresses simulator artifacts and user-specific fidelity, but fidelity does not necessarily establish outcome calibration. CUE connects continuous user representations to simulation.

Interactive Benchmarks

  • Multi-turn benchmarks often depend on user simulators, while prior work reports realism and aggregate-success gaps and fidelity is not necessarily a proxy for outcome calibration.The paper argues that outcome calibration requires separate evaluation.Source: §2 · pp. 2–3

User Simulation

  • One line of work reduces generic simulator artifacts or optimizes realism and coverage; another conditions simulation on profiles extracted from observed users.The latter includes structured persona manuals and simulators trained to reproduce implicit profiles.Source: §2 · p. 3

Continuous User Representations

  • CUE uses a continuous latent representation to encode observed users, sample additional session representations, and produce natural-language prompts usable with arbitrary LLMs.This differs from discrete profiles, command lists, or simulator-specific parameterizations.Source: §2 · p. 3

3. The Calibrated User Embedding (CUE) Framework

CUE provides a shared representation and prompt interface for replaying observed sessions or sampling user representations. Decoded commands and retrieved examples condition the simulator prompt.

3. The Calibrated User Embedding (CUE) Framework

  • CUE encodes behavior and style from sessions and supports both observed-session replay and sampling additional representations.The pipeline constructs persona-command targets, trains an encoder and decoder, and fits a sampler to encoded sessions.Source: §3 · p. 3, Figure 2 · p. 4

3.1. User-Conditioning for Replay as Observed Users A

  • For replay, user turns are encoded separately from preceding assistant turns, which enter the session representation through attention.Content-suppression losses reduce, but cannot guarantee removal of, task and outcome information.Source: §3.1 · p. 3
  • A prompt decoder turns a CUE into persona commands using a fixed soft memory bank and typed slots, then concatenates emitted commands into a natural-language prompt.The natural-language interface decouples the learned representation from a specific simulator LLM.Source: §3.1 · p. 4

3.2. Sampling Plausible Session Representations B

  • A latent diffusion model samples new CUEs from the learned distribution rather than selecting nearest vectors from the empirical bank.Generated vectors receive LayerNorm-based normalization before decoding.Source: §3.2 · p. 4
  • Sampling can be unconditional or conditioned on a neighborhood of observed users, but the session-weighted training mixture need not match a target population.Unless otherwise noted, evaluations use unconditional sampling without neighborhood conditioning or classifier-free guidance.Source: §3.2 · p. 5

4. Experimental Setup

CUE is trained on a mixed conversation corpus with contrast-based persona-command supervision. Evaluation tests replay and sampled simulation against human episodes using fidelity and outcome metrics in a shared benchmark harness.

4.1. Training

  • Training uses approximately 170k English multi-turn conversations from 24 DialogStudio datasets, LMSYS-Chat-1M, and WildChat-1M, capped at 10k samples per dataset.Only sessions with accepted persona annotations supervise CUE.Source: §4.1 · p. 5
  • A proposal LLM contrasts each observed user with four simulators’ counterfactual replies and other human sessions to propose behavior and style commands.Lexical and consistency gates filter targets; two evaluated base models occur in the contrast set, while Gemini 3.5 Flash Lite represents an unseen family.Source: §4.1 · p. 5
  • The encoder, decoder, and sampler use ModernBERT, Qwen 3 0.6B, and an approximately 162M-parameter MLP; inference adds eight retrieved examples to decoded commands.The evaluated treatment therefore includes both learned commands and retrieval.Source: §4.1 · p. 5

4.2. Evaluation

  • The primary evaluation uses 483 human airline and retail episodes from the τ-USI study with τ2-Bench, alongside SimulatorArena and PRISM.In replay, CUE encodes the completed trajectory including assistant context; sampled evaluation instead uses a latent vector without the target trajectory.Source: §4.2 · p. 5
  • Fidelity is assessed through naturalness, mimicry, and coverage, while outcome measures include success-rate error, paired-episode F1 scores, conditional failure TVD, and user-share gap.Baselines run under a shared harness with matched tasks, agents, and turn budgets; these are end-to-end comparisons, not matched-supervision controls.Source: §4.2 · p. 6

5. Fidelity to Real Users

CUE is competitive on established user-fidelity measures, but results vary by axis: mimicry metrics disagree, while coverage scores and sampling results point to increased output variance.

Naturalness

  • CUE is competitive with current state-of-the-art user simulation techniques on established fidelity criteria.PPol leads S2R-N, but its evaluator uses the same trajectory taxonomy as its prompt optimization; excluding PPol, CUE performs far better on S2R-N in all but one setting and improves TT over no-persona across six settings.Source: §5 · p. 6, Table 2 · p. 7
  • CUE improves on RealUserSim’s naturalness, but differences in example access and distribution domain prevent attributing this solely to representation learning.Source: §5 · pp. 6–7

Mimicry

  • Paired CUEs score 0.38 versus 0.21 for shuffled CUEs on PT3 and 0.27 versus 0.20 on AVA, showing pairing sensitivity despite disagreement in method rankings.Episodes were shuffled within a single domain across tasks; the two mimicry metrics are not consistent in differentiating methods.Source: §5 · p. 7

Coverage

  • CUE attains best or tied-best reported coverage; on S2R-C, it scores 0.83 versus None’s 0.64 with GPT 5.4 Mini, and 0.63 versus None’s 0.01 with Llama 3.1 8B.UserLM is also competitive; for Gemini 3.5 Flash Lite, unconditional diffusion sampling outperforms resampling from the training session bank, though this does not establish diffusion is necessary.Source: §5 · p. 7, Table 2 · p. 7

6. Calibration to Real Outcomes

CUE reproduces specific user-task failure modes well, but success-rate and attribution comparisons vary across simulator base models. Fidelity rankings do not consistently identify the best outcome-calibrated simulator.

Failure Reproduction

  • CUE achieves the highest user-task failure mode macro-F1 with GPT or Gemini and falls within the confidence interval of the highest value with Llama.Source: Table 3 · p. 8, §6 · p. 8

Outcome Evaluation

  • LLM-assisted thematic analysis assigned user, agent, or environment attribution and agent-failure modes, with the taxonomy refined over three researcher-reviewed batches of 100 episodes.The final batch required one correction.Source: §6 · p. 8

Role Adherence

  • Paired attribution macro-F1 is mixed: CUE beats None on Llama (0.38 vs 0.35), ties on Gemini (0.33), and trails on GPT (0.31 vs 0.36).CUE has the smallest user-share gap among persona-based methods on each base model, but the gap is larger than None on GPT (0.10 vs 0.08).Source: §6 · p. 8, Table 3 · p. 8

Failure Reproduction

  • CUE has the lowest sampled agent-failure TVD on GPT (0.18) and Gemini (0.28), while UserLM leads on Llama (0.31).Figure 3 shows conditional failure-mode proportions, including authentication deadlocks and unnecessary escalation.Source: §6 · p. 8, Figure 3 · p. 9

Success Alignment

  • Against the human success rate of 0.61, None is closest on Llama, while CUE beats all methods on Gemini and has the highest persona-method replay outcome macro-F1 on every base model.Compared with None, replay performance is better on the two stronger simulator base models.Source: Table 3 · p. 8, §6 · p. 9

Fidelity and Outcome Agreement

  • Fidelity rankings do not identify a uniformly best outcome simulator; persona conditioning can improve behavioral resemblance while shifting task difficulty or cooperation and worsening agreement with human outcomes.Source: §6 · p. 9

7. Transfer to Held-Out Benchmarks

The fixed CUE model is evaluated on held-out SimulatorArena and PRISM benchmarks without retraining. Results show lower document-quality error on SimulatorArena and a tradeoff between generated-user proximity and human-population coverage on PRISM.

Held-out evaluation

  • The same CUE model is tested without retraining on held-out SimulatorArena co-writing and tutoring tasks and PRISM casual conversation, despite training data being mostly task-oriented.The training mixture includes approximately 20k general-conversation samples from LMSYS and WildChat.Source: §7 · p. 9

SimulatorArena

  • 2.21 versus 5.80 document-quality MAE for CUE and RealUserSim, respectively, on GPT 5.4 Mini.CUE also has 1.33 versus 1.39 interaction-quality MAE; its document error remains above the human comparison range.Source: §7 · pp. 9–10, Table 4 · p. 10

PRISM

  • At radius 0.05 on PRISM, 81% of CUE trajectories have a human neighbor versus 67% for the nearest alternative; USP and UserLM instead achieve 80% and 78% human-to-generated coverage, versus CUE’s 69%.All reported methods reach at least 0.86 on behavioral coverage scores; proximity to human samples and coverage of the human population are distinct goals.Source: §7 · pp. 10–11, Figure 4 · p. 10

8. Conclusion

CUE provides a shared representation and prompt interface for replaying observed sessions and sampling users, with competitive fidelity and generally improved outcome agreement across three benchmarks. The results also show that stronger persona resemblance alone need not improve benchmarking validity, and leave several calibration questions unresolved.

Conclusion

  • Across τ2-Bench, SimulatorArena, and PRISM, CUE is competitive on user fidelity and generally improves outcome agreement over other persona-based methods.The evaluation does not establish that the continuous representation or diffusion sampler individually causes these gains.Source: §8 · p. 11
  • The results do not establish that sampled sessions match every target population or that retrospective replay predicts future behavior; testing mechanisms and agent-ranking effects requires matched-supervision controls and studies across multiple agents.Improved outcome reproduction is not shown to preserve comparative agent rankings.Source: §8 · p. 11
  • On several τ2-Bench settings, the no-persona baseline beats several baseline methods, so a more human-like persona does not necessarily yield a better benchmarking proxy.The authors recommend evaluating outcome calibration directly rather than treating naturalness, mimicry, or coverage as sufficient evidence of benchmarking validity.Source: §8 · p. 11

Limitations

The paper identifies limits in outcome-calibration evidence and in the scope of its training data. Its training pipeline constructs persona-command supervision from contrastive examples and filters candidate manuals through explicit gates.

Limitations

  • Outcome calibration rests on two human studies; comparative agent rankings are untested, correlation analysis is descriptive, and replay does not establish prospective user-behavior prediction.Other limitations include LLM-based failure attribution, English-only data, some proprietary simulator models, and calibration that remains imperfect.Source: Limitations · p. 11

A.1. CUE Training Data

  • The training inventory mixes naturalistic human–LLM conversations, elicited or role-play dialogue, human-human negotiations, and authored or paraphrased task data; not all are real-user interactions with deployed agents.Only trajectories with the specified supervision are used for training, and related dataset versions in the inventory are not necessarily unique conversations.Source: §A.1 · p. 17, Table A1 · p. 18

A.2. Proposal Prompts & Gating for Training

  • Persona manuals draw on simulator contrasts from four LLMs and five retrieved real-user conversations, aiming to distinguish simulator behavior from differences among humans.The proposal model generates five simulator-contrast, five user-specific, and five style commands, each linked to evidence turns.Source: §A.2 · p. 17
  • Prompts exclude task-specific content and require conditional rules for history-dependent behaviors, while deterministic gates check command coverage, evidence grounding, leakage, and distinctive human contrasts.Additional gates check emotion consistency and answer-lag behavior and coverage.Source: §A.2 · pp. 17–18
  • Failed manuals are regenerated with feedback for up to three attempts; the resulting rollout set contains approximately 170k labeled examples for CUE training.The encoder first forms turn embeddings and then aggregates them into a session-level CUE.Source: §A.2 · p. 18, Figure A1 · p. 19

A.3. Evaluation

This section evaluates CUE across task-oriented dialogue, open-ended collaboration, and general conversation using replay and sampled-user settings, fidelity metrics, and success estimates; it also documents training and sampling mechanisms, comparator implementations, and simulator-model effects on naturalness. Bootstrap intervals capture finite-item uncertainty conditional on fixed seeds, not seed-to-seed variability.

A.3. Evaluation

  • Evaluation compares simulated users with 483 human episodes from τ2-Bench and also tests collaboration and conversation using SimulatorArena’s 450 mathematics and 495 writing interactions and PRISM’s 5,284 conversations.The evaluated agent is matched to the original interaction as closely as possible; τ2-Bench episodes use GPT 5.2 with high reasoning.Source: §A.3 · p. 19

B.1. Joint Encoder–Decoder Training

  • CUE jointly learns slot-conditioned persona commands and representations that preserve user behavior and style while limiting content leakage and representation collapse.Its objectives include content-span perturbation invariance, session-level InfoNCE, style-aware contrastive learning, and VICReg penalties; command losses use injective slot-command assignments and head separation.Source: §B.1 · p. 20

B.2. Sampler

  • The latent diffusion sampler generates novel CUE representations by conditioning noise prediction on a center embedding’s local neighborhood, with auxiliary losses encouraging local coherence and separation.Training selects one of the center’s eight nearest neighbors as the target, conditions on a random non-empty subset of the others, and uses null conditioning with probability 0.15.Source: §B.2 · pp. 21, 23

B.4. Base Model Flexibility

  • Across SimulatorArena and PRISM, CUE naturalness increases as the simulator base model strengthens, allowing stronger or closed-source models without CUE retraining.The CUE model remains fixed; the passage does not provide numerical results.Source: §B.4 · p. 23

C. Baselines

  • The four comparators are UserLM, USP, PPol, and RealUserSim, evaluated under a shared closed-loop protocol with common episode construction and generation wrapping.UserLM is sampled-only; USP supports target-profile replay and sampled profiles; PPol evolves task personas and is evaluated only on τ2-Bench, with its results favorable to in-domain optimization; RealUserSim samples external WildChat profiles or infers a target-user profile without example utterances, unlike CUE’s learned bottleneck and continuous sampler.Source: C. Baselines · p. 25, §C.1 · p. 25, §C.2 · p. 25

D.1. Naturalness

  • Naturalness metrics assess behavioral-fingerprint resemblance to human trajectories and raw-trajectory discrimination, but S2R-N is not calibrated human-likeness or proof of statistical indistinguishability.The S2R-N probe is trained against vanilla-simulator artifacts and held fixed; TT uses an LLM judge with paired trajectories, few-shot examples, and both presentation orders.Source: §D.1 · p. 27

D.2. Mimicry

  • Mimicry is measured through human-calibrated style similarity (AVA) and paired-trajectory judgments (PT3) across persona, style, competency, interaction habits, and pacing.AVA’s threshold is calibrated only on human data; higher AVA and PT3 scores indicate stronger alignment with the paired human trajectory.Source: §D.2 · pp. 27–28

D.3. Coverage

  • Coverage metrics use bidirectional Chamfer scores to compare simulated and human populations in behavioral-fingerprint space (S2R-C) and human-defined stylistic space (SD-C).SD-C fits PCA on human embeddings alone and retains 16 components, making it sensitive to whether simulated users span real users’ principal stylistic variation.Source: §D.3 · p. 28

E.1. Finite-Sample Uncertainty

  • Tables A5–A6 report percentile-bootstrap intervals for fidelity and success estimates, capturing finite-item uncertainty conditional on three fixed seeds, not seed-to-seed variability.Metrics are computed within each subdomain and seed, macro-averaged across airline and retail, and averaged across seeds; bootstrap replicates resample items within subdomains and recompute metrics over the three fixed seeds.Source: §E.1 · p. 29

E.2. Expanded Outcome Calibration Evaluation

The expanded evaluation tests whether mimicry depends on paired user sessions and whether continuous sampling improves coverage. CUE does not consistently lead adjusted success, and the ablations do not establish that its representation or diffusion mechanism is necessary for full-system performance.

E.2. Expanded Outcome Calibration Evaluation

  • Failure-mode support varies by method: the Llama USP mode score uses 19 cases, compared with 118 for CUE.Attribution macro-F1 uses paired human-and-simulator failures; failure-mode macro-F1 further requires both failures to be agent-attributed.Source: §E.2 · p. 29
  • RealUserSim has the smallest adjusted-success gap on GPT, while CUE does not consistently lead this metric against a human reference of 0.75.Adjusted success compares separately selected samples because exclusions depend on each population’s outcomes, rather than evaluating agent performance on one common subset.Source: §E.2 · p. 29
  • AVA falls from 0.27 to 0.20 and PT3 from 0.38 to 0.21 when CUEs are shuffled across same-domain sessions, indicating sensitivity to paired-user information.The authors interpret this as evidence that these metrics respond to user-specific pairing rather than domain-level behavior alone.Source: Table A11 · p. 30, §E.2 · p. 30
  • Unconditional diffusion sampling scores S2R-C/SD-C 0.83/0.57, compared with 0.77/0.56 for direct resampling from the training-session pool.The authors conclude that sampling from the continuous distribution can increase behavioral coverage relative to discrete resampling.Source: Table A12 · p. 30, §E.2 · p. 30
  • Neighbor-conditioned sampling yields S2R-C 0.84 but SD-C 0.49, so local conditioning does not uniformly improve coverage.Its S2R-C is similar to unconditional sampling, while its SD-C is lower.Source: Table A12 · p. 30, §E.2 · p. 30
  • The ablations support retained user-specific information and broader coverage from continuous sampling, but do not establish either component as necessary for full-system performance.The authors qualify the conclusion because the ablations do not isolate the representation or diffusion mechanism as necessary.Source: §E.2 · p. 30

F. Failure Mode Analysis

This section presents a failure taxonomy for τ²-Bench and explains how failures are attributed to agents, simulators, or the evaluation environment, labeled, and aggregated, including measurement of conditional user-side attribution gaps.

F. Failure Mode Analysis

  • The analysis identifies 16 primary failure modes spanning agent errors, simulator errors, and evaluation-environment failures.Researchers iteratively reviewed three batches of 100 failed trajectories; in the third batch, they revised only 1 of 100 LLM-assigned labels, then fixed the taxonomy for labeling remaining failures.Source: F. Failure Mode Analysis · p. 31

F.1. User Data Leakage

  • Agent failures include cross-account disclosure, unnecessary human transfer, and missed or ignored task-relevant information.Examples include revealing another account’s order after an identity switch, transferring instead of resolving a request, and repeatedly requesting a passenger’s birth date despite its presence in retrieved profile data.Source: §F.1 · p. 31, §F.2 · p. 32, §F.3 · pp. 33–34

F.4. Wrong Action Parameter

  • Other agent failures include using the wrong action parameter, executing policy-forbidden actions, and refusing authentication despite sufficient credentials.The categories cover selecting the wrong replacement item, acting on an unverified claim that an airline cancelled a flight, and failing to attempt a lookup when the user ID and other supplied information suffice.Source: §F.4 · p. 34, §F.5 · p. 35, §F.6 · p. 35

F.7. Failure to Provide Correct Options

  • Agents also fail by withholding available choices or acting before required follow-up steps are resolved.The analysis defines undisclosed or misstated choice sets and irreversible writes that change system state and block completion of a remaining task.Source: §F.7 · pp. 35–36, §F.8 · p. 36

F.9. Unrequested Action

  • Unrequested actions and incorrect policy guidance can create unnecessary state changes or steer users away from an allowed resolution.The categories include making an unrelated booking as a workaround and incorrectly claiming that a qualifying exchange is unavailable, leading the user toward a return.Source: §F.9 · p. 37, §F.10 · p. 37

F.11. Premature User Stop

  • Simulator failures include stopping before the agent can complete a task and abandoning the assigned identity or task.A premature stop can prevent execution of an already confirmed action, while identity or task derailment includes unrelated requests that prevent task completion.Source: §F.11 · pp. 37–38, §F.12 · p. 38

F.13. Other Simulator Error

  • The taxonomy also distinguishes other simulator faults that prevent success without premature stopping or wholesale identity or task derailment.Examples in the category definition include inventing constraints, contradicting task information, choosing an option inconsistent with the assigned user, or refusing a required confirmation.Source: §F.13 · p. 38
  • The simulator caused failure by requesting four bags instead of three, omitting a passenger change, and supplying incorrect reservation information that the agent followed.Source: §F.13 · p. 39

airline_17_ann3

  • In airline_17_ann3, the transcript reached the conversation cap before graded resolution despite consistent agent behavior, so the failure was attributed to the harness.Source: airline_17_ann3 · p. 39

F.15. Unexecuted Action After Confirmation

  • The agent failed to execute authorized address updates, continuing to request redundant confirmations after the user supplied the remaining fields.Source: §F.15 · p. 40

F.16. Incorrect Payment Method or Amount

  • The agent routed multiple cabin-downgrade refunds to a user-selected payment method instead of each reservation’s original method, producing incorrect payment_id writes.Source: §F.16 · p. 40
  • The analysis pools tagged failures across seeds, excludes uncategorized labels, and reports conditional user-side shares and their absolute gap from human attribution; Table A13 gives a human reference share of 0.21.User share includes environment failures in its denominator, while agent-only TVD excludes user-side and environment failures; the gap is on the proportion scale and does not measure unconditional user-error incidence.Source: §F.16 · p. 40, Table A13 · p. 41

G. Fidelity Outside of Task-Oriented Dialogue

Out-of-domain fidelity metrics behave differently across task-focused and conversational settings, and stronger simulator bases generally preserve CUE’s naturalness. A selected retail replay also shows CUE reproducing a human’s specific constraint-loss failure.

G. Fidelity Outside of Task-Oriented Dialogue

  • On SimulatorArena, CUE’s sampled S2R-N rises from 0.27 →0.47 →0.80 as the simulator base strengthens, while TT falls from 0.21 →0.15 →0.09.On PRISM, S2R-N is 0.70 →0.67 →0.89 and TT is 0.19 →0.13 →0.12 across the same bases.Source: G. Fidelity Outside of Task-Oriented… · p. 41, Table A14 · p. 42
  • PT3 is uniformly low on SimulatorArena (0.01–0.15) versus PRISM (0.21–0.51), which the authors attribute to limited user-specific signal in task-focused interactions.A simulator can have lower observed outcome error even when its trajectory provides too little identifying signal to match an individual user.Source: G. Fidelity Outside of Task-Oriented… · p. 41
  • On SimulatorArena, CUE scores below RealUserSim on AVA at every base (0.44/0.33/0.42 versus 0.50/0.43/0.51), despite matching or exceeding it on PT3.The paired-versus-shuffled ablation changes AVA only from 0.27 to 0.20, motivating greater weight on PT3 for paired-session fidelity.Source: G. Fidelity Outside of Task-Oriented… · pp. 41–42
  • S2R-C separates methods on task-like SimulatorArena (CUE 0.83–0.90; UserLM 0.84) but saturates on PRISM, where all methods score 0.86–0.96.The underlying behavioral taxonomy comes from task-oriented interactions, making it less discriminative on open-ended conversation.Source: G. Fidelity Outside of Task-Oriented… · p. 42
  • On PRISM, SD-C’s narrow 0.68–0.77 range obscures finer differences: at radius 0.05, 67%–81% of sampled users have at least one real-user neighbor.Bidirectional nearest-neighbor aggregation can reward broad coverage even when few simulated users closely match individual human neighborhoods.Source: G. Fidelity Outside of Task-Oriented… · pp. 42–43
  • USP’s strongest S2R-N scores occur on PRISM, whose open-ended conversations most resemble its LMSYS-Chat training data, so they do not establish domain-independent generalization.USP scores 0.26/0.27 on τ²-Bench, 0.41/0.38 on SimulatorArena, and 0.82/0.82 on PRISM.Source: G. Fidelity Outside of Task-Oriented… · p. 43

H. Single-Task Case Study: Reproducing a Human Interaction’s Constraint Loss

  • In the selected retail replay, CUE executes the same incorrect catalog-wide replacement as the human, while the no-persona baseline chooses the correct within-order replacement.The comparison concerns the concrete tool action, not merely matching the binary failure outcome.Source: H. Single-Task Case Study: Reproducing… · p. 44

H.1. Task and human reference

The retail task required exchanging an IPX7 earbud for the cheapest IPX4 earbud already in the order. In a selected replay, CUE reproduced the human’s specific constraint loss, while other trajectories varied in whether they recovered the restriction or failed before an exchange.

H.1. Task and human reference

  • The task restricts the replacement to earbuds already in the order: item 1646531091 costs $232.49, while the cheaper $226.49 catalog variant is outside the order.Choosing the catalog-wide cheaper item satisfies the water-resistance preference but violates the within-order constraint.Source: §H.1 · p. 44

H.2. Complete logged CUE instruction

  • The saved CUE command contains conversational and user-response behaviors but no earbuds, item identifiers, water resistance, or within-order instruction.The logged command includes style sketches and user-specific response guidance; task instructions are supplied separately.Source: Complete CUE description for… · p. 45, §H.2 · p. 45

H.3. Simulated interaction and executed outcomes

  • CUE reproduces the human’s catalog-wide incorrect exchange, while None selects the required within-order item; across three GPT seeds, CUE succeeds twice and None succeeds all three times.Both the human and CUE execute the same incorrect exchange in the selected interaction.Source: §H.1 · p. 44, §H.3 · p. 46

H.4. Additional baseline trajectories on the same episode

  • UserLM and USP runs fail before an exchange, receiving reward 0; these failures do not reproduce the human’s specific incorrect exchange.The baseline comparison includes saved closed-loop interactions with their own agent histories.Source: §H.4 · pp. 47–48
  • PPol initially follows the catalog-wide choice, then restores the within-order restriction and executes the correct exchange, but its persona record describes an airline-refund customer.That mismatch prevents treating the trajectory as a verified comparison under task-appropriate persona conditioning.Source: §H.4 · p. 47

Related papers

Loading 2610.02460v1…