On Evaluating and Improving Conversational Agents in Production

1 Introduction

• A modified response makes later customer turns in a production log invalid, while stochastic agents, changing products, prices, and personalization signals cause run-to-run variation.Aggregate measures combine distinct behaviors and do not indicate what failed or how to correct it.Source: §1 · pp. 1–2
• The framework guided production investigations of an assistant serving more than twenty European markets and millions of monthly conversations, and changes to the deployed system resulted.Assertion profiles located carousel failures at particular positions, while audits found defects in the evaluation itself.Source: §1 · pp. 3–4
• The harness begins from a reported problem or low-quality interactions flagged by automated assessments, after which a human or the harness identifies a behavior to investigate.Assessments can rank candidate behaviors for a human to choose from.Source: §1 · p. 3
• The simulator generates each customer turn in response to the system under test, conditioned on recorded customer messages and context, rather than replaying the logged conversation.Context can include personalization signals such as preferred sizes and brands.Source: §1 · p. 4
• Assertion results support hypotheses about system changes, and a modification that regresses another targeted behavior or guardrail is not advanced.A favorable result is evaluated again before human review and a possible online A/B test.Source: §1 · p. 4
• The framework commissions behavior-specific assertions and customer scenarios, then uses grounded simulation and paired repeated comparisons against a stored baseline to test isolated modifications.The unchanged system is run repeatedly to capture its run-to-run variation; the harness may be revised for future investigations only with human approval.Source: §1 · pp. 3–4

2 Related Work

• Conversational search makes fixed test collections difficult because later requests depend on preceding turns; user simulation instead lets each system generate its own conversation [7].Prior simulation work covers conversational recommendation, query refinement, and feedback generation.Source: §2 · p. 4
• Unlike domain-wide evaluation sets, this framework builds a separate cohort and assertion set for each behavior, grounds simulation in production conversations, and compares versions with paired repeated runs.Raju et al. construct a single domain-covering evaluation set by clustering prompts and sampling across clusters.Source: §2 · p. 5
• Behavior-specific assertions build on behavioral testing and assertion-generation work, but here they form diagnostic profiles for whole simulated conversations alongside grounded simulation and repeated paired comparison.Related approaches include CheckList, AdaTest, EvalLM, SPADE, and Eval-Gen.Source: §2 · p. 5
• Prior work on failure attribution and prompt optimization informs the Orchestrator, while this harness revises only after an investigation, with human approval and recorded decisions unchanged.Using the same examples for improvement and assessment can overstate progress in harness-evolution methods.Source: §2 · p. 5

3 System Under Test

• Production conversations are mainly product searches, followed by style advice and product questions; about one in ten concerns an order, and stated requirements are led by category.Topics vary by season, and the reported investigations examine category and gender of the shopping context.Source: §3 · pp. 6–7
• An LLM router selects a domain vertical after guards process the request; the fashion vertical handles shopping tasks, while customer care retrieves help content and queries order systems.The fashion vertical may start speculatively while routing continues, lowering latency at the cost of wasted work.Source: Figure 2 · p. 6, §3 · p. 7
• Two guards block malicious or unrelated requests and moderate generated answers; the paper reports that observed malicious probes had not succeeded.Refusal of a message class can be evaluated with an assertion like other system behaviors.Source: §3 · p. 7
• Conversation history, structured search state, request context, and consented customer memory supply information to the fashion agent, while responses can include text, carousels, cards, or comparisons.Customers can also send images for visual retrieval, and configuration determines which models and optional components are active.Source: §3 · pp. 7–8
• Product search generates one to three queries and structured filters, retrieves and personalizes products, then deduplicates and curates them into a carousel.If curation returns too few products, remaining positions are filled from the pre-curation ranking without a further check.Source: §3 · p. 8
• The system's many configurable LLM calls, scale, varied requests, and changing assortment make a single benchmark or comparison insufficient to characterize behavior.Daily changes in products, prices, stock, and personalization signals motivate stored baselines and paired repeated runs.Source: §3 · p. 8

4 Commissioning the Evaluation

• Local reproduction must make the reported behavior appear often enough for comparison; repeated unchanged-system runs establish the stored baseline, and failure to reproduce ends the investigation.When source conversations exist, the local failure rate should also be similar to the reported rate.Source: §4 · pp. 8–9

4.1 Cohort Construction

• The cohort is a fixed scenario set drawn from reconstructable production conversations or constructed goals, and its sampling strategy determines what results can show.Except under random sampling, cohort failure rates describe the selected scenarios, not production prevalence.Source: §4.1 · p. 9

4.2 Behavioral Assertions

• Assertions encode pass/fail conditions, while guardrails protect behavior that should remain unchanged; profiles can reveal whether a carousel failure affects all products or particular positions.A human reviews generated assertions before they are frozen, and the judge’s evidence fields are fixed before modified versions are evaluated.Source: §4.2 · pp. 9–10

4.3 Grounded User Simulation

• Grounded simulation generates customer turns in response to each evaluated system rather than replaying logged continuations, using recorded messages or a customer goal as its starting point.Validation found that simulated sessions reproduced reported failures across cohorts spanning languages, customer contexts, and interaction settings.Source: §4.3 · p. 10

4.4 Paired Decision Rule

• Paired comparisons average repeated sessions within each scenario, subtract baseline from modified scores, and bootstrap scenario-level differences; repeated sessions improve precision but are not independent observations.The frozen decision rule uses 2,000 percentile-bootstrap resamples and requires a target signal+ with no guardrail signal- in a separate second evaluation before advancement.Source: §4.4 · p. 11

5.1 Hypothesis-Driven Improvement

• The orchestrator forms diverse failure-mechanism hypotheses and tests isolated modifications against frozen assertions, using detailed results to refine, extend, or abandon each hypothesis.A human-set autonomy budget limits comparisons, and reproducible improvements are evaluated again before human review and possible online testing.Source: §5.1 · p. 12

5.2 Post-Investigation Harness Revision

• Post-investigation reflection may revise scenario selection, customer simulation, or result interpretation, but approved changes create a new harness version for future investigations only.Production investigations led to paired scenario-level intervals after pooled intervals obscured small improvements, as well as revisions following sampling and judge audits.Source: §5.2 · pp. 12–13

6 Discussion and Lessons Learned

• In the gender-context investigation, h2’s improvement reproduced in three further runs, and the guardrail assertion passed in every run.The cohort contained nine gender-silent requests and one women’s-context guardrail scenario.Source: §6 · p. 13
• Targeted assertions make evaluations actionable by testing a selected behavior from several angles and strictness levels, unlike broad quality scores that offer little guidance on what to change.Their profiles can also check afterward whether a modification affected the assertions it was predicted to change.Source: §6 · pp. 14–15
• The category change improved first-product and any-product assertions, while the 75% and all-products assertions changed little, indicating its effect was concentrated in prominent carousel positions.The category cohort was selected to show the failure, so its pass rates do not estimate overall quality or production frequency.Source: §6 · pp. 13, 15, Table 1 · p. 14
• Comparing each scenario with itself separates repeated-run variation from differences in scenario difficulty, which pooled sessions would count as uncertainty.The cohort approach also stratifies by customer intent because embedding distance reflected vocabulary, product category, and language more readily than intent.Source: §6 · p. 15
• Plausible judge outputs can conceal decision-changing defects in evidence inputs or applied settings, so judge inputs, executed configuration, and statistical procedures require direct tests.The audit also found a separate benchmark report that measured between-scenario differences instead of repeated-run variation.Source: §6 · p. 15
• Frozen cohorts and criteria can become outdated as systems, products, or scenarios change, while safeguards for sequential hypotheses lower—but do not bound—the risk of false signals.The authors describe anytime-valid inference with e-values and report a post hoc betting interval whose lower end was 0.00–0.01, versus 0.067 for the percentile bootstrap; at eighteen scenarios it rose to 0.08.Source: §6 · pp. 15–16

• Offline improvements do not establish customer-satisfaction or business-outcome gains in live traffic; those require controlled A/B tests.Source: §7 · p. 17
• The simulator was checked against source conversations, but its match to real customers across many intents and scenarios was not measured.Source: §7 · p. 17
• Paired bootstrap intervals treat the judge and production environment as fixed, excluding differences between judge configurations from reported uncertainty.Source: §7 · p. 17
• Multiple assertions and hypotheses were tested without formal multiplicity correction; safeguards reduce but do not bound false discoveries, and the anytime-valid analysis was post hoc.Source: §7 · p. 17
• With nine paired scenarios, the gender case had limited power, and percentile bootstrap intervals under-cover at that sample size.Source: §7 · p. 17

8 Conclusion

• Logs cannot be replayed after modifications, and unchanged agents vary as their LLM components and environments change.Source: §8 · p. 17
• Each investigation freezes behavior-specific assertions and a scenario cohort, then uses grounded simulation and paired repeated runs against a stored baseline.Source: §8 · p. 17
• One reported modification was deployed, while another was confirmed offline; assertion profiles localized failures and repeated measurements separated improvements from fluctuations.Source: §8 · p. 17
• Auditing the judge uncovered defects hidden by plausible-looking scores, underscoring that evaluation itself must be tested as carefully as the system.Source: §8 · p. 17
• The harness can learn from completed investigations under human approval without changing their recorded decisions; broader application depends on simulating users closely enough to reproduce observed behaviors.Source: §8 · p. 17