Source-linked AI summary
Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents
Kaiwen Luo, Ming Gao
TL;DR
Multi-turn business agents require evaluation beyond fixed outputs because correctness depends on business-specific facts and procedures, and outcomes emerge across conversations. The paper integrates specification, modular judges, intent-preserving simulation, and audited human feedback; production studies report improving system-level fidelity and strongest descriptive fidelity for the combined workflow in both reported task-completion settings, while remaining observational.
Problem
Method
The methodology connects conversation-level specification, modular judges with shared versioned evidence and explicit aggregation, intent-preserving user simulation, and audited human feedback.
Source: §1.2 · p. 2, §5 · p. 5, §6.2 · p. 8Results
Production studies report improved system-level NDR across repeated audits and strongest descriptive fidelity for the combined human–judge workflow in both reported task-completion settings.
Source: §9 · p. 12Takeaways & Limitations
The findings support treating LLM-based evaluation as a maintained measurement system whose fidelity must be demonstrated as the agent, evidence, and traffic evolve.
Source: §9 · p. 12Human-reference labels were revised during disagreement review, and reported human–judge and co-pilot differences lack confidence intervals or paired significance tests.
Source: §9 · p. 12
Abstract
from arXiv · showhide
Many LLM-as-a-judge evaluations score fixed outputs under a fixed task definition. Production multi-turn business agents instead require a maintained measurement system: correctness depends on business-specific facts and procedures, outcomes emerge across turns, and failures must be attributed to either agent capability or missing business knowledge before they are actionable. We present an integrated methodology spanning evaluation specification, modular LLM judges, intent-preserving user simulation, and human-in-the-loop governance. The specification defines conversation-level end states and actionable failure ownership. Atomic judges share versioned evidence and feed an explicit aggregation graph. The simulator is released only after task-preservation and stability checks. Independent human audits estimate measurement fidelity, renew tiered reference sets, and route disagreements to label correction, guideline revision, or judge improvement. Production studies show that system-level fidelity improved across repeated audits, that human reviewers and automated judges improved together under the shared feedback loop, and that their combined workflow had the strongest descriptive performance in both reported task-completion settings. Because the studies are observational and the human reference itself required revision, these findings demonstrate operational usefulness rather than causal or universal superiority. The contribution is a practical framework for making multi-turn agent measurement reliable, actionable, and maintainable as the evaluated system and its evidence evolve.
1 Introduction
Multi-turn business-agent evaluation requires a maintained measurement system because correctness depends on business-specific evidence and procedures, while outcomes and failure ownership emerge across conversations. The paper connects specification, modular judging, user simulation, and audited human feedback into an end-to-end methodology.
1 Introduction
- Business-agent evaluation must handle instance-specific facts, merchant procedures, and conversation-level outcomes rather than score fixed outputs against one task definition.Failures also need to distinguish model shortcomings from missing business data.Source: §1 · pp. 1–2
- Human review and direct user feedback are insufficient as standalone measurement sources: feedback can be sparse, and human-review fidelity also requires independent auditing.The introduction positions audited references as part of the measurement process.Source: §1 · p. 2
1.1 Proposed measurement-system methodology
- The components are coupled through shared interfaces: disagreements can update specifications and labels, while audited human references provide a system-level fidelity signal.The same judge implementation runs offline and online, and simulator fidelity is evaluated separately from agent quality.Source: §1.1 · p. 2
1.2 Contributions
- The proposed lifecycle connects evaluation specification, judge development, intent-preserving user simulation, and human-in-the-loop evaluation and improvement.The contribution is the lifecycle and its cross-component constraints, not an individual judge technique.Source: §1.2 · p. 2, §1.1 · p. 2
- Production studies examine longitudinal measurement-fidelity improvement and complementarity between human raters and automated judges.The introduction frames these as empirical studies of the proposed lifecycle.Source: §1.2 · p. 3
2 Problem formulation
The problem formulation identifies four properties that make generic judge recipes insufficient for multi-turn business agents: business-specific truth, procedure adherence, actionable failure attribution, and conversation-scoped defects. These requirements shape the metric taxonomy and subsequent methodology.
2 Problem formulation
- Business-specific correctness has no single golden answer: equivalent responses can be valid, while the same statement can be correct for one business and false for another.Reference-based metrics and cross-instance rubrics are insufficient on their own.Source: §2 · p. 3
- Quality includes following third-party, business-authored procedures such as ordered information-gathering and authentication rules.Judgment requires applying the relevant business document to each conversation, and this is a frequent source of human–judge disagreement.Source: §2 · p. 3
- Failures must distinguish model-capability gaps from knowledge-coverage gaps because they have different owners and remediations.The distinction requires an explicit aggregation layer, and exclusive attribution is difficult to measure because it is a conjunction with a negation.Source: §2 · p. 3
- Conversation-scoped labels are needed for cross-turn state defects such as repeated questions, contradicted commitments, and failure to respond to customer repetition.These failures are not visible in any single turn, and turn-level rubrics require a counterfactual judgment about possible harm.Source: §2 · p. 3
3 Related work
The related work establishes that judge decomposition, multi-turn evaluation, user simulation, and human-in-the-loop alignment are existing research areas. The paper positions its contribution as governing their integration and maintenance for business-agent measurement.
3.1 Prior work by measurement-system function
- Prior work covers decomposed rubrics, explicit aggregation choices, multi-turn evaluation, simulated users, and human-in-the-loop judge alignment.The paper claims novelty neither for these individual techniques nor for conversation-level evaluation or user simulation alone.Source: §3.1 · p. 3, §3 · p. 3, §3.2 · p. 4
- The paper frames its distinction from prior work as a lifecycle in which task preservation, information boundaries, and simulator stability are release criteria.These criteria qualify the use of user simulation within measurement, rather than claiming simulation itself as new.Source: §3.1 · p. 4
3.2 From judge methods to measurement systems
- The claimed contribution integrates per-business facts and procedures, conversation-level outcomes, failure attribution, simulator validation, composite-node fidelity, and recurring human audit as one governed system.The authors explicitly disclaim being first to introduce a decomposed judge, evaluation graph, simulator, or continuously maintained evaluator.Source: §3.2 · p. 4
4 Evaluation specification
The specification evaluates complete conversations, assigns failures to actionable owners, and measures label fidelity against an independent audit rather than relying on agreement alone. Net detection rate (NDR) is the system-level fidelity objective, with its interpretation depending on stable error rates.
4.1 From evaluation requirements to metric definition
- The specification separates model-capability gaps from missing business knowledge and grounds correctness and procedure adherence in each business’s own evidence.A hierarchy retains finer root causes beneath topline success or failure rates.Source: §4.1 · p. 4
- Conversation-level evaluation asks whether valid user tasks were completed and, if not, what prevented completion, assigning each conversation to one of seven mutually exclusive end states.Invalid or unfinished interactions handled well are distinguished from agent failures.Source: §4.1 · p. 4
4.2 Reference-label reliability and measurement fidelity
- Agreement is a reliability diagnostic, not correctness: fidelity is measured against an independently audited reference because agreement can fall while audited label quality improves.Precision, recall, F1, and inter-rater agreement remain guardrails rather than substitutes for NDR.Source: §4.2 · p. 5
- NDR is TPR − FPR, ranges from −1 to 1, and distinguishes perfect measurement, no information beyond chance, and reversed signal direction.The positive class is defect presence for gap metrics and successful completion or resolution for success metrics.Source: §4.2 · p. 5
- NDR preserves more underlying metric movement when error rates are stable, supports tighter corrected-estimate intervals, and does not change mechanically with defect prevalence.Its attenuation interpretation depends on stable error rates, so it must be re-estimated after changes to the product, traffic, guidelines, reference process, judge prompt, or foundation model.Source: §4.2 · p. 5
5 LLM-as-a-judge architecture
The judge architecture implements the specification through independent atomic judges, shared versioned evidence, and a separately versioned aggregation graph. This separation lets metric and attribution definitions change without silently changing judge behavior.
5 LLM-as-a-judge architecture
- The architecture integrates modular judges, governed inputs, and explicit aggregation logic to implement the evaluation specification.It focuses on integrating established decomposition into a versioned measurement architecture.Source: §5 · p. 5
- Independent judges assess dimensions in parallel, while a separately versioned aggregation layer changes metric definitions without silently changing judge behavior.All judge evidence resolves through one versioned context store, avoiding different factual bases for the same conversation.Source: §5 · p. 5
- Composite dimensions can be decomposed into binary judgments, with missing-data detection first summarizing discrete user asks and excluding greeting turns.The registry is shared across agents, with a scoped subset invoked for each conversation.Source: §5 · p. 5
- For valid conversations, parallel branches judge agent capability, business information, and outcomes after gates assess intent clarity, understanding, and handling of invalid or spammy interactions.Capability checks include hallucination, action accuracy, procedure following, and tool or retrieval failures; business-information checks distinguish conflicting from missing data.Source: §5 · p. 6, Figure 1 · p. 6
- Boolean operators and explicit precedence rules map overlapping atomic evidence to conversation-level outcomes, making reported metrics specific to the graph version.Changing a judge, operator, or overwrite order can change results on unchanged conversations.Source: §5 · p. 6
6 Simulation with an LLM-as-a-user
The paper treats simulated users as measurement components that must preserve tasks and remain stable, rather than as unrestricted role-play. It specifies two operating settings, fidelity checks, and release requirements while acknowledging that simulation is not equivalent to real users.
6 Simulation with an LLM-as-a-user
- LLM-as-a-user simulation adapts to changed agent trajectories, unlike fixed replay, and is intended to be specified and validated before evaluating the agent.Human role-play is slow and costly, while fixed responses can become incoherent when the agent changes its questions or answer order.Source: §6 · p. 6
6.1 Two LAAU operating settings
- Conversation-conditioned simulation preserves original intents for regression benchmarks, while instruction-conditioned simulation enacts specified tasks and personas, including cases absent from history.The simulator produces only user-visible utterances and ends after required intents have been exercised.Source: §6.1 · p. 7
6.2 Intent-preserving simulator design
- Conversation-conditioned LAAU must express every original intent without adding one; persona, tone, wording, and turn order are secondary fidelity targets.It answers only from curated information, minimally adapts context-dependent messages, and terminates after required intents are addressed.Source: §6.2 · p. 7
6.3 Simulator fidelity specification
- Simulator fidelity is evaluated independently through intent consistency, persona consistency, comprehensibility, and stability across repeated runs and downstream metrics.Naturalness preference over exact replay is only diagnostic; it cannot substitute for these fidelity criteria.Source: §6.3 · p. 8
6.4 Validation requirements and limitations
- Release requires audited evidence for intent preservation, comprehensibility, run-to-run stability, and downstream metric stability, but LAAU is not assumed equivalent to real users.Minimal rephrasing, ambiguous messages, and different handoff points can affect tone or downstream metrics without a changed simulated intent.Source: §6.4 · p. 8
7 Human-in-the-loop evaluation and improvement
The deployed feedback loop monitors fidelity and renews references while improving both human raters and judges. Independent review, escalation, and audits route disagreements to label correction, guideline revision, or judge improvement.
7 Human-in-the-loop evaluation and improvement
- The feedback system connects fidelity monitoring, tiered golden-set renewal, rater quality management, and judge benchmarking to maintain measurement after deployment.A fidelity breach can trigger root-cause analysis of both raters and judges rather than assigning fault in advance.Source: §7 · p. 8, Figure 2 · p. 9
7.1 Independent review and escalation
- Judges and initial human reviewers label independently; disagreements go to quality-stratified escalation, with top-annotator audits and an expert tier able to override decisions.Blind initial review mitigates anchoring, while stronger reviewers focus on disputed items.Source: §7.1 · p. 8
7.2 Reference maintenance and feedback
- Adjudicated disagreements route to rater correction, guideline revision, or judge improvement rather than presuming either labeler is correct.The workflow adds an explicit specification-revision path to annotation-error handling.Source: §7.2 · p. 9
- Audits renew tiered golden sets for rater testing and allocation, while a separate expert-aligned set supports judge-release regression testing.Because finite release sets cannot guarantee fidelity on changing live traffic, reference labels must be versioned and re-audited.Source: §7.2 · p. 9
8 Empirical Studies
The production studies examined whether measurement fidelity improved over repeated audits and how human and judge workflows aligned. They report gains in system-level fidelity and descriptive complementarity, while remaining observational.
8 Empirical Studies
- The combined review workflow had the strongest descriptive point estimate in both reported task-completion settings.The table does not establish statistical superiority because confidence intervals and paired significance tests were unavailable.Source: Table 2 · p. 10
8.1 Improvement in system-level measurement fidelity
- At the final reported audit, model-capability NDR was 0.550 and knowledge-coverage NDR was 0.686.The initial model-capability NDR was −0.022; the table reports −0.022 for model capability and 0.299 for knowledge coverage at the initial audit.Source: §8.1 · p. 10
- A judge panel in production for over a year had model-capability NDR of −0.022 at its first audit, despite acceptable component-level metrics.The authors argue that only the loop-level metric revealed that it was not detecting model-capability gaps better than chance.Source: §8.1 · p. 10
- At the final snapshot, the judge’s knowledge-coverage NDR was 0.686 versus 0.553 for the human co-pilot, an observed comparison rather than a general superiority claim.On model capability, the judge had better recall but worse precision; the human values are observed comparators, not theoretical ceilings.Source: §8.1 · p. 10
8.2 Alignment and complementarity with human raters
- Over four and a half months, judge NDR rose on all four dimensions, while human-reviewer NDR also rose by 0.11 to 0.16.Figure 3 tracks a single agent and locale; the shared feedback loop and concurrent changes prevent attributing the co-movement to either arm alone.Source: §8.2 · p. 10, Figure 3 · p. 11
- The feedback loop also improved human review: biweekly golden-set testing drove more than 70 reviewer reallocations, and breach investigations recovered recall on two products.L1-failure recall rose from 0.58 to 0.80 on one product; knowledge-gap recall rose from 0.24 to 0.81 on another, as incident recoveries rather than controlled effects.Source: §8.2 · p. 11
9 Discussion & Conclusion
The paper presents a connected measurement-system methodology and reports improved fidelity and human–judge performance in production studies. The evidence is observational and bounded in duration and scope.
9 Discussion & Conclusion
- The production studies support operational usefulness but do not isolate the causal effect of any individual design or workflow choice.They are observational, not a controlled comparison against an alternative methodology.Source: §9 · p. 12
- Figure 3 shows a period of coupled improvement followed by a plateau, not a sustained trend.In the ten weeks after the plotted interval, both arms were mostly flat; later human NDR became unmeasurable when labels stopped disagreeing with the audit reference.Source: §9 · p. 12
- The human reference was revised during disagreement review, and human–judge differences lack confidence intervals or paired significance tests.The human comparator values are observed values rather than ceilings; static-golden-set precision and recall are reported separately from live-traffic NDR.Source: §9 · p. 12
- Evidence is limited to one organisation and the reported agent, task, and locale samples; LAAU evidence uses internal benchmarks and traffic-derived samples.These constraints do not establish universal superiority over human evaluation or every alternative system design.Source: §9 · p. 12
- The methodology connects conversation-level specification, business-grounded atomic judges and aggregation, intent-preserving simulation, and an independently audited human feedback loop.The production studies report improved system-level NDR across audits, strongest descriptive combined-workflow fidelity in both reported task-completion settings, and improvement in both human and automated evaluation.Source: §9 · p. 12