Inquesto Score: A reliability Protocol For Voice Agents

1. INTRODUCTION

• Transcript-only evaluation can miss talk-over, delayed responses, lost corrections, and acoustic failures that affect whether a consequential interaction succeeds.Some verifier cues can be obscured by narrowband telephone audio.Source: §1 · p. 1
• Existing end-to-end benchmarks leave open what a single reported score should mean, while composite metrics can conceal severe functional failures.The passage notes that evaluations can also be internally consistent while measuring the wrong thing.Source: §1 · p. 1
• Inquesto Score measures the percentage of protocol-defined calls that reach the caller’s goal without a functional failure or worse.The protocol evaluates the deployed pipeline and checks each failure type against evidence suited to it.Source: §1 · pp. 1–2
• The protocol uses a versioned fixed-population pass rate, explicit failure severities, audio-based timing checks, and separate diagnostic slices.The diagnostic views cover behavior, acoustic robustness, identity handling, and speaker groups without being combined into the score.Source: §1 · p. 2

2. THE INQUESTO PROTOCOL V0.1

• Inquesto is instantiated by supplying use-case scenarios and predicates while population construction, scoring, failure semantics, evidence requirements, and reporting remain fixed.The paper uses billing support because it combines task completion, conversational, timing, and authorization failures in one controlled setting.Source: §2 · p. 2

2.1. Evaluation Population

• The billing-support population has 30 scenarios and 306 calls per agent across three acoustic conditions and four speaker groups.It includes goal-directed and identity scenarios, with each call weighted one; scores are comparable only under the same protocol version and population.Source: §2.1 · p. 2

2.2. Clean Success and Severity

• Each call receives a goal predicate and severity-rated failure events; in v0.1, S3 is the lowest severity that makes a call fail.The goal is state-first: required actions must appear in tool traces, and impostor success requires no changes or disclosures.Source: §2.2 · p. 2

2.3. Failure Taxonomy

• Failure events use evidence suited to their type: timing is measured from audio timelines, trace events from tool calls, and judged events by a pinned open model.Table 1 defines the v0.1 events and observable criteria; the S3-and-above threshold determines call failure.Source: §2.3 · p. 2, Table 1 · p. 2

2.4. Diagnostic Views

• Four diagnostic views report pass rates on defined slices with their own intervals and sample sizes, without combining them into the score.They cover behavior, degraded-condition robustness, identity handling, and the worst speaker group.Source: §2.4 · pp. 2–3

2.5. Identity Evaluation and Reporting

• The identity view evaluates how the agent uses an authentication signal, not the verifier’s intrinsic accuracy.The verifier threshold is calibrated on the v0.1 population; scores are reported with 95% Wilson intervals, sample sizes, and a versioned run record.Source: §2.5 · p. 3

3.1. Pipeline

• The open audio pipeline synthesizes caller turns, applies acoustic conditions, segments speech using the endpointing window, transcribes it, and measures reply start time from synthesized speech.This makes latency and talk-over properties of real audio rather than transcript inferences.Source: §3.1 · p. 3

3.2. Agents as instruments

• Thirteen configurations vary the language model and endpointing window while keeping recognition, synthesis, caller simulation, population, taxonomy, and judge identical.The controlled setup tests the scoring protocol, not the agents; differences between configurations reflect agent behavior and timing.Source: §3.2 · p. 3

4.2. Deployment Parameters Change Reliability

• Qwen2.5-7B scores rose from 24 at 400 ms to 39 at 700 ms and 45 at 1100 ms as endpointing changed.With the same Llama-3.1-8B weights, local versus hosted serving scored 36 versus 20.Source: §4.2 · p. 3

4.3. Diagnostic Views Expose Hidden Differences

• The worst speaker group scored as much as 15.3 points below the reference voice, while identity-handling scores ranged from 0 to 44.Combining diagnostic views would shift agents’ scores, so v0.1 keeps them separate.Source: §4.3 · p. 3

4.1. Audio Reveals Failures Transcript Evaluation Misses

• 33% of failed calls had only audio-derived failures; transcript-only scoring passed them and averaged 24 points higher.Claude-Haiku-4.5h / 700 ms scored 66 with audio events removed but 26 on the phone.Source: §4.1 · p. 3, Table 2 · p. 4

4.4. Threshold Sensitivity and the Judge

• 12.8 points lower on average was the score when clean success required freedom from S2 events; S4- and S5-only thresholds raised it by 24.5 and 25.0 points.Configuration ordering remained stable across thresholds.Source: §4.4 · p. 3, Table 3 · p. 4
• 82% agreement on goal decisions and 91% on wrong-information decisions fell to 68% on context-loss decisions across the two judges.This variability motivates pinning one judge and rubric per protocol version.Source: §4.4 · pp. 3–4
• A judge-free score ranked configurations almost identically (Spearman ρ = 0.90) but sat 12 points higher.The judge adds semantic failures that state predicates, audio events, and traces alone cannot detect.Source: §4.4 · p. 4

• Inquesto Score reports an interpretable rate over a fixed population, with explicit success and failure definitions and diagnostics kept outside the score.The protocol measures interaction failures from audio.Source: §5 · p. 4
• Version 0.1 covers one domain, synthesized callers, one judge, and one verifier; scores are comparable only within the same protocol version.The score is not a real-world success probability.Source: §5 · p. 4
• The authors identify hosted-deployment scope, judge variability, and the single-domain synthesized-caller population as limitations shaping v0.2.Planned extensions include a second domain, human callers, a verifier-cloning attack, and a human-calibrated rubric.Source: §5 · p. 4