PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes

Shivam Negi, Arpit Rawat, Rashi Jain

arXiv:2609.30798v1cs.AI

TL;DR

Real-time voice-agent research spans speech generation, turn-taking, and agentic evaluation, but its measures do not combine into one account of interaction quality. This survey organises 38 primary sources and argues that architecture remains a deployment choice, evaluation is moving toward grounded outcomes, and reporting should consider timing, recovery, and verified results together.

  • Problem

    Research on real-time voice agents is divided across architecture, turn-taking, and task evaluation, whose separate measures do not describe overall interaction quality.

    Source: §12 · p. 9

  • Method

    The survey organises 38 primary sources into six application-centric categories and structures retrieval and verification before synthesis.

    Source: §12 · pp. 9–10

  • Results

    The survey concludes that architecture remains an open deployment question and that evaluation has shifted toward grounded outcomes verified against backend state.

    Source: §8 · p. 5, §12 · p. 9

  • Takeaways & Limitations

    The survey proposes characterising real-time voice agents jointly by timing, post-disruption recovery, and grounded outcomes, with multiparty appropriateness where applicable.

    Source: §12 · p. 9

    The 38-source corpus is focused rather than exhaustive, with possible topical bias from the seed list, vocabulary bias from keyword search, and recency bias from follow-up selection.

    Source: §11 · p. 9

Abstract

from arXiv · show

Real-time voice agents have moved from research prototypes to production deployments, yet the literature describing them is fragmented across three communities that rarely cite one another: speech foundation modelling, turn-taking psycholinguistics, and agentic evaluation. Architecture papers report latency, turn-taking papers report prediction accuracy, and agentic benchmarks report task success, so no single number describes whether a deployed agent is actually good. We address that gap with three evidence-based claims, each traceable to a corpus of 38 primary sources organised into an application-centric taxonomy of six categories. First, architecture choice is a deployment constraint rather than a settled verdict: a 2026 enterprise tutorial reports that no fully self-hostable end-to-end system yet meets production constraints, while a chunked cascade independently reaches state-of-the-art duplex behaviour, showing duplex behaviour is separable from duplex architecture. Second, evaluation has shifted decisively from component quality toward grounded outcomes, with recent benchmarks verifying backend state rather than trusting what the agent claims to have done. Third, the dyadic assumption in most models and benchmarks is breaking down: multiparty turn-taking and multi-speaker reasoning benchmarks show that deciding when not to speak, and reasoning about who may be told what, are first-class capabilities two-participant framings cannot measure. For each source we state the problem it targets, its mechanism, and its reported evidence, alongside the search strategy, inclusion criteria, and a verification step that caught a misattributed arXiv identifier in circulation. We propose TRG (Timing-Recovery-Grounded), a reporting standard characterising an agent by timing, post-disruption recovery, and state-verified outcome together, with a conditional fourth axis for multiparty deployments.

1 Introduction

Real-time voice-agent research is split across speech generation, turn-taking, and task evaluation, leaving no single measure of overall interaction quality. This survey connects those concerns through a 38-source review, a six-category taxonomy, and a proposal to report timing, recovery, and grounded outcomes together.

1 Introduction

  • Pipeline systems separate voice activity detection, speech recognition, dialogue decisions, and speech synthesis, but their modularity comes with latency and limits on representing overlap and non-linguistic information.Conversation often involves simultaneous listening and speaking, turn projection, and contributions that begin before silence.Source: §1 · p. 1
  • Speech generation, turn-taking, and agentic evaluation measure different parts of interaction, so strong component quality does not establish that an agent completed an authorized task.The survey frames these as three largely parallel research communities.Source: §1 · p. 1
  • The survey advances three claims: architecture choice remains a deployment constraint, evaluation is shifting toward grounded outcomes, and dyadic assumptions are breaking down.Each claim is argued with its strongest supporting evidence and a serious complication.Source: §1 · p. 1
  • The review documents its search strategy, inclusion criteria, and programmatic verification step to keep its claims traceable to checked sources.It also positions its interactive-systems scope against surveys of Speech Language Models and direct speech-to-speech translation [3] [4].Source: §1 · pp. 1–2
  • The supporting literature is organised into six application-centric categories and compared by architecture class, evaluation axis, and reported metrics.The survey proposes TRG, reporting timing, recovery, and grounded outcomes jointly, with multiparty appropriateness where applicable.Source: §1 · p. 2

2 Survey Methodology

The survey constructs a reproducible corpus through systematic retrieval, screening, and PDF verification, then organises retained work by the question it answers. Its taxonomy separates speech production, turn-taking, and interaction evaluation rather than grouping papers by venue or date.

2 Survey Methodology

  • Candidates came from a 15-paper seed list and 20 papers found by systematic arXiv queries, then were ranked and screened by title and abstract.The themes included full-duplex dialogue, turn-taking, streaming synthesis, and named benchmark families.Source: §2 · p. 2
  • The review included work on interactive spoken interaction and excluded vendor comparisons, unavailable URLs, and an unpublished local document; retained items required an obtainable, verifiable PDF.Source: §2 · p. 2
  • PDF verification checked file structure and whether first-page text matched the recorded paper, catching a circulated misattribution of Full-Duplex-Bench-v3’s identifier.The correct identifier is arXiv:2604.04847, rather than arXiv:2602.05105; all 41 retained PDFs passed the check.Source: §2 · p. 2
  • The final corpus contains 38 primary sources, weighted toward 2024–2026 because the full-duplex literature is recent.It includes 9 architecture sources, 16 turn-taking and conversational or multi-speaker benchmark sources, 7 agentic and tool-use evaluation sources, and 6 synthesis and codec sources.Source: §2 · p. 2

3 Taxonomy

  • The taxonomy groups works by the question they answer—how speech is produced, when the system should speak, or whether interaction succeeded—rather than venue or chronology.These are presented as orthogonal concerns whose conflation contributes to apparent disagreement in the field.Source: §3 · p. 2

4 End-to-End Speech-to-Speech Architectures

End-to-end speech-to-speech systems remove text as an obligatory intermediate and can represent parallel speech streams, overlap, and interruption. The surveyed work also addresses timing, retained text bottlenecks, translation policy, and the scarcity of speaker-separated training data.

4 End-to-End Speech-to-Speech Architectures

  • Moshi models its speech and the user’s speech in parallel codec-token streams, removing explicit speaker turns so overlap and interruption can be represented natively [1].It became a reference point for subsequent full-duplex work and supplies the Mimi codec analysed by later papers.Source: §4 · p. 3
  • Synchronous LLMs integrate time information so speech units, including silence, are generated in lockstep with real-world clock time [2].The work identifies the absence of a sense of time in pretrained LLMs as an obstacle to synchrony.Source: §4 · p. 3
  • MOSS-Speech combines modality-based layer splitting with frozen pretraining to add native speech capability while retaining a pretrained text LLM’s reasoning [5].The authors report state-of-the-art spoken question answering and speech-to-speech performance comparable to text-guided systems.Source: §4 · p. 3
  • OmniFlatten uses progressive multi-stage post-training to flatten parallel speech and text streams, targeting interruptions, backchannels, and overlap without changing the underlying GPT architecture [6].Source: §4 · p. 3
  • Hibiki-Zero eliminates the need for word-level aligned training data in simultaneous translation, easing training and scaling across languages with different grammatical structures [7].Source: §4 · p. 3
  • Nakata et al. construct speaker-separated full-duplex dialogue speech at scale to address the scarcity of training audio where speakers are mixed on one channel [8].Source: §4 · p. 3

5 Cascaded and Hybrid Pipelines

The surveyed evidence challenges the view that cascades are obsolete: they remain a practical self-hosted option, and chunked cascades can achieve state-of-the-art duplex turn-taking. The adjacent turn-taking literature develops self-supervised prediction and extends it to real-time and multiparty settings.

5 Cascaded and Hybrid Pipelines

  • The surveyed cascade evidence challenges the legacy-technology framing and supports separating duplex behaviour from duplex architecture.Source: §5 · p. 3
  • Qiu et al. report that local Transformers deployment of Qwen3-Omni takes roughly 146 s for the full pipeline, while their cascade achieves 755 ms time-to-first-audio [9].They conclude that cascaded streaming remains the practical architecture for self-hosted real-time agents; the cloud-only Realtime API is approximately 702 ms audio-to-audio but not self-hostable.Source: §5 · p. 4
  • Chunk-wise microturns and conversational control tokens let DuplexCascade achieve state-of-the-art full-duplex turn-taking on Full-Duplex-Bench while retaining strong VoiceBench conversational intelligence [10].The system uses 50k multi-turn text dialogues and lightweight LoRA adaptation; its authors attribute duplex behaviour to the control policy rather than end-to-end audio modelling.Source: §5 · p. 4

6 Turn-Taking and Voice Activity Projection

  • VAP predicts both interlocutors’ joint future voice activity from audio without labelled turn-taking data, and is evaluated on turn-shift and backchannel prediction [12].Its authors argue that projection-window events should be modelled dependently rather than independently.Source: §6 · p. 4
  • Follow-up VAP studies test prosodic contribution, cross-language transfer, real-time continuous prediction, and multimodal inputs [13] [14] [15] [16].Together, these studies examine signal source, language, latency, and modality.Source: §6 · p. 4
  • MuVAP maps N-speaker interactions to a fixed current-versus-next floorholder state using face tracks and monaural audio, and contributes a 31-hour unedited single-camera conversation corpus [18].The approach addresses multiparty deployment settings that lack microphone arrays or multi-camera rigs.Source: §6 · p. 4

7 Conversational Benchmarks

Conversational benchmarks increasingly test behavior under realistic interaction conditions, extending evaluation from spoken output toward overlap, interruptions, multiple rounds, and multiple speakers. They also examine when an agent should speak and whether it respects speaker identity and disclosure boundaries.

Benchmark trajectory

  • Benchmarks are presented as a progression from scoring what agents say toward testing overlap, interruption, multiple rounds, and multiple speakers.The sequence is organized by how much conversational reality each benchmark admits.Source: §7 · p. 4

VoiceBench

  • VoiceBench evaluates real and synthetic spoken instructions across speaker characteristics, environmental variation, and content factors.It addresses prior evaluation centered on ASR accuracy or spoken QA synthesized with high-quality TTS.Source: §7 · pp. 4–5

The Full-Duplex-Bench family

  • Full-Duplex-Bench v1.5 adds overlap handling, while v2 adds multi-turn evaluation and later work finds performance can degrade unevenly across rounds and dimensions.The later benchmark segments continuous full-duplex audio into discrete turns for scoring and reports difficulty maintaining consistent performance across conversational features, dialogue quality, instruction following, and safety.Source: §7 · p. 5

Multi-speaker interaction

  • 1,152 test cases split evenly between Mandarin and English show that fluent speech understanding does not ensure correct reasoning about who is speaking.The benchmark tests multi-speaker memory, instruction-following, and reasoning, including who said what, to whom, and with what authority.Source: §7 · p. 5

FLEXI and MP-Bench

  • FLEXI tests whether agents interrupt users in emergencies and reports significant gaps between open-source and commercial models in emergency awareness, turn termination, and latency.MP-Bench evaluates whether an agent behaves appropriately among several participants, including whether it should speak at all.Source: §7 · p. 5

8 Agentic and Tool-Use Evaluation

Agentic and tool-use evaluation increasingly checks task outcomes and tool behavior rather than relying on conversational quality or the agent’s own claims. The section covers grounded verification, recovery after disruption, integrated duplex systems, and evaluation or training approaches that isolate speech-related tool-use challenges.

Grounded task completion

  • Grounded-task evaluation combines conversational dynamics with multi-turn task completion under real-world navigation complexity.Ray et al. identify the separation of conversational dynamics and task completion in existing evaluations as the gap addressed.Source: §8 · p. 5

Verifying against state

  • A benchmark grader checks binary assertions against complete traces for 100 scenarios, including tool calls, arguments, and returned rows, rather than trusting self-reported success.Each scenario seeds a PostgreSQL backend and simulated caller with a private goal; roughly one-third apply adversarial pressure. The grader can catch false claims of database changes and correct changes that disclose protected information.Source: §8 · p. 5

Disfluency and recovery

  • Closed-weight models degrade roughly 3.3× more slowly as conversations lengthen, while open-weight models lose ground on all three evaluated axes.Across 27 audio-language model configurations in 10 enterprise domains, the study also reports no audio-versus-text modality gap for closed-weight models and finds recovery quality to be a largely distinct capability axis.Source: §8 · p. 5

A convergent system

  • On Full-Duplex-Bench 1.0, NVIDIA’s system has the lowest pause false-takeover rate among tested open-weight systems and a 100% correct-takeover rate after genuine interruptions.It resumes after backchannels in 93% of cases on v1.5 and reaches 82.5% tool-selection F1 on FDB-v3, while argument accuracy and end-to-end tool execution remain weaker.Source: §8 · p. 5

Isolating the decision-maker

  • A cascade-focused benchmark isolates the language model’s reasoning under transcription noise and split utterances, separating it from recognition errors while retaining phone-call challenges.It targets a gap between end-to-end benchmarks that mix recognition and reasoning errors and LLM benchmarks that isolate the model but omit phone-call difficulty.Source: §8 · pp. 5–6

Training, not just measuring

  • Audio-native training uses two omni-modal models conversing in native audio without external ASR or TTS, and per-turn process rewards restore variance when outcome-only GRPO is gradient-starved.The paper attributes speech-related tool failures to perceptual errors, such as mishearing a value despite selecting the right tool and argument slot.Source: §8 · p. 6

9 Streaming Synthesis and Neural Codecs

Streaming synthesis research addresses responsiveness through latency measurement and methods that begin speech earlier, alter decoding, align text and audio, or sustain generation over long conversations. A codec study questions whether standard ABX evaluation captures linguistically meaningful properties of learned audio representations.

Responsiveness as a first-class metric

  • Tail latency, latency distributions, and intelligibility are evaluated across 13 open-source TTS models, with inputs ranging from single words to sentence-length utterances.The benchmark reports tail latency because users experience worst-case rather than only mean latency.Source: §9 · p. 6

Streaming synthesis

  • Four streaming approaches target when synthesis starts, how decoding proceeds, how text and audio align, and whether performance holds over a conversation.They respectively overlap speech generation with ongoing text generation, use autoregressive discrete audio codes, use CTC alignment in a dual-streaming architecture, and target sustained low-latency generation.Source: §9 · p. 6

What codecs encode

  • The standard ABX evaluation fails to capture aspects of Mimi’s 2048-token semantic codebook representation, whose codec uses 80 ms frames under real-time constraints.The study asks whether learned audio tokens partition the signal in linguistically meaningful ways and notes that end-to-end systems inherit what their codec preserves or discards.Source: §9 · p. 6

10 Discussion: Three Central Claims

The discussion argues that architecture trade-offs, grounded evaluation, and multiparty capabilities cannot be captured by one score. It proposes TRG to report timing, recovery, and grounded outcomes jointly, adding multiparty appropriateness when applicable.

10.1 Architecture choice is a deployment constraint, not a verdict

  • The corpus yields no clean architecture winner: Qiu et al. [9] did not meet production requirements, while DuplexCascade [10] achieved state-of-the-art duplex behaviour using a cascade.Together, the findings show duplex behaviour can be supplied as a control policy layered on a conventional pipeline, rather than requiring duplex architecture.Source: §10.1 · p. 6
  • The architecture trade-off remains inferred across separate studies; self-hosting and controllability favour cascades, while paralinguistic applications favour end-to-end modelling despite current hosting costs.No system in the corpus is evaluated on both paralinguistic content and duplex behaviour simultaneously.Source: §10.1 · pp. 6, 8

10.2 Evaluation has shifted, and should finish shifting, from claims to state

  • Recent evaluation verifies database rows [29], post-interruption workflow position [31], and tool-call correctness under disfluency [30], exposing failures component metrics cannot detect.Fan et al. [34] report that many speech-agent failures are perceptual, such as mishearing a value, rather than reasoning failures.Source: §10.2 · p. 8
  • Grounded evaluation is costly and unevenly adopted, so the paper recommends joint reporting while expecting cheaper component metrics to persist.VAmoS Bench required a seeded PostgreSQL backend and hand-designed adversarial goals for 100 scenarios [29].Source: §10.2 · p. 8

10.3 The dyadic assumption is breaking down, ahead of the models that assume it

  • MP-Bench [27], triadic VAP [17], and MuVAP [18] treat silence as a first-class behaviour, while MSI-Bench [25] tests authority-aware reasoning across 1,152 Mandarin and English cases.A fluent, correct answer can still disclose a parent’s private instruction to a child.Source: §10.3 · p. 8
  • The benchmarks expose a modelling gap: systems are primarily trained and evaluated on dyadic dialogue, with no corpus system tested at MSI-Bench’s scale for multiparty authority reasoning.For deployments with multiple legitimate speakers, the paper recommends specifying who may be told what alongside task success.Source: §10.3 · p. 8

10.4 TRG: a joint reporting standard follows from all three claims

  • TRG requires reporting latency under intended deployment conditions, post-disruption recovery separately from undisturbed performance, and task success verified against external state.The proposed axes are timing, recovery, and grounded outcome.Source: §10.4 · p. 8
  • For deployments with more than two legitimate speakers, TRG adds multiparty appropriateness; it prescribes required axes, not specific metrics.The paper keeps this fourth axis conditional because strictly dyadic systems do not exercise that capability.Source: §10.4 · p. 8

11 Limitations of This Review

The review states boundaries on corpus coverage, source status, cross-paper comparability, recency, and language access. These limits constrain how broadly its synthesis can be interpreted.

11 Limitations of This Review

  • The 38-source corpus is focused rather than exhaustive, and seed-list selection, keyword search, and a follow-up pass may bias coverage.Research using different terminology may be under-represented.Source: §11 · p. 9
  • Several 2026 entries have not completed peer review, and the review reports source claims without independently verifying their experimental results.The corpus is predominantly arXiv preprints.Source: §11 · p. 9
  • Reported latencies and accuracies use incompatible hardware, audio conditions, and prompt regimes, so Table 1 should not be read as a ranking.The review deliberately does not aggregate numbers across papers.Source: §11 · p. 9
  • Coverage is current only to the retrieval date; the Full-Duplex-Bench line released four versions in roughly eighteen months, requiring revision on a months—not years—timescale.Source: §11 · p. 9
  • Only English-language, machine-readable text sources were retained, excluding non-English work and scanned documents.Source: §11 · p. 9

12 Open Challenges and Conclusion

The review identifies unresolved evaluation and deployment challenges, including perceptual robustness, recovery, multiparty interaction, codec interpretation, and tail latency. It argues for joint reporting and verification of consequences rather than conversational self-report.

Open challenges

  • Fan et al. [34] find speech-agent failures are often perceptual: mishearing an argument value can cascade into a failed call.Text-domain evaluation cannot surface this failure mode.Source: §12 · p. 9
  • Salimi et al. [31] find post-interruption recovery largely independent of other measured axes, with the open-weight gap widening as conversations grow.Source: §12 · p. 9
  • Dyadic assumptions remain embedded in most models and datasets [18] [27], while deciding when not to speak remains under-measured.Source: §12 · p. 9
  • End-to-end systems inherit codec representational choices, but standard evaluation does not capture what those tokens encode [40].Source: §12 · p. 9
  • Dinh et al. argue responsiveness deserves first-class status, since mean latency hides worst-case experience and users notice.Source: §12 · p. 9

Future directions

  • Future work calls for joint reporting of timing, recovery, and grounded outcomes, audio-based training to address text/audio asymmetry, and controlled architecture comparisons.The architecture trade-off is currently inferred because cascaded and end-to-end systems are typically evaluated by different groups on different tasks.Source: §12 · p. 9

Conclusion

  • Meyer et al. [29] grade against database state because fluent self-report can describe an action the agent never performed.The conclusion argues that evaluation must verify consequences as agents gain the ability to act.Source: §12 · p. 9

Methodological note

  • The survey verified PDFs against first-page text after finding a circulated Full-Duplex-Bench-v3 identifier resolved to an unrelated simulator.It recommends structuring retrieval and verification before synthesis to reduce plausible but ungrounded claims.Source: §12 · pp. 9–10

Data Availability

The corpus is released as structured metadata with verification information and tools for rebuilding and auditing it. The papers themselves are not redistributed.

Data Availability

  • The release provides one structured metadata record per source, including its identifier, taxonomy category, verification status, and analyzed-PDF SHA-256 hash.Source: Data Availability · p. 10
  • A fetch script rebuilds a local corpus copy from original hosts and checks each hash, making the corpus independently auditable.Source: Data Availability · p. 10
  • The papers are not redistributed because a licence audit found only one of the 41 sources had an explicit open-licence grant.The release also includes a fillable TRG report template and compliance validator.Source: Data Availability · p. 10
Loading 2609.30798v1…