PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction

Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He

arXiv:2512.14865v1cs.SDcs.CLcs.LG

TL;DR

Realistic multi-turn spoken dialogue remains underexplored by benchmarks centered on synthetic speech and single-turn tasks. Audio MultiChallenge evaluates human-recorded conversations across four capabilities, and reports substantial gaps in current end-to-end systems. The evaluation also finds that performance shows no meaningful relationship with turn count under its fixed-context protocol.

  • Problem

    Existing evaluation emphasizes synthetic speech and single-turn tasks, leaving realistic multi-turn audio interaction underexplored.

    Source: §1 · p. 1

  • Method

    Audio MultiChallenge extends evaluation to human-recorded speech and four conversational capabilities, using multi-turn examples and rubric-based scoring.

    Source: §2 · p. 3, §3 · p. 6, §5 · p. 12

  • Results

    The evaluation finds substantial gaps in end-to-end capabilities, while turn count shows no meaningful relationship with performance under the fixed-context protocol.

    Source: §3.2 · p. 9, §5 · p. 12

  • Takeaways & Limitations

    The open-source benchmark provides a reproducible testbed for evaluating multi-turn spoken dialogue systems.

    Source: §5 · p. 12

    The evaluation does not operate directly on raw audio outputs.

    Source: §5 · p. 12

Abstract

from arXiv · show

End-to-end (E2E) spoken dialogue systems are increasingly replacing cascaded pipelines for voice-based human-AI interaction, processing raw audio directly without intermediate transcription. Existing benchmarks primarily evaluate these models on synthetic speech and single-turn tasks, leaving realistic multi-turn conversational ability underexplored. We introduce Audio MultiChallenge, an open-source benchmark to evaluate E2E spoken dialogue systems under natural multi-turn interaction patterns. Building on the text-based MultiChallenge framework, which evaluates Inference Memory, Instruction Retention, and Self Coherence, we introduce a new axis Voice Editing that tests robustness to mid-utterance speech repairs and backtracking. We further augment each axis to the audio modality, such as introducing Audio-Cue challenges for Inference Memory that require recalling ambient sounds and paralinguistic signals beyond semantic content. We curate 452 conversations from 47 speakers with 1,712 instance-specific rubrics through a hybrid audio-native agentic and human-in-the-loop pipeline that exposes model failures at scale while preserving natural disfluencies found in unscripted human speech. Our evaluation of proprietary and open-source models reveals that even frontier models struggle on our benchmark, with Gemini 3 Pro Preview (Thinking), our highest-performing model achieving a 54.65% pass rate. Error analysis shows that models fail most often on our new axes and that Self Coherence degrades with longer audio context. These failures reflect difficulty of tracking edits, audio cues, and long-range context in natural spoken dialogue. Audio MultiChallenge provides a reproducible testbed to quantify them and drive improvements in audio-native multi-turn interaction capability.

1. Introduction

End-to-end spoken dialogue systems process speech directly, but their evaluation needs to reflect extended, spontaneous interaction rather than isolated prompts or short synthetic inputs. Realistic multi-turn audio remains understudied, despite evidence that strong single-turn performance does not guarantee robust multi-turn behavior.

1. Introduction

  • End-to-end systems map speech directly to conversational responses, unlike cascaded pipelines that combine speech recognition, text-based language models, and speech synthesis.They may reduce latency and error propagation while better preserving paralinguistic and acoustic information.Source: §1 · p. 1
  • Realistic multi-turn audio remains understudied: existing benchmarks often use synthetic speech and single-turn tasks, while recent multi-turn efforts cover only short dialogues.Audio’s continuous, high-frame-rate token streams also consume context quickly, making long-horizon state tracking harder than in text.Source: §1 · p. 1

I. Mid-Utterance Edits II. Previous-Turn Edits

Audio MultiChallenge tests interaction patterns in natural speech, including retaining acoustic cues, maintaining consistency, and handling user edits. Evaluation exposes failures on these challenges, with self-coherence also declining on longer audio contexts.

I. Mid-Utterance Edits II. Previous-Turn Edits

  • Audio MultiChallenge targets four capabilities: Inference Memory, Instruction Retention, Self Coherence, and Voice Editing.Its tasks include recalling ambient sounds or paralinguistic signals, following conditional instructions, maintaining consistency, and resolving speech repairs.Source: Self Coherence · p. 2
  • The benchmark uses human-in-the-loop conversations generated from agent-produced blueprints, preserving unscripted speech and natural disfluencies.The blueprints are strategic guidance rather than verbatim scripts.Source: Self Coherence · p. 2
  • 1,712 atomic, instance-specific rubrics decompose ideal responses into sub-requirements; the LLM judge agrees with human evaluators 93% of the time.A task passes only when all corresponding rubrics are satisfied.Source: Self Coherence · p. 2

Self Coherence

  • 36.5% lower performance on Audio-Cue than semantic Inference Memory, while Self Coherence falls from 33.3% on shorter tasks to 20.0% on 3–5 minutes of audio.Voice Editing is the most challenging axis; mid-utterance self-corrections and prior-turn edits frequently trigger failures.Source: Self Coherence · p. 3

2. Audio MultiChallenge

Audio MultiChallenge extends multi-turn dialogue evaluation to human-recorded, non-scripted speech, which captures disfluencies, backtracking, interruptions, and acoustic cues that text-derived or synthetic benchmarks miss. It tests four conversational capabilities and uses a hybrid agentic and human-led pipeline to create and review conversations and rubrics.

2. Audio MultiChallenge

  • The benchmark evaluates Inference Memory, Instruction Retention, Self Coherence, and Voice Editing using human-recorded, non-scripted examples.Audio recordings preserve disfluencies, backtracking, third-party interruptions, accents, and paralinguistic cues that text-derived benchmarks do not capture.Source: §2 · p. 3

2.1 Axes

  • Inference Memory tests recall and synthesis of earlier details, including ambient sounds and paralinguistic signals that implicitly constrain a later response.Instruction Retention tests consistent application of stacked and conditional instructions; Self Coherence targets unwarranted contradictions of prior statements.Source: §2.1 · pp. 3–4
  • Voice Editing tests whether a model recognizes spontaneous speech repairs and applies the user’s final intent across mid-utterance or multi-turn edits.The axis addresses audible revisions in continuous speech, where retracted content must be filtered out.Source: §2.1 · p. 4

2.2 Dataset Curation

  • A multi-agent pipeline explores Audio LM failures and converts them into human-friendly blueprints, which guide human contributors in recording realistic conversations.Agents escalate conversation complexity until failure or a turn limit; contributors improvise from blueprints rather than following scripts.Source: §2.2 · pp. 4–5
  • Human contributors create atomic binary rubrics for the final response, limited to the assigned axis and final user turn.Decomposing ideal responses into discrete criteria supports LLM-as-a-judge verification of each criterion.Source: §2.2 · p. 5
  • Each task passes a three-stage review covering conversation naturalness, failure validity, prompt diversity, audio quality, and rubric quality.Figure 3 presents the dataset-generation process and quality check.Source: §2.2 · p. 5, Figure 3 · p. 5

3. Evaluation

The evaluation uses fixed-context conversations and rubric-based scoring to compare end-to-end spoken dialogue models. Results show low overall pass rates, particular difficulty with audio cues, conditional instructions, voice edits, and longer audio histories, alongside distinct effects from output modality and evaluation setup.

3. Evaluation

  • The fixed-context protocol evaluates only the final response using pre-seeded user-audio and assistant-text histories; APR requires satisfying all task rubrics, while ARS measures the average share satisfied.Providing identical histories avoids conversation drift from sequential generation and supports reproducible comparisons.Source: §3 · p. 6

3.1 Results

  • 54.65% APR is the highest overall score, achieved by Gemini 3 Pro Preview (Thinking); Voice Editing and Inference Memory average 17.99% and 21.55% across frontier models.GPT 4o Audio Preview scores 25.44% APR, while Voxtral Small 24B scores 26.33%.Source: §3.1 · p. 6, Table 2 · p. 7
  • Audio output averages 16.76% versus 19.36% for text output, and Audio-Cue Inference Memory has 36.5% lower ARS than semantic memory on average.Audio-Cue tasks require recalling paralinguistic or ambient information conveyed by audio rather than spoken words.Source: §3.1 · p. 7, Figure 5 · p. 8
  • Conditional instruction rubrics score worse than fixed rubrics across models; frontier models average 25.08% APR and 46.11% ARS on Instruction Retention tasks.Conditional rules apply only in specific future-turn scenarios, unlike persistent fixed constraints.Source: §3.1 · p. 7, Figure 6 · p. 8
  • Voice Editing averages 17.99% APR, with failures concentrated in conversations requiring integration of multiple mid-utterance repairs and prior-turn edits.These tasks may end with a request to summarize or revise all previously specified changes.Source: §3.1 · p. 8, Figure 7 · p. 9

3.2 Analysis

  • Performance stays largely stable as turn count rises, while longer audio histories coincide with lower Self Coherence and Voice Editing scores; Instruction Retention stays steady or improves.Self Coherence APR falls from 33.3% at 0–60 seconds to 20.0% at 3–5 minutes; Voice Editing falls from 17.5% at 3–5 minutes to 1.2% and 0% in longer bins with low sample counts.Source: §3.1 · p. 8, §3.2 · pp. 9–10, Figure 8 · p. 9

3.3 LLM-as-a-Judge

  • Across 1,712 rubric observations, judges achieve extremely high agreement with human graders, and o4 Mini has the highest Macro F1 score at 0.937.Evaluated audio-language models capable of text-based prompts were excluded from judging to mitigate self-enhancement bias.Source: §3.3 · p. 10

3.4 Ablation Studies

  • TTS improves text-output performance by 7.5% relatively but lowers audio-output performance by 2.5%; cascading lowers Gemini 3 Pro Preview and Gemini 2.5 Pro performance by 10.3% and 4.9%, respectively.On the audio-specific subset, cascading produces relative declines of 30.3% and 24.1% for those Gemini models, while Gemini 2.5 Flash Thinking shows no change.Source: §3.4 · p. 11, Table 4 · p. 11, Table 5 · p. 11

4. Related Work

Prior speech benchmarks largely emphasize short or single-turn tasks, while text-only work has explored multi-turn evaluation more extensively. Audio MultiChallenge addresses the comparatively underexplored speech-native setting, including extended interaction and multi-turn Voice Editing.

Single-Turn Speech Evaluation

  • Single-turn speech benchmarks cover speech, sound, and paralinguistic understanding, while long-form audio benchmarks test isolated comprehension rather than conversational state tracking.The cited long-form benchmarks show performance degradation as recordings grow longer and sparser.Source: §4 · p. 12

Multi-Turn LLM Evaluation

  • Text-only MultiChallenge tests instruction retention, information recall, versioned editing, and self-coherence, and studies report declines from single-turn to multi-turn settings.The reported declines span long-context generation, state tracking, reasoning, and aggregation.Source: §4 · p. 12

Multi-Turn Speech Evaluation

  • Speech-native multi-turn evaluation remains comparatively underexplored; MTalk-Bench uses scripted dialogues limited to 2–3 turns, and no known benchmark systematically tests multi-turn Voice Editing.Audio MultiChallenge targets extended speech interaction, long-horizon retention and memory, and non-monotonic user behavior.Source: §4 · p. 12

5. Conclusion and Future Work

Audio MultiChallenge evaluates four competencies in realistic multi-turn spoken dialogue and reveals gaps in current models. The conclusion also identifies a boundary in the evaluation: it does not assess raw audio outputs directly.

Conclusion

  • The benchmark evaluates Inference Memory, Instruction Retention, Self Coherence, and Voice Editing across 452 conversations using 1,712 instance-specific rubrics.The reported failures include audio-cue retention, conditional instruction following, long-context self-consistency, and mid-utterance edits.Source: §5 · p. 12
  • Models show substantial gaps in audio-cue retention, conditional instruction following, long-context self-consistency, and mid-utterance edits.The benchmark is open-source to support reproducible evaluation.Source: §5 · p. 12

Future Work

  • The evaluation does not operate directly on raw audio outputs, leaving open-ended audio-native assessment for future work.The authors also call for longer contexts and improved hybrid data curation.Source: §5 · p. 12

A.1.1 Inference Memory

The example tests whether systems can use a sound from an earlier turn to identify an item referenced later. Only Gemini correctly uses the buzzing cue; other models guess from the transcript.

A.1.1 Inference Memory

  • The final prompt refers to a drugstore item without naming it, requiring the system to connect the request with the earlier vibrating sound.The user asks how to use the item after saying they are going to bed soon.Source: §A.1.1 · pp. 17–18
  • Gemini’s answer explicitly grounds its electric-toothbrush identification in the buzzing sound played earlier.It then provides steps for using the toothbrush.Source: §A.1.1 · p. 18
  • Only Gemini 3 Pro Preview identifies the item from the earlier buzzing sound; other models guess face mist, skincare products, or bug spray from the transcript.The task expects an electric razor or toothbrush, and Gemini identifies an electric toothbrush.Source: §A.1.1 · pp. 17–18

A.1.2 Instruction Retention

The example tests whether models retain a roleplay instruction to provide a counterargument and follow-up for every question. Models often address the sensor-size question while missing the video-resolution question’s requirements.

A.1.2 Instruction Retention

  • The rubric separately checks a counterargument and follow-up for each of the two questions, making omission of either question a failure.The questions concern lower-resolution video formats and sensor sizes in DSLRs versus mirrorless cameras.Source: §A.1.2 · pp. 19–20
  • Gemini handles the sensor-size requirements but misses both the counterargument and follow-up for the video-resolution question.Its result is marked FAIL FAIL PASS PASS across the four requirements.Source: §A.1.2 · pp. 19–20
  • Most models fail to satisfy the roleplay instruction for both questions, often ignoring the video-resolution question while focusing on sensor size.The instruction requires a counterargument and follow-up for each individual question.Source: §A.1.2 · pp. 19–21

A.1.3 Self Coherence

In the calorie-plan example, most models fail to maintain the established target after calorie-dense foods are replaced. Gemini recognizes the shortfall and increases portions to compensate.

A.1.3 Self Coherence

  • Replacing calorie-dense snacks with fruit makes reaching approximately 3,000 calories harder and requires carrying more physical weight.Gemini responds by increasing portions, including six tablespoons of peanut butter.Source: §A.1.3 · p. 22
  • One reported total is approximately 2,300–2,500 calories, below the 3,000-calorie target; when calorie-dense snacks totaling 1,550 calories are swapped for apples and oranges (about 300–400), most models fail to compensate.Source: §A.1.3 · pp. 22–23
  • Only Gemini recognizes the deficit from the substitutions and increases portions to maintain the 3,000+ calorie target.It explicitly notes that fruit replaces calorie-dense snacks and gives a plan totaling approximately 3,240 calories.Source: §A.1.3 · pp. 22–23

A.1.4 Voice Editing

The Voice Editing task tests whether models track multiple spoken corrections to travel dates and resort choice. Models selected the correct resort but often mishandled the exact dates.

A.1.4 Voice Editing

  • The task evaluates whether models track multiple verbal booking corrections: departure changes from the 13th to the 12th, return to “Monday before Thanksgiving,” and a resort switch, then provide accurate final details.Source: §A.1.4 · p. 23
  • One response gives November 14 as Wednesday and November 20 as the departure date; both are marked incorrect in the response.The response is from Voxtral (Text).Source: §A.1.4 · p. 25
  • All models identified the correct resort but struggled with exact dates, including weekday confusion for November 14 and checkout dates of 20th, 25th, or 26th instead of 24th.The analysis attributes these errors to challenges processing incremental voice edits.Source: §A.1.4 · p. 25

A.2 Sample Human-Friendly Blueprints

This appendix section presents human-friendly dialogue probes and axis-specific task templates, alongside evaluation metrics, dataset diversity analysis, the synthetic failure-discovery process, and the LLM-as-a-Judge format. It highlights Inference Memory blueprints that reuse personal context in future suggestions and identifies a taxonomy of Self Coherence topics and instructions.

Inference Memory; Instruction Retention; Self Coherence; Voice Editing

  • The blueprints probe memory for a once-mentioned hobby, retention of a devil’s-advocate role, consistency of trading figures, and handling of mid-sentence recipe corrections.The examples test whether models confuse a speaker’s interests with their child’s, preserve a role without contradictory instruction updates, keep previously stated trade details consistent, and incorporate spontaneous corrections into a cooking timeline.Source: Inference Memory · p. 26, Instruction Retention · p. 26, Self Coherence · p. 26

A.3 LLM-as-a-Judge Prompt

  • The LLM-as-a-Judge prompt scores the conversation’s last assistant response against a rubric item and requests JSON fields for an explanation and a Boolean indicating whether criteria were met.The prompt is adapted from Arora et al., 2025 [35], with further implementation details in the dataset card.Source: §A.3 · p. 27

A.4 Evaluation Metrics

  • Average Pass Rate (APR) requires every rubric criterion on a task to be met, while Average Rubric Score (ARS) averages each task’s proportion of criteria met with equal task weight.APR averages binary task-level pass indicators; ARS provides a more granular measure of partial success.Source: §A.4 · pp. 27–28

A.5 Dataset Diversity

  • 0.57 average pairwise cosine distance between HuBERT embeddings indicates moderate acoustic variability across tasks, speakers, and audio cues.The analysis samples one utterance per task, mean-pools final-layer HuBERT features, and projects embeddings with UMAP using cosine distance.Source: §A.5 · p. 28, Figure 10 · p. 28

A.6 Synthetic Audio LM Failure Discovery

  • The synthetic failure-discovery loop samples an axis, topic, and persona, then iteratively generates tester turns, synthesizes audio, and queries a target audio model until a failure is found or turns run out.When it finds a failure, the planner distills the interaction history and failure reason into a human-friendly blueprint; otherwise, it returns null.Source: §A.6 · p. 29

A.7 Conversation Topics

  • The planner receives a randomly sampled conversation topic, with the appendix introducing the topic seeds used for its demonstrations.The supplied passage does not enumerate those seeds.Source: §A.7 · p. 29

VOICE EDITING

  • Voice-editing templates span corrections to schedules and numbers, shopping-list changes, message revisions, event details, to-do lists, and travel plans.Examples include changing a time mid-speech, correcting a phone-number segment, switching a message recipient, revising a deadline, and changing a destination.Source: VOICE EDITING · pp. 29–30

INSTRUCTION RETENTION; INFERENCE MEMORY

  • Instruction-retention templates cover response structure and style, personalization, role play, and content restrictions, while inference-memory templates include dietary restrictions, cuisine and taste preferences, and event dates.Examples include adopting a tutor persona, avoiding a specified word, explaining at a beginner level, remembering a dietary restriction, and recalling an event date.Source: INSTRUCTION RETENTION · p. 31, INFERENCE MEMORY · p. 31

INFERENCE MEMORY

  • Schedule, travel, and work examples call for remembering availability, recurring events, destinations, distance preferences, deadlines, and task status.These details inform future suggestions, comparisons, and time-management advice.Source: INFERENCE MEMORY · pp. 31–32
  • Blueprints use personal details, hobbies, ongoing projects, and emotional context to tailor later suggestions and support.Examples include recalling relationship or gift preferences, personal projects, mood, and mental-health goals.Source: INFERENCE MEMORY · p. 32
  • Shopping examples carry brand, price, and prior-purchase feedback preferences into future recommendations.Source: INFERENCE MEMORY · p. 32
  • Behavioral blueprints reuse hesitation points, preferred problem-solving styles, and knowledge gaps to offer reassurance or clarify proactively.The examples also include recalling hobbies and personal projects when offering related ideas.Source: INFERENCE MEMORY · p. 32

SELF COHERENCE

  • Table 6 is titled “Taxonomy of Topics and Instructions” under Self Coherence.Source: Table 6 · p. 33
Loading 2512.14865v1…