PaperlayerView exact revision on arXiv

Source-linked AI summary

Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap

Yueqian Lin, Zhengmian Hu, Qinsi Wang, Yudong Liu, Hengfan Zhang, Jayakumar Subramanian, Nikos Vlassis, Hai Helen Li, Yiran Chen

arXiv:2509.26542v1eess.AScs.MMcs.SD

TL;DR

Existing voice benchmarks have not evaluated general-purpose reasoning through a real-time conversational interface. VERA provides a benchmark for that setting and supports text–voice comparisons; its evaluation finds a broad performance gap, while the study’s heterogeneous commercial systems limit causal conclusions about modality alone.

  • Problem

    Existing voice benchmarks focus on acoustic understanding or spoken-language tasks, leaving general-purpose reasoning through a real-time conversational interface unevaluated.

    Source: §1 · p. 1

  • Method

    VERA is a benchmark for evaluating reasoning under real-time constraints, designed to enable controlled comparisons on identical tasks across voice and text systems.

    Source: §1 · p. 1, §3.1 · pp. 3–4

  • Results

    Across 2,931 episodes, voice models achieve 42% lower accuracy on average, with gaps exceeding 68% on complex domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Distinct architecture-linked failure signatures make VERA a diagnostic tool for measuring progress toward voice systems that reason while speaking.

    Source: §1 · p. 2, §7 · p. 10

    Because the evaluated commercial systems differ in architecture and training objectives, the study cannot isolate the causal impact of modality alone.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

We present Voice Evaluation of Reasoning Ability (VERA), a benchmark for evaluating reasoning ability in voice-interactive systems under real-time conversational constraints. VERA comprises 2,931 voice-native episodes derived from established text benchmarks and organized into five tracks (Math, Web, Science, Long-Context, Factual). Each item is adapted for speech interaction while preserving reasoning difficulty. VERA enables direct text-voice comparison within model families and supports analysis of how architectural choices affect reliability. We assess 12 contemporary voice systems alongside strong text baselines and observe large, consistent modality gaps: on competition mathematics a leading text model attains 74.8% accuracy while its voice counterpart reaches 6.1%; macro-averaged across tracks the best text models achieve 54.0% versus 11.3% for voice. Latency-accuracy analyses reveal a low-latency plateau, where fast voice systems cluster around ~10% accuracy, while approaching text performance requires sacrificing real-time interaction. Diagnostic experiments indicate that common mitigations are insufficient. Increasing "thinking time" yields negligible gains; a decoupled cascade that separates reasoning from narration improves accuracy but still falls well short of text and introduces characteristic grounding/consistency errors. Failure analyses further show distinct error signatures across native streaming, end-to-end, and cascade designs. VERA provides a reproducible testbed and targeted diagnostics for architectures that decouple thinking from speaking, offering a principled way to measure progress toward real-time voice assistants that are both fluent and reliably reasoned.

1 INTRODUCTION

The introduction presents VERA as a benchmark for measuring reasoning under real-time voice constraints and documents a substantial Voice Reasoning Gap. It argues that the gap is linked to a latency–accuracy trade-off and that voice architectures exhibit distinct failure patterns.

1 INTRODUCTION

  • GPT-realtime reaches 6.1% accuracy on mathematical problems, versus 74.8% for GPT-5, a 68.7-point difference.The paper describes this as representative of a broader pattern in which low-latency streaming models perform consistently worse.Source: §1 · p. 1
  • VERA measures reasoning under real-time constraints, and its analysis reveals a latency–accuracy trade-off.The benchmark is introduced to investigate the hypothesis that real-time voice architecture conflicts with complex reasoning.Source: §1 · p. 1
  • Fast voice models plateau around 10% accuracy at ≤1.5s response time, while approaching the ∼54% text upper bound requires sacrificing real-time interaction.Figure 1 plots first response time against accuracy and marks the Pareto frontier.Source: Figure 1 · p. 2, §1 · p. 2
  • Native streaming models tend to produce fluent but incorrect responses, while decoupled cascades are more prone to grounding errors.Even a powerful text reasoner decoupled from the voice interface improves accuracy but remains significantly below its native text performance.Source: §1 · p. 2
  • Across 2,931 episodes, the paper reports that voice models achieve 42% lower accuracy on average, with gaps exceeding 68% on complex domains.VERA is presented as a unified protocol for comparing native, cascade, and end-to-end voice architectures.Source: §1 · p. 2

2 RELATED WORK

Prior voice benchmarks largely evaluate acoustic understanding, spoken-language comprehension, or dialogue mechanics rather than general-purpose reasoning during real-time conversation. VERA targets the untested intersection of reasoning, real-time constraints, and direct text–voice comparison on identical tasks.

2 RELATED WORK

  • Existing benchmarks assess audio-content understanding or spoken-language comprehension, while full-duplex work focuses on dialogue mechanics rather than substantive reasoning during conversation.The cited areas include acoustic-scene and audio questions, speech-to-meaning tasks, and turn-taking or interruption handling.Source: §2 · pp. 2–3
  • Table 1 compares representative benchmarks by primary focus and marks capabilities as present, partial, or not included.The passage supplies the table’s scope and legend, not individual benchmark cell values.Source: Table 1 · p. 3
  • No existing benchmark combines multi-step general-purpose reasoning, explicit real-time latency constraints, and direct text–voice comparison on identical tasks.VERA is presented as the first benchmark at this intersection.Source: §2 · p. 3

3 THE VERA BENCHMARK

VERA is a benchmark for studying reasoning in voice systems under real-time conversational constraints. It formalizes the text–voice accuracy gap and builds a voice-adapted, five-track dataset from established benchmarks.

3 THE VERA BENCHMARK

  • Five representative panels show how Math, Web, Science, Long-Context, and Factual items are rewritten for voice while preserving reasoning difficulty.Source: Figure 2 · p. 3

3.1 FORMAL DEFINITION AND DIAGNOSTIC FRAMEWORK

  • The Voice Reasoning Gap is the expected difference in accuracy between text and voice across a distribution of reasoning tasks.Source: §3.1 · p. 3
  • The framework compares accuracy-oriented text models with top-performing voice models, using models from the same family where possible to avoid confounding modality with latency policy.Source: §3.1 · p. 3

3.2 VOICE ADAPTATION PIPELINE

  • VERA uses a reproducible, multistage pipeline to adapt established text benchmarks while preserving task semantics and enforcing voice-native constraints.The pipeline screens for voice suitability, rewrites questions for speech, and applies held-out quality validation.Source: §3.2 · p. 4
  • Accepted episodes meet track-specific quality thresholds before validated prompts are rendered as acoustically diverse 24kHz audio.Acceptance requires Qoverall ≥ τ and Qreason ≥ 7.0; speech generation varies timbre, tone, and emotion.Source: §3.2 · pp. 4–5

3.3 DATASET COMPOSITION

  • VERA contains 2,931 voice-optimized episodes derived from five benchmarks, spanning math, web-grounded synthesis, science, long-context memory, and factual recall.The 2,931 episodes were curated from approximately 22,000 source items; a manual audit of 200 episodes confirmed preservation of semantic and logical structures.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

The experimental setup evaluates voice reasoning with normalized speech transcripts, automated accuracy and failure judgments, and human calibration. It compares several voice-system designs with text baselines and a cascade that separates reasoning from narration.

4.1 EVALUATION METHODOLOGY

  • Voice transcripts and ground truth are normalized to canonical mathematical notation before Word Error Rate is calculated.Examples include converting “f of sixteen equals fifty four” to “f(16) = 54” and “twenty twenty-four” to “2024”.Source: §4.1 · p. 5
  • GPT-4o grades normalized voice transcripts and other predictions in three independent evaluations, with majority vote assigning the final label.The labels are Correct, Incorrect, and Not Attempted.Source: §4.1 · p. 5
  • GPT-5 classifies incorrect predictions across 16 error categories, distinguishing voice transcription artifacts from content errors.The categories span knowledge, reasoning, and understanding errors.Source: §4.1 · p. 6
  • On 1,000 sampled predictions, GPT-4o judgments agreed with human evaluation 97.8% of the time, with agreement ranging from 100% on Math to 84.3% on Science.Gemini-2.5-Flash achieved 98.7% agreement with humans and 98.1% with GPT-4o.Source: §4.1 · p. 6

4.2 MODEL CONFIGURATIONS

  • The evaluation benchmarks commercial real-time APIs, open voice models, and end-to-end speech generators against text-only references and a reasoning–narration cascade.LiveAnswer uses GPT-5 for reasoning and Llama-3.3-70B-Instruct to synthesize a concise spoken response.Source: §4.2 · p. 6

5 RESULTS AND ANALYSIS

Voice systems show a large, complexity-sensitive reasoning gap relative to text, and diagnostic experiments find that added thinking time and cascade designs do not eliminate it. Failure patterns also vary by architecture.

5.1 WHAT IS THE GAP AND HOW DOES IT VARY BY TASK?

  • Voice accuracy drops 40.4 percentage points on average, with GPT-5 at 74.8% versus GPT-realtime at 6.1% on Math.The gap widens with reasoning complexity; Table 3 reports macro-averaged accuracies across tracks.Source: §5.1 · p. 6, Table 3 · p. 7
  • Text models outperform voice within GPT and Gemini families, while several voice architectures remain below 5% on reasoning tasks.Across voice models, Math-score variance is 171× smaller than the between-modality variance.Source: §5.1 · pp. 7–8, Figure 4 · p. 7

5.2 WHY DOES THE GAP EXIST?

  • Extending Audio Flamingo 3 thinking time from 2.40s to 15.14s lowered overall accuracy from 1.7% to 1.5%, while LiveAnswer still trailed text on Math by 15.7 points.The cascade's reasoning-to-speech translation introduced inconsistencies and was especially detrimental to exact-match Context tasks, where accuracy was 0.2%.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 HOW DO THE MODELS FAIL DIFFERENTLY?

  • Native streaming systems tend to complete fluent but incorrect answers, cascades show more grounding and logical-consistency errors, and end-to-end models vary between off-target answers and disengagement.The reported error deviations include LiveAnswer's +0.27 for UNSUPPORTED FACT and Moshi's +0.52 for OFF TARGET.Source: §5.3 · pp. 8–9

6 FUTURE DIRECTIONS

The paper argues that incremental improvements are unlikely to close the voice reasoning gap and recommends architectures that separate internal reasoning from speech output. It also identifies latency and cross-stage consistency as design challenges.

6 FUTURE DIRECTIONS

  • The authors propose an editable internal state separate from the speech output buffer to decouple thinking from speaking.Suggested directions include asynchronous backend reasoning with conversational frontends and chunked reasoning in parallel with audio playback.Source: §6 · p. 9
  • LiveAnswer's cascade still has a 15.7-point Math penalty and +0.27 UNSUPPORTED FACT deviation, motivating attention to latency and cross-stage consistency.The authors suggest streaming ASR with confidence-gated handoff and answer-first narration as engineering directions.Source: §6 · p. 9
  • Figure 6 encodes each model's error-category deviation from the global baseline, with cool colors marking above-average error production and warm colors below-average production.The authors use these architecture-specific signatures to motivate changes beyond incremental improvements.Source: Figure 6 · p. 9, §6 · p. 9

7 CONCLUSION

The conclusion presents VERA as a benchmark for measuring the performance drop between text and voice reasoning. The authors report that the gap persists across tested mitigations and that error patterns differ by architecture.

7 CONCLUSION

  • The voice reasoning gap persists with extended thinking time, high audio fidelity, and a cascade that separates reasoning from audio input and output.The paper attributes the pattern to tension between low-latency streaming and iterative, revisable computation.Source: §7 · p. 9
  • Native streaming systems tend toward fluent but incorrect responses, whereas decoupled systems show grounding and consistency errors.The authors present VERA as a diagnostic tool for measuring progress toward voice systems that reason reliably.Source: §7 · p. 10

A PREVIOUS BENCHMARKS

The supplied appendix passages identify a voice-benchmark comparison table and describe normalization of spoken mathematical expressions for WER evaluation. The normalization standardizes notation while preserving meaning and context.

A PREVIOUS BENCHMARKS

  • Table 4 is identified as a comparison of voice benchmarks.Source: Table 4 · p. 14

B ASR TRANSCRIPT NORMALIZATION

  • The normalization procedure converts ASR transcripts and reference texts to canonical mathematical notation before WER is computed.It is intended to distinguish transcription errors from superficial formatting differences.Source: B ASR TRANSCRIPT NORMALIZATION · p. 15, §B.2 · p. 15

B.1 NORMALIZATION APPROACH

  • The approach uses GPT-4o with a deterministic prompt to standardize spoken mathematical expressions.The instructions include converting spoken numbers to digits, preserving semantic meaning, and leaving non-mathematical context unchanged.Source: §B.1 · p. 15

B.2 REPRESENTATIVE NORMALIZATION EXAMPLES

  • The same normalization is applied to ground truth and ASR output, and Table 5 is identified as showing representative examples.Source: §B.2 · p. 15, Table 5 · p. 15

C HUMAN EVALUATION AND JUDGE VALIDATION

The evaluation checks automated judging against human ratings, while the model suite spans commercial, open, end-to-end, text-only, and cascade systems. Proprietary systems are treated as black boxes, and open models are described through their published work when available.

C HUMAN EVALUATION AND JUDGE VALIDATION

  • Across 1,000 sampled outputs, human agreement was near-perfect on Math, Web, and Factual, and 84.3–92.9% on Science.The authors report that the lower Science agreement reflects the greater interpretive complexity of graduate-level scientific reasoning; they conclude that LLM-based judgments align reliably with human assessment.Source: C HUMAN EVALUATION AND JUDGE VALIDATION · p. 15, Table 6 · p. 15

D.1 COMMERCIAL VOICE APIS

  • Commercial voice baselines include streaming, low-latency systems such as GPT-realtime, Gemini-2.5-Flash-audio, and Nova-Sonic.GPT-realtime represents a native full-duplex baseline; vendor defaults are used for Gemini-2.5-Flash-audio and Nova-Sonic.Source: §D.1 · p. 16

D.2 OPEN VOICE MODELS

  • Open voice baselines include speech-and-text or audio-language models, plus UltraVox’s streaming ASR-to-LLM-to-TTS stack and the compact Phi-4-multimodal.Audio Flamingo 3 is tested both in its standard setting and with a thinking mode.Source: §D.2 · p. 16

D.3 END-TO-END VOICE MODELS

  • Moshi, Freeze-Omni, and Qwen2.5-Omni provide end-to-end voice baselines that directly support speech interaction.The set is used to examine reasoning under speech pressure across different architectural choices.Source: §D.3 · p. 16

D.4 TEXT-ONLY UPPER BOUNDS

  • Text-only GPT and Gemini systems receive the same tasks and answer formats as voice counterparts, isolating reasoning capacity from voice constraints.The text results serve as modality upper bounds; GPT-5 is tested at low and high effort, with high effort using more decode-time compute and incurring greater latency.Source: §D.4 · pp. 16–17

D.5 CASCADE BASELINE: LIVEANSWER

  • LiveAnswer separates GPT-5 problem-solving from user-facing narration, with a faster Llama-3.3-70B synthesizer narrating the reasoner’s streamed thoughts.The pipeline transcribes speech, runs reasoning and narration in parallel, then renders narration as audio.Source: §D.5 · p. 17

D.6 EVALUATION INFRASTRUCTURE

The evaluation infrastructure standardizes accuracy judgments, speech transcription scoring, and modality settings across systems.

D.6 EVALUATION INFRASTRUCTURE

  • Accuracy is judged by GPT-4o three times per item, with majority voting.The configuration is held out from the evaluated systems and is described as the automatic grader.Source: §D.6 · p. 17
  • Generated speech is scored after ASR, with an LLM-based normalizer canonicalizing spoken mathematics and notation.Source: §D.6 · p. 17
  • Voice-native systems use streaming and full-duplex when supported, while text upper bounds use matching prompts and answer formats.Web tools or retrieval are disallowed unless natively exposed; text and voice differ only by modality and, for high-effort settings, decode-time budget.Source: §D.6 · pp. 17–18

E STATISTICAL VALIDATION

The paper tests the robustness of voice–text comparisons statistically and describes prompts for filtering, adapting, checking, grading, and classifying benchmark items and model errors.

E STATISTICAL VALIDATION

  • All primary comparisons are significant at p < 0.001, and the gap persists with identical text models wrapped in voice I/O.Comparisons use paired McNemar tests and bootstrap confidence intervals from 10,000 resamples; the authors attribute the persistent difference to modality constraints rather than model capacity.Source: Table 7 · p. 18, E STATISTICAL VALIDATION · p. 18
  • The Web comparison is not significant (p = 0.636), with both modalities near 12%; LiveAnswer’s 0.2% Context result is flagged as a possible system-specific failure.The authors state that the Context result warrants investigation.Source: E STATISTICAL VALIDATION · p. 18, §F.1 · p. 18

F.1 FILTER PROMPT

  • The filter prompt screens for speech feasibility, task suitability, and task-type requirements, including spoken clarity and reasonable audio memory load.Source: §F.1 · pp. 18–19

F.2 ADAPTATION PROMPT

  • The adaptation prompt converts questions into natural, TTS-friendly speech while preserving the task, difficulty, constraints, and expected answer.It directs writers to spell out numbers and symbols and avoid including answers or hints.Source: §F.2 · pp. 19–20

F.3 QUALITY CHECK PROMPT

  • The quality check scores TTS optimization, conversational quality, task preservation, and voice clarity, with scores below 6 requiring revision.Source: §F.3 · pp. 20–21

F.4 GRADING PROMPT

  • The grading prompt assigns correct, incorrect, or not-attempted labels using general and task-specific criteria.For retrieval, it requires an exact ground-truth string; for mathematics, it judges the final numeric answer strictly.Source: §F.4 · pp. 21–22

F.5 FAILURE ANALYSIS PROMPT

  • The failure-analysis prompt classifies errors across knowledge, reasoning, output, and understanding, and requests confidence, rationale, and answer evidence.Source: §F.5 · p. 22

G DATASET SELECTION CRITERIA

The five dataset tracks are selected and adapted to retain reasoning tasks that can be presented and answered through speech, with exclusions for visual dependence and other voice constraints.

G.1 MATHEMATICAL REASONING (AIME)

  • The AIME math track retains 115 of 120 problems, excluding five requiring geometric diagrams or extensive symbolic manipulation.Retained answers are integers from 0 to 999 for pronunciation clarity, and notation is verbalized for speech.Source: §G.1 · p. 23

G.2 WEB-GROUNDED SYNTHESIS (BROWSECOMP)

  • BrowseComp contributes 1,107 of 1,255 multi-hop questions after removing items with temporal instability, visual dependencies, or speech-unfriendly evidence chains.URL citations are adapted into spoken attributions.Source: §G.2 · p. 23

G.3 SCIENTIFIC EXPERTISE (GPQA DIAMOND)

  • The GPQA Diamond science track retains 161 of 198 questions after excluding 37 with visual dependencies.The subset covers physics, chemistry, and biology; chemical notation is verbalized for speech.Source: §G.3 · p. 23

G.4 LONG-CONTEXT MEMORY (MRCR)

  • The long-context track retains 548 episodes from 2,400 synthetic multi-round coreference conversations, with contexts up to 100,000 characters.Random identifiers are adapted into natural ordinal references, and source materials are limited to 2022–2025.Source: §G.4 · p. 23

G.5 FACTUAL RECALL BASELINE (SIMPLEQA)

  • The factual-recall control track retains 1,000 of 4,326 fact-seeking questions, selecting for brief, clear, stable answers suited to audio.Selection criteria include responses under 10 spoken words, unambiguous pronunciation, temporal stability, and acoustic distinctiveness.Source: §G.5 · p. 23

Related papers

Loading 2509.26542v1…