Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems

1 Introduction

• A comprehensive benchmark for mathematical reasoning from spoken input is still missing, although spoken math matters in classrooms, tutoring, and collaborative discussions.Prior spoken-input studies chiefly probe basic comprehension or factual question answering, offering limited insight into multi-step reasoning; mathematical reasoning tests step-by-step deduction and sensitivity to fine-grained errors.Source: §1 · pp. 1–2
• Spoken-MQA evaluates cascade models and end-to-end speech LLMs across arithmetic, contextual reasoning, and mathematical-knowledge problems in unambiguous natural spoken language.The benchmark combines existing human-recorded problems with newly generated spoken data; GPT-4o verbalizes math expressions, and humans help filter for ambiguity before speech synthesis.Source: §1 · p. 2
• Speech-based models still struggle with direct arithmetic, verbalized mathematical expressions, and knowledge-oriented reasoning, despite some competitive contextual-reasoning performance.Cascade models remain effective but can lose performance through ASR transcripts, especially for mathematical symbols and formula-heavy problems; speech LLM training often emphasizes general conversation or audio understanding.Source: §1 · p. 2
• The paper introduces Spoken-MQA for systematic evaluation and compares cascade models with end-to-end speech LLMs to analyze their mathematical-reasoning limitations.The authors present the comparison and analysis as guidance for future research on improving spoken mathematical reasoning.Source: §1 · p. 2

2 Related Work

• Speech LLMs process raw speech end-to-end, whereas cascade models first transcribe speech to text with ASR before passing it to an LLM.Speech LLMs use a speech encoder and projection module; the passage says this design reduces cascade error propagation and retains paralinguistic cues.Source: §2 · p. 2
• Text and visual benchmarks cover mathematical reasoning, but speech and audio benchmarks have largely assessed general auditory understanding, arithmetic within logical reasoning, or spoken knowledge understanding.The cited examples include StepEval-Audio-360, which incorporates arithmetic, and VoxEval, which includes certain math-related queries.Source: §2 · p. 3
• Existing speech and audio evaluations fall short of systematically assessing mathematical reasoning, motivating a benchmark focused solely on spoken input.The cited speech work includes general auditory understanding, arithmetic in a logical-reasoning framework, and spoken knowledge understanding.Source: §2 · p. 3

3.1 Benchmark Taxonomy

• Arithmetic tests fundamental operations with minimal contextual understanding or external mathematical knowledge, using human-recorded SimpleMath problems with integer and decimal calculations.Selected problems include moderate-difficulty computations involving numbers greater than four digits.Source: §3.1 · pp. 3–4
• Contextual Reasoning covers everyday word problems requiring one or more arithmetic steps, split into single-step and multi-step subsets for finer analysis by reasoning depth.The source datasets include AddSub, SingleOp, SVAMP, and GSM8K; SVAMP complexity is partitioned heuristically by the number of arithmetic operations in a solution.Source: §3.1 · p. 4
• Knowledge-oriented problems require multi-step reasoning with mathematical domain knowledge, including number theory, algebra, or modular arithmetic.This category draws on MATH, whose high-school competition problems span areas such as algebra, number theory, geometry, and probability and often contain extensive LaTeX symbols and formulas.Source: §3.1 · p. 4

3.2 Speech Conversion and Disambiguation

• Text-based problems are converted to speech with TTS, but symbol-heavy expressions pose both verbalization and ambiguity challenges, especially in Knowledge-Oriented Reasoning.For example, “9!” should be spoken as “nine factorial”; a phrase such as “the magnitude of z minus w” can map to |z−w| or |z|−w. The passage says simpler arithmetic and contextual problems are unaffected by these issues.Source: §3.2 · pp. 4–5
• 32 of 100 sampled MATH problems were judged potentially ambiguous to humans when verbalized without visual or text context.This analysis motivates filtering and manual validation when constructing an unambiguous spoken-math benchmark.Source: §3.2 · p. 5
• GPT-4o verbalizes math problems and flags potentially ambiguous cases; the remaining examples are filtered and manually verified, yielding 500 curated unambiguous spoken problems for Knowledge-Oriented Reasoning.The curated problems are converted to speech using Coqui TTS2.Source: §3.2 · p. 5

4.1 Experimental Setup

• Evaluation compares Whisper-based ASR cascades with end-to-end speech LLMs; Phi-4 was fine-tuned on 500k OpenMathInstruct-1 questions synthesized into speech.Models are evaluated zero-shot with chain-of-thought prompting, using final-answer accuracy.Source: §4.1 · pp. 5–6

4.2 Experimental Result & Analysis

• Whisper-Qwen2.5-7B-Instruct achieves the best overall cascade performance, while GPT-4o-audio is the best end-to-end speech LLM; open-source speech LLMs generally trail cascades.Among speech LLMs, Ultravox and Phi-4 perform relatively strongly on Contextual Reasoning, but most speech LLMs lag on Arithmetic.Source: §4.2 · p. 6, Table 2 · p. 6

4.2.1 Arithmetic Gap

• Speech LLMs underperform cascades on both short- and long-digit arithmetic, so large numbers alone do not explain their Arithmetic gap.The authors suggest limitations in speech encoders or arithmetic capability may contribute; contextual-task strength may instead reflect surface-pattern matching, an open question for future work.Source: §4.2.1 · p. 7, Figure 3 · p. 7

4.2.2 Impact of Input Format on LLMs

• Ground-truth text generally yields higher accuracy than ASR transcripts, especially for Knowledge-oriented Reasoning; verbalized math also scores below ground-truth text despite preserving meaning.The results indicate that transcription errors can propagate to downstream reasoning, while LLMs show a strong preference for symbolic formulas over spoken-language formulations.Source: §4.2.2 · p. 7, Table 3 · p. 8

4.2.3 Domain-specific Fine-tuning

• Ultravox performs comparably to its cascade counterpart on most tasks except Arithmetic, while other speech LLMs significantly underperform cascades.Ultravox’s training methodology is undisclosed; the authors suggest speech LLMs may lack domain-specific reasoning optimization or lose such capabilities during training.Source: §4.2.3 · p. 8
• Fine-tuning Phi-4 on spoken mathematical data substantially improves Arithmetic and Knowledge-oriented Reasoning, with more modest gains in Contextual Reasoning.The authors suggest the smaller contextual gain may reflect an already strong baseline, comparable to most cascades and approaching GPT-4o-audio.Source: §4.2.3 · p. 8

• Spoken-MQA evaluates cascade and end-to-end speech models across arithmetic, contextual, and knowledge-oriented mathematical reasoning.Source: §5 · p. 8
• Current end-to-end speech LLMs generally lag cascades, especially on arithmetic and knowledge-intensive tasks, and cascade LLMs prefer symbolic LaTeX expressions to verbalized math.Source: §5 · p. 8
• Domain-specific fine-tuning boosts arithmetic and knowledge-reasoning performance, while the findings point to improving verbal-math understanding and domain-adaptive training.Source: §5 · p. 8

• TTS generates the Contextual Reasoning and Knowledge-Oriented Reasoning subsets, not all Spoken-MQA subsets.Source: Limitations · p. 9
• Synthesized speech lacks the natural variability and imperfections of human speech, potentially affecting generalizability to real-world applications.Source: Limitations · p. 9
• The authors call for more human-spoken data across all subsets to make evaluation more realistic and comprehensive.Source: Limitations · p. 9

• Figure 4 presents instructions used to generate verbalized math questions.Source: A Prompts for Generating Verbalized… · p. 11
• The instructions also address assessing ambiguity in the questions.Source: A Prompts for Generating Verbalized… · p. 11
• The figure concerns question-generation instructions rather than model-evaluation results.Source: A Prompts for Generating Verbalized… · p. 11

• Figure 4 presents examples of math problems that exhibit ambiguity after verbalization.Source: B Ambiguous Examples · p. 11

• The base model and speech encoder remain frozen while only the audio project and LoRA_A adapters are trained.Source: C Training Strategy · p. 11
• Training uses 3 epochs and a learning rate of 4e-5.Source: C Training Strategy · p. 11