PaperlayerView exact revision on arXiv

Source-linked AI summary

$S^3$-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants

Heyang Liu, Jiayi Huang, Wenyang Xiao, Ziyang Cheng, Lixin Zhang, Zhen Liu, Miao He, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang

arXiv:2609.09852v1cs.CL

TL;DR

Scientific speech-interaction models perform well in general voice-assistant settings, but their reliability in scientific conversations remains insufficiently characterized because these interactions combine rare terminology, specialized abbreviations, symbolic expressions, and multi-turn adaptation. S3-Bench evaluates this problem across scientific question answering and progressive dialogue by decomposing turns into recognition, perception, reasoning, and pronunciation stages. The benchmark reveals that perception is usually strong but does not guarantee reasoning, while speech generation can conflict with structured symbolic outputs and multi-turn systems remain limited in factual consistency, audience adaptation, and inquiry efficiency.

  • Problem

    Scientific speech interaction remains underexplored despite general voice-assistant progress, with challenges from technical terminology, abbreviations, symbolic expressions, and complex conversational demands.

  • Method

    S3-Bench evaluates 10 scientific domains through S3-Knowledge and multi-turn S3-Dialogue, decomposing interactions into recognition, perception, reasoning, and pronunciation stages.

  • Results

    Perception is generally strong but exceeds knowledge-utilization performance, while generation and pronunciation expose failures with complex terminology and symbolic expressions.

  • Takeaways & Limitations

    S3-Bench identifies scientific voice-assistant bottlenecks in factual consistency, audience adaptation, inquiry efficiency, and representation-to-pronunciation alignment.

  • Takeaways & Limitations

    Multi-turn scientific dialogues still show persistent limitations in factual consistency, audience adaptation, and efficient information acquisition.

Abstract

from arXiv · show

The advance of multimodal large language models (MLLMs) has fundamentally reshaped the paradigm of human-computer interaction, especially speech interaction models capable of seamless conversations. Despite remarkable performance as general voice assistants, their performance in specialized domains remains underexplored, particularly in scientific areas. Scientific interactions introduce formidable challenges, involving rare technical terminology, spoken norms of abbreviations, and the natural verbalization of symbolic special expressions. In this paper, we introduce S$^3$-Bench, a systematic evaluation framework covering 10 major disciplines, consisting of a Knowledge set for speech question-answering and a Dialogue set for multi-turn progressive interactions with simulated user agents. By decomposing a complete atomic turn into stages of speech recognition, perception, knowledge utilization with reasoning, and response pronunciation, we systematically characterize the common challenges and performance tradeoffs of existing approaches. Furthermore, experiments on multi-turn interactions reveal persistent limitations in user adaptation and the generation of accurate, comprehensive, and efficient responses.

Introduction

Scientific speech interaction remains underexplored despite strong general voice-assistant performance, because scientific turns require accurate recognition, concept perception, reasoning, and professional pronunciation. S3-Bench addresses this gap with a dedicated benchmark for scientific interaction scenarios.

  • Motivation: Scientific conversational turns require recognizing key expressions, mapping acoustic fragments to scientific concepts, reasoning over knowledge, and producing professionally pronounced responses.These stages extend beyond generic speech processing and are illustrated as necessary components of scientific question answering.
  • Benchmark: S3-Bench evaluates speech-interactive models across 10 major scientific domains and their sub-fields.The benchmark is designed specifically for scientific interaction scenarios.
  • Benchmark: The benchmark contains S3-Knowledge for specialized scientific question answering and S3-Dialogue for progressively advancing multi-turn conversations.S3-Knowledge targets terminology, abbreviations, and domain-specific expressions, while S3-Dialogue uses simulated user profiles.
  • Evaluation design: S3-Bench decouples atomic interaction into Recognition, Perception, Knowledge Utilization & Reasoning, and Generation & Pronunciation to expose performance bottlenecks.This decomposition supports systematic diagnosis of speech-interaction limitations in scientific contexts.
  • Evaluation design: Multi-turn dialogues with diverse personas evaluate scientific factuality, audience adaptation, and inquiry efficiency.The design targets models’ adaptation across users with different academic backgrounds and profiles.

S3-Bench

S3-Bench combines a four-stage scientific speech interaction process with curated knowledge and dialogue data spanning 10 domains. Its construction addresses terminology, abbreviations, symbolic expressions, pronunciation quality, and user-profile variation.

  • Challenges: Scientific speech interaction is difficult because professional terminology is sparse in pretraining data, abbreviations have domain-specific pronunciations, and discourse contains structured symbolic expressions.Examples include chemical formulas, mathematical equations, genomic sequences, and complex measurement units.
  • Interaction process: An atomic scientific interaction proceeds through Recognition, Perception, Knowledge Utilization & Reasoning, and response generation with precise pronunciation.The stages move from speech-text alignment and concept association to scientifically accurate answering and articulation.
  • S3-Knowledge: S3-Knowledge is constructed from college-level and domain-specific scientific benchmarks, with overly long or acoustically unsuitable entries removed.The pipeline uses existing text-based knowledge benchmarks and applies pruning for spoken interaction compatibility.
  • S3-Knowledge: Specialized vocabulary pronunciation is calibrated through manual review, online videos, authoritative disciplinary texts, and repeated re-synthesis checks.The process distinguishes initialisms from acronyms and adjusts representations for text-to-speech systems.
  • S3-Dialogue: S3-Dialogue converts representative concepts from research articles into structured conversations paired with synthetic users varying in expertise and character traits.Expertise levels range from Beginner to Expert, while traits include curious, patient, anxious, overconfident, and practical.
  • Coverage and statistics: The benchmark covers 10 scientific domains, with S3-Knowledge containing 1,980 simulated queries sourced from 17 public evaluation datasets.The knowledge set includes multiple-choice and open-ended questions, while S3-Dialogue is distributed across academic disciplines.

S3-Knowledge Experiments

S3-Knowledge evaluates scientific speech interaction across recognition, perception, reasoning, and generation, exposing terminology, abbreviation, and pronunciation bottlenecks. Results show positive coupling among early stages but a trade-off between perception or recognition and generation.

  • Evaluation Setup: The benchmark evaluates six Speech LLMs, two Omni-LLMs, and one proprietary API using ASR-derived metrics and standard accuracy.The evaluation includes terminology EER, abbreviation AER, relevant rate, and question-answering accuracy.
  • Recognition: Recognition errors concentrate on scientific terminology and abbreviations, with the best offline model exceeding 36% error on terminology.Approximately half of the evaluated models also show more than double strict AER relative to loose AER.
  • Perception: Most models achieve RR > 85%, but perception consistently exceeds knowledge-utilization performance, making perception a threshold rather than a guarantee.Qwen3-Omni and Fun-Audio-Chat-8B reach 97.42% and 96.26% RR, respectively.
  • Knowledge Utilization and Reasoning: Knowledge utilization differentiates models: Qwen3.5-Omni-Flash leads with 63.62% average accuracy, while special expressions remain difficult for all systems.The leading API reaches 66.20% on special expressions, and around half of offline models fall below 50%.
  • Capability Transfer and Consistency Analysis: The front three stages are positively coupled, while generation correlates negatively with perception and recognition, evidencing a cross-stage trade-off.The strongest reported positive correlation is perception–knowledge utilization at ρ=0.786; generation correlates with perception at ρ=−0.595.
  • Generation and Pronunciation: Generation and pronunciation expose a representation-pronunciation trade-off, as stronger perception captures domain-specific content that downstream vocoders may articulate poorly.VocalNet, Qwen2.5-Omni, and Qwen3.5-Omni-Flash achieve WER below 6% with low EER/AER, whereas other models struggle with complex terminology.

S3-Dialogue Experiments

S3-Dialogue evaluates scientific speech interaction through multi-turn conversations, measuring factuality, audience adaptation, and inquiry efficiency. Results show persistent difficulty with strict scientific correctness and user adaptation, while Qwen3-Omni performs comparatively well on factuality and dialogue efficiency.

  • Evaluation framework: The evaluation uses dual-agent multi-turn interactions and measures Scientific Factuality, Audience Adaptation, and Inquiry Efficiency.The User agent combines Qwen3.7-Plus, Qwen3-TTS, text normalization, and speech prompts; Table 4 distinguishes Base/Hard difficulty and Loose/Strict criteria.
  • Scientific factuality: 94.84% loose and 50.99% strict factuality make Qwen3-Omni the strongest offline end-to-end model, while most systems score below 20% under strict evaluation.The loose–strict gap indicates that scientifically rigorous answers are substantially harder than broadly plausible responses.
  • Audience adaptation: 63.38% AAR is the highest audience-adaptation score, achieved by the cascade system, whereas offline models remain around 45–54%.Current speech models tend toward relatively fixed explanation styles rather than dynamically adjusting to audience expertise.
  • Inquiry efficiency: Qwen3-Omni averages 5.85 turns and completes over 95% of conversations within eight turns, while earlier speech-language models often require over ten turns.The results associate efficient inquiry with identifying missing information and progressively resolving users’ questions.
  • Overall findings: The benchmark identifies three challenges: maintaining factual consistency across turns, adapting explanations to diverse expertise, and efficiently acquiring missing information.The paper points toward tighter knowledge grounding, explicit user modeling, and dialogue-policy optimization as relevant research directions.

1 Appendix Introduction

The appendices provide the paper’s supporting documentation, covering related work, benchmark construction, test-set statistics, implementation, additional results, and human evaluation.

  • Appendix contents: Appendix 2 reviews speech-interactive models, speech evaluation frameworks, and scientific capability benchmarks.It situates the paper within recent work across these three areas.
  • Appendix contents: Appendix 3 details construction of the S3-Knowledge and S3-Dialogue benchmarks, while Appendix 4 reports complete final-test-set statistics.These appendices expand the dataset and distribution information beyond the main text.
  • Reproducibility: Appendix 5 documents model access, inference hyperparameters, and representative responses for implementation and reproducibility.Appendix 7 adds experimental results omitted from the main text.
  • Evaluation support: Appendix 6 presents fine-tuned speech-recognition variants for pronunciation evaluation, including Qwen3-ASR-Terminology and Qwen3-ASR-Abbs.Appendix 8 describes the human-evaluation protocol for speech quality and validation.

2 Related Works

Prior speech benchmarks increasingly evaluate holistic and multi-turn interaction, while scientific benchmarks assess knowledge, reasoning, and multimodal understanding. However, these lines of work leave specialized scientific speech interaction insufficiently evaluated.

  • Speech interaction models: Speech interaction systems include cascade architectures, aligned multimodal models, and native multimodal models with different trade-offs between modularity, semantic alignment, and joint acoustic-text modeling.Cascade systems can accumulate latency and errors, while native models jointly represent linguistic and acoustic information.
  • Speech evaluation: Speech evaluation has progressed from task-specific understanding toward interactive speech generation and conversational evaluation in diverse scenarios.Existing benchmarks cover instruction following, broad audio understanding, multi-turn reasoning, consistency, speech quality, and safety alignment.
  • Research gap: No existing evaluation framework is specifically tailored to specialized scientific dialogue settings, despite the domain’s expertise, structural complexity, and reasoning demands.Scientific and speech-interaction benchmarks have largely developed separately, leaving spoken scientific expression recognition and spoken knowledge delivery underexplored.
  • Scientific LLM evaluation: Scientific LLM evaluation commonly targets expert knowledge comprehension, scientific reasoning and problem solving, and multimodal scientific-data understanding.Benchmarks span general academic knowledge, domain-specific reasoning, and figures, tables, charts, and other visual-textual inputs.
  • Scientific LLM evaluation: Most scientific reasoning benchmarks remain offline and task-isolated, limiting assessment of interactive clarification, adaptive explanation, and error correction.This leaves communication behaviors that emerge during scientific dialogue underrepresented.

3 Detailed Construction Pipeline

The construction pipeline assembles college-level scientific content across ten domains, calibrates and verifies its spoken form, and builds progressive dialogues around user profiles and complementary topic aspects.

  • Scientific content selection: The benchmark selects college-level or harder scientific material that can plausibly occur in speech, excluding visual-only samples and excessively complex structured text.The selection emphasizes fine-grained disciplinary coverage and specialized domains.
  • Speech preparation: Terminology pronunciation is calibrated through pronunciation-equivalent replacements informed by authentic speech recordings and professional academic sources.Abbreviations with no apparent lexical structure may be pronounced letter by letter.
  • Speech verification: Synthesized queries undergo automatic ASR-based verification and further manual comparison against text, spoken annotations, and reference videos.Mismatched samples are regenerated, and remaining pronunciation inconsistencies are corrected manually.
  • Perception evaluation: Relevant Rate evaluates whether responses identify the core entity, understand the queried concept, and connect them to relevant domain knowledge.Keyword candidates are extracted from questions, answers, explanations, and reasoning chains, then filtered for traceability and informativeness.
  • Dialogue construction: Each dialogue sample centers on one topic and includes a user profile, an initial question, follow-ups, and reference answers.Profiles vary across Beginner, Undergraduate, Graduate, and Expert difficulty and curious, patient, anxious, overconfident, and practical interaction styles.
  • Dialogue construction: Progressive questions cover complementary aspects including concept, mechanism, and milestone, with expected depth adjusted to the user profile.Concept questions establish definitions and scope; mechanism questions test processes and principles; milestone questions address developments that increased importance.

4 Benchmark Statistics

S3-Knowledge spans 10 scientific domains and 1,980 test samples, while S3-Dialogue organizes domain topics into progressive multi-turn interactions. The benchmark varies both scientific coverage and conversational depth.

  • S3-Knowledge covers 10 major scientific domains and 1,980 test samples drawn from 17 publicly available benchmark datasets.Average query duration is 12.74 seconds, and the longest samples exceed 20 seconds.
  • S3-Dialogue contains mostly 20 dialogue topics per domain, with Computer Science containing the most at 32 dialogues.The topics are distributed across five progressive aspects.
  • S3-Dialogue query lengths remain broadly consistent across the first four aspects, whereas the final Application aspect generally has longer queries.

5 Evaluated Models

S3-Bench evaluates speech, multimodal, and proprietary systems across scientific domains and model configurations. Its response examples expose tradeoffs between concise speech and structured reasoning outputs.

  • Model inventory: S3-Bench evaluates six Speech LLMs, two Omni-LLMs, and one proprietary API.Most multimodal models use native architectures with shared speech-text backbones and either shared or parallel decoding heads.
  • Scientific coverage: The benchmark’s scientific coverage includes domains such as Physics, Computer Science, Biology, Medicine, Geology, Climate Science, Astronomy, and Material Science.The supplied domain examples include specialized concepts ranging from Lattice QCD and graph neural networks to signaling pathways and nanoparticle materials.
  • Response behavior: Model response examples show a tradeoff between concise, clear speech and structured outputs whose complexity can create pronunciation ambiguities.Qwen3-Omni’s structured textual outputs are associated with substitution errors, truncation, and distortion in the talker output.
  • Dialogue examples: S3-Dialogue examples are composite responses stitched across evaluation aspects, so they illustrate responsiveness and style rather than complete continuous conversations.

6 Qwen3-ASR Models for Evaluation

The evaluation uses Qwen3-ASR variants adapted to scientific terminology and abbreviations to reduce transcription bias. The abbreviation-focused corpus provides broad coverage of scientific acronym forms.

  • Terminology recognition: Qwen3-ASR-1.7B is fine-tuned on the full Spoken Darwin-Science training split for terminology recognition.The selected model demonstrates improved transcription accuracy on real-world acoustic recordings.
  • Corpus construction: The corpus is curated from academic papers across nine scientific fields and filtered with rule-based matching for targeted abbreviations.
  • Abbreviation coverage: The abbreviation training corpus contains approximately 33K distinct abbreviations, with more than one-fifth assigned plausible acronym pronunciations.Most abbreviations are 4 to 5 characters long, matching typical scientific-domain structure.
  • Evaluation setup: The fine-tuned model is trained for one epoch on a single A100 GPU, while strict and loose AER distinguish exact formatting from case-insensitive uppercase alignment.Scientific abbreviation audio is collected from YouTube and segmented into clips under 30 seconds.

7 Additional Experiment Results

Additional experiments show that speech-enabled models lose knowledge accuracy relative to text backbones, especially on open-ended questions, while acronym pronunciation creates a retrieval bottleneck. Cross-stage analyses further separate terminology and abbreviation effects.

  • Perception: Perception is measured by keyword coverage across core entities, target concepts, and knowledge anchors, with Strict RR requiring all three categories.Nearly all responses cover at least one category, but simultaneous coverage remains limited.
  • LLM performance: More than 10% accuracy drops occur for VITA-Audio and MiMo-Audio relative to their text backbones, while other models decline by more than 5%.
  • Task formats: OpenQA accuracy consistently falls below 20% for spoken QA, whereas most text backbones exceed 30%; MCQA degradation is more moderate.The textual answer choices in MCQA may provide crucial semantic cues.
  • Abbreviations: Most models improve mildly when acronym pronunciations are replaced by full terminology forms, especially on OpenQA.The comparison indicates that spoken domain-specific formatting can obstruct grounding acronyms in their underlying semantics.
  • Cross-stage analysis: For terminology, perception strongly links to knowledge (P→K: ρ=0.643, r2=0.41, OR=5.44), while recognition-to-knowledge coupling is weaker than on the full set.Abbreviation recognition and knowledge are tightly coupled at entity level (R →K: ρ=0.571, r2=0.36).

8 Human Evaluation

Human evaluation covers both synthetic-speech quality and the reliability of automated judgments. The dialogue evaluation uses simulated learners whose personas, conversation state, and next-step decisions are explicitly prompted across progressive interactions.

  • Evaluation design: Human evaluation has two phases: assessing synthetic speech quality and validating the consistency of LLM-as-a-judge metrics.Five English-fluent university students reviewed text outputs, while reference videos and online dictionaries supported pronunciation verification.
  • Speech quality: 400 S3-Knowledge audio instances were randomly sampled and evenly assigned among four English-fluent annotators for clarity and phonetic-accuracy judgments.Technical-term pronunciations were checked against mainstream online dictionaries and annotated video resources; reported sample judgments were 98%, 99%, 100%, and 99%.
  • Dialogue evaluation: 400 S3-Dialogue conversations were sampled for Audience Adaptation evaluation, with each dialogue covering five target aspects and up to 15 interaction turns.Annotators inferred users’ academic backgrounds from stylistic shifts across model responses, and agreement rates with the LLM judge were assessed.
  • Automated metrics: Automated evaluation prompts separately target knowledge-set accuracy, scientific factuality, and audience adaptation.These prompts define distinct judging targets for the benchmark’s knowledge and dialogue assessments.
  • User-agent design: The simulated learner is constrained by a fixed academic background and personality, speaks English in the first person, and remains in character throughout the conversation.The user-agent prompt establishes these persona constraints for multi-turn interaction.
  • User-agent design: The user agent generates an initial query, evaluates the assistant’s last reply, and decides whether to advance, ask a follow-up, or terminate.Prompts track the topic, progressive science dimension, dimension index, inner turn, and prior conversation history; separate prompts generate turn-one and follow-up questions.
Loading 2609.09852v1…