PaperlayerView exact revision on arXiv

Source-linked AI summary

$S^3$-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants

Heyang Liu, Jiayi Huang, Wenyang Xiao, Ziyang Cheng, Lixin Zhang, Zhen Liu, Miao He, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang

arXiv:2609.09852v1cs.CL

TL;DR

Whether speech interaction models can remain effective in complex scientific conversations is still an open question. S3-Bench evaluates them across scientific domains and interaction stages, revealing positive coupling among upstream capabilities but negative coupling between speech generation and recognition and perception.

  • Problem

    Evidence remains limited on whether speech interaction frameworks sustain their efficacy in complex scientific conversational settings.

    Source: Introduction · p. 1

  • Method

    S3-Bench evaluates scientific speech interaction across 10 domains using atomic question-answering and multi-turn dialogue sets, decomposing turns into four stages.

    Source: Introduction · p. 1

  • Results

    Evaluations show upstream capabilities form a positively coupled chain, while speech generation couples negatively with recognition and perception.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The authors position S3-Bench as a diagnostic benchmark for research on scientific voice assistants.

    Source: Conclusion · p. 7

    In multi-turn scientific dialogue, current models struggle to maintain factual consistency, adapt explanations to users’ expertise, and efficiently acquire missing information.

    Source: Evaluation Methods · p. 7

Abstract

from arXiv · show

The advance of multimodal large language models (MLLMs) has fundamentally reshaped the paradigm of human-computer interaction, especially speech interaction models capable of seamless conversations. Despite remarkable performance as general voice assistants, their performance in specialized domains remains underexplored, particularly in scientific areas. Scientific interactions introduce formidable challenges, involving rare technical terminology, spoken norms of abbreviations, and the natural verbalization of symbolic special expressions. In this paper, we introduce S$^3$-Bench, a systematic evaluation framework covering 10 major disciplines, consisting of a Knowledge set for speech question-answering and a Dialogue set for multi-turn progressive interactions with simulated user agents. By decomposing a complete atomic turn into stages of speech recognition, perception, knowledge utilization with reasoning, and response pronunciation, we systematically characterize the common challenges and performance tradeoffs of existing approaches. Furthermore, experiments on multi-turn interactions reveal persistent limitations in user adaptation and the generation of accurate, comprehensive, and efficient responses.

Introduction

S^3-Bench addresses whether speech interaction models can function effectively in scientific conversations, where accurate recognition, scientific reasoning, professional speech, and audience adaptation are required. It evaluates these capabilities across 10 scientific domains through question answering and progressive multi-turn dialogue.

Introduction

  • Scientific interaction remains an open challenge because models must gauge users’ expertise, explain intellectually challenging ideas, address cutting-edge science, and provide heuristic guidance across turns.The paper asks whether speech interaction frameworks can sustain their effectiveness in complex scientific conversational settings.Source: Introduction · p. 1
  • S3-Bench is a comprehensive framework for scientific interaction across 10 domains and sub-fields, with college-level question answering and multi-turn dialogue subsets.S3-Knowledge targets professional terminology, colloquial abbreviations, and domain-specific linguistic expressions; S3-Dialogue focuses on frontier sciences and cutting-edge techniques.Source: Introduction · p. 1
  • The benchmark decomposes scientific speech responses into Recognition, Perception, Knowledge Utilization & Reasoning, and Generation & Pronunciation to expose performance bottlenecks.Scientific turns require recognizing key expressions, mapping acoustic segments to scientific concepts, deriving correct responses, and producing clear speech that follows professional conventions.Source: Introduction · pp. 1–2
  • S3-Dialogue uses progressively advancing conversations and simulated users with varied personas and academic backgrounds to assess Scientific Factuality, Audience Adaptation, and Inquiry Efficiency.The dialogue benchmark examines models’ adaptation limits and deployment potential as scientific voice assistants.Source: Introduction · pp. 1–2

S3-Bench

S3-Bench frames scientific voice interaction as a four-stage task and builds knowledge-QA and multi-turn dialogue datasets across 10 scientific domains. Its design targets rare terminology, community pronunciation norms, and structured scientific expressions.

Key Scientific Expressions

  • Scientific speech challenges include sparse technical terminology, domain-specific abbreviation pronunciations, and structured notations such as formulas, equations, genomic sequences, and measurement units.Rendering such notations as natural speech requires contextual reasoning and multimodal alignment.Source: Key Scientific Expressions · p. 2

Atomic Interactive Turn Process

  • S3-Bench defines a scientific speech turn as recognition, concept perception, knowledge use and reasoning, then spoken response generation with accurate pronunciation.Specialized terminology, colloquial abbreviations, and domain-specific expressions carry critical semantic weight throughout the stages.Source: Atomic Interactive Turn Process · p. 2

Construction Pipeline

  • The Knowledge pipeline curates about 2,200 text entries, calibrates scientific pronunciations for speech synthesis, and refines samples until 99% have pristine clarity and phonetic accuracy.Entries with synthetic errors or unfounded pronunciations are pruned, yielding 1,980 high-quality speech samples.Source: Construction Pipeline · p. 3
  • Dialogue topics from research articles are organized into progressive queries about concepts, mechanisms, milestones, characteristics, and applications, with user profiles varying by expertise and character trait.Questions are generated by GPT-5.5 Pro and cross-validated by Gemini3.5-flash and Qwen3.7-Max.Source: Construction Pipeline · p. 3

Benchmark Statistics

  • The benchmark spans 10 scientific domains: Knowledge includes 1,980 MCQA and open-ended queries from 17 public datasets, while Dialogue retains 213 multi-turn dialogues.The Knowledge instances are balanced and diverse, and the Dialogue set is balanced across academic disciplines.Source: Benchmark Statistics · p. 3

S3-Knowledge Experiments

S3-Knowledge evaluates scientific speech models across recognition, perception, reasoning, and spoken response, finding that reasoning and pronunciation expose major weaknesses despite generally strong perception. Cross-stage analysis shows that generation is a bottleneck and that stronger reasoning does not ensure intelligible speech.

Experiment Setup

  • The benchmark covered six Speech LLMs, two Omni-LLMs, Qwen3.5-Omni-flash, and a Whisper-large-v3, Qwen3-8B, and Qwen3-TTS cascade.Most models used 7–9B backbones; Qwen3-Omni used a 30B MoE Thinker-Talker architecture.Source: Experiment Setup · p. 4

Evaluation Indicators

  • Evaluation measured general transcription with WER, rare terminology with EER, abbreviations with strict and loose AER, perception with RR, and response correctness with accuracy.Terminology meant lexical items occurring fewer than 10 times in GigaSpeech; loose AER standardized case and spaces, whereas strict AER required exact case and format.Source: Evaluation Indicators · pp. 4–5

Fine-tuned ASR Models

  • Fine-tuned ASR variants addressed terminology and abbreviation recognition, with a pronunciation lexicon of around 230K abbreviations supporting Qwen3-ASR-Abbs.Qwen3-ASR-1.7B was used for standard WER, while fine-tuned variants handled terminology and abbreviation evaluation; the decline in strict AER indicated better contextual differentiation of words and abbreviations.Source: Fine-tuned ASR Models · p. 5

Performance Analysis (Table 2)

  • Qwen3.5-Omni-Flash (8.42%) and Kimi-Audio (9.62%) had the strongest query-WER, but scientific terminology and abbreviations remained difficult to recognize.The best offline model had terminology error above 36%, and approximately half of evaluated models had more than double AER in strict versus loose mode.Source: Performance Analysis (Table 2) · p. 5
  • RR exceeded 85% for most models, with Qwen3-Omni at 97.42% and Fun-Audio-Chat-8B at 96.26%, but perception consistently outperformed knowledge utilization.Restating a perceived concept did not guarantee successful reasoning, so perception acted as a threshold rather than a guarantee.Source: Performance Analysis (Table 2) · p. 5
  • 63.62% average accuracy led Qwen3.5-Omni-Flash in knowledge utilization, ahead of Cascade (60.35%), Fun-Audio-Chat-8B (57.58%), and VocalNet (56.21%).Terminologies and special expressions remain difficult: only Fun-Audio-Chat exceeded 60% accuracy on terminology among offline models, and around half scored below 50% on special expressions.Source: Performance Analysis (Table 2) · p. 5, Table 2 · p. 6
  • VocalNet, Qwen2.5-Omni, and Qwen3.5-Omni-Flash achieved WER below 6% with low EER/AER, while several models struggled to pronounce complex terminology.Although 84.2% of correctly answered turns contained special-symbol tokens, speech generation often failed to render structured outputs intelligibly.Source: Performance Analysis (Table 2) · p. 5
  • Cross-stage correlations linked perception and knowledge most strongly (ρP K=0.786), while generation correlated negatively with perception (ρP G=−0.595) and recognition (ρRG=−0.429).Recognition-to-perception coupling was weaker (ρRP=0.393, r2=0.11); knowledge reasoning and generation were nearly independent, making speech generation the main bottleneck.Source: Performance Analysis (Table 2) · pp. 5–6, Table 3 · p. 6

S3-Dialogue Experiments

S3-Dialogue evaluates scientific voice assistants in multi-turn interactions, finding persistent challenges in factual correctness, audience adaptation, and inquiry efficiency. Qwen3-Omni leads offline end-to-end models in factuality and efficient completion, while the cascade system achieves the highest audience adaptation rate.

Evaluation Methods

  • The dual-agent evaluation uses generated speech queries and conversation histories, scoring factuality, audience adaptation, and inquiry efficiency.The User agent follows up on ambiguous or problematic responses for clarification, with up to three turns per aspect; GPT-5.6 Luna serves as judge, with human consistency evaluations in Appendix 8.Source: Evaluation Methods · p. 6, Figure 5 · p. 7
  • Qwen3-Omni achieves 94.84% loose and 50.99% strict scientific factuality, while most systems score below 20% strict.The gap between loose and strict scores indicates that scientifically rigorous responses remain harder to produce than merely plausible ones.Source: Evaluation Methods · pp. 6–7, Table 4 · p. 7
  • The cascade system reaches the highest audience adaptation rate at 63.38%, while offline models achieve only around 45–54%.Speech models tend to use relatively fixed explanation styles regardless of audience expertise.Source: Evaluation Methods · p. 7
  • Qwen3-Omni averages 5.85 turns and completes over 95% of dialogues within eight turns; Qwen3.5-Omni-Flash has the highest completion rate and lowest average length.Earlier speech-language models often require over ten turns and fail to converge within the dialogue budget; Qwen3.5-Omni-Flash has lower factuality than Qwen3-Omni.Source: Evaluation Methods · p. 7

Conclusion

  • S3-Bench identifies multi-turn factual consistency, audience adaptation, and inquiry efficiency as persistent bottlenecks in scientific dialogue.The paper points to tighter scientific knowledge grounding, explicit user modeling, and dialogue policy optimization as directions for addressing them.Source: Evaluation Methods · p. 7, Conclusion · p. 7

Appendix · 1 Appendix Introduction

The appendix supplements the main text with related work, benchmark and dataset details, implementation and reproducibility information, additional experiments, evaluation protocols, and prompts.

Appendix · 1 Appendix Introduction

  • Appendix 2 reviews recent advances in speech interactive models, speech evaluation frameworks, and scientific capability benchmarks.It provides the related-work context omitted from the main text.Source: §1 · p. 10
  • Appendices 3 and 4 describe construction of the S3-Knowledge and S3-Dialogue datasets and report final test-set statistical distributions.These sections provide benchmark construction and test-set statistics.Source: §1 · p. 10
  • Appendix 5 documents model access, inference hyperparameters, and representative model responses for implementation and reproducibility.It supplies practical details for reproducing the work.Source: §1 · p. 10
  • Appendix 6 introduces pronunciation-evaluation fine-tuning variants Qwen3-ASR-Terminology and Qwen3-ASR-Abbs.The variants are fine-tuned speech recognition models.Source: §1 · p. 10
  • Appendix 7 presents experimental results omitted from the main text.It extends the reported experiments with additional results.Source: §1 · p. 10
  • Appendix 8 describes a human evaluation protocol for assessing speech quality and validating alignment with LLM-as-a-judge metrics.The protocol addresses both speech quality and metric alignment.Source: §1 · p. 10
  • Appendix 9 lists prompt templates for automated capability evaluation and S3-Dialogue system execution.The templates cover both automated evaluation and dialogue-system operation.Source: §1 · p. 10

2 Related Works

Related work covers general-purpose speech interaction and scientific language-model evaluation, but neither evaluates scientific dialogue through spoken interaction. The paper positions S^3- and related evaluation as addressing this gap in specialized knowledge, spoken expressions, and adaptive communication [s21.p5#b0e17c, s22.p6#242ccb].

2.1 Speech Interaction Models

  • Speech interaction progresses from cascade ASR–LLM–TTS systems [1] [2] [3] toward aligned [5] [8] [9] and native [10] [11] [12] [13] [14] multimodal models [4].Cascades incur latency, error propagation, and loss of paralinguistic cues; aligned models condition speech on intermediate text, while native models jointly model modalities with greater optimization and data demands.Source: §2.1 · p. 10
  • Recent systems scale training data [15], decouple duplex speech streams [16] [17], add expressive singing [18], or are closed-source [19]; this study focuses on general-purpose models for multi-turn conversation.Source: §2.1 · p. 10

2.2 Speech Interaction Evaluation

  • Speech-interaction evaluation has progressed from task-oriented understanding through interactive speech generation to diverse conversational scenarios, while early benchmarks assessed text responses to vocal queries [21] [22].AIR-Bench, AudioBench, and OpenAudioBench assess instruction following, broad audio understanding, or knowledge-intensive QA and reasoning, but omit generated-speech consistency and naturalness.Source: §2.2 · pp. 10–11
  • Later benchmarks assess dual-modal consistency, speech quality, multi-turn reasoning, paralinguistic awareness, diverse conversational settings, robustness, and system-level assistant capabilities [23] [24] [25] [26] [27] [28].The cited approaches include expressive interactive pairs, noisy acoustic conditions, and evaluation across listening, speaking, and viewing.Source: §2.2 · p. 11

2.3 Scientific LLM Evaluation

  • Scientific and speech-interaction evaluation have largely developed separately, leaving scientific speech recognition, spoken explanations, technical terminology, abbreviations, and user-adapted reasoning underexplored [s21.p5#b0e17c, s22.p6#242ccb].The paper motivates a dedicated framework for evaluating speech models as scientific assistants.Source: §2.2 · p. 11, §2.3 · p. 12
  • Scientific LLM evaluation spans expert knowledge, reasoning, and multimodal understanding [29]; knowledge benchmarks range from MMLU and C-Eval [31] to MMLU-Pro [32], GPQA, and SuperGPQA.These standardized benchmarks mainly use single-turn multiple-choice questions and final-answer accuracy, offering limited insight into intermediate reasoning.Source: §2.3 · p. 11
  • Scientific reasoning benchmarks assess broad competencies [35] or discipline-specific skills such as physical derivation and chemical reasoning [37] [38], but remain largely offline and task-isolated.That setup limits assessment of interactive clarification, adaptive explanation, and error correction during scientific communication.Source: §2.3 · pp. 11–12
  • Multimodal scientific benchmarks test visual, textual, symbolic, or structured inputs [39] [40] [41], but rarely assess specialized scientific information conveyed and understood through speech.ScienceQA uses diagrams, images, and explanations; MMMU and MMMU-Pro test expert reasoning over figures, tables, charts, and interleaved visual-textual information.Source: §2.3 · p. 12

3 Detailed Construction Pipeline

The construction pipeline selects spoken-suitable scientific benchmarks, calibrates and verifies synthesized query pronunciations, and builds perception annotations and multi-turn scientific dialogues. Dialogue topics are balanced across ten domains, while dialogue profiles and question types structure the interactions.

3.1.1 Public Scientific Benchmarks

  • S3-Knowledge draws on scientific benchmarks selected for disciplinary breadth, spoken-conversation suitability, concise content, and challenging terminology or verbalized special expressions.The selection excludes tasks requiring visual modalities, excessively complex structured text, and over-lengthy formats.Source: §3.1.1 · p. 12

3.1.2 Text Pronunciation Calibration

  • Pronunciation calibration manually annotates abbreviations and special expressions whose spoken conventions differ from their written forms.Annotators compare candidate forms with online speech recordings and, when resources are limited, infer pronunciations from professional academic websites; consonant-only abbreviations are pronounced letter by letter.Source: §3.1.2 · p. 13

3.1.3 Verification Process

  • Synthesized queries undergo iterative verification: specialized ASR models identify mismatches for regeneration, followed by manual comparison with reference speech and correction as needed.Manual verification uses the original query, its spoken-form annotation, synthesized audio, and a timestamped reference video.Source: §3.1.3 · p. 13

3.1.4 Relevant Word Generation

  • Relevant Rate evaluates whether a response identifies the core entity, understands the queried concept, and connects them to domain knowledge.Keyword candidates come from the question, answer, explanation, and MiniMax-M3 reasoning chain, weighted 5, 5, 3, and 2; selected terms are classified as core entities, target concepts, or knowledge anchors and must be traceable to source text. In open-ended perception evaluation, any listed keyword in the response counts as successful perception.Source: §3.1.4 · pp. 13, 16

3.2.1 Topic Selection

  • Topic selection balances ten scientific domains and prioritizes specific concepts, methods, mechanisms, techniques, metrics, and applications suited to multi-turn dialogue.Candidates are extracted as phrases rather than complete paper titles and assessed for specificity, mechanism interpretability, and application scope.Source: §3.2.1 · p. 16
  • The current topic set contains 213 topics, with at least 20 in each of ten scientific domains.Topics are selected for domain coverage, specificity, interpretable mechanisms, application scope, and suitability for dialogue; short, information-dense topics are prioritized. The domain distribution is summarized in Table 2.Source: §3.2.1 · p. 16, Table 2 · p. 16

3.2.2 Dialogue Construction

  • Each S3-Dialogue sample centers on one topic and uses a user profile to shape difficulty and style across five complementary question aspects.Profiles combine four difficulty levels with five user characters; questions address concept, mechanism, milestones, characteristics, and application, with depth adapted to the profile.Source: §3.2.2 · pp. 16–17

4 Benchmark Statistics

S^3-Knowledge combines samples from public benchmarks across 10 scientific domains, while S^3-Dialogue covers domain-specific topics and five progressive aspects. Their statistics show variation in query duration and length, as well as in domain and source coverage.

4 Benchmark Statistics

  • S^3-Knowledge spans 10 scientific domains, drawing 1,980 test samples from 17 public datasets, with 12.74-second average queries and some exceeding 20 seconds.Some domains draw on multiple benchmarks: Life Sciences includes four sources. A benchmark can also contribute to several domains; EarthSE spans Life Sciences, Earth Sciences, and Climate Science [42].Source: §4 · p. 17
  • Most S^3-Dialogue domains contain 20 topics, while Computer Science has 32; query lengths stay similar across the first four aspects and generally increase in Application.The five progressive aspects are summarized in Table 4, and all dialogue topics are listed in Table 5.Source: §4 · p. 17

5 Evaluated Models

S3-Bench evaluates a mix of speech and omni-language models alongside a cascaded baseline, then examines their configurations and example responses across knowledge and dialogue tasks. The examples reveal tradeoffs in spoken output and illustrate multi-turn exchanges, while dialogue examples are stitched rather than continuous.

5.1 Model Overview

  • The benchmark includes six Speech LLMs, two Omni-LLMs, and one proprietary API; its cascaded baseline uses Whisper-Large-v3 for ASR and Qwen3-TTS with the default Ryan voice for speech generation.Most multimodal models use a shared speech-and-text backbone, with speech tokens decoded through a shared or separate head.Source: §5.1 · p. 17

5.2 Model Configuration

  • Inference uses default generation and voice settings when available, a Common Voice male utterance as the speech prompt otherwise, and NVIDIA L20 GPUs except for Kimi-Audio on an A100.ASR prompts and MCQA text prompts are specified; MCQA options are provided in an A/B/C format.Source: §5.2 · p. 20

5.3.1 S3-Knowledge

  • Some S3-Knowledge models produce concise, clear speech, while Qwen3-Omni’s structured text can cause pronunciation ambiguity, substitutions, truncation, and distortion on long symbolic expressions.MiMo-Audio’s speech-adapted backbone yields relatively clear, natural speech despite less readable text responses.Source: §5.3.1 · p. 20

5.3.2 S3-Dialogue

  • S3-Dialogue examples show follow-up exchanges about vector-index recall, progressing from foundational assumptions and experiments to questions about monitoring silent degradation.The examples are composite dialogues stitched across models and evaluation aspects, not complete continuous conversations.Source: §5.3.2 · pp. 22, 24
  • Another S3-Dialogue example follows rare-variant burden testing from sensitivity–specificity tradeoffs to weighting variants by predicted functional impact during collapsing.The exchange also notes filtering low-quality or low-effect variants as a way to reduce neutral noise.Source: §5.3.2 · p. 25

6 Qwen3-ASR Models for Evaluation

To reduce transcription-error bias when evaluating speech generation, the paper uses Qwen3-ASR variants fine-tuned for scientific terminology and abbreviations, whose contextual reasoning can produce semantically coherent transcriptions.

6 Qwen3-ASR Models for Evaluation

  • Speech outputs are transcribed and aligned with generated text, using fine-tuned Qwen3-ASR variants because open-source ASR models generalize poorly to scientific terminology and standardized expressions.The evaluation selects LLM-based ASR for its contextual reasoning in scientific dialogues.Source: §6 · p. 26

6.1 Qwen3-ASR-Terminology

  • Qwen3-ASR-1.7B fine-tuned on the full Spoken Darwin-Science training split is used for terminology recognition and improves transcription accuracy on real-world acoustic recordings.Source: §6.1 · p. 26

6.2.1 Corpus Construction

  • The abbreviation corpus contains 887K instances, built by filtering Darwin-Science sentences for targeted abbreviations and sampling at most five sentences per abbreviation.Pronunciations follow a hierarchy of acronym forms, canonical words or homophones, and spelled-out initialisms, with length-dependent processing; high-quality Qwen3-TTS clips serve as speech prompts.Source: §6.2.1 · p. 26

6.2.2 Corpus Distribution

  • The abbreviation vocabulary covers approximately 33K distinct abbreviations, with over one-fifth assigned plausible acronym pronunciations and most abbreviations four or five characters long.Source: §6.2.2 · p. 27, Table 15 · p. 27

6.2.3 Model Training and Evaluation

  • Qwen3-ASR-Abbs achieves 2.72% loose AER and 16.45% strict AER, versus base-model 3.83% and 28.36%, and Whisper-large-v3 strict AER of 24.08%.Its loose AER is comparable to Whisper-large-v3 (2.72% vs. 2.59%). The relative gains over the base model are 29.0% for loose AER and 42.0% for strict AER.Source: §6.2.3 · p. 27

7 Additional Experiment Results

Speech-enabled models underperform their text backbones, with spoken OpenQA particularly difficult and full terminology generally helping. Cross-stage analyses show distinct terminology and abbreviation links between recognition, perception, knowledge, and generation, while most responses cover at least one keyword category.

7.1 Perception

  • Although responses containing all three keyword categories remain limited, nearly all responses include at least one, indicating broad coverage of relevant scientific concepts.Perception is measured by keyword coverage across core entities, target concepts, and knowledge anchors; Strict RR requires all three categories simultaneously.Source: §7.1 · p. 27

7.2.1 LLM Performance

  • VITA-Audio and MiMo-Audio lose over 10% accuracy versus their text backbones, while every other evaluated speech-enabled model declines by more than 5%.The comparison evaluates text backbones on textual questions; Figure 3 illustrates the degradation.Source: §7.2.1 · p. 28, Figure 3 · p. 28, Table 17 · p. 28

7.2.2 Task Format Performance

  • Most text backbones exceed 30% on OpenQA, whereas spoken QA accuracy consistently falls below 20%; MCQA degrades more moderately, probably because textual choices provide semantic cues.S3-Knowledge includes both multiple-choice QA and open-ended QA.Source: §7.2.2 · p. 28, Table 18 · p. 28, Table 19 · p. 29

7.2.3 Abbreviations to Full Name

  • Most models improve mildly when spoken acronyms are replaced by full canonical terms, especially on OpenQA, indicating that acronym grounding can bottleneck scientific knowledge retrieval.The comparison contrasts Spoken acronym pronunciations with Academic full terminology forms.Source: §7.2.3 · p. 29, Table 20 · p. 29

7.3 Capability Transfer and Consistency Analysis

  • On terminology questions, perception predicts knowledge correctness (P→K: ρ=0.643, r2=0.41, OR=5.44), but recognition predicts it weakly (ρ=0.179), and knowledge is nearly uncoupled from generation (K→G: ρ=−0.024).Terminology also shows weak recognition–generation coupling (R→G: ρ=0.286, OR=3.89) and negative perception–generation coupling (P→G: ρ=−0.238, OR=0.18); the analysis attributes the broader knowledge–pronunciation tradeoff to structured expressions rather than terminology entities.Source: §7.3 · pp. 29–30, Table 21 · p. 29
  • For abbreviations, recognition and knowledge are tightly coupled (R→K: ρ=0.571, r2=0.36), while recognition–generation coupling is negative (R→G: ρ=−0.250, OR=0.60).Perception also predicts knowledge (P→K: ρ=0.548, r2=0.28), but knowledge–generation coupling is weak (K→G: ρ=0.095, r2=0.13); the passage identifies abbreviation pronunciation as the principal carrier of negative generation coupling.Source: Table 21 · p. 29, §7.3 · p. 30

8 Human Evaluation · 9 Evaluation Prompt

Human evaluation checks synthetic-speech quality and the consistency of LLM-as-a-judge ratings, while the evaluation-prompt section presents prompts for knowledge accuracy, dialogue-agent behavior, and dialogue evaluation.

8 Human Evaluation

  • Human evaluation covered synthetic-speech phonetic quality and review of text outputs to validate LLM-as-a-judge metrics.Five English-fluent university students conducted the evaluation with reference videos and online dictionaries available for pronunciation verification.Source: §8 · p. 30

8.1 Speech Quality

  • Annotators judged 98%, 99%, 100%, and 99% of sampled S3-Knowledge speech phonetically accurate, respectively.Four English-fluent annotators assessed 400 randomly sampled audio instances, using dictionaries and annotated videos as references for technical pronunciations.Source: §8.1 · p. 30

8.2 LLM Judge Consistency

  • Human–LLM agreement on Audience Adaptation was 80%, 83%, 82%, and 79% across four annotators, averaging 81%.For disagreements, the average categorical distance was 1.092, indicating they were predominantly between neighboring academic-background tiers.Source: §8.2 · p. 30

9.1 Knowledge Accuracy Prompt

  • The Knowledge Accuracy prompt section includes a multiple-choice-answer evaluator and a prompt for knowledge-set accuracy.The supplied passages identify the prompt’s task and scope but do not provide its full contents.Source: Figure 4 · p. 31

9.2 Dialogue Agent Prompt

  • The Dialogue Agent Prompt section presents prompts for a user-agent persona, next-step strategy, initial query, and follow-up query.The supplied passages identify these four prompt functions but do not provide their full contents.Source: Figure 5 · p. 31, Figure 6 · p. 32, Figure 7 · p. 32

9.3 Dialogue Evaluation Prompt

  • The Dialogue Evaluation Prompt section includes prompts for evaluating scientific factuality and audience adaptation.The supplied passages identify the two evaluation targets but do not provide the full prompt text.Source: Figure 9 · p. 33, Figure 10 · p. 33

Related papers

Loading 2609.09852v1…