PaperlayerView exact revision on arXiv

Source-linked AI summary

VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing

Ke Wang, Houxing Ren, Zimu Lu, Mingjie Zhan, Hongsheng Li

arXiv:2509.22651v1cs.CLcs.AIcs.CVcs.HCcs.SD

TL;DR

Existing benchmarks cover only partial aspects of voice assistants’ listening, speaking, and viewing abilities. VoiceAssistant-Eval evaluates these abilities jointly, and its results show that most models perform better at speaking than listening, while smaller specialized models can outperform larger ones on some tasks.

  • Problem

    Existing benchmarks cover only partial aspects of voice assistants’ listening, speaking, and viewing abilities.

    Source: §1 · pp. 2–3

  • Method

    VoiceAssistant-Eval jointly evaluates listening, speaking, and viewing across 10,497 curated instances in 13 subsets.

    Source: §1 · pp. 2–3

  • Results

    20 of 22 models scored higher on Speaking than Listening, and smaller, specialized models can outperform larger models on some tasks.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Rich audio understanding, multimodal integration, and balancing response fidelity with voice naturalness in role-play remain challenges for current models.

    Source: §5 · p. 10

    The evaluation relies on automated measures and GPT-based scoring, which may introduce noise and bias.

    Source: A LIMITATIONS AND FUTURE WORK · p. 21

Abstract

from arXiv · show

The growing capabilities of large language models and multimodal systems have spurred interest in voice-first AI assistants, yet existing benchmarks are inadequate for evaluating the full range of these systems' capabilities. We introduce VoiceAssistant-Eval, a comprehensive benchmark designed to assess AI assistants across listening, speaking, and viewing. VoiceAssistant-Eval comprises 10,497 curated examples spanning 13 task categories. These tasks include natural sounds, music, and spoken dialogue for listening; multi-turn dialogue, role-play imitation, and various scenarios for speaking; and highly heterogeneous images for viewing. To demonstrate its utility, we evaluate 21 open-source models and GPT-4o-Audio, measuring the quality of the response content and speech, as well as their consistency. The results reveal three key findings: (1) proprietary models do not universally outperform open-source models; (2) most models excel at speaking tasks but lag in audio understanding; and (3) well-designed smaller models can rival much larger ones. Notably, the mid-sized Step-Audio-2-mini (7B) achieves more than double the listening accuracy of LLaMA-Omni2-32B-Bilingual. However, challenges remain: multimodal (audio plus visual) input and role-play voice imitation tasks are difficult for current models, and significant gaps persist in robustness and safety alignment. VoiceAssistant-Eval identifies these gaps and establishes a rigorous framework for evaluating and guiding the development of next-generation AI assistants. Code and data will be released at https://mathllm.github.io/VoiceAssistantEval/ .

1 INTRODUCTION

VoiceAssistant-Eval addresses gaps in existing benchmarks by jointly evaluating AI assistants’ listening, speaking, and viewing abilities. Its evaluation finds that model strengths and weaknesses vary across these capabilities and tasks.

1 INTRODUCTION

  • Step-Audio-2-mini achieves the highest overall score and listening accuracy of 40.06 vs. 16.00 for 32B LLaMA-Omni2.The paper reports that the 7B model achieves more than double the listening accuracy of LLaMA-Omni2-32B-Bilingual.Source: Figure 1 · p. 2, §1 · p. 3
  • Existing benchmarks inadequately assess voice personalization, hands-free interaction, varied everyday audio contexts, and joint vision-audio understanding.The paper connects these omissions to personalized interaction, safety-critical and accessibility contexts, everyday reliability, and multimodal applications.Source: §1 · p. 2
  • VoiceAssistant-Eval contains 10,497 curated instances across 13 subsets and jointly tests listening, speaking, and viewing, covering task combinations missing from prior benchmarks.Its representative tasks include personalized voice imitation, natural hands-free audio interaction, vision-audio understanding, and audio QA with audio context.Source: §1 · pp. 2–3, Table 1 · p. 3
  • The benchmark draws visual and auditory inputs from 37 datasets, curates role-play speech for 100 roles, and balances examples across 13 task categories.Curation includes deduplication and filtering; approximately 10% of candidates are removed for issues such as missing media, ill-posed questions, or incorrect answers.Source: §1 · p. 3
  • GPT-4o-Audio fails to surpass open-source models in 4 of 13 tasks, while 20 of 22 models score higher on Speaking than on Listening.The paper identifies the mismatch as evidence of a need for more balanced development.Source: §1 · p. 3
  • Step-Audio scores 33.2 for content and 75.1 for speaker similarity but 56.0 for Speech Naturalness; Qwen2.5-Omni-7B drops 16.3 points on image+audio versus image+text queries.The paper describes role-play and vision-audio integration as challenging for current models.Source: §1 · p. 3
  • Some smaller or unaligned models, including the Moshika family, have robustness below 1 and safety below 28.The paper identifies safety alignment and robustness as areas requiring further improvement.Source: §1 · p. 3

2 RELATED WORKS

Existing evaluations cover speech interaction, audio and vision understanding, and multimodal models, but leave richer audio–visual contexts and complex multi-turn multimodal instructions untested. VoiceAssistant-Eval addresses these gaps across listening, speaking, and viewing.

Speech Interaction Benchmarks

  • VoiceBench (Chen et al., 2024a) scores text only, while other speech benchmarks omit audio/visual context and voice imitation; VoiceAssistant-Eval covers listening, speaking, viewing, and these capabilities.SOVA-Bench also uses text instructions, limiting hands-free evaluation.Source: §2 · p. 4

Audio and Vision Understanding Benchmarks

  • Audio, music, and vision benchmarks follow audio/image-in, text-out paradigms, whereas VoiceAssistant-Eval is presented as the first to evaluate models in rich audio–visual contexts.Source: §2 · p. 4

MultiModal Large Language Models

  • Audio models span speech-focused, sound-and-music, general-audio, and omni-modal systems, but complex multi-turn multimodal instruction performance remains untested.Vision multimodal LLMs show strong perception but are tested on narrow tasks.Source: §2 · p. 4

3 DATASET

VoiceAssistant-Eval is a voice-first benchmark built to test personalized speech, diverse listening conditions, and audio-visual integration. Its data are synthesized and curated into 10,497 high-quality samples across 13 task categories.

3.1 DATA COLLECTION

  • The design includes speaker-voice imitation, spoken prompts and responses, realistic diverse audio contexts, and tasks combining auditory data with visual inputs.These choices target voice personalization, hands-free interaction, challenging acoustic scenes, and joint speech-and-vision processing.Source: §3.1 · pp. 4–5
  • Listening combines sound, music, speech, and general audio from 23 datasets; Speaking defines eight voice-driven subtasks; Viewing uses MMMU images spanning 30 disciplines and 30 formats.Listening sources comprise 10 speech, 6 sound, 4 music, and 3 general datasets; role-play uses RoleBench profiles and questions (Wang et al., 2023c).Source: §3.1 · p. 5

3.2 SPEECH SYNTHESIS

  • Three TTS models synthesize question audio, which is preprocessed, screened with UTMOS Strong above 3.8, and selected by lowest Whisper-Large-v3 word error rate, yielding 13,471 files.Preprocessing replaces image tokens with descriptions, rewrites text for accurate speech, and rephrases multiple-choice options for readability.Source: §3.2 · pp. 5–6

3.3 DATA CURATION

  • After deduplication, validation, categorization, and task balancing, curation yields 10,497 high-quality samples across 13 categories.Deduplication leaves 15,359 samples; approximately 10% of candidates are then discarded during validation.Source: §3.3 · p. 6

4 EXPERIMENTS

Across the benchmark, models generally perform better on speaking than listening, but strong results from mid-sized models and persistent role-play, multimodal, safety, and robustness challenges complicate that pattern. Error analysis further identifies distinct failure modes across listening, speaking, and viewing.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers 22 models, including six that accept audio and visual inputs, and scores response content, speech quality, and text-speech consistency.The final task score multiplies the three scores and converts the result to a percentage; role-play additionally incorporates speaker similarity measured with Wespeaker.Source: §4.1 · pp. 6–7, §4 · p. 6
  • The model set spans small, medium-scale, and large open-source models, alongside proprietary GPT-4o-Audio.The benchmark includes seven small models, eleven medium-scale models, and two large models; the paper provides evaluation hyperparameters in Appendix K.1.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS

  • GPT-4o-Audio fails to outperform open-source models on 4 of 13 tasks, including Listening Sound and Speech, yet leads the best open-source model by 16.1 points on Speaking Robustness.GPT-4o-Audio scores 47.7 vs. Step-Audio-2-mini’s 52.0 in Listening Sound, 37.4 vs. 46.5 in Listening Speech, and 54.7 vs. Qwen2.5-Omni-7B’s 38.6 in Speaking Robustness.Source: §4.2 · p. 7
  • 20 of 22 models score higher on Speaking than Listening, while Step-Audio-2-mini scores 40.06 on Listening and 31.30 on Speaking and achieves the highest overall score.Step-Audio-2-mini’s listening accuracy is more than double LLaMA-Omni2-32B’s (40.06 vs. 16.00).Source: §4.2 · pp. 7–8, Table 4 · p. 8
  • On identical image questions, all models perform worse with spoken than written input; Qwen2.5-Omni-7B drops 16.3 points, from 59.2% to 42.9%.The results point to limited audio-visual integration and motivate improved multimodal architectures or more integrated training data.Source: Figure 3 · p. 8, §4.2 · pp. 8–9
  • Role-play results expose a tradeoff: Step-Audio leads in content (33.2) and speaker similarity (75.1), while Step-Audio-2-mini has higher speech naturalness (76.0 vs. 56.0).The results indicate that content accuracy and audio fidelity need to be balanced in role-play output.Source: §4.2 · p. 8, Table 5 · p. 8
  • Safety and robustness vary widely: Moshika models score below 1 in robustness and below 28 in safety, while Freeze-Omni scores 79.8 in safety.The passage attributes improved performance to explicit alignment training and diverse data exposure.Source: §4.2 · p. 9

4.3 HUMAN EVALUATION FOR ERROR ANALYSIS

  • For Qwen2.5-Omni-7B, analysis of 593 errors finds listening failures tied mainly to context loss, speaking failures to incomplete or divergent responses, and viewing failures to visual misinterpretation.Listening context loss accounts for 46% of errors; speaking errors include insufficient answers (25%) and requirement deviation (23%); vision perception errors make up 50% of viewing errors.Source: §4.3 · p. 9

5 CONCLUSION

VoiceAssistant-Eval evaluates voice assistants across listening, speaking, and viewing, revealing strengths in fluent speech and straightforward conversation but persistent difficulties with rich audio understanding and multimodal integration. The findings point to improving audio perception, multimodal understanding, safety, and robustness.

5 CONCLUSION

  • Most models perform better on speaking than listening, and joint audio–image queries score significantly lower than text–image queries.Smaller specialized models can outperform larger models on certain tasks.Source: §5 · p. 10
  • Current models generate fluent speech and handle straightforward conversational queries but struggle with rich audio understanding and multimodal integration.The benchmark systematically evaluates integrated listening, speaking, and viewing tasks.Source: §5 · p. 10
  • Role-play remains challenging because balancing response fidelity with voice naturalness is difficult.The conclusion recommends refining training and alignment methods to improve safety and robustness.Source: §5 · p. 10
  • Future research should enhance audio perception and memory, strengthen multimodal understanding, and improve safety and robustness.The stated aim is to close the gap between written and spoken inputs.Source: §5 · p. 10

6 REPRODUCIBILITY STATEMENT · APPENDIX

The paper supports reproducibility by promising to release the dataset and evaluation code and documenting its models, prompts, data sources, curation, metrics, and reliability checks.

6 REPRODUCIBILITY STATEMENT

  • The authors will release the dataset and evaluation code used in the work.Source: §6 · p. 10
  • Appendix K.1 documents evaluated models, including timestamps where available and generation configurations.Source: §6 · p. 10
  • All evaluation prompts are listed in Appendix K.2.Source: §6 · p. 10
  • Data sources are specified in Section 3.1, with further details in Table 8, and the curation tool is described in Appendix J.Source: §6 · p. 10
  • Evaluation metrics, protocols, and relevant equations are outlined in Section 4.Source: §6 · p. 10
  • Reliability support includes human-evaluation validation in Appendix B, evaluation-stability results in Appendix C, and statistical-significance testing in Appendix D.Source: §6 · p. 10

A LIMITATIONS AND FUTURE WORK

VoiceAssistant-Eval is presented as a comprehensive, reliable framework, while its current boundaries motivate broader data, improved metrics, and more realistic challenges.

Dataset diversity

  • Although the dataset contains 10.5K curated QA items and five Chinese role-play roles, other prompts and evaluations are English, limiting diversity and generalization.Future iterations could add non-English languages to better reflect global usage.Source: A LIMITATIONS AND FUTURE WORK · p. 21

Model coverage

  • The study evaluates primarily open-source models and only GPT-4o-Audio among proprietary systems, offering a narrow comparison that future work could broaden.The authors invite the community to extend benchmarking to more models.Source: A LIMITATIONS AND FUTURE WORK · p. 21

Evaluation methodology

  • Automated measures and GPT-based scoring may introduce noise or bias, but validation showed strong alignment with human judgments and statistically significant conclusions.Future work could refine evaluation with more robust automatic metrics.Source: A LIMITATIONS AND FUTURE WORK · p. 21

Bias and realism

  • Academic or synthetic source data and a fixed set of 100 role-play personas may encode stereotypes or omit users, motivating more realistic, fairness-focused tests.The authors encourage analysis of model biases and unsafe behavior, along with additional ethical test cases.Source: A LIMITATIONS AND FUTURE WORK · p. 21

Task coverage and realism

  • The 13-category benchmark omits video-audio integration and continuous audio-visual streaming, so future versions should add dynamic tasks that test memory and adaptability.The authors warn that controlled evaluations may overestimate capabilities on messy, continuous real-world input.Source: A LIMITATIONS AND FUTURE WORK · p. 21

Interactive evaluation

  • Offline evaluation leaves latency, partial-utterance handling, barge-in, and error recovery unmeasured; future work could test real-time conversations or user-in-the-loop feedback.Real-world assistants must handle continuous listening and incremental speech processing.Source: A LIMITATIONS AND FUTURE WORK · p. 21

B HUMAN EVALUATION VALIDATION

The automated evaluation scheme closely aligns with human judgments, with high overall correlation and agreement across listening and speaking tasks. Human judges independently evaluated a sample of Qwen2.5-Omni-7B outputs to validate the scheme.

B HUMAN EVALUATION VALIDATION

  • Human judges independently evaluated 1,050 randomly sampled responses, representing 10% of Qwen2.5-Omni-7B’s outputs from each of 13 tasks (Xu et al., 2025).The sample was used to verify whether the automatic evaluation scheme aligned with human preferences.Source: B HUMAN EVALUATION VALIDATION · p. 22
  • Overall Pearson correlation was 0.92 and overall agreement was 96.1%, showing close alignment between automated evaluations and human judgments.Correlations ranged from 0.73 to 1.00 across tasks; agreement rates were often above 90%, with several categories reaching 100%.Source: B HUMAN EVALUATION VALIDATION · p. 22
  • Correlation was lowest for robustness tests (RBT) at 0.73, while several categories reached a perfect correlation of 1.00.The content-quality metric derived from gpt-oss-20b reflected human perceptions across listening and speaking tasks.Source: B HUMAN EVALUATION VALIDATION · p. 22

C STABILITY OF THE EVALUATION

Repeated runs produced highly consistent evaluation scores across tasks, supporting the stability and reproducibility of the automated evaluation.

C STABILITY OF THE EVALUATION

  • The analysis repeated scoring ten times on a random 10% sample of Qwen2.5-Omni-7B responses from each of 13 tasks, totaling 1,050 examples.Source: C STABILITY OF THE EVALUATION · p. 22
  • Across ten runs, Listening Sound scored 61.5 (variance = 0, standard deviation = 0.0), while Speaking Robustness ranged from 44.8 to 51.0 (mean = 47.8, standard deviation = 2.3).The boxplots show narrow interquartile ranges across tasks.Source: C STABILITY OF THE EVALUATION · p. 22, Figure 5 · p. 23
  • Scores were nearly identical across tasks, with standard deviations mostly below 3, indicating that the automated evaluations were reproducible and reliable.Source: C STABILITY OF THE EVALUATION · p. 22

D STATISTICAL SIGNIFICANCE TESTING · E SOURCE DATASETS · F DATASET INTENDED USAGE AND LICENSE

Statistical testing confirms a much larger listening advantage for Qwen2.5-Omni-7B than its modest, mixed speaking differences against LLaMA-Omni2-32B-Bilingual. The appendices describe source-dataset coverage and document intended use, licensing, and safeguards for voice-imitation research.

D STATISTICAL SIGNIFICANCE TESTING

  • The comparison used ten repeated scoring runs on a random 10% sample of each of 13 tasks (1,050 examples), paired two-sided t-tests, and Bonferroni correction at p < 0.0042.For each metric, the authors report the runs’ mean and standard deviation and test matched score distributions.Source: D STATISTICAL SIGNIFICANCE TESTING · p. 23
  • The corrected tests support reliable listening advantages, while speaking differences are smaller and mixed in direction; Listening Sound’s constant score across runs yielded p < 10−10.The passage interprets these results as showing that smaller, carefully trained models can rival or surpass larger ones on some tasks, while large models tend to perform better on speaking tasks.Source: D STATISTICAL SIGNIFICANCE TESTING · pp. 23–24
  • 40.5 ± 0.6 vs. 19.5 ± 0.6 average listening accuracy: Qwen2.5-Omni-7B gains +21.0 points over LLaMA-Omni2-32B-Bilingual, with every listening task significantly improved (p < 0.001).Listening Sound, Music, and General gains were +51.5, +19.6, and +9.0 points, respectively; all were significant after correction.Source: D STATISTICAL SIGNIFICANCE TESTING · p. 23, Table 7 · p. 24

E SOURCE DATASETS

  • The source-dataset overview lists the number of VoiceAssistant-Eval samples selected from each dataset and describes coverage across varied tasks and modalities.The table caption specifies that it overviews the included source datasets and their evaluation-sample counts; no specific counts are given in the supplied text.Source: Table 8 · p. 24, E SOURCE DATASETS · p. 24

F DATASET INTENDED USAGE AND LICENSE

  • VoiceAssistant-Eval and reproduction code are intended for evaluating voice assistants, not training models to cheat, and the dataset and code are under an MIT license.The authors state that they bear responsibility for rights violations.Source: F DATASET INTENDED USAGE AND LICENSE · pp. 24–25
  • For role-play voice imitation and speaker-similarity evaluation, the authors specify consent, provenance, and anti-impersonation safeguards.They prohibit releasing or training models to reproduce identifiable speakers without documented consent, require downstream consent gating and watermarking or traceability, provide safety cards, and maintain audit and asset-removal processes.Source: F DATASET INTENDED USAGE AND LICENSE · p. 25

G DETAILED ASSESSMENT OF MODEL CAPABILITIES … I.2 AUDIO AND VISION UNDERSTANDING BENCHMARKS.

The assessment finds that model strengths are uneven: GPT-4o-Audio leads on some content tasks, while open-source models often produce more natural speech, and specialization and scale affect performance differently. The related-work sections describe benchmarks spanning speech interaction, audio understanding, and multimodal visual reasoning.

G DETAILED ASSESSMENT OF MODEL CAPABILITIES

  • Step-Audio-2-mini pairs speech quality scores often reaching 90 with Speaking Assistant consistency of 77.9, while Qwen2.5-Omni variants often score in the high 90s for consistency and mid-80s for speech quality.The results indicate a trade-off between naturalness and fidelity to intended text.Source: G DETAILED ASSESSMENT OF MODEL… · p. 25
  • GPT-4o-Audio scores 90.2 in Speaking Assistant content, 24.9 points above MiniCPM-o-2_6, but its speech naturalness score is 76 versus about 90 for Step-Audio-2-mini.GPT-4o-Audio’s consistency score is 91.4, 5.5 points below Qwen2.5-Omni’s 96.9; open-source models’ speech scores are typically higher across tasks.Source: G DETAILED ASSESSMENT OF MODEL… · p. 25, Table 9 · p. 26, Table 10 · p. 27
  • Task strengths vary: audio specialists lead on listening, GPT-4o leads on multi-turn, reasoning, and instruction-following, and larger models do not automatically produce better speech.GPT-4o trails the best open models by 2.9 points on listening sound and 10.1 on listening speech; a 130B Step-Audio model has worse speech quality than Step-Audio-2-mini. The findings motivate routing audio perception and dialogue to specialized models, and targeted audio training for voice quality.Source: G DETAILED ASSESSMENT OF MODEL… · p. 26

H USAGE OF LARGE LANGUAGE MODELS IN PAPER WRITING

I.1 SPEECH INTERACTION BENCHMARKS.

  • Speech-interaction benchmarks assess complementary abilities: VoiceBench (Chen et al., 2024a) covers spoken instructions, while VocalBench (Liu et al., 2025) and SOVA-Bench (Hou et al., 2025) include vocal quality and conversational skills.SD-Eval (Ao et al., 2024) targets paralinguistic speech understanding, and WildSpeech-Bench (Zhang et al., 2025) tests natural multi-turn dialogue and speech-specific phenomena; VoiceBench and WildSpeech-Bench evaluate textual content rather than synthesized speech acoustics.Source: §I.1 · pp. 27–29

I.2 AUDIO AND VISION UNDERSTANDING BENCHMARKS.

  • LibriSpeech, Common Voice (Ardila et al., 2020), Clotho (Drossos et al., 2020), and AudioCaps support speech recognition or audio captioning, while AVQA and MUSIC-AVQA (li et al., 2022) require audio-visual reasoning.The audio-caption datasets are limited by moderate size and domain bias; the audiovisual QA datasets focus on short clips in everyday or music domains.Source: §I.2 · p. 29
  • SUPERB (Yang et al., 2021) evaluates speech representations across discriminative tasks, HEAR (Baur et al., 2024) tests embeddings on 19 downstream tasks without fine-tuning, and AIR-Bench (Yang et al., 2024) evaluates generated answers to audio queries.AIR-Bench includes about 19K multiple-choice questions and 2K open-ended QA pairs, but scores textual output rather than synthesized speech.Source: §I.2 · pp. 29–30
  • MMMU (Yue et al., 2024) tests multidisciplinary multimodal reasoning, while MathVista (Lu et al., 2024) and MATH-Vision (Wang et al., 2024a) target visual mathematical reasoning, where current models remain unreliable.MMMU contains 11.5K questions and GPT-4V and Gemini Ultra score around 56–59%; MathVista’s best accuracy is about 50% versus about 60% for humans.Source: §I.2 · p. 30

J DATA CURATION TOOL … L.7 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING INSTRUCTION FOLLOWING TASK

These appendices describe standardized curation and evaluation procedures, then illustrate Qwen2.5-Omni-7B errors in listening, speaking, and instruction following.

J DATA CURATION TOOL

  • The curation tool standardizes item review across modalities, with curators checking media integrity, task clarity, and answer validity.It records Good/Bad decisions and timestamps, exports approved items to clean JSONL shards, and the review process discards about 10% of candidates; figures show listening, speaking, role-play, and viewing interfaces.Source: J DATA CURATION TOOL · p. 30, Figure 6 · p. 31, Figure 7 · p. 32

K.1 MODEL SETUP; K.2 EVALUATION PROMPTS

  • The experiment-details appendix summarizes model-generation configurations and presents 13 task-specific evaluator prompts.The prompts cover listening, speaking, and viewing dimensions such as correctness, helpfulness, empathy, and safety; instruction-following judgments assess literal adherence, while safety judgments assess refusal or safe redirection.Source: Table 12 · p. 35, §K.2 · p. 35, Figure 16 · p. 39

L.1 ERROR CASES OF QWEN2.5-OMNI-7B ON THE LISTENING GENERAL TASK

  • Listening-general examples show knowledge errors, failure to follow constraints, and speech-perception mistakes.The model gives an incorrect location name, mentions specific sounds despite an instruction not to, and confuses “isodesmic” with “isometric.”Source: Figure 23 · p. 44, §L.1 · p. 45, Figure 24 · p. 45

L.2 ERROR CASES OF QWEN2.5-OMNI-7B ON THE LISTENING MUSIC TASK

  • Listening-music examples show context loss and mistaken genre or mood classification.The model asks for more details despite supplied audio, infers Experimental instead of Blues, and selects relaxing instead of Christmas.Source: Figure 26 · p. 47, Figure 27 · p. 48, Figure 28 · p. 49

L.3 LISTENING SOUND; L.4 LISTENING SPEECH

  • Listening-sound and speech examples feature sound misclassification, equivocal answers, and failures to use audio context.Examples include confusing a sigh with a cough, giving no definite recording location, and saying it cannot listen when asked to identify a speaker’s gender or language.Source: Figure 29 · p. 50, Figure 30 · p. 51, Figure 31 · p. 52

L.5 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING ASSISTANT TASK

  • Speaking-assistant examples show vague or unsupported answers, inaccurate recommendations, and misinterpretation of the requested task.One reply omits requested reasoning and evidence; another lists potentially irrelevant brands, while a third asks for clarification instead of completing a company description.Source: Figure 36 · p. 57, Figure 37 · p. 58, Figure 38 · p. 59

L.6 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING EMOTION TASK

  • Speaking-emotion examples show insufficient empathy and responses that fail to match the user’s expressed emotion.The assistant gives advice without acknowledging anger, and responds neutrally and happily to an utterance conveying fear.Source: Figure 39 · p. 60, Figure 40 · p. 61

L.7 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING INSTRUCTION FOLLOWING TASK

  • Speaking instruction-following examples show failures to satisfy requested actions, exact format constraints, and resource limits.The assistant lists teaching ideas without starting the class, produces six rather than five sentences, and recommends online resources despite a no-online-access constraint.Source: Figure 41 · p. 62, Figure 42 · p. 63, Figure 43 · p. 64

L.8 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING MULTI ROUND TASK … L.12 ERROR CASES OF QWEN2.5-OMNI-7B ON THE VIEWING MULTI DISCIPLINE TASK

The examples show Qwen2.5-Omni-7B making speech-perception, knowledge, reasoning, instruction-following, and safety errors, as well as misreading images in viewing tasks.

L.8 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING MULTI ROUND TASK

  • In a multi-round speaking example, the model misinterprets monetary values and omits a spoken number, producing an incorrect answer.Other examples in this task show a mistaken synonym and an incorrect physics explanation.Source: §L.8 · p. 65, Figure 44 · p. 65, Figure 45 · p. 66

L.9 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING REASONING TASK

  • Speaking-reasoning examples show incorrect answers involving the Emancipation Proclamation’s global impact, navigation instructions, and a normal-force question.The model overgeneralizes the Proclamation’s impact, answers “No” when the path returns to its start, and chooses B instead of C for the normal-force question.Source: Figure 47 · p. 68, Figure 48 · p. 69, Figure 49 · p. 70

L.10 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING ROBUSTNESS TASK

  • On speaking robustness examples, the model drifts from the user’s statement about council membership and gives generic, inaccurate commentary on a transcript.In the council example, it presumes the user intends to join and asks why they decided to join.Source: Figure 50 · p. 71, Figure 51 · p. 72

L.11 ERROR CASES OF QWEN2.5-OMNI-7B ON THE SPEAKING SAFETY TASK

  • Speaking-safety examples show the model speculating about a resident’s identity, supplying insults, and giving terrorism-related details instead of refusing.The examples are judged unsafe or bad, while the references refuse and redirect.Source: Figure 52 · p. 73, Figure 53 · p. 74, Figure 54 · p. 75

L.12 ERROR CASES OF QWEN2.5-OMNI-7B ON THE VIEWING MULTI DISCIPLINE TASK

  • In a viewing example, the model misreads a river-width diagram and answers 123 m against the reference answer of 120 m; other examples show image-based errors in labor-choice and heat-transfer questions.The river-width example’s plausible reasoning uses an incorrect similar-triangles setup.Source: §L.12 · p. 76, Figure 55 · p. 76, Figure 56 · p. 77
  • Further viewing examples show a refusal to compute a bond value, misinterpretation of a circuit diagram, and a color-wheel answer that contradicts the model’s own calculation.The circuit response selects B instead of reference C, while the color-wheel response computes 0.25 but selects A (0.125).Source: Figure 58 · p. 79, Figure 59 · p. 80, Figure 60 · p. 81
Loading 2509.22651v1…