Source-linked AI summary
NVMOS: Non-Verbal Vocalization Quality Assessment in Speech
Jialong Mai, Jinxin Ji, Xiaofen Xing, Wencui Liu, Xiangmin Xu
TL;DR
Non-verbal vocalizations carry emotional and intentional cues, but existing evaluations emphasize controllability or overall speech quality rather than the perceptual quality of local NV events. The paper builds an expert-rated dataset and proposes NVMOS, whose local event-focused design reaches expert-level or stronger agreement with human MOS.
Problem
Existing NV evaluations mainly assess whether an event is present, correctly categorized, and correctly positioned, while local NV perceptual quality remains underexplored.
Source: §1 · p. 1Method
The paper constructs NV-MOS from synthetic and natural speech with expert ratings of marked NV events, and proposes NVMOS to predict their perceptual quality.
Source: §1 · p. 1, §2.2 · p. 2Results
Takeaways & Limitations
With local NV-event focusing, NVMOS is a reliable model for assessing NV quality in speech.
Source: §6 · p. 4General multimodal models are not calibrated for local NV dimensions such as intensity, duration, expressive function, and connection with surrounding speech.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent. Existing speech quality assessment methods typically focus on overall naturalness, while non-verbal TTS evaluations mainly examine whether a target NV appears with the correct type and position. However, the perceptual quality of NV events themselves remains underexplored. To address this gap, we construct an NV-MOS dataset containing outputs from multiple NV-TTS systems and naturally occurring NV samples, with ratings collected from three acoustic experts on a perceptual quality scale. We further analyze audio-capable multimodal large language models such as Gemini and find clear inconsistencies between their scores and expert ratings. These results suggest that general-purpose multimodal models cannot reliably replace human judgments for NV quality assessment. We then propose NVMOS, to our knowledge the first model that can reliably predict the perceptual quality of NV events in speech. Experimental results show that, with a local NV-event focusing module, NVMOS reaches expert-level or stronger agreement with human MOS.
1. INTRODUCTION
NV quality is distinct from controllability and utterance-level naturalness, yet scalable perceptual assessment remains lacking. The paper introduces expert-rated NV-MOS and NVMOS, while finding audio-capable multimodal judges unreliable.
1. INTRODUCTION
- NV evaluations often check presence, category, and position, while subjective MOS tests are difficult to scale and do not establish whether an event sounds natural, expressive, and integrated.Source: §1 · p. 1
- Utterance-level quality methods do not explicitly model local NV quality, obscuring abrupt events or unnatural sounds despite acceptable overall speech.Source: §1 · p. 1
- NV-MOS brings together synthetic NV-TTS outputs and naturally occurring NV samples, each rated by three acoustic experts.Source: §1 · pp. 1–2
- Audio-capable multimodal judges can underestimate high-quality NVs, confuse their presence, misidentify types, and explain judgments inconsistently with human perception.Source: §1 · pp. 1–2
- NVMOS uses a local NV-event focusing module and achieves expert-level or stronger agreement with human MOS.The module emphasizes time regions related to the target NV.Source: §1 · pp. 1–2
2. NV-MOS DATASET
NV-MOS combines synthetic and natural speech, uses expert ratings of marked NV events, and splits samples to limit speaker and synthesis-system leakage.
2.1. Data Sources
- NV-MOS contains 7,784 samples spanning 16 NV categories, including 2,655 synthetic and 5,129 natural samples.The dataset totals about 9.51 hours and is designed to span the NV support range of existing NV-TTS systems.Source: §2.1 · p. 2
2.2. Annotation Protocol
- Experts rate the perceptual quality of the marked NV event, considering its position, naturalness and clarity, and connection to surrounding speech.Source: §2.2 · p. 2
- Ratings use a 0–5 scale, and three PhD-level acoustic experts rate each sample; inconsistent scores are reviewed before their average becomes the continuous label.Predictions are evaluated with error metrics and correlation with expert judgments.Source: §2.2 · p. 2
2.3. Data Split
- The split keeps natural-speech speakers within one set and synthetic samples from the same system together as much as possible, while balancing sources and NV categories.The training, validation, and test sets contain 7,006, 389, and 389 samples, respectively.Source: §2.3 · p. 2
3. ARE MULTIMODAL JUDGES RELIABLE FOR NV QUALITY?
The section tests whether general-purpose audio multimodal judges can assess target NV quality against expert ratings. Even the best judge falls below inter-expert agreement, with manual review identifying errors in judging NV presence and type.
3. ARE MULTIMODAL JUDGES RELIABLE FOR NV QUALITY?
- The LLM protocol scores marked-NV presence, naturalness, clarity, and connection to surrounding speech on a 0–5 scale aligned with expert MOS.Five settings were evaluated: Gemini 2.5 Pro, Gemini 3 Flash, MOSS-Audio 4B, MOSS-Audio 8B, and Qwen-Omni 30B.Source: §3 · p. 2
- 0.468 Pearson correlation for Gemini 3 Flash remains below the 0.589–0.699 range of pairwise expert Pearson correlations.Gemini 3 Flash is the highest-correlating LLM judge among the five evaluated settings.Source: §3 · pp. 2–3, Table 1 · p. 3
- Manual review found judges sometimes score high-quality present NVs low, absent NVs high, or confuse NV categories.Examples include interpreting a sneeze as laughter.Source: §3 · p. 3
- NV quality depends on local event properties and connection to surrounding speech, dimensions for which general multimodal models are not calibrated.The target may be short, low-energy, or ambiguous, and models may rely instead on global speech naturalness, textual semantics, or category priors.Source: §3 · p. 3
4. NVMOS: LOCAL EVENT-FOCUSED NV QUALITY ASSESSMENT
NVMOS predicts the perceptual quality of a marked NV event from audio and text, while accounting for its coherence with surrounding speech. Its text-queried focusing module uses the marked tag to guide attention over frame-level audio features before score regression.
4. NVMOS: LOCAL EVENT-FOCUSED NV QUALITY ASSESSMENT
- NVMOS scores the target NV event rather than the whole utterance, using marked text and audio while considering coherence with surrounding speech.Source: §4 · p. 3
4.1. Text-Queried Local Focusing
- Frame-level WavLM Large or SPEAR representations preserve temporal information so the model can attend to audio regions relevant to the target NV.Source: §4.1 · p. 3
- The model averages hidden states for tokens overlapping the marked NV tag, then projects the query to center it explicitly on that NV.The text encoder is XLM-R Large, used for the dataset’s bilingual English and Chinese text.Source: §4.1 · p. 3
- Cross-attention uses the text query and frame-level audio features to focus on frames useful for judging the NV and its connection to neighboring speech.After two layers, the output states are averaged and passed to a feed-forward regression head trained with robust regression loss against expert MOS.Source: §4.1 · p. 3
5. EXPERIMENTS
NVMOS is evaluated on speech representations using a text-queried local focusing module, and its test predictions agree strongly with expert ratings. Ablations show that focusing the query on the marked NV improves over generic full-sentence queries.
5.1. Experimental Setup
- Evaluation uses train, validation, and test splits, with Pearson correlation primary and Spearman, Kendall’s tau-b, and MAE also reported; marked text guides cross-attention over frame-level speech representations.The text query supports local NV quality prediction.Source: §5.1 · p. 3, Figure 1 · p. 4
5.2. Main Results
- 0.697 test Pearson with WavLM Large layer 7 is comparable to inter-expert agreement; SPEAR Large layer 9 reaches 0.690, with stronger test agreement than pairwise experts across three correlations.Table 2 reports both representation results.Source: Table 2 · p. 4, §5.2 · p. 4
5.3. Ablation Study
- Full-sentence queries add little over audio-only, whether mean-pooled or kept token-level, whereas the tag-context query improves correlation metrics and score calibration.The ablation compares systems using the same frame-level audio features, training split, and cross-attention scorer.Source: Table 3 · p. 4, §5.3 · p. 4
6. CONCLUSION
The paper presents an expert-rated dataset and a text-queried cross-attention framework for perceptual NV quality assessment. It reports expert-level or stronger agreement with human MOS when NVMOS uses local NV-event focusing.
6. CONCLUSION
- With local NV-event focusing, NVMOS achieves expert-level or stronger agreement with human MOS and is described as reliable for assessing NV quality in speech.The conclusion identifies the local focusing module as part of this result.Source: §6 · p. 4
- General audio-capable LLM judges remain poorly aligned with expert NV quality ratings, motivating a dedicated task-specific predictor.Source: §6 · p. 4
- The paper presents NV-MOS, an expert-rated dataset and modeling framework for perceptual quality assessment of non-verbal vocalizations in speech.Source: §6 · p. 4