Source-linked AI summary
On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Chan-Jan Hsu, Liang-Hsuan Tseng, Yi-Cheng Lin, Yen-Chun Kuo, Ju-Chieh Chou, Kai-Wei Chang, Hung-yi Lee, Carlos Busso
TL;DR
Global token perplexity may overlook speech-specific characteristics and misalign with human judgments. The paper introduces likelihood- and generation-based evaluations emphasizing local context, finding stronger MOS alignment and a reshaped performance landscape. These evaluations reduce the apparent gap between leading spoken language models and the human topline, while benchmark scope and speech dimensions remain important boundaries.
Problem
Global token perplexity may overlook speech-specific characteristics and misalign with human perception when evaluating spoken language model quality.
Method
The paper introduces localized and normalized likelihood-based evaluations alongside generation-based methods that emphasize local context sensitivity.
Results
The proposed evaluations correlate more closely with MOS and reshape spoken language model rankings, with the best-performing model closing 83% of the gap to the human topline on SALMon.
Takeaways & Limitations
Appropriate evaluation can substantially change conclusions about spoken language model progress and better reflect human-perceived continuation quality.
Takeaways & Limitations
The evaluation scope remains constrained by existing benchmarks, and SALMon does not systematically test compounded variations such as speaker changes under noisy background conditions.
Abstract
from arXiv · showhide
Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving attributes like speaker and emotion, serving as foundation models for spoken dialogue. In prior literature, these models are often evaluated using ``global token perplexity'', which directly applies the text perplexity formulation to speech tokens. However, this practice overlooks fundamental differences between speech and text modalities, possibly leading to an underestimation of the speech characteristics. In this work, we propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity. We demonstrate that the proposed evaluations more faithfully reflect perceived generation quality, as evidenced by stronger correlations with human-rated mean opinion scores (MOS). When assessed under the new metrics, the relative performance landscape of spoken language models is reshaped, revealing a significantly reduced gap between the best-performing model and the human topline. Together, these results suggest that appropriate evaluation is critical for accurately assessing progress in spoken language modeling.
1 Introduction
Spoken language models are commonly evaluated with global token perplexity, but speech-specific temporal and acoustic properties can make this measure misaligned with human perception. The paper proposes localized and alternative evaluations that better capture continuation quality and substantially reshape model comparisons.
- Speech generation quality is difficult to evaluate because judging whether outputs are plausible or coherent is inherently subjective.
- Global token perplexity applies sequence-level likelihood scoring to discrete speech tokens, extending a text-modeling practice to spoken language models.
- Speech acoustics favor short-span conditioning, while text coherence relies on long-range dependencies; positive–negative likelihood differences therefore concentrate near divergence onset.
- The proposed likelihood- and generation-based evaluations emphasize local context sensitivity and yield substantially different conclusions from naive global token perplexity.
- Pearson correlation with human MOS increases from 0.64 to 0.8, while Spearman correlation increases from 0.67 to 0.8 under the proposed methods.
- Under the revised evaluation, the best-performing model closes 83% of the gap to the human topline on SALMon and achieves a new SOTA.
2 Background
Global token perplexity evaluates speech-token sequences through a holistic likelihood estimate, but this formulation treats tokens across channels and time equally. SALMon compares shared-prefix positive and negative continuations using perplexity or equivalent NLL rankings.
- Likelihood modeling evaluates sequence models through perplexity, where higher perplexity indicates lower likelihood and PPL(s) = exp(NLL(s)).
- Global token perplexity computes NLL over sequences of discrete speech tokens and reports the result in NLL terms.
- Multi-channel speech outputs are flattened into one serialized stream, treating tokens across channels and time steps as equally informative.
- In SALMon, each positive–negative pair shares a prefix, and a perception-faithful model should assign lower perplexity, equivalently lower NLL, to the positive sequence.
- Calibration methods can preserve rankings or alter model selection; this work extends calibration to speech through localization and normalization methods.
3 Proposed Method
The paper replaces global token perplexity with likelihood-based variants that emphasize local acoustic discontinuities and generation-based evaluations of SLM continuations. These methods use shared prompts and contrastive responses, human ratings, or qualified embedding judges to assess acoustic consistency.
- Global token perplexity can be disproportionately influenced by semantic factors, limiting its ability to express acoustic features.
- Localized and normalized perplexity variants focus evaluation on the target attribute distinguishing positive and negative speech samples.
- Windowed perplexity slides a short temporal window over the sequence and uses the maximum perplexity as a local anomaly measure.
- Each SALMon pair is decomposed into a shared prompt S and positive and negative responses P and N for localized comparison.
- Generation-based evaluation samples continuations from a speech prompt, enabling MOS collection and model-as-a-judge comparisons.
- Embedding judges are selected using labeled contrastive pairs and retained when their development-set accuracy exceeds the human qualification threshold.
4 Experimental Setup
Experiments evaluate SLMs on SALMon acoustic-consistency subsets using likelihood estimators, human MOS ratings, and embedding-based judges. The setup compares multiple models and evaluators, including 450 human-rated generations.
- SALMon contains six acoustic-consistency subsets covering gender, speaker identity, sentiment, background conditions, and room attributes.
- Each SALMon data point pairs a positive sample with a negative sample containing an inconsistency in one speech attribute.
- The evaluation covers GSLM, TWIST, pGSLM, Spirit-LM, TASTE, Flow-SLM, and Llama-Mimi.
- Figure 3 compares alternative likelihood estimators, MOS, and embedding-as-a-judge scores with Global-PPL, whose horizontal spread indicates evaluator disagreement.
- 450 generations from 9 models were independently rated by five proficient English-speaking annotators on a five-point Likert scale.
- Embedding-based judges use diverse models, including TITANET, CAM++, CLAP, and AudioSet-trained models, selected on SALMon development prompts.
5 Experimental Results
Experiments compare likelihood- and generation-based evaluators against global token perplexity on SALMon. Alternative metrics better align with human MOS, alter model rankings, and reveal tokenizer- and model-specific differences in continuation quality.
- Likelihood-based evaluation: Global token perplexity disagrees systematically with alternative likelihood methods, with deviation rising from +6.63% at 50% accuracy to +8.62% at 80%.The bias is usually one-sided for each SLM configuration and depends strongly on token type: proposed methods score HuBERT-based models lower and Mimi-based models higher.
- Experimental setup: The evaluation compares likelihood-based estimators, generation-based judge scoring, MOS ratings, and model consistency across SALMon.Experiments include actual model continuations and correlations between evaluator scores and human ratings.
- Human evaluation: Llama-Mimi achieves the highest MOS at 3.29, followed by Flow-SLM, while HuBERT-token models perform substantially worse.Stronger models mainly improve speech-centric attributes such as sentiment, speaker identity, and gender; ambience remains difficult.
- Generation-based evaluation: Model-as-a-judge results place Flow-SLM and Llama-Mimi near the human topline, whereas most HuBERT-based models perform close to random choice.TASTE performs relatively well, supporting the importance of speaker vectors during token-to-speech conversion; continuation failures principally arise from information lost during speech encoding.
- Correlation with MOS: Evaluator agreement with MOS improves from Kendall τ = 0.444 to 0.480 with normalized perplexity, 0.502 with windowed perplexity, and 0.674 with model-as-a-judge scoring.Across acoustic-consistency tasks, normalized and windowed likelihood metrics and model-as-a-judge methods produce stronger rankings than the baseline.
- Reshaped performance landscape: Normalized perplexity raises Llama-Mimi from 80.92 to 90.42, closing 84.1% of the human-topline gap, while its benefits vary by token composition and model.For Llama-Mimi, normalized and localized settings improve average scores by +2.4 and +2.8 points, with final accuracy gains of +9.42 and +11.08 points; Spirit-LM-Expressive does not improve.
6 Conclusion
The paper introduces likelihood- and generation-based alternatives to global token perplexity for evaluating spoken language models. These methods better reflect human perception and reshape the observed performance landscape.
- The proposed evaluation methods target speech characteristics that global token perplexity may miss.The methods emphasize localization and normalization to better reflect speech properties.
- Correlations with mean opinion scores indicate that the proposed methods better reflect human perception.
- Under the re-evaluation, the previously best-performing model closes 83% of the gap to the human topline on SALMon.
- The findings reshape the spoken language model performance landscape and establish a new evaluation paradigm for future studies.
Limitations
The proposed evaluators remain constrained by the benchmarks and speech dimensions they address. Their demonstrated scope centers on acoustic continuity rather than all aspects of speech.
- Because the methods are applied to existing benchmarks, their scope remains constrained by benchmark limitations.
- SALMon does not systematically test compounded variations such as speaker changes under noisy background conditions.This restricts characterization of model performance in those settings, even with improved evaluators.
- The discussion focuses on acoustic continuity, while global perplexity may remain appropriate for dimensions such as semantics.
- Different aspects of speech are inherently entangled, motivating continued scrutiny of the broader notion of speech perplexity.
A Further Evaluations of Spoken Language Models
Spoken language model evaluation follows a curriculum that mirrors model training progression, with different metrics emphasized for pretrained and finetuned models.
- For pretrained spoken language models, likelihood-based evaluation is prominent.
- For finetuned models, task-based metrics assess utility through speech recognition, emotion recognition, keyword spotting, and speaker identification.
B Verbose Evaluation Settings
The verbose evaluation setting asks annotators to compare generated continuations with prompts on a specified target feature. Scores from 1 to 5 represent increasing similarity, with explicit handling for missing features.
- Annotators compare a generated continuation and its audio prompt only on the specified target feature.Target features include gender, speaker, sentiment, and background.
- A score of 5 denotes a perfect match on the target feature, while a score of 1 denotes complete mismatch.
- A target feature is missing when the audio is silent or lacks the relevant speech or acoustic source.
- The scoring guidelines define 5 as indistinguishable from the prompt on the target attribute.
- Scores of 2 through 4 range from clearly distinguishable with minor overlap to distinguishable only with close attention.
C.1 Likelihood-Based Evaluations
Alternative uncertainty methods reshape SALMon evaluations across spoken language models, with model-family differences and occasional scores above the human topline. Semantic-acoustic alignment remains difficult, with accuracies below 60% across models and methods.
- Model-family results: Alternative uncertainty methods score HuBERT-based models lower and Mimi-based models higher than global token perplexity on SALMon.For top-performing models, proposed methods occasionally place performance above the human topline, especially for speaker information.
- Model-family results: Some proposed evaluations place top-performing models above the human topline, with these cases concentrated around speaker information.
- Semantic-acoustic alignment: Semantic-acoustic alignment accuracies remain below 60% regardless of the model or evaluation method.Only Global-PPL and Windowed-PPL are supported because these tasks lack a common speech prompt.
- Semantic-acoustic alignment: The consistent low alignment accuracies suggest that current spoken language models do not capture this trait, motivating future work.
C.2 Model-as-a-Judge for Measuring Generation Consistency
Model-as-a-judge evaluations measure generation consistency across speech attributes and show that top models approach the human topline on several human-speech traits. They also expose reconstruction distortions and attribute-specific differences that global uncertainty measures can miss.
- Embedding judges: Embedding judges show that no single embedding model performs best across all six SALMon attributes.The results motivate more generalizable audio embedding approaches.
- Generation judgments: True-continuation judgments place top-performing models near the human topline, especially for sentiment, speaker, and gender.Table 7 compares continuation samples with reconstructed audio and marks scores exceeding the human topline.
- Generation judgments: HuBERT-based models perform close to chance, while Spirit-LM-Expressive reaches 72% on sentiment with added pitch and style tokens.
- Reconstruction analysis: Reconstructed audio often achieves near-chance scores because it lacks relevant content, despite original audio receiving near-ceiling benchmark performance.Inspection finds semantic pronunciations largely preserved while speaker and background information collapse.
- Reconstruction analysis: Global uncertainty measures can show moderate performance without generalizing to continuation tasks requiring local acoustic fidelity at each generation step.
- Token contributions: Token ablations show Spirit-LM-Expressive relies most on HuBERT tokens, whereas Llama-Mimi relies most on layer 1.The strongest cross-setting differences concentrate on sentiment, speaker, and gender.
- Likelihood-response analysis: Speaker-related attributes show the largest separations between positive and negative NLL responses, which largely correlate with resulting accuracy.This pattern appears across the per-task breakdowns for the evaluated models.