PaperlayerView exact revision on arXiv

Source-linked AI summary

DAMOS: Learning Distortion-Aware Speech Quality Assessment through Explicit Distortion Localization

Naiyuan Li, Li Dong, Diqun Yan

arXiv:2608.21176v1cs.SDcs.AI

TL;DR

Existing speech quality assessment mainly relies on utterance-level MOS, which provides coarse supervision for locally occurring distortions. DAMOS adds explicit distortion localization through an annotated partially distorted dataset and localization-guided MOS modeling. Across multiple public benchmarks, it consistently outperforms existing methods and shows strong cross-dataset generalization.

  • Problem

    Utterance-level MOS does not explicitly indicate where perceptually important speech distortions occur, limiting fine-grained quality modeling.

  • Method

    DAMOS trains a localization model on a partially distorted speech dataset and integrates its frame-level distortion cues throughout the MOS prediction pipeline.

  • Results

    DAMOS consistently outperforms existing state-of-the-art methods on multiple benchmarks and achieves strong cross-dataset generalization.

  • Takeaways & Limitations

    Explicit distortion localization provides complementary information beyond conventional MOS supervision for speech quality assessment.

Abstract

from arXiv · show

Automatic speech quality assessment aims to predict Mean Opinion Scores (MOS) consistent with human subjective perception and is essential for evaluating speech generation, enhancement, and communication systems. For speech signals, especially synthetic speech, distortions often occur locally, and overall perceptual quality is usually dominated by a small number of perceptually salient distortion regions. However, most existing methods are primarily optimized with utterance-level MOS, which provides only coarse-grained supervision and offer no explicit indication of where perceptually important distortions occur. To address this limitation, we introduce explicit distortion localization as auxiliary knowledge for speech quality assessment. We construct the first partially distorted speech dataset with frame-level distortion annotations and train a localization model to generate distortion cues. Building on these cues, we propose DAMOS, a distortion-aware speech quality assessment framework that integrates localization information into the MOS prediction pipeline. Experiments on multiple public benchmarks demonstrate that DAMOS consistently outperforms existing methods and exhibits strong cross-dataset generalization, validating the effectiveness of explicit distortion localization for speech quality assessment.

I. INTRODUCTION

Existing speech quality assessment relies mainly on utterance-level MOS, which lacks temporal distortion information despite local degradations strongly affecting perceived quality. DAMOS addresses this gap by adding explicit distortion localization, a partially distorted annotated dataset, and localization-guided MOS prediction.

  • Speech quality assessment supports evaluating and improving speech generation, enhancement, and communication systems.
  • Utterance-level MOS provides global quality supervision but does not identify which temporal regions cause perceived degradation.
  • Local distortions in synthetic speech can disproportionately degrade overall listening quality, making their locations and perceptual effects important.
  • DAMOS introduces explicit distortion localization as auxiliary knowledge to guide MOS prediction and fine-grained quality modeling.
  • The authors construct the first partially distorted speech dataset with explicit distortion-location annotations for training localization models.
  • Experiments on multiple public benchmarks show that DAMOS consistently outperforms existing advanced methods and generalizes across datasets.

II. EXPLICIT DISTORTION LOCALIZATION

The paper constructs a partially distorted speech dataset whose distortion boundaries are controllable and whose synthesized degradations are designed to approximate practical speech conditions. Known injection locations enable automatic frame-level labels without manual annotation.

  • The dataset uses automatically generated frame-level labels because distortion locations are known during synthesis.
  • Its design targets controllable distortion boundaries and realistic degradation characteristics so localization knowledge can generalize to quality assessment.
  • The construction begins with LibriSpeech utterances resampled to 16 kHz and cropped or cyclically repeated to durations between 5 and 15 seconds.
  • Voice activity preprocessing removes low-energy frames, contracts speech boundaries by 20 ms, and retains segments longer than 0.3 s to improve annotation reliability.
  • Distortions are organized into global, additive, and signal-dependent categories according to their interaction with speech activity.
  • Each utterance receives one to three non-overlapping distortion segments lasting 0.5–3 s, separated by at least 0.3 s.
  • Frames are 160 ms long and labeled distorted when more than half their duration overlaps an injected distortion segment.
  • The dataset contains 15,000 utterances with an 8:1:1 train/validation/test split and approximately 3% clean samples.

B. Distortion Localization Model

The localization model learns temporal boundaries between clean and degraded speech, then supplies frozen frame-level distortion cues to the MOS prediction framework. DAMOS uses these cues throughout the pipeline rather than only at a final fusion stage.

  • The localization model learns transferable distortion-aware representations rather than directly predicting speech quality.
  • A Boundary-Aware Model serves as the localization backbone because of its capability to model boundary transitions.
  • Given waveform x, BAM predicts a binary frame-level map M indicating distorted and clean frames.
  • The localization model is trained independently, remains frozen during quality assessment, and provides distortion-aware guidance aligned with SSL representations.
  • Directly fusing localization with quality representations at one network stage cannot fully exploit its guidance throughout MOS prediction.
  • DAMOS progressively uses localization to extract distortion-sensitive representations, modulate features, and preserve localized cues until utterance-level estimation.

B. Distortion-Sensitive Layer Adaptation

DAMOS addresses the uneven distribution of distortion information across SSL transformer layers by learning adaptive, distortion-sensitive layer weights. Its DSLA module aggregates hidden representations according to their estimated relevance for distortion perception.

  • Motivation: SSL hidden layers encode hierarchical information, so distortion-relevant quality cues may be distributed across multiple layers.Existing methods commonly use the final layer or fixed aggregation, which can overlook intermediate layers and may not transfer equally across datasets.
  • DSLA: DSLA automatically estimates the importance of each transformer layer and adaptively aggregates distortion-sensitive representations.The module extracts hidden representations from all L layers before learning layer-specific importance weights.
  • DSLA: Temporal average pooling and a linear layer with sigmoid activation produce normalized importance weights for individual transformer layers.Each learned weight represents the estimated importance of its corresponding layer.
  • DSLA: The weighted aggregation produces a distortion-sensitive representation that becomes the basis for subsequent distortion-guided feature conditioning.Unlike fixed aggregation, DSLA emphasizes layers that are more informative for distortion perception.

C. Distortion-Guided Feature Modulation

DAMOS injects localization cues into distortion-sensitive features and preserves them through frame-level quality regression before utterance-level pooling. This design selectively emphasizes degraded regions while avoiding their dilution through early temporal aggregation.

  • Distortion-Guided Feature Modulation: Local-global coupling can obscure short distorted regions because SSL encoders model entire sequences containing many clean frames.The resulting coupling makes localized distortion cues harder to distinguish under utterance-level supervision alone.
  • Distortion-Guided Feature Modulation: DistortionFiLM uses predicted localization maps to dynamically modulate feature responses without increasing representation dimensionality.The localization map is aligned to the SSL feature sequence, projected through an MLP, and converted into a frame-wise modulation gate.
  • Distortion-Guided Feature Modulation: Element-wise gating conditions distortion-sensitive representations so distorted regions are emphasized and distortion-irrelevant regions are relatively suppressed.This produces distortion-aware features before MOS regression.
  • Localized Quality Regression: Early temporal pooling can weaken localized degradation cues by mixing distortion-enhanced frames with abundant irrelevant regions.The framework therefore performs quality regression before temporal aggregation.
  • Localized Quality Regression: A lightweight prediction head independently estimates quality for every temporal frame before temporal average pooling produces utterance-level MOS.The frame-level prediction uses the distortion-aware representation at each time step.
  • Localized Quality Regression: Training uses utterance-level MOS MSE, while delayed aggregation preserves localization knowledge until final quality estimation.This allows localized cues to contribute directly to the quality estimate despite the absence of frame-level MOS supervision.

IV. EXPERIMENTS

The experiments evaluate DAMOS on BVCC using standard speech-quality metrics and a fixed train/validation/test split. The reported setup uses WavLM-Large and selects the checkpoint with the highest validation SRCC.

  • Evaluation Setup: BVCC evaluation uses a predefined 4974/1066/1066 train/validation/test split after resampling all waveforms to 16 kHz.Performance is assessed with MSE, LCC, and SRCC.
  • Implementation: The default SSL backbone is WavLM-Large, trained with grouped learning rates and checkpoint selection by validation SRCC.Training runs for 30 epochs with batch size 16 on a single GPU.

B. Results

DAMOS consistently improves speech quality assessment across in-domain, diverse-dataset, and cross-dataset evaluations, while ablations show that all three proposed components contribute to performance.

  • Comparison with State-of-the-Art Methods: DAMOS improves BVCC utterance-level SRCC from 0.878 to 0.885 over UTMOS while also reducing prediction errors.
  • Comparison with State-of-the-Art Methods: At the system level, DAMOS achieves the most accurate quality ranking and the lowest prediction error among compared methods.
  • Comparison with State-of-the-Art Methods: DAMOS obtains the best performance on most non-synthetic datasets and superior average performance across telephone, conversational, and noisy speech conditions.
  • Cross-Dataset Generalization: DAMOS achieves the highest SRCC on TCD-VoIP, NISQA_FOR, and SOMOS, plus the best average performance across seven additional evaluation datasets.
  • Ablation Study: Removing DistortionFiLM lowers utterance-level SRCC from 0.885 to 0.877 and raises MSE from 0.191 to 0.205.
  • Ablation Study: Replacing DSLA with final-layer features causes the largest ablation degradation, while replacing LQR with utterance-level regression also consistently reduces performance.
  • Ablation Study: Removing any component degrades performance, indicating that DSLA, DistortionFiLM, and LQR are complementary.

3) Analysis of Distortion-Sensitive Layer Adaptation:

DSLA outperforms fixed hidden-layer selection and uniform aggregation on average by adapting layer contributions across datasets, although uniform averaging can win on some individual datasets.

  • Analysis of Distortion-Sensitive Layer Adaptation: DSLA achieves the best average performance across all evaluated datasets compared with alternative hidden-layer aggregation strategies.
  • Analysis of Distortion-Sensitive Layer Adaptation: Selecting the fifth hidden layer improves in-domain BVCC performance over the final layer but degrades on several unseen datasets relative to DSLA and averaging.
  • Analysis of Distortion-Sensitive Layer Adaptation: Uniform averaging slightly outperforms DSLA on NISQA_FOR, Tencent, and PSTN, showing that adaptive weighting does not dominate on every dataset.
  • Analysis of Distortion-Sensitive Layer Adaptation: Hidden layers contribute unequally to quality assessment, so fixed selection and uniform aggregation may miss representations relevant across distortion distributions and acoustic conditions.
  • Analysis of Distortion-Sensitive Layer Adaptation: DSLA dynamically estimates each hidden layer’s contribution, balancing generally useful early and final layers with dataset-dependent intermediate representations.

4) Analysis of Distortion Localization:

The localization model provides reliable frame-level supervision but varies by distortion type, while modulation remains stable at moderate bias values and degrades when global features dominate.

  • Localization Performance: The localization model achieves overall frame-level F1 of 0.807, accuracy of 0.903, and boundary-level F1 of 0.742.
  • Localization Performance: Pitch shifting, clipping, MNRU, and vocoder mismatch are localized more accurately than reverberation, codec tandeming, and Mel-spectrogram reconstruction.
  • Localization Performance: High precision but lower recall indicates conservative localization that limits false positives while sometimes missing ambiguous distortion boundaries.
  • Modulation Sensitivity: Moderate modulation bias values from 0 to 1 preserve relatively stable performance, indicating robustness to the local-versus-global feature balance.
  • Modulation Sensitivity: At m = 10.0, utterance-level SRCC falls from 0.885 to 0.862 as uniformly preserved features weaken localization guidance.
  • Modulation Sensitivity: A small bias of m = 0.2 causes negligible variation, supporting localization as a complementary perceptual prior rather than a strict constraint.

6) Mask Perturbation Analysis of Distortion Localization Guidance:

Mask perturbation shows that DAMOS tolerates missing localization cues better than false-positive distortion masks, consistent with localization also shaping representations during training.

  • Mask Perturbation Analysis: Replacing predicted masks with all-zero masks causes only marginal SRCC degradation on BVCC and TCD-VoIP.
  • Mask Perturbation Analysis: All-one masks cause larger drops, reducing BVCC SRCC from 0.885 to 0.870 and TCD-VoIP SRCC from 0.837 to 0.818.
  • Mask Perturbation Analysis: The all-zero condition remains comparatively robust because training encourages DSLA representations to encode distortion-sensitive characteristics beyond the runtime modulation gate.
  • Mask Perturbation Analysis: The perturbation profile aligns with the localization detector’s conservative high-precision, lower-recall behavior across distortion categories.
  • Mask Perturbation Analysis: Explicit localization provides both frame-level inference guidance and training-time supervision that persists when localization is degraded or unavailable at test time.

V. CONCLUSION

DAMOS uses explicit distortion localization to enrich MOS prediction with distortion-aware information. It combines a partially distorted speech dataset, a dedicated localization model, and progressive integration of localization cues, while identifying several directions for improving robustness, joint training, and interpretability.

  • The authors construct a partially distorted speech dataset with automatically generated frame-level annotations and train a dedicated localization model to provide transferable distortion-aware cues.
  • DAMOS progressively exploits explicit distortion information throughout the MOS prediction pipeline.Its design includes DSLA, DistortionFiLM, and LQR for identifying, injecting, and preserving localized distortion information.
  • DAMOS consistently outperforms existing state-of-the-art methods on multiple benchmarks and achieves strong cross-dataset generalization.The results indicate that explicit distortion localization provides complementary information beyond conventional MOS supervision.
  • The framework relies on synthetically generated distortion masks, while localization and quality prediction models are trained separately with the localization backbone frozen during MOS training.The authors propose evaluation on real non-synthetic distortions and joint optimization as directions for assessing robustness and improving interaction between objectives.
  • Fully frame-level, interpretable quality assessment remains an open direction for exposing predicted distortion locations as evidence for quality scores.
Loading 2608.21176v1…