PaperlayerView exact revision on arXiv

Source-linked AI summary

DAMOS: Learning Distortion-Aware Speech Quality Assessment through Explicit Distortion Localization

Naiyuan Li, Li Dong, Diqun Yan

arXiv:2608.21176v1cs.SDcs.AI

TL;DR

Existing speech quality assessment mainly relies on utterance-level MOS, which provides coarse supervision for locally occurring distortions. DAMOS adds explicit distortion localization through an annotated partially distorted dataset and localization-guided MOS modeling. Across multiple public benchmarks, it consistently outperforms existing methods and shows strong cross-dataset generalization.

Abstract

from arXiv · show

Automatic speech quality assessment aims to predict Mean Opinion Scores (MOS) consistent with human subjective perception and is essential for evaluating speech generation, enhancement, and communication systems. For speech signals, especially synthetic speech, distortions often occur locally, and overall perceptual quality is usually dominated by a small number of perceptually salient distortion regions. However, most existing methods are primarily optimized with utterance-level MOS, which provides only coarse-grained supervision and offer no explicit indication of where perceptually important distortions occur. To address this limitation, we introduce explicit distortion localization as auxiliary knowledge for speech quality assessment. We construct the first partially distorted speech dataset with frame-level distortion annotations and train a localization model to generate distortion cues. Building on these cues, we propose DAMOS, a distortion-aware speech quality assessment framework that integrates localization information into the MOS prediction pipeline. Experiments on multiple public benchmarks demonstrate that DAMOS consistently outperforms existing methods and exhibits strong cross-dataset generalization, validating the effectiveness of explicit distortion localization for speech quality assessment.

I. INTRODUCTION

Existing speech quality assessment relies mainly on utterance-level MOS, which lacks temporal distortion information despite local degradations strongly affecting perceived quality. DAMOS addresses this gap by adding explicit distortion localization, a partially distorted annotated dataset, and localization-guided MOS prediction.

  • Speech quality assessment supports evaluating and improving speech generation, enhancement, and communication systems.Source: §I · p. 1
  • Utterance-level MOS provides global quality supervision but does not identify which temporal regions cause perceived degradation.Source: §I · p. 1
  • Local distortions in synthetic speech can disproportionately degrade overall listening quality, making their locations and perceptual effects important.Source: §I · p. 1
  • DAMOS introduces explicit distortion localization as auxiliary knowledge to guide MOS prediction and fine-grained quality modeling.Source: §I · p. 2
  • The authors construct the first partially distorted speech dataset with explicit distortion-location annotations for training localization models.Source: §I · p. 2
  • Experiments on multiple public benchmarks show that DAMOS consistently outperforms existing advanced methods and generalizes across datasets.Source: §I · p. 2

II. EXPLICIT DISTORTION LOCALIZATION

The paper constructs a partially distorted speech dataset whose distortion boundaries are controllable and whose synthesized degradations are designed to approximate practical speech conditions. Known injection locations enable automatic frame-level labels without manual annotation.

B. Distortion Localization Model

The localization model learns temporal boundaries between clean and degraded speech, then supplies frozen frame-level distortion cues to the MOS prediction framework. DAMOS uses these cues throughout the pipeline rather than only at a final fusion stage.

  • The localization model learns transferable distortion-aware representations rather than directly predicting speech quality.Source: B. Distortion Localization Model · p. 3
  • A Boundary-Aware Model serves as the localization backbone because of its capability to model boundary transitions.Source: B. Distortion Localization Model · p. 3
  • Given waveform x, BAM predicts a binary frame-level map M indicating distorted and clean frames.Source: B. Distortion Localization Model · p. 3
  • The localization model is trained independently, remains frozen during quality assessment, and provides distortion-aware guidance aligned with SSL representations.Source: B. Distortion Localization Model · p. 3
  • Directly fusing localization with quality representations at one network stage cannot fully exploit its guidance throughout MOS prediction.Source: A. Overview · p. 3
  • DAMOS progressively uses localization to extract distortion-sensitive representations, modulate features, and preserve localized cues until utterance-level estimation.Source: A. Overview · p. 3

B. Distortion-Sensitive Layer Adaptation

DAMOS addresses the uneven distribution of distortion information across SSL transformer layers by learning adaptive, distortion-sensitive layer weights. Its DSLA module aggregates hidden representations according to their estimated relevance for distortion perception.

Motivation

  • SSL hidden layers encode hierarchical information, so distortion-relevant quality cues may be distributed across multiple layers.Existing methods commonly use the final layer or fixed aggregation, which can overlook intermediate layers and may not transfer equally across datasets.Source: B. Distortion-Sensitive Layer Adaptation · pp. 3–4

DSLA

  • DSLA automatically estimates the importance of each transformer layer and adaptively aggregates distortion-sensitive representations.The module extracts hidden representations from all L layers before learning layer-specific importance weights.Source: B. Distortion-Sensitive Layer Adaptation · p. 4
  • Temporal average pooling and a linear layer with sigmoid activation produce normalized importance weights for individual transformer layers.Each learned weight represents the estimated importance of its corresponding layer.Source: B. Distortion-Sensitive Layer Adaptation · p. 4
  • The weighted aggregation produces a distortion-sensitive representation that becomes the basis for subsequent distortion-guided feature conditioning.Unlike fixed aggregation, DSLA emphasizes layers that are more informative for distortion perception.Source: B. Distortion-Sensitive Layer Adaptation · p. 4

C. Distortion-Guided Feature Modulation

DAMOS injects localization cues into distortion-sensitive features and preserves them through frame-level quality regression before utterance-level pooling. This design selectively emphasizes degraded regions while avoiding their dilution through early temporal aggregation.

Distortion-Guided Feature Modulation

  • Local-global coupling can obscure short distorted regions because SSL encoders model entire sequences containing many clean frames.The resulting coupling makes localized distortion cues harder to distinguish under utterance-level supervision alone.Source: C. Distortion-Guided Feature Modulation · p. 4
  • DistortionFiLM uses predicted localization maps to dynamically modulate feature responses without increasing representation dimensionality.The localization map is aligned to the SSL feature sequence, projected through an MLP, and converted into a frame-wise modulation gate.Source: C. Distortion-Guided Feature Modulation · pp. 4–5
  • Element-wise gating conditions distortion-sensitive representations so distorted regions are emphasized and distortion-irrelevant regions are relatively suppressed.This produces distortion-aware features before MOS regression.Source: C. Distortion-Guided Feature Modulation · p. 5

Localized Quality Regression

  • Early temporal pooling can weaken localized degradation cues by mixing distortion-enhanced frames with abundant irrelevant regions.The framework therefore performs quality regression before temporal aggregation.Source: D. Localized Quality Regression · p. 5
  • A lightweight prediction head independently estimates quality for every temporal frame before temporal average pooling produces utterance-level MOS.The frame-level prediction uses the distortion-aware representation at each time step.Source: D. Localized Quality Regression · p. 5
  • Training uses utterance-level MOS MSE, while delayed aggregation preserves localization knowledge until final quality estimation.This allows localized cues to contribute directly to the quality estimate despite the absence of frame-level MOS supervision.Source: D. Localized Quality Regression · p. 5

IV. EXPERIMENTS

The experiments evaluate DAMOS on BVCC using standard speech-quality metrics and a fixed train/validation/test split. The reported setup uses WavLM-Large and selects the checkpoint with the highest validation SRCC.

Evaluation Setup

  • BVCC evaluation uses a predefined 4974/1066/1066 train/validation/test split after resampling all waveforms to 16 kHz.Performance is assessed with MSE, LCC, and SRCC.Source: 1) Dataset · p. 5

Implementation

  • The default SSL backbone is WavLM-Large, trained with grouped learning rates and checkpoint selection by validation SRCC.Training runs for 30 epochs with batch size 16 on a single GPU.Source: 2) Implementation Details · p. 5

B. Results

DAMOS consistently improves speech quality assessment across in-domain, diverse-dataset, and cross-dataset evaluations, while ablations show that all three proposed components contribute to performance.

Comparison with State-of-the-Art Methods

Cross-Dataset Generalization

Ablation Study

  • Removing DistortionFiLM lowers utterance-level SRCC from 0.885 to 0.877 and raises MSE from 0.191 to 0.205.Source: 2) Ablation Study · p. 6
  • Replacing DSLA with final-layer features causes the largest ablation degradation, while replacing LQR with utterance-level regression also consistently reduces performance.Source: 2) Ablation Study · p. 6
  • Removing any component degrades performance, indicating that DSLA, DistortionFiLM, and LQR are complementary.Source: 2) Ablation Study · p. 6

3) Analysis of Distortion-Sensitive Layer Adaptation:

DSLA outperforms fixed hidden-layer selection and uniform aggregation on average by adapting layer contributions across datasets, although uniform averaging can win on some individual datasets.

Analysis of Distortion-Sensitive Layer Adaptation

4) Analysis of Distortion Localization:

The localization model provides reliable frame-level supervision but varies by distortion type, while modulation remains stable at moderate bias values and degrades when global features dominate.

Localization Performance

Modulation Sensitivity

6) Mask Perturbation Analysis of Distortion Localization Guidance:

Mask perturbation shows that DAMOS tolerates missing localization cues better than false-positive distortion masks, consistent with localization also shaping representations during training.

Mask Perturbation Analysis

V. CONCLUSION

DAMOS uses explicit distortion localization to enrich MOS prediction with distortion-aware information. It combines a partially distorted speech dataset, a dedicated localization model, and progressive integration of localization cues, while identifying several directions for improving robustness, joint training, and interpretability.

  • The authors construct a partially distorted speech dataset with automatically generated frame-level annotations and train a dedicated localization model to provide transferable distortion-aware cues.Source: §V · p. 10
  • DAMOS progressively exploits explicit distortion information throughout the MOS prediction pipeline.Its design includes DSLA, DistortionFiLM, and LQR for identifying, injecting, and preserving localized distortion information.Source: §V · pp. 10–11
  • DAMOS consistently outperforms existing state-of-the-art methods on multiple benchmarks and achieves strong cross-dataset generalization.The results indicate that explicit distortion localization provides complementary information beyond conventional MOS supervision.Source: §V · p. 11
  • The framework relies on synthetically generated distortion masks, while localization and quality prediction models are trained separately with the localization backbone frozen during MOS training.The authors propose evaluation on real non-synthetic distortions and joint optimization as directions for assessing robustness and improving interaction between objectives.Source: §V · p. 11
  • Fully frame-level, interpretable quality assessment remains an open direction for exposing predicted distortion locations as evidence for quality scores.Source: §V · p. 11

Related papers

Loading 2608.21176v1…