PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features

Ryandhimas E. Zezario, Szu-Wei Fu, Fei Chen, Chiou-Shann Fuh, Hsin-Min Wang, Yu Tsao

arXiv:2111.02363v5eess.AScs.LGcs.SD

TL;DR

Subjective speech ratings require substantial listening tests, while existing deep-learning assessment models often use a single feature type and objective. MOSA-Net combines cross-domain features with multi-objective prediction, and its latent representations guide QIA-SE; reported comparisons show benefits across assessment and enhancement tasks.

  • Problem

    Subjective ratings require many listeners and diverse utterances, while deep-learning assessment models commonly use a single acoustic feature type and one objective.

    Source: §I · p. 2, A. Deep Learning-based Assessment… · p. 3

  • Method

    MOSA-Net uses spectral, learnable-filter, and self-supervised features for multi-objective assessment, and its latent representations are incorporated into QIA-SE.

    Source: §I · p. 2, A. Multi-Target Speech Assessment… · p. 4, B. QIA-SE Model · p. 4

  • Results

    MOSA-Net performs better than single-task assessment baselines and supports adaptation to subjective ratings, while QIA-SE outperforms prior assessment-guided enhancement models with less storage and online computation.

    Source: 7) Adapting MOSA-Net to predict human… · p. 11, §V · pp. 13–14

  • Takeaways & Limitations

    The reported QIA-SE comparison indicates that directly incorporating MOSA-Net representations can deliver better enhancement performance with lower storage and online-computation requirements than SSEMS and ZMOS.

    Source: §V · p. 14

    Improving MOSA-Net's robustness in real-world scenarios remains future work.

    Source: §V · p. 14

Abstract

from arXiv · show

In this study, we propose a cross-domain multi-objective speech assessment model called MOSA-Net, which can estimate multiple speech assessment metrics simultaneously. Experimental results show that MOSA-Net can improve the linear correlation coefficient (LCC) by 0.026 (0.990 vs 0.964 in seen noise environments) and 0.012 (0.969 vs 0.957 in unseen noise environments) in perceptual evaluation of speech quality (PESQ) prediction, compared to Quality-Net, an existing single-task model for PESQ prediction, and improve LCC by 0.021 (0.985 vs 0.964 in seen noise environments) and 0.047 (0.836 vs 0.789 in unseen noise environments) in short-time objective intelligibility (STOI) prediction, compared to STOI-Net (based on CRNN), an existing single-task model for STOI prediction. Moreover, MOSA-Net, originally trained to assess objective scores, can be used as a pre-trained model to be effectively adapted to an assessment model for predicting subjective quality and intelligibility scores with a limited amount of training data. Experimental results show that MOSA-Net can improve LCC by 0.018 (0.805 vs 0.787) in mean opinion score (MOS) prediction, compared to MOS-SSL, a strong single-task model for MOS prediction. In light of the confirmed prediction capability, we further adopt the latent representations of MOSA-Net to guide the speech enhancement (SE) process and derive a quality-intelligibility (QI)-aware SE (QIA-SE) approach accordingly. Experimental results show that QIA-SE provides superior enhancement performance compared with the baseline SE system in terms of objective evaluation metrics and qualitative evaluation test. For example, QIA-SE can improve PESQ by 0.301 (2.953 vs 2.652 in seen noise environments) and 0.18 (2.658 vs 2.478 in unseen noise environments) over a CNN-based baseline SE model.

I. INTRODUCTION

Human listening tests are burdensome, and signal-level metrics may not fully reflect speech quality or intelligibility. The paper proposes MOSA-Net for multi-objective assessment and uses its latent representations in QIA-SE, reporting improvements over baseline enhancement systems.

I. INTRODUCTION

  • Human listening tests require many listeners and diverse utterances, making subjective assessment prohibitive and not always feasible.Objective metrics have therefore been developed as surrogates for human listening tests –.Source: §I · pp. 1–2
  • Non-intrusive objective metrics avoid requiring clean reference speech, making them more suitable for real-world use, but generally have lower assessment capability than intrusive metrics.PESQ and POLQA are examples of intrusive metrics.Source: §I · p. 2
  • MOSA-Net combines spectral, learnable-filter, and self-supervised features to predict multiple objective speech assessment scores.Its targets include speech quality, intelligibility, and distortion scores; the model uses CNN-BLSTM with attention.Source: §I · p. 2
  • Adapting an objective-score-pretrained MOSA-Net to subjective quality and intelligibility prediction can work with limited training data.The paper distinguishes this adaptation from pseudo-labeling on the same tasks, such as DNSMOS.Source: §I · p. 2
  • QIA-SE improves performance over baseline and several existing speech enhancement systems by incorporating MOSA-Net latent representations.The introduction describes these results as confirming the advantage of combining assessment-model knowledge with enhancement.Source: §I · pp. 2–3

II. RELATED WORK

Prior work predicts subjective ratings or objective assessment scores using varied architectures, targets, and acoustic features. Speech assessment models have also been incorporated into enhancement systems, either as training targets or for model and output selection; the proposed-methods section introduces MOSA-Net and assessment-guided enhancement.

A. Deep Learning-based Assessment Metrics

  • Subjective deep-learning assessment models predict listener ratings, but unbiased training and extension to new domains require additional listening tests.Examples include MOSNet, DNSMOS, and MBNet for subjective quality; the review notes less work on subjective intelligibility prediction.Source: A. Deep Learning-based Assessment… · p. 3
  • Objective assessment models use quality targets such as PESQ – and POLQA, or intelligibility targets such as STOI – and HASPI.Reported model choices include BLSTM, CNN, CNN-BLSTM, attention, multi-task learning, and spectral, waveform-derived, or self-supervised features.Source: A. Deep Learning-based Assessment… · p. 3

B. Incorporating Speech Assessment Metrics to SE

  • Assessment-guided enhancement methods either train the enhancement system with assessment metrics as targets or use metrics to select an architecture or output.The review cites target-based approaches,, and selection approaches,,.Source: B. Incorporating Speech Assessment… · p. 3

III. PROPOSED METHODS

  • The proposed-methods section presents MOSA-Net and explains how its latent representations are used to pursue better speech quality or intelligibility.Source: §III · p. 4

A. Multi-Target Speech Assessment Model with Cross-Domain Features (MOSA-Net)

MOSA-Net combines cross-domain acoustic features to predict multiple speech assessment scores, using both utterance-level and frame-level estimation. QIA-SE incorporates MOSA-Net latent representations into speech enhancement to guide the conversion of noisy speech features into enhanced features.

A. Multi-Target Speech Assessment Model with Cross-Domain Features (MOSA-Net)

  • MOSA-Net combines STFT, learnable-filter-bank, and self-supervised features to predict PESQ, STOI, and SDI scores through metric-specific attention layers.The feature streams are aligned in dimension and processed by a bidirectional layer before framewise scores are averaged into utterance-level predictions.Source: A. Multi-Target Speech Assessment… · p. 4
  • MOSA-Net’s objective combines utterance-level and frame-level losses for PESQ, STOI, and SDI estimation.The motivation is that noise may vary across frames, making direct utterance-level estimation less accurate; the experiments set the loss weights to 1.Source: A. Multi-Target Speech Assessment… · p. 4

B. QIA-SE Model

  • QIA-SE converts noisy-speech spectral features into enhanced features while conditioning on MOSA-Net latent representations concatenated into its middle layer.The representation is taken from the fully connected layer after attention in MOSA-Net.Source: B. QIA-SE Model · p. 4
  • QIA-SE uses a CNN enhancement model while MOSA-Net remains frozen; the enhancement parameters are trained with an MSE-based loss.The implementation uses 12 convolutional layers followed by a 128-neuron fully connected layer, and noisy-speech phase is retained for waveform reconstruction.Source: B. QIA-SE Model · p. 5

IV. EXPERIMENTS

The experiments examine how MOSA-Net performance varies with input features, model architectures, and output labels, alongside comparisons with related neural evaluation metrics. The setup uses noisy and enhanced WSJ utterances and evaluates predictions on seen and unseen noise conditions.

A. Analysis of MOSA-Net

  • The analysis investigates MOSA-Net performance across input features, model architectures, and output labels, and compares it with related neural evaluation metrics.Source: A. Analysis of MOSA-Net · p. 5

1) Experimental setup:

  • For WSJ, 37,416 noisy-clean pairs, contaminated with 100 noise types at 31 SNR levels (-10 to 20 dB, 1-dB intervals), trained a BLSTM enhancement system; MOSA-Net used PESQ, STOI, and SDI scores from 1,500 clean, 15,000 noisy, and 15,000 enhanced utterances.Source: 1) Experimental setup · p. 5
  • The seen test set used utterances outside the MOSA-Net training set, while the unseen set used four unseen noise types and speakers excluded from training.Both sets included clean, noisy, and enhanced utterances.Source: 1) Experimental setup · p. 5
  • Evaluation measured prediction error with MSE and correlation with LCC and SRCC, where lower MSE and higher correlations indicate better predictions.Source: 1) Experimental setup · p. 5

2) MOSA-Net with different model architectures:

MOSA-Net’s CRNN with attention outperformed the CNN, BLSTM, and CRNN architectures, and it also compared favorably with Quality-Net and STOI-Net. The comparisons evaluate prediction quality across speech metrics and noise conditions.

2) MOSA-Net with different model architectures:

  • CRNN slightly outperformed CNN and BLSTM for PESQ and STOI prediction on seen and unseen test sets, while CRNN+AT outperformed CRNN.The authors attribute the attention model’s improvement to its ability to focus on more important regions.Source: 2) MOSA-Net with different model… · p. 6, Table I · p. 6
  • Fig. 3 compares MOSA-Net, Quality-Net, and STOI-Net predictions, with MOSA-Net predictions more densely distributed along the diagonal.Diagonal concentration indicates stronger agreement between predicted and true scores.Source: 2) MOSA-Net with different model… · p. 6, Figure 3 · p. 6
  • MOSA-Net outperformed Quality-Net and STOI-Net on almost all LCC, SRCC, and MSE comparisons; all reported differences were significant except PESQ MSE in unseen noise.The statistical tests used 20 matched pairs and found p < 0.05 for the performance differences in Table II.Source: 2) MOSA-Net with different model… · p. 6, Table II · p. 6

3) MOSA-Net with single- and multi-task training:

MOSA-Net’s multi-task training often improves assessment predictions, although gains depend on the target metric and noise condition. Its multi-task model also shows more similar attention patterns across metric branches than single-task models.

3) MOSA-Net with single- and multi-task training:

  • For seen-noise PESQ prediction, Q+I and Q+D beat single-task Q across LCC, SRCC, and MSE, while Q+I+D performed best; multi-task gains did not hold for unseen-noise PESQ.For STOI, similar trends were observed; for SDI, adding PESQ or STOI improved over single-task SDI in both noise conditions, though using both was not best.Source: 3) MOSA-Net with single- and… · pp. 6–7, Table III · p. 7
  • Multi-task MOSA-Net estimates assessment scores more accurately than single-task models in Fig. 4, with points more densely distributed along the diagonal.The reported PESQ SRCC example is 0.977 for Q+I+D versus 0.965 for Q under seen noise.Source: 3) MOSA-Net with single- and… · p. 7, Figure 4 · p. 7
  • Single-task attention representations differ across PESQ, STOI, and SDI, whereas the triple-task model yields similar patterns across its metric branches.The authors suggest the shared patterns may reflect useful shared representations and more general weights.Source: 3) MOSA-Net with single- and… · p. 7

4) Comparison with another multi-task method:

MOSA-Net combines utterance-level and frame-level losses in its objective function and outperformed AMSA across PESQ, STOI, and SDI predictions. The section also presents attention-layer representations for single- and multi-task models.

4) Comparison with another multi-task method:

  • Figures 5 and 6 plot attention weights against frame index for PESQ, STOI, and SDI representations from single-task and multi-task MOSA-Net.Figure 5 shows the single-task model and Figure 6 the multi-task model.Source: Figure 5 · p. 8, Figure 6 · p. 8
  • MOSA-Net combines estimated utterance-level and frame-level losses, whereas AMSA combines utterance-level regression and classification-aided loss.The comparison used the same model architecture and number of assessment targets.Source: 4) Comparison with another multi-task… · p. 8
  • MOSA-Net consistently outperformed AMSA in PESQ, STOI, and SDI prediction across LCC, SRCC, and MSE.The authors attribute the advantage to combining utterance-level and frame-level scores in the objective function.Source: 4) Comparison with another multi-task… · p. 8, Table IV · p. 8

5) MOSA-Net with cross-domain features:

Cross-domain features provide complementary information for MOSA-Net, and the PS+LFB+SSL(Hub) model significantly improves on PS-only predictions across metrics and noise conditions. On TIMIT, the cross-domain model also performs best in almost all testing conditions.

5) MOSA-Net with cross-domain features:

  • Individual features differ by target: PS tended to do better for PESQ, SSL(Hub) for STOI, and PS for SDI, while complex features retained useful phase information, particularly in seen environments.SSL(Hub) generally performed better than SSL(W2V).Source: 5) MOSA-Net with cross-domain features · pp. 8–9
  • Complex+SSL(Hub) outperformed either feature alone for PESQ, STOI, and SDI in seen and unseen environments; three-feature combinations performed better than two-feature combinations in unseen environments.Complex+Hub was best among combinations for seen-environment STOI and SDI prediction.Source: 5) MOSA-Net with cross-domain features · p. 9, Table VI · p. 9
  • PS+LFB+SSL(Hub) significantly improved over PS-only MOSA-Net in PESQ, STOI, and SDI predictions across seen and unseen conditions, for LCC, SRCC, and MSE.The corresponding prediction scatter plots also show more accurate estimation with cross-domain features.Source: 5) MOSA-Net with cross-domain features · p. 9, Table VI · p. 9, Figure 7 · p. 9

6) MOSA-Net tested on the TIMIT dataset:

  • On TIMIT, MOSA-Net Cross-Domain achieved the best performance in almost all testing conditions, except SDI MSE, and beat AMSA significantly in 8 of 9 conditions.The evaluation tested WSJ-trained models on TIMIT; the authors conclude cross-domain features improve generalization to new datasets.Source: 6) MOSA-Net tested on the TIMIT dataset · pp. 9–10, Table VII · p. 10

7) Adapting MOSA-Net to predict human subjective ratings:

The section tests whether objective-score-pretrained MOSA-Net can be adapted to predict human subjective ratings; the adapted model with fine-tuned SSL features performs best among the evaluated models.

  • Subjective-score prediction is challenging because human ratings vary, and collecting them requires multiple listeners and is time-consuming and expensive.The section motivates adapting an objective-score model using a limited amount of subjective-score training data.Source: 7) Adapting MOSA-Net to predict human… · p. 10
  • The evaluation used 15,000 single-subject-rated utterances for training and 1,900 multi-subject-rated utterances for testing, with no overlap between the sets.Test ground-truth scores were averages across subjects; the dataset included clean, noisy, and enhanced utterances.Source: 7) Adapting MOSA-Net to predict human… · p. 10
  • The comparison included MOSNet, MOS-SSL, scratch-trained MOSA-Net, and MOSA-Net adapted from WSJ pretraining, with and without fine-tuned SSL features.MOSNet was trained from scratch, while MOS-SSL fine-tuned its wav2vec module.Source: 7) Adapting MOSA-Net to predict human… · p. 11
  • MOSA-Net(Adapt)FT−SSL outperformed MOSNet and MOS-SSL, with statistically significant improvements over MOS-SSL for MOS and subjective intelligibility prediction across all metrics.The results support using MOSA-Net(WSJ) as a pre-trained model for predicting human subjective ratings.Source: Table VIII · p. 10, 7) Adapting MOSA-Net to predict human… · p. 11

B. Experiments of SE with assessment information

On WSJ, QIA-SE directly incorporates a speech-assessment model’s latent representation into the enhancement model and outperforms the baseline and model-selection systems across the reported metrics and noise conditions.

1) Experiments on the WSJ dataset:

  • The WSJ evaluation used 37,416 noisy-clean training pairs and 330 seen-noise plus 330 unseen-noise test utterances across six SNR levels.Seen and unseen sets each covered four noise types.Source: 1) Experiments on the WSJ dataset · p. 11
  • QIA-SE significantly outperformed SSEMS and ZMOS in seen and unseen environments, and achieved the best CSIG and SSNRI performance in both.It also outperformed the other systems in STOI; SSEMS and ZMOS both exceeded the CNN baseline.Source: Table IX · p. 12, 1) Experiments on the WSJ dataset · p. 12
  • QIA-SE directly incorporates the assessment model’s latent representation into the SE model, unlike SSEMS and ZMOS, which prepare component models offline and select among them online.The comparison systems and QIA-SE used the same CNN architecture.Source: 1) Experiments on the WSJ dataset · p. 12

2) Experiments on the TMHINT dataset:

On TMHINT, QIA-SE using multi-task-trained MOSA-Net performs better than variants using single-task-trained MOSA-Net and the CNN baseline under both seen and unseen noise conditions.

2) Experiments on the TMHINT dataset:

  • The TMHINT experiment tested cross-language transfer from English to Mandarin and compared speech-assessment codes from single-task and multi-task training criteria.The test used separate speakers from training and included seen and unseen noise conditions.Source: 2) Experiments on the TMHINT dataset · p. 12
  • Figs. 9 and 10 show that QIA-SE(M) outperformed the CNN baseline and both single-task QIA-SE variants in PESQ and STOI.QIA-SE(S-PESQ) and QIA-SE(S-STOI) use MOSA-Net trained on PESQ and STOI, respectively.Source: Figure 9 · p. 12, Figure 10 · p. 12, 2) Experiments on the TMHINT dataset · p. 12
  • The results connect enhancement performance with assessment-model quality: multi-task learning more accurately predicted assessment scores, and its QIA-SE system performed better.The reported comparison is among QIA-SE systems using the same CNN architecture.Source: 2) Experiments on the TMHINT dataset · pp. 12–13

3) Qualitative Analysis:

The qualitative examples show QIA-SE retaining more speech detail than the CNN baseline, while the conclusion highlights MOSA-Net’s cross-domain assessment and QIA-SE’s enhancement efficiency.

3) Qualitative Analysis:

  • QIA-SE preserved more detail than the CNN baseline in waveform and spectrogram examples, and its enhanced-speech amplitude envelope more closely resembled the clean waveform.The envelope comparison focuses on the 457–1202 Hz band, whose middle-frequency components are noted as relevant to intelligibility.Source: 3) Qualitative Analysis · p. 13, Figure 13 · p. 13

V. CONCLUSION

  • MOSA-Net combines cross-domain acoustic features, and the conclusion reports that multi-task training and the CRNN with attention achieved stronger assessment performance.The feature types include spectral, learnable-filter waveform, and SSL representations.Source: §V · p. 13
  • QIA-SE directly combines MOSA-Net latent representations and outperformed SSEMS and ZMOS while requiring less model storage and online computation.The conclusion also reports better enhancement when QIA-SE is combined with a better speech-assessment model.Source: §V · pp. 13–14
  • The authors identify improving MOSA-Net’s robustness in real-world scenarios as future work.Source: §V · p. 14

Related papers

Loading 2111.02363v5…