PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks

Szu-Wei Fu, Tao-Wei Wang, Yu Tsao, Xugang Lu, Hisashi Kawai

arXiv:1709.03658v2stat.MLcs.LGcs.SD

TL;DR

Speech enhancement often optimizes MMSE while evaluation uses perception-based measures such as STOI, creating a mismatch between training and assessment. The paper proposes an end-to-end utterance-based FCN that directly optimizes STOI, with reported improvements in STOI, human intelligibility, and ASR robustness over MMSE-based enhancement.

  • Problem

    Speech enhancement training commonly minimizes MMSE even when intelligibility is evaluated with STOI, so the optimization and evaluation criteria differ.

  • Method

    An end-to-end utterance-based FCN directly optimizes STOI over complete variable-length waveform utterances.

  • Results

    The proposed STOI-optimized enhancement improves STOI, human intelligibility, and ASR robustness under noisy conditions compared with MMSE-based enhancement.

  • Takeaways & Limitations

    Aligning the training objective with the evaluation metric can improve speech-enhancement intelligibility outcomes for listeners and noisy-condition ASR.

  • Takeaways & Limitations

    A conditional MSE-STOI objective produced a trivial low-energy output because STOI is insensitive to scaling through its correlation-coefficient basis.

Abstract

from arXiv · show

Speech enhancement model is used to map a noisy speech to a clean speech. In the training stage, an objective function is often adopted to optimize the model parameters. However, in most studies, there is an inconsistency between the model optimization criterion and the evaluation criterion on the enhanced speech. For example, in measuring speech intelligibility, most of the evaluation metric is based on a short-time objective intelligibility (STOI) measure, while the frame based minimum mean square error (MMSE) between estimated and clean speech is widely used in optimizing the model. Due to the inconsistency, there is no guarantee that the trained model can provide optimal performance in applications. In this study, we propose an end-to-end utterance-based speech enhancement framework using fully convolutional neural networks (FCN) to reduce the gap between the model optimization and evaluation criterion. Because of the utterance-based optimization, temporal correlation information of long speech segments, or even at the entire utterance level, can be considered when perception-based objective functions are used for the direct optimization. As an example, we implement the proposed FCN enhancement framework to optimize the STOI measure. Experimental results show that the STOI of test speech is better than conventional MMSE-optimized speech due to the consistency between the training and evaluation target. Moreover, by integrating the STOI in model optimization, the intelligibility of human subjects and automatic speech recognition (ASR) system on the enhanced speech is also substantially improved compared to those generated by the MMSE criterion.

I. INTRODUCTION

The paper targets a mismatch between training objectives, typically MMSE, and perception-based evaluation metrics such as STOI and PESQ. It proposes utterance-based FCN enhancement to optimize evaluation criteria directly and reports gains in objective intelligibility, human intelligibility, and ASR robustness.

  • MMSE-based training commonly differs from human perception-based evaluation metrics, including STOI and PESQ.
  • Frame-wise DNN optimization limits direct use of metrics requiring whole clean and processed utterances.
  • FCNs enable utterance-based enhancement despite variable utterance lengths because they contain only convolutional layers.
  • The proposed framework directly optimizes final evaluation criteria across local frame-scale to global utterance-scale objectives, with STOI as the demonstrated example.
  • Incorporating STOI improves the corresponding metric, human intelligibility, and ASR robustness under noisy conditions.

II. END-TO-END WAVEFORM BASED SPEECH ENHANCEMENT

The paper motivates waveform enhancement by avoiding frequency-domain processing and preserving phase and local sample relationships. An utterance-based FCN processes complete variable-length waveforms end to end, producing one clean utterance.

  • Conventional DNN enhancement often processes magnitude spectra while retaining noisy phase and requires framing, DFT, and waveform reconstruction.
  • Waveform-based FCN avoids frequency-domain transformation and has improved over LPS-based DNN models.
  • FCN convolutional layers preserve local neighboring-sample relationships that represent waveform frequency structure.
  • Removing fully connected layers lets FCN process complete utterances of different lengths without fixed-length framing.
  • The proposed architecture maps a noisy waveform utterance directly to a clean waveform utterance, with one final filter producing the single-channel output.

III. OPTIMIZATION FOR SPEECH INTELLIGIBILITY

Utterance-based optimization treats each utterance as a whole, addressing frame-boundary effects and enabling objective functions defined over complete utterances. This supports direct optimization for speech intelligibility rather than relying only on MSE.

  • Utterance-based optimization addresses frame-boundary problems and permits objective functions computed over entire utterances.
  • Treating each utterance as a whole can make its global optimal solution more accessible during optimization.
  • The section motivates speech-intelligibility optimization by showing that MSE may not suit human hearing characteristics.

A. Problems of Applying MSE as an Objective Function

The section argues that MSE is not reliably aligned with human speech perception or intelligibility. Signals with lower MSE can perform worse perceptually, motivating STOI-based evaluation and optimization.

  • A. Problems of Applying MSE as an Objective Function: MSE compares signal similarity without modeling human perception, so lower MSE does not guarantee better speech performance.
  • A. Problems of Applying MSE as an Objective Function: Frequency-domain MSE treats attenuation and amplification distortions without assuming their perceptual effects are equivalent.
  • A. Problems of Applying MSE as an Objective Function: A waveform, its negative version, and an amplitude-shifted version can sound identical to people despite very large MSE.
  • A. Problems of Applying MSE as an Objective Function: The relation between waveform MSE and human perception is non-monotonic, allowing larger optimization MSE to produce perceptually more similar speech.
  • A. Problems of Applying MSE as an Objective Function: STOI evaluates correlation between clean and degraded speech temporal envelopes over short segments such as 30 frames.

B. Introduction of STOI

STOI is a perception-oriented intelligibility measure computed from clean and degraded speech, using short-time temporal-envelope correlations across frequency bands and frames. Its short-segment correlation basis creates a mismatch with traditional frame-wise enhancement.

  • STOI measures speech intelligibility by comparing clean and degraded speech through a human-hearing-oriented objective.Its score ranges from 0 to 1, with higher values indicating better speech intelligibility.
  • STOI processing removes silent frames, applies STFT, groups frequencies into 15 one-third-octave bands, and performs subsequent normalization and clipping.The signals use 50% overlapping Hann-windowed frames of 256 samples, zero-padded to 512.
  • The clean-speech envelope is represented over N = 30 frames, corresponding to an analysis length of 384 ms.The representation indexes one-third-octave bands and frames across the utterance.
  • STOI is computed by averaging intermediate intelligibility measures over frequency bands and frames.The intermediate measure is based on normalized temporal-envelope comparisons.
  • Because STOI uses temporal-envelope correlation over short segments, it cannot be optimized by a traditional frame-wise enhancement scheme.The correlation is computed for short segments such as 30 frames.

C. Maximizing STOI for Speech Intelligibility

The proposed training objective embeds STOI computation after the FCN enhancement model and optimizes utterances directly. Because most STOI computation is differentiable, FCN weights can be updated by gradient descent.

  • STOI is cascaded after the FCN model and used as the objective function for utterance-based optimization.The objective function is formed from the STOI computation applied to the model output.
  • Most STOI computation is differentiable, allowing automatic differentiation and gradient-descent updates of the FCN weights.Keras and Theano are used to perform automatic differentiation without explicitly computing the cost-function gradients.
  • The training objective uses clean and estimated utterances, with STOI evaluated for each training utterance and aggregated across the training set.The clean and estimated utterances are indexed by u, and U denotes the total number of training utterances.

IV. EXPERIMENT

Experiments evaluate enhancement models and objective functions on TIMIT, MHINT, and CHiME2. The FCN structure is specified with dataset-dependent K and F settings.

  • Three corpora are used: TIMIT, MHINT, and CHiME2, covering comparison, listening-test, and more difficult noisy-speech conditions.CHiME2 includes both additive and convolutive noise.
  • The frame-based and utterance-based FCN models share the same structure, differing in whether they process fixed-length waveform segments or utterances.The frame-based input is a fixed-length waveform segment containing 512 sample points.
  • For TIMIT, the FCN uses K=5 and F=15; for MHINT and CHiME2, it uses K=7 and F=30.

A. Experiment on the TIMIT data set

The TIMIT experiment trains and tests enhancement models under multiple noise types and SNR levels, then compares frame-based and utterance-based FCN objectives using STOI and PESQ. Utterance-based STOI optimization improves intelligibility, especially at low SNR, while reducing average PESQ.

  • TIMIT data set: 600 TIMIT training utterances are corrupted with five noise types across five SNR levels, while 100 different utterances form the test set.The noise types are Babble, Car, Jackhammer, Pink, and Street; SNRs range from -10 dB to 10 dB.
  • TIMIT data set: The evaluation reports STOI for intelligibility and PESQ for speech quality, comparing frame-based FCN with utterance-based FCN objectives.The compared utterance-based objectives include MSE and STOI.
  • TIMIT data set: An average STOI improvement of 0.04 results when the utterance-based FCN uses STOI as its objective, especially under low-SNR conditions.The average PESQ decreases, while STOI increases as the study’s primary target.
  • TIMIT data set: Utterance-based FCN with MSE outperforms frame-based FCN on both PESQ and STOI.The passage attributes this improvement mainly to solving the frame-boundary problem in frame-based optimization.

B. Experiment on the MHINT data set

The MHINT experiments compare FCN enhancement models trained with MSE, STOI, or both objectives against DNN and BLSTM baselines. STOI optimization improves intelligibility-oriented scores, but combining STOI and MSE better balances intelligibility, quality, noise suppression, and ASR performance.

  • Experiment setup: The FCN architecture uses eight convolutional layers with zero padding, 30 filters per intermediate layer, and no pooling, while DNN and BLSTM provide baselines.The FCN output layer uses one filter; its waveform samples are constrained with tanh.
  • Objective evaluation scores: FCN trained with MSE achieves higher SSNRI and STOI but lower PESQ than BLSTM, using roughly 7% and 23% of BLSTM and DNN parameters, respectively.Changing FCN training from MSE to STOI considerably improves STOI while decreasing PESQ.
  • Learned-filter analysis: STOI-optimized first-layer filters concentrate most pass-bands below 4 kHz, compared with MSE filters covering nearly the entire frequency range.Above 4 kHz, the energy ratio is 21% for STOI-optimized filters versus 31% for MSE-optimized filters.
  • Listening test: STOI-optimized speech has better intelligibility but worse quality than MMSE-based speech, confirming that intelligibility and quality are related but distinct dimensions.Speech quality improved over noisy speech for all enhanced samples, whereas intelligibility was harder to improve.
  • Listening test: Joint MSE and STOI optimization appears to achieve the best combined intelligibility and quality, with statistically significant intelligibility improvement at -6 dB SNR but limited improvement at -3 dB.The limited -3 dB gain is attributed to human robustness to moderate noise; the -6 dB result has p<0.05.
  • ASR evaluation: All FCN-enhanced samples reduce ASR WER relative to noisy speech, but STOI alone performs worse than MSE, whereas combining STOI with MSE provides considerable WER benefits.The authors attribute STOI-only limitations to insufficient noise suppression because STOI ignores non-speech regions and uses correlation-based intermediate intelligibility.

C. Experiment on the CHiME-2 data set

The CHiME-2 experiment evaluates enhancement under simultaneous additive and convolutive noise using the same FCN structure as the MHINT experiment. Performance trends broadly resemble MHINT, while STOI optimization achieves the highest STOI for reverberant speech.

  • Experiment setup: CHiME-2 contains noisy and reverberant mixtures created from WSJ0-5k speech, binaural room impulse responses, and genuine room noise across six SNR levels from -6 dB to 9 dB.The dataset includes 7138 training, 2460 development, and 1980 evaluation utterances.
  • Experiment setup: The CHiME-2 models use the same structure as in the MHINT experiment, with the optimal training epoch selected using the development set.Figure 13 reports average objective evaluation scores for the different models.
  • Objective evaluation results: STOI optimization achieves the highest STOI score for reverberant speech, while SSNR and PESQ improvements are less pronounced than in MHINT.The reduced SSNR and PESQ gains are associated with the presence of convolutive noise.
  • Objective evaluation results: FCN trained with MSE has a PESQ score that can outperform BLSTM in the reverberant-noise task, unlike the corresponding MHINT comparison.The authors note that convolutional reverberation resembles a single-layer FCN without activation, but call for more rigorous verification.

V. DISCUSSION

The discussion argues that directly optimizing STOI improves intelligibility but that STOI alone is insufficient because it leaves silent and high-frequency regions unconstrained. Combining MSE and STOI, while retaining an utterance-based FCN framework, provides a broader optimization strategy subject to differentiability.

  • Objective-function design: Direct STOI optimization improves speech intelligibility, but using STOI as the only objective can produce unexpected processing in silent and high-frequency regions.The authors therefore describe combined MSE-and-STOI optimization as a reasonable solution.
  • Objective-function design: Jointly optimizing MSE and STOI achieved the best performance in listening tests and ASR experiments.The comparison is reported for the combined objective rather than STOI alone.
  • Objective-function design: A conditional objective applies MSE only in silent regions to reduce its influence on speech regions, but preliminary experiments show that this formulation performs poorly.Because silent-region targets are near zero, the model can learn to scale down its weights, yielding a trivial low-energy output similar to STOI-optimized speech.
  • Framework scope: The proposed utterance-based waveform FCN can be extended from local to global time scales and from time-domain to frequency-domain evaluation metrics.STOI is presented as one example; other differentiable metrics can be optimized directly, while nondifferentiable metrics require continuous approximations.
  • Implications: Listening tests and ASR results confirm that changing the objective function can substantially alter enhancement outcomes even when the model structure remains fixed.The authors note that future, more accurate evaluation metrics could further reduce the mismatch with human perception.

VI. CONCLUSION

The paper concludes that an end-to-end utterance-based raw-waveform FCN can train enhancement models with evaluation metrics, addressing mismatched objectives and several limitations of conventional frame-based spectral processing.

  • Conclusion: The proposed framework incorporates testing evaluation metrics into model-parameter training instead of using objectives inconsistent with evaluation.This directly targets the mismatch between training objectives and evaluation criteria.
  • Conclusion: Utterance-based waveform optimization addresses objective mismatch, avoids time-frequency preprocessing, retains phase information, and removes discontinuities from frame-based processing.The conclusion attributes these advantages to direct waveform denoising and treating each utterance as a whole.
Loading 1709.03658v2…