Source-linked AI summary
On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression
Jun Qi, Jun Du, Sabato Marco Siniscalchi, Xiaoli Ma, Chin-Hui Lee
TL;DR
DNN-based vector-to-vector regression lacks a theoretical comparison of MAE and MSE's generalization and noise robustness. The paper derives MAE-based bounds, links MAE to Laplacian errors, and evaluates the losses in speech enhancement, where MAE-trained models perform better on loss, PESQ, and STOI.
Problem
The paper addresses the missing theoretical comparison between MAE and MSE for generalization capabilities and robustness to noise in DNN-based vector-to-vector regression.
Method
The paper derives bounds using MAE's Lipschitz continuity, extends them to additive noise, connects MAE with Laplacian distribution, and tests MAE and MSE in speech enhancement.
Results
MAE-trained models achieve lower MAE and MSE values and higher PESQ and STOI scores than MSE-trained models in speech enhancement.
Takeaways & Limitations
The results support MAE as an effective objective function for DNN-based vector-to-vector regression in the evaluated speech enhancement setting.
Abstract
from arXiv · showhide
In this paper, we exploit the properties of mean absolute error (MAE) as a loss function for the deep neural network (DNN) based vector-to-vector regression. The goal of this work is two-fold: (i) presenting performance bounds of MAE, and (ii) demonstrating new properties of MAE that make it more appropriate than mean squared error (MSE) as a loss function for DNN based vector-to-vector regression. First, we show that a generalized upper-bound for DNN-based vector- to-vector regression can be ensured by leveraging the known Lipschitz continuity property of MAE. Next, we derive a new generalized upper bound in the presence of additive noise. Finally, in contrast to conventional MSE commonly adopted to approximate Gaussian errors for regression, we show that MAE can be interpreted as an error modeled by Laplacian distribution. Speech enhancement experiments are conducted to corroborate our proposed theorems and validate the performance advantages of MAE over MSE for DNN based regression.
I. INTRODUCTION
The paper addresses the missing theoretical comparison of MAE and MSE for DNN-based vector-to-vector regression. It develops MAE bounds and distributional interpretations, then evaluates them in speech enhancement.
- The paper targets the lack of theory comparing MAE and MSE in generalization and noise robustness for DNN-based vector-to-vector regression.
- MAE is shown to support upper bounds through Lipschitz continuity, including a new bound for regression with additive noise.
- The paper connects MAE with Laplacian errors, contrasting MSE's association with Gaussian errors.
- Speech enhancement experiments assess the theoretical derivations and compare MAE with MSE as regression losses.Speech enhancement is modeled as an unbounded mapping from R^d to R^q.
- The paper is organized around mathematical preliminaries, MAE properties, the Laplacian connection, speech enhancement experiments, and conclusions.
2. Lipschitz continuity
This section defines MAE and MSE for vector-to-vector regression using L1 and L2 norms, respectively.
- MAE measures the average magnitude of absolute differences between predicted and target vectors.
- MSE is a quadratic scoring rule that measures the average magnitude of differences between predicted vectors and actual observations.
5. Empirical Rademacher Complexity
The section uses empirical Rademacher complexity to bound DNN regression estimation error and shows that MAE, unlike MSE, supplies the required Lipschitz property.
- Empirical Rademacher complexity measures a hypothesis space using Rademacher random variables over observed samples.
- Larger empirical Rademacher complexity is associated with greater likelihood of overfitting to training data.
- The Lipschitz property is fundamental for deriving an upper bound on estimated regression error.
- MAE is 1-Lipschitz continuous, whereas MSE cannot satisfy Lipschitz continuity.
- The relevant function family consists of DNN-based vector-to-vector functions, with a corresponding family of generalized MAE loss functions.
- The estimation error for DNN-based vector-to-vector functions can be upper-bounded by empirical Rademacher complexity.
AGAINST ADDITIVE NOISES
The MAE loss provides bounded sensitivity to additive noise in DNN-based vector-to-vector regression, unlike MSE. This supports robustness guarantees when the imposed noise is small.
- MAE enables an upper bound on regression errors, supporting DNN robustness against additive noise.
- Theorem 1 applies to MAE because it is Lipschitz continuous, whereas it does not apply to MSE.The DNN regression function is treated as Lipschitz continuous, and the proof uses the triangle inequality and a Lipschitz-continuity lemma.
- For MSE, differences caused by additive noise are unbounded, while MAE guarantees an upper bound.
- Small additive noise cannot lead to significant performance degradation under the derived MAE upper bound.
V. CONNECTION OF MAE LOSS FUNCTION TO LAPLACIAN DISTRIBUTION
The paper connects MAE with Laplacian-distribution loss and MSE with Gaussian-distribution loss. Under equal variance-related terms, MAE retains the reported regression advantage over MSE.
- MAE and MSE are linked to Laplacian-distribution and Gaussian-distribution losses, respectively.The paper modifies both losses with variance-related terms to establish these connections.
- MAE and MSE become special cases of the corresponding distribution-based losses when variance terms are omitted.
- When every dimension has constant variance, the Laplacian- and Gaussian-based losses exactly correspond to MAE and MSE, respectively.
- MAE retains an advantage over MSE when the variance-related terms are the same.Speech enhancement experiments are used to verify this relationship.
VI. EXPERIMENTS
Speech enhancement experiments assess the theoretical derivations and compare MAE with MSE under various noisy conditions.
- The experiments use speech enhancement as a regression task to corroborate the paper’s theorems.Regression-based speech enhancement maps input features from R^d to R^q and aims to approximate clean speech features.
- The experiments compare MAE and MSE under various noisy conditions to test whether MAE achieves better regression performance.
A. Data Preparation
The data preparation uses the Edinburgh noisy speech database, synthesizing noisy training utterances across multiple signal-to-noise ratios and noise types.
- The Edinburgh noisy speech database contains 23075 clean training utterances and 824 clean testing utterances.
- Noisy training data are synthesized at 15 dB, 10 dB, 5 dB, and 0 dB SNR levels.
- The training synthesis uses 40 noisy types spanning domestic, office, public-space, transportation, and street environments.
B. Experimental Setup
The experiments use feed-forward DNNs to map noisy-speech LPS feature vectors to clean or enhanced-speech LPS vectors. Evaluation combines MAE and MSE with perceptual quality and intelligibility measures.
- Feed-forward DNNs take normalized noisy-speech LPS feature vectors as inputs and produce clean or enhanced-speech LPS features.Clean LPS vectors serve as training targets, while the top layer generates enhanced LPS vectors at test time.
- Evaluation uses MAE, MSE, PESQ, and STOI to assess regression error, speech quality, and intelligibility.PESQ ranges from −0.5 to 4.5, with higher scores indicating better perceived quality; STOI ranges from 0 to 1 and measures intelligibility.
C. Evaluation Results
On the Edinburgh speech corpus, MAE-trained DNNs outperform MSE-trained DNNs on regression losses and speech-quality and intelligibility metrics. The observed gains are presented as consistent with the paper’s robustness-bound and Laplacian-distribution arguments.
- Regression losses: MAE-trained DNNs achieve lower MAE than MSE-trained DNNs under the same noisy condition in every Table I column.A reported comparison is 0.7812 for DNN-MAE versus 0.8278 for DNN-MSE.
- Regression losses: 0.7954 vs. 0.8371: DNN-MAE achieves a lower MSE score than DNN-MSE.
- Speech quality and intelligibility: 2.93 vs. 2.85 for PESQ and 0.8509 vs. 0.8317 for STOI: MAE-trained DNNs outperform MSE-trained DNNs on both speech metrics.
- Interpretation: The performance advantage of DNN-MAE is reported as consistent with the additive-noise upper bound and the MAE–Laplacian connection.
- Conclusion: The conclusion reports lower loss values and better enhanced-speech PESQ and STOI scores for MAE-optimized DNN regression.