PaperlayerView exact revision on arXiv

Source-linked AI summary

A Model Compression Method with Matrix Product Operators for Speech Enhancement

Xingwei Sun, Ze-Feng Gao, Zhong-Yi Lu, Junfeng Li, Yonghong Yan

arXiv:2010.04950v1cs.SDcs.LGeess.ASquant-ph

TL;DR

DNN speech enhancement can perform well, but its parameter and memory demands constrain use on resource-limited devices. The paper replaces linear-transformation weights in MLP and LSTM models with MPO representations for compression. MPO compression outperforms pruning at equal compression rates, with further gains at low rates.

  • Problem

    DNN speech enhancement has promising performance, but its large parameter counts and memory requirements are a concern for resource-limited applications.

    Source: §I · p. 2

  • Method

    The method replaces weight matrices in MLP and LSTM linear transformations with MPO decompositions before training, and uses the models to estimate the ideal ratio mask.

    Source: §I · p. 2

  • Results

    MPO-based compression outperforms pruning in speech-enhancement performance at the same compression rates for MLP and LSTM models, with further improvement at low rates.

    Source: §I · p. 2, §VI · p. 9

  • Takeaways & Limitations

    The authors consider MPO compression more suitable for platforms with limited memory and high computational ability for tensor products, especially cloud-free applications.

    Source: §VI · p. 9

Abstract

from arXiv · show

The deep neural network (DNN) based speech enhancement approaches have achieved promising performance. However, the number of parameters involved in these methods is usually enormous for the real applications of speech enhancement on the device with the limited resources. This seriously restricts the applications. To deal with this issue, model compression techniques are being widely studied. In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement. In this method, the weight matrices in the linear transformations of neural network model are replaced by the MPO decomposition format before training. In experiment, this process is applied to the causal neural network models, such as the feedforward multilayer perceptron (MLP) and long short-term memory (LSTM) models. Both MLP and LSTM models with/without compression are then utilized to estimate the ideal ratio mask for monaural speech enhancement. The experimental results show that our proposed MPO-based method outperforms the widely-used pruning method for speech enhancement under various compression rates, and further improvement can be achieved with respect to low compression rates. Our proposal provides an effective model compression method for speech enhancement, especially in cloud-free application.

I. INTRODUCTION

DNN speech enhancement performs well, but its parameter and memory demands constrain use on resource-limited devices. The paper applies MPO compression to MLP and LSTM models and reports better enhancement performance than pruning at equal compression rates.

I. INTRODUCTION

  • MPO compression outperforms pruning for MLP and LSTM speech enhancement at the same compression rates, with further gains at low rates.The models estimate the ideal ratio mask, and the paper evaluates compression rates from 5 to 100 times.Source: §I · p. 2
  • The method replaces linear-transformation weight matrices with MPO representations before training, including both input-to-hidden and hidden-to-hidden LSTM matrices.The paper distinguishes this LSTM application from earlier work that compressed only the input-to-hidden matrix using Tensor-Train format.Source: §I · p. 2
  • MPO compression can be integrated into network models with linear transformations and trained alongside the models’ other parts.The paper identifies this as a method benefit beyond the evaluated MLP and LSTM models.Source: §I · p. 2
  • DNN speech enhancement has achieved promising performance, but its large parameter counts and memory requirements are a concern for resource-limited applications.The paper positions model compression as a way to address the trade-off between performance and compact model size.Source: §I · p. 2

II. DNN-BASED MONAURAL SPEECH ENHANCEMENT

The system estimates a time-frequency mask from noisy-speech features and uses it to reconstruct enhanced speech. It describes the noisy-speech signal model, acoustic features, and the ideal ratio mask training target.

II. DNN-BASED MONAURAL SPEECH ENHANCEMENT

A. Signal Model

  • The received noisy signal is modeled as target speech plus additive background noise, and the enhancement objective is to remove noise while retaining target speech.The signal model is also transformed to the frequency domain using the short-time Fourier transform.Source: A. Signal Model · p. 3

B. Acoustic Features and Training Target

  • The DNN estimates the ideal ratio mask (IRM), a masking-based training target, using a complementary feature set that includes AMS, RASTA-PLP, MFCC, cochleagram response, and their deltas.During reconstruction, the estimated IRM is applied to noisy-speech magnitude and the noisy phase is used with inverse STFT.Source: B. Acoustic Features and Training Target · p. 3

C. DNN Model Architecture

The section outlines MLP and LSTM architectures used for mask estimation. MLP uses fully connected layers, while LSTM models temporal dynamics through memory and gates.

1) Basic Architecture of MLP Model:

  • The MLP uses multiple fully connected layers; each layer applies a trainable weight matrix and bias to its input before an activation function.The described MLP is trained using back propagation.Source: 1) Basic Architecture of MLP Model · p. 3

2) Basic architecture of LSTM Model:

  • The LSTM models speech temporal dynamics with a memory cell and input, forget, and output gates that regulate cell updates and information passed onward.The input gate controls information added to the cell, the forget gate controls what is erased, and the output gate controls what is transformed to the next layer.Source: 2) Basic architecture of LSTM Model · p. 3

III. MPO-BASED MODEL COMPRESSION

The section introduces MPO-based compression by representing model linear transformations with a compact product of local tensors. Tunable factorization and bond dimensions set the compression rate and trade off model size against fitting ability.

III. MPO-BASED MODEL COMPRESSION

  • Replacing model linear transformations with MPO representations yields compact models that can be trained end-to-end alongside the rest of the model.The section states that no extra training process is needed.Source: §III · p. 3

A. Matrix Product Operators

  • An MPO is constructed by reshaping and transposing a weight matrix into a higher-dimensional tensor, then expressing it as a product of local tensors.Fig. 2 visualizes how a single matrix element maps into the higher-dimensional representation.Source: A. Matrix Product Operators · p. 4
  • MPO represents a higher-order tensor as a sequential product of local tensors, with tunable factorization and bond dimensions controlling the compression rate.Reducing bond dimensions removes redundant parameters, but must be balanced against fitting ability.Source: A. Matrix Product Operators · p. 4
  • The MPO structure reduces parameter-count scaling with tensor dimensions from exponential to polynomial.Source: A. Matrix Product Operators · p. 4

B. MPO-based Compression of MLP Model

The MLP compression method replaces the fully connected layer's weight matrix with an MPO representation. The resulting linear transformation operates on a reshaped input tensor, and the MPO parameters remain trainable by back propagation.

B. MPO-based Compression of MLP Model

  • The fully connected layer's weight matrix is factorized into a product of MPO tensors, each with a four-dimensional shape and tunable hyperparameters.The factorization is applied to the linear transformation, apart from its activation function.Source: B. MPO-based Compression of MLP Model · p. 4
  • The MPO linear transformation operates on the input after it is reshaped and transposed into an N-dimensional tensor.Source: B. MPO-based Compression of MLP Model · p. 4
  • Before training, the MLP weight matrix is replaced by an MPO representation, whose parameters are trained with the rest of the model using back propagation.The passage describes the parameter compression rate for one fully connected layer.Source: B. MPO-based Compression of MLP Model · p. 4

C. MPO-based Compression of LSTM Model

The LSTM compression method applies MPO decomposition to its input-to-hidden and hidden-to-hidden weight matrices. It reorganizes the LSTM operators and expresses the transformations in MPO format.

C. MPO-based Compression of LSTM Model

  • The method reorganizes the LSTM's basic operators, following, before expressing the transformations in MPO format.The example is a first-layer LSTM with Nx input units and Nh hidden units.Source: C. MPO-based Compression of LSTM Model · pp. 4–5
  • Fig. 2 depicts how a standard weight-matrix element is transformed into a higher-dimensional representation and a sequence of MPO matrices.Source: Figure 2 · p. 5
  • The LSTM's input-to-hidden and hidden-to-hidden weight matrices are compressed using MPO decomposition.The matrices have dimensions 4Nh × Nx and 4Nh × Nh, respectively.Source: C. MPO-based Compression of LSTM Model · p. 5
  • MPO tensors are used for both input-to-hidden and hidden-to-hidden matrices, with dimensions factored according to the hidden-unit size Nh.Source: C. MPO-based Compression of LSTM Model · p. 5

A. Data Preparation

The experiments use separate training and test speech utterances and evaluate two monaural enhancement tasks with different noise splits. Baseline MLP and LSTM systems estimate ideal ratio masks using the described acoustic features and training setup.

A. Data Preparation

  • The study uses 4,620 Chinese speech utterances for training and 100 separate utterances for testing, with 20 noise types divided into three datasets.The noise types come from NOISEX92, DEMAND, and CHiME-III.Source: A. Data Preparation · p. 5
  • Task I trains and tests on separate halves of noise dataset C, while Task II trains on datasets A and B and tests on C.Both tasks use the same speech datasets.Source: A. Data Preparation · p. 5
  • Training and test mixtures use −5, 0, and 5 dB SNRs, producing 55,440 training and 1,200 test noisy utterances per task.Utterances are expanded or truncated to five seconds before mixing.Source: A. Data Preparation · p. 6

B. Baseline and Training Details

  • The baseline MLP has 3.54 million parameters, while the three-layer, 512-unit LSTM with a 256-unit output layer has 5.90 million.The MLP uses three prior frames and the current frame as input; the LSTM uses complementary features without frame expansion.Source: B. Baseline and Training Details · p. 6
  • The systems use MSE loss, Adam, dropout at 0.3, and 50 training epochs.The learning rate starts at 0.0005 and decays during training.Source: B. Baseline and Training Details · p. 6

C. Comparison Methods and Performance Measures

The experiments compare MPO compression with pruning in MLP and LSTM speech-enhancement models using objective speech-quality and intelligibility measures. For MLPs, MPO outperforms pruning across compression rates in both evaluated tasks, with especially small losses at high compression and further PESQ and SNR gains at low compression.

C. Comparison Methods and Performance Measures

  • The comparison evaluates pruning and MPO at compression rates from 5× to 100× using STOI, PESQ, and global SNR.Pruning iteratively removes low-magnitude connections; the experiments apply both methods to MLP and LSTM speech-enhancement models.Source: C. Comparison Methods and Performance… · p. 6

V. EXPERIMENTAL RESULTS AND ANALYSIS

  • The evaluation compares compressed and uncompressed MLP and LSTM models across two monaural speech-enhancement tasks and multiple compression rates.For MLP MPO settings, the factorization dimensions are fixed by weight-matrix shape while the bond dimension is tuned for compression rate.Source: §V · p. 6, 1) Detailed Settings of the MPO and… · p. 6, Table II · p. 7

2) Evaluation Results in Speech Enhancement:

  • In Task II, MPO outperforms pruning at the same compression rate in matched dataset B and mismatched dataset C, with performance declining more slowly as compression increases.Figure 3 averages results over noises in each dataset and all mixing SNR levels.Source: Figure 3 · p. 7, 2) Evaluation Results in Speech… · p. 7
  • At 100× compression, the MLP incurs losses of 0.85% STOI, 0.05 PESQ, and 0.14 dB SNR; MPO outperforms pruning at all rates in Task I.The uncompressed MLP improves on noisy speech by 9.09% STOI, 0.58 PESQ, and 8.89 dB SNR on average across mixing SNR levels.Source: 2) Evaluation Results in Speech… · pp. 6–7
  • At low compression rates, MPO-compressed MLPs further improve PESQ and SNR; at higher rates, their performance declines relatively little compared with pruning.The authors describe MPO compression as a possible regularization of redundant parameters.Source: 2) Evaluation Results in Speech… · p. 7

B. Evaluation of the Compressed LSTM Models

For LSTM compression, the method decomposes both input-to-hidden and hidden-to-hidden weight matrices. The experiments fix factorization dimensions and tune bond dimensions to reach different compression rates, using layer-specific decomposition solutions.

1) Detailed Settings of the MPO and Pruning-based Compression:

  • MPO compression decomposes both the input-to-hidden weight matrix W and the hidden-to-hidden weight matrix U in each LSTM layer.The decomposition also introduces adjustable factorization and bond-dimension parameters.Source: 1) Detailed Settings of the MPO and… · p. 7
  • The LSTM settings use layer-specific decomposition solutions and identical bond-dimension factors for W and U within each LSTM layer, tuned for the target compression rate.The solutions cover LSTM1, LSTM2, LSTM3, and the fully connected layer.Source: 1) Detailed Settings of the MPO and… · p. 7, Table V · p. 8, Table VI · p. 8
  • The authors chose the decomposition for convenience and state that different factorization manners should produce almost the same result when the network is away from under-fitting.Pruning instead requires setting weight-matrix sparsity.Source: 1) Detailed Settings of the MPO and… · pp. 7–8

2) Evaluation Results in Speech Enhancement:

Across speech-enhancement evaluations, MPO compression outperforms pruning overall, including at high compression rates. Its computational cost depends on the MPO solution and can be kept near or below that of uncompressed models.

2) Evaluation Results in Speech Enhancement:

  • The uncompressed LSTM improves on noisy speech by 10.8% STOI, 0.77 PESQ, and 9.34 dB SNR on average across mixing SNR levels.In Task I, the LSTM also outperforms the MLP.Source: 2) Evaluation Results in Speech… · p. 8
  • In Task I, MPO beats pruning at all compression rates and evaluation measures; in Task II, it wins overall despite isolated exceptions, including mismatched-noise SNR at 50 times.In Task II, the LSTM model also outperforms the MLP in matched and mismatched noise, though a performance gap remains between those conditions.Source: 2) Evaluation Results in Speech… · pp. 8–9
  • At 100 times compression, the MPO model has only 1.97% STOI, 0.2 PESQ, and 0.5 dB SNR losses, with 59.0 thousand parameters.These losses are reported relative to the uncompressed model.Source: 2) Evaluation Results in Speech… · p. 8

C. Computational Complexity Analysis of the Compressed Model

  • MPO computational cost varies by decomposition solution; it approaches uncompressed-model cost at about 100 times compression for MLPs and at several tested rates for LSTMs.Within a fixed LSTM solution, computational cost decreases as compression increases; the authors say solution selection can maintain or reduce computation while reducing parameters.Source: C. Computational Complexity Analysis… · p. 9

VI. CONCLUSION AND DISCUSSION

  • MPO compression replaces linear-transformation weight matrices in MLP and LSTM models, and the authors identify limited-memory platforms with high tensor-product compute capacity as a suitable setting.They also state it can be integrated into networks with linear transformations and trained end-to-end.Source: §VI · p. 9
Loading 2010.04950v1…