Source-linked AI summary
An Investigation of Incorporating Mamba for Speech Enhancement
Rong Chao, Wen-Huang Cheng, Moreno La Quatra, Sabato Marco Siniscalchi, Chao-Han Huck Yang, Szu-Wei Fu, Yu Tsao
TL;DR
The paper examines whether Mamba, presented as an efficient long-sequence model, can be applied to speech enhancement and compared with Transformer-based solutions. The authors deploy Mamba in basic and advanced speech-enhancement models, test causal and non-causal configurations and different losses, and evaluate PCS and ASR preprocessing. On VoiceBank-DEMAND, SEMamba with PCS achieved a SOTA PESQ score of 3.69; in the advanced configuration, replacing Transformers with Mamba reduced FLOPs by up to ∼12% with a similar amount of parameters.
Problem
The paper examines whether Mamba, presented as an efficient long-sequence model, can be applied to speech enhancement and compared with Transformer-based solutions.
Source: §1 · p. 1Method
The authors deploy Mamba in basic and advanced speech-enhancement models, test causal and non-causal configurations and different losses, and evaluate PCS and ASR preprocessing.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
This work aims to investigate the use of a recently proposed, attention-free, scalable state-space model (SSM), Mamba, for the speech enhancement (SE) task. In particular, we employ Mamba to deploy different regression-based SE models (SEMamba) with different configurations, namely basic, advanced, causal, and non-causal. Furthermore, loss functions either based on signal-level distances or metric-oriented are considered. Experimental evidence shows that SEMamba attains a competitive PESQ of 3.55 on the VoiceBank-DEMAND dataset with the advanced, non-causal configuration. A new state-of-the-art PESQ of 3.69 is also reported when SEMamba is combined with Perceptual Contrast Stretching (PCS). Compared against Transformed-based equivalent SE solutions, a noticeable FLOPs reduction up to ~12% is observed with the advanced non-causal configurations. Finally, SEMamba can be used as a pre-processing step before automatic speech recognition (ASR), showing competitive performance against recent SE solutions.
1. INTRODUCTION
The introduction frames speech enhancement as neural mapping from noisy speech to cleaner speech and motivates Mamba as a resource-efficient alternative for sequence modeling. It introduces SEMamba configurations and compares them with Transformer-based solutions.
1. INTRODUCTION
- Speech enhancement has been modeled with neural architectures including CNNs, recurrent networks, Transformers, and Conformers, while training objectives range from signal-level distances to learned metric-oriented scores.PCS is described as stretching target-feature contrast according to perceptual importance without a notable runtime-cost increase.Source: §1 · p. 1
- Mamba is presented as a state-space model with long-range modeling capability and computational scaling linear in sequence length, unlike Transformers’ quadratic scaling.The introduction positions these properties as motivation for testing Mamba in speech enhancement.Source: §1 · p. 1
- The study compares Transformer- and Mamba-based speech enhancement using basic and advanced, causal and non-causal configurations, with distance-based and metric-oriented losses.The paper also evaluates consistency loss, PCS, and SEMamba as preprocessing for ASR.Source: §1 · pp. 1–2
- PESQ 3.55 is reported for advanced non-causal SEMamba on VoiceBank-DEMAND, while SEMamba with PCS reaches a state-of-the-art PESQ of 3.69.The introduction also reports up to ∼12% fewer FLOPs for advanced non-causal Mamba configurations and competitive ASR performance.Source: §1 · p. 2
2. MAMBA: LINEAR-TIME SEQUENCE MODELING WITH SELECTIVE STATE SPACES
This section describes state-space sequence modeling and Mamba’s selective, hardware-aware design. It explains the latent-state mapping and the architecture choices used to improve computational efficiency.
2. MAMBA: LINEAR-TIME SEQUENCE MODELING WITH SELECTIVE STATE SPACES
- Mamba adds input-dependent selection and a hardware-aware scan algorithm, with computation scaling linearly in sequence length.The section describes SSMs as handling long-dependent sequences with low computation and memory requirements.Source: §2 · p. 2
- An SSM maps input x through a higher-dimensional latent state h to output y using discretized state matrices.The passage gives the recurrence h_n = Āh_(n−1) + B̄x_n and output y_n = Ch_n.Source: §2 · p. 2
- Mamba stacks SSM components with linear projections, concentrating most parameters in those projections and using SiLU/Swish activation, normalization, and residual connections.The passage says the design reduces parameters in the inner SSM.Source: §2 · p. 2
3. MAMBA IN SPEECH ENHANCEMENT
SEMamba is deployed in two architectures: a basic speech-enhancement design with Mamba and an advanced design that replaces MP-SENet’s attention-based blocks with Mamba.
3. MAMBA IN SPEECH ENHANCEMENT
- SEMamba uses Mamba in either a basic speech-enhancement architecture or an MP-SENet-based advanced architecture that replaces attention-based blocks.Source: §3 · p. 2
3.1. SEMamba-basic
SEMamba-basic is causal and processes noisy speech through spectral magnitude enhancement before waveform reconstruction with the noisy phase.
3.1. SEMamba-basic
- SEMamba-basic transforms noisy speech with an STFT, compresses its magnitude, enhances it with four convolutional layers, two uni-directional Mamba blocks, and a decoder, then restores the waveform using noisy phase.It uses magnitude mean absolute error for training; a Transformer-based version follows the same basic architecture for comparison.Source: §3.1 · p. 2
3.2. SEMamba-advanced
SEMamba-advanced removes the causality constraint and uses a more sophisticated enhancement structure. Its MP-SENet backbone processes magnitude and phase through Time-Frequency Mamba blocks and separate decoders.
3.2. SEMamba-advanced
- Causality is removed in SEMamba-advanced, which uses a more sophisticated speech-enhancement structure.The architecture is shown in Fig. 2.Source: §3.2 · p. 2, Figure 2 · p. 3
- Noisy-speech STFT magnitude and phase are stacked and processed by a feature encoder built around a dilated DenseNet.The encoder is flanked by two convolutional layers.Source: §3.2 · p. 3
- Four Time-Frequency Mamba blocks transform the encoded features before separate magnitude and phase decoders.Each decoder uses a dilated DenseNet followed by a deconvolutional block and 2D-convolution output layer.Source: §3.2 · p. 3
- Training combines a PESQ-based GAN discriminator with time, magnitude, complex, and phase losses.The loss combination follows MP-SENet.Source: §3.2 · p. 3
3.3. SEMamba-advanced & additional designs
The additional designs explore bidirectional Mamba, consistency loss, and perceptual contrast stretching. The supplied comparison tables cover basic and advanced Mamba-versus-Transformer architectures.
3.3.1. From uni- to bi-directional Mamba
- Bidirectional Mamba processes each input sequence in its original and reversed forms, concatenates the outputs, and applies Conv1D.The equation defines x and y as input and output, respectively.Source: §3.3.1 · p. 3
3.3.2. Consistency loss (CL)
- Consistency loss minimizes the gap between the model’s direct complex-spectrum output and the spectrum obtained by iSTFT followed by STFT.The process returns the spectrum to the STFT domain, helping ensure complex-domain enhancements translate to the time-frequency domain.Source: §3.3.2 · p. 3
- Table 1 compares basic Mamba and Transformer SE architectures, while Table 2 compares them in an advanced architecture excluding CL and PCS.Source: Table 1 · p. 4, Table 2 · p. 4
3.3.3. Perceptual contrast stretching (PCS)
- PCS stretches the enhanced signal’s magnitude spectrum by frequency band according to perceived importance, as an auxiliary post-enhancement step to refine perceptual quality.Source: §3.3.3 · p. 4
3.4. Dataset
The study uses VoiceBank-DEMAND, a benchmark dataset of noisy speech created by mixing VoiceBank clean recordings with DEMAND noise.
3.4. Dataset
- VoiceBank-DEMAND contains 30 speakers, with 28 assigned to training and 2 to testing.The dataset includes 11,572 training utterances and 824 test utterances.Source: §3.4 · p. 4
- Training mixtures use SNRs of 0, 5, 10, and 15 dB, while test mixtures use 2.5, 7.5, 12.5, and 17.5 dB.Source: §3.4 · p. 4
3.5. Experimental Setup
Experiments evaluate SEMamba configurations, losses, and ASR preprocessing on VoiceBank-DEMAND using standard speech-enhancement metrics. Table 3 reports results for SEMamba and other established enhancement methods on that dataset.
3.5. Experimental Setup
- Enhancement is evaluated with wide-band PESQ, CSIG, CBAK, COVL, and STOI, with higher values indicating better outcomes.Recordings are downsampled from 48 kHz to 16 kHz.Source: §3.5 · p. 4
- Table 3 reports VoiceBank-DEMAND results for SEMamba and several well-known speech-enhancement methods.Source: Table 3 · p. 4
- Experiments assess SEMamba-basic, an advanced Mamba architecture, advanced SEMamba with consistency loss and PCS, and SEMamba preprocessing for ASR.Source: §3.5 · p. 4
4. EXPERIMENTS
Experiments compare Mamba- and Transformer-based speech enhancement systems, then evaluate SEMamba against established methods and as ASR preprocessing. SEMamba reaches a state-of-the-art PESQ of 3.69 with PCS, while advanced Mamba configurations also show lower computational costs and improved ASR word error rates.
4.1. Experimental Results
- 66.37% fewer FLOPs and 60.22% fewer parameters are reported for the causal Mamba block than the Transformer block in the basic architecture.For non-causal configurations, the reported reductions are 53.09% fewer FLOPs and 28.28% fewer parameters; Mamba's performance is comparable or superior in both configurations.Source: §4.1 · p. 4
- 11.88% fewer FLOPs than Conformer are reported for Mamba in the advanced, non-causal architecture, alongside superior performance.The compared advanced systems are non-causal; Mamba (Bi) is bidirectional and Mamba (Uni) is unidirectional.Source: §4.1 · pp. 4–5
- 3.69 PESQ is reported for SEMamba with PCS, achieving state-of-the-art results on VoiceBank-DEMAND.With consistency loss, SEMamba reaches a PESQ of 3.55 on the same dataset.Source: §4.1 · p. 5
- 12.22% and 12.90% relative WER reductions are reported for SEMamba on Whisper-Base and Whisper-LargeV3, respectively.The comparison is shown in Fig. 3; the tested SEMamba-Advanced configuration had PESQ=3.55 and was evaluated without retraining.Source: §4.1 · p. 5, Figure 3 · p. 5
5. CONCLUSION
The study investigates Mamba, a selective state-space model, for speech enhancement using basic and advanced schemes and signal-level and metric-oriented objectives. SEMamba with PCS achieves a VoiceBank-DEMAND PESQ of 3.69, and SEMamba preprocessing is reported to yield lower WERs overall; future work will explore other speech-generation tasks.
5. CONCLUSION
- SEMamba with PCS achieves a state-of-the-art PESQ score of 3.69 on VoiceBank-DEMAND.Source: §5 · p. 5
- The study investigates Mamba, a selective state-space model, for speech enhancement using basic and advanced schemes and signal-level and metric-oriented objectives.Source: §5 · p. 5
- SEMamba used as preprocessing before ASR with advanced technologies allows, overall, lower WERs.Source: §5 · p. 5
- Future research will explore Mamba’s potential in other speech-generation tasks.Source: §5 · p. 5