SE-MSB: End-to-End Unpaired Speech Enhancement using Mamba Schrödinger Bridges

• Paired enhancement trains on clean speech and corresponding synthetically degraded examples, but paired data collection for real-world audio may be infeasible or impossible.The figure contrasts paired clean/degraded samples with independent samples from the clean and degraded distributions.Source: §1 · p. 1, Figure 1 · p. 2
• SE-MSB is on par with or outperforms state-of-the-art unpaired enhancement methods while running orders of magnitude faster at inference.The introduction also reports that it outperforms paired methods in some cases while remaining highly competitive in efficiency.Source: §1 · p. 2
• SE-MSB combines Diffusion Schrödinger Bridges, which learn stochastic maps between arbitrary distributions, with an efficient Mamba model operating on raw audio waveforms.The introduction attributes its speed to Mamba’s linear sequence-length scaling and the model’s few-step diffusion.Source: §1 · p. 2

• A2ASB learns paired audio-distribution maps and achieves state-of-the-art results on paired restoration tasks, while CycleGAN audio mappings are described as unstable and requiring four jointly optimized networks.The passage gives bandwidth extension and inpainting as A2ASB examples, and whisper-to-normal speech conversion as a CycleGAN example.Source: §2 · p. 2
• Gaussian Flow Bridges map samples through an intermediate Gaussian, which the authors say yields non-optimal transport; diffusion posterior sampling requires knowledge of the degradation process.The cited passage describes these as prior unpaired enhancement approaches.Source: §2 · p. 3
• DSB Matching learns maps between arbitrary distributions from unpaired samples, while prior Mamba-plus-DSB speech enhancement uses STFT representations with fixed transformation hyperparameters.The authors identify the fixed time-frequency trade-off as a limitation of those representations.Source: §2 · p. 3
• Table 1 compares current paired and unpaired diffusion models for speech enhancement; its caption notes that GFB uses internal STFT encoding and ISTFT decoding despite processing raw audio.Source: Table 1 · p. 3

3 Method

• The method learns forward and backward stochastic processes that transport samples between clean and degraded speech distributions.A shared neural network parameterizes both drifts; the backward process maps degraded speech to clean speech, and the forward process does the reverse.Source: §3 · p. 3
• DSB Matching pre-trains with independently sampled clean and degraded examples, then fine-tunes using couplings obtained by simulating the current processes.The objective in both phases is to predict the forward and backward drift at sampled intermediate times.Source: §3 · p. 4

Mamba Diffusion Model architecture

• The Mamba Diffusion Model encodes raw waveforms with a 1D convolution, mixes in conditioning, and processes the representation with Mamba Diffusion Blocks.The architecture uses an internal learned representation rather than a fixed time-frequency representation.Source: Mamba Diffusion Model architecture · p. 4, Figure 3 · p. 6
• At the same compute budget, SE-MSB outperforms unpaired state-of-the-art models; its metrics are also nearly unchanged across diffusion-step counts.Figure 2 compares total compute with SISDRi and FAD; point area scales with mean inference time.Source: Figure 2 · p. 5
• The paper estimates selective-scan FLOPs as 2BTN(HQ + 3D), with a separate 1D causal-convolution cost of 2BTDK.The selective scan and convolution account for most of the Mamba2 FLOP count described in the passage.Source: Mamba Diffusion Model architecture · p. 5
• SE-MSB is almost unaffected by the number of diffusion steps, indicating it can perform few-step generation.Figure 4 plots metrics as a function of diffusion-step count.Source: Figure 4 · p. 6

4 Experiments

• SE-MSB is evaluated on dereverberation, denoising, declipping, and mixtures of reverberation, noise, and clipping.The clean speech comes from VCTK, which is also used to simulate degraded speech, but the model never sees clean/degraded pairs during training.Source: §4 · pp. 6–7
• Evaluation includes VCTK with unseen room impulse responses and an independent Libri ASR Corpus dataset to assess generalizability.The experiments also report content-fidelity and perceptual-quality metrics, including WER, SR-CS, SISDRi, pMOS, and FAD.Source: §4 · pp. 7–8
• Denoising uses WHAM! noise mixed at 5 dB SNR, while clipping removes a randomly selected 0–1 dB of signal power.Dereverberation applies room impulse responses from public datasets to clean speech.Source: §4 · p. 7
• Dereverberation comparisons include unpaired and paired enhancement methods, an unprocessed-audio baseline, and computational-efficiency measures.The listed unpaired baselines include GFB, BUDDy, and WPE; paired baselines include SGMSE and A2SB. Efficiency is assessed using FLOPS, inference time, and trainable parameters.Source: §4 · p. 7, Table 3 · p. 8

5 Results

• SE-MSB is around two orders of magnitude more efficient than BUDDy in FLOPS, while paired methods outperform unpaired methods on most dereverberation metrics, with negligible gaps on some.The authors attribute the FLOPS difference to the Mamba diffusion model’s linear-time sequence scaling.Source: §5 · p. 8
• SE-MSB outperforms BUDDy by a large margin on mixed reverberation, noise, and clipping, and is comparable to supervised Sepformer.Table 4 reports results for WHAM! noise with reverberation, with and without amplitude clipping.Source: §5 · pp. 8–9, Table 4 · p. 9
• SE-MSB performance is almost unaffected by fewer sampling steps, whereas GFB performance degrades significantly; more SE-MSB steps negatively affect WER and, to a lesser extent, SISDRi.The authors suggest the metric changes may indicate a perception-distortion trade-off.Source: §5 · p. 9

6 Conclusion

• SE-MSB combines Diffusion Schrödinger Bridges with an efficient Mamba diffusion architecture that operates on raw waveforms.The model maps degraded-speech distributions to clean-speech distributions without paired training examples.Source: §6 · p. 9
• SE-MSB performs on par with or better than state-of-the-art unpaired methods while being orders of magnitude faster during inference.The authors report that paired methods moderately outperform SE-MSB overall, with negligible gaps on some metrics.Source: §6 · p. 9
• The authors identify Lombard speech, style transfer, and historical-recording reconstruction as domains where paired data may be unavailable or prohibitively expensive.They present these as potential applications for efficient and robust speech enhancement.Source: §6 · p. 9

A Training

• Training uses AdamW with a constant learning rate of 10^-4, β1 = 0.9, β2 = 0.999, and zero weight decay.The models train for 250k pre-training and 100k fine-tuning iterations, using an L1 loss and gradient clipping at norm 1.0.Source: A Training · p. 12
• Audio is resampled to 16 kHz, normalized to -20 dBFS, and cropped or extended to 1.00 seconds for training and 5.12 seconds for testing.Training samples are multiplied by 20 after normalization.Source: A Training · p. 12
• Fine-tuning simulates forward and backward processes with 20 diffusion steps, and each model trains on one NVIDIA GeForce RTX 4090 GPU for about 1 day and 9 hours.Training uses mixed precision, a batch size of 8 per process (effective batch size 16), and EMA decay 0.9999.Source: A Training · p. 12

A.1 Dataset descriptions

• VCTK supplies clean speech from 109 English speakers, with approximately 400 sentences per speaker and 44 hours total, resampled from 48 kHz to 16 kHz.The authors selected it because of its prominence in unpaired speech enhancement, enabling fairer comparisons.Source: VCTK · p. 12

Room Impulse Responses (RIRs)

• The room-impulse-response collection combines several datasets and contains approximately 50 minutes of audio.The same RIR datasets as the GFB model are used for fairer comparison.Source: Room Impulse Responses (RIRs) · p. 12

WSJ0 Hipster Ambient Mixtures (WHAM!)

• WHAM! provides approximately 78 hours of real-world environmental noise, recorded at 48 kHz and resampled to 16 kHz for training and evaluation.Source: WSJ0 Hipster Ambient Mixtures (WHAM!) · p. 12

Libri Speech

• Dereverberation is evaluated on LibriSpeech test-clean, comprising approximately 5.4 hours of clean read English speech from 40 speakers.Source: Libri Speech · p. 12

Mamba Diffusion Model

• The Mamba Diffusion Model takes waveform, timestep, and forward/backward conditioning inputs, then returns a waveform of the same shape using 10 Mamba blocks and approximately 45 million parameters.An initial convolution reduces sequence length to T/16 and produces 512 channels; each Mamba block uses model dimension 512 and state-space dimension 128.Source: Mamba Diffusion Model · pp. 12–13

Gaussian Flow Bridge (GFB)

• GFB maps distributions to a shared Gaussian latent space and uses reverberation time and clarity as explicit conditioning descriptors.Its roughly 44 million parameters are comparable to SE-MSB’s 45 million, and it uses VCTK speech and the same reverberation datasets.Source: Gaussian Flow Bridge (GFB) · p. 13

Blind Unsupervised Dereverberation with Diffusion Models (BUDDy)

• BUDDy uses a pretrained diffusion prior for clean speech and jointly estimates room-acoustic parameters and the clean signal.It is not explicitly trained for dereverberation; it assumes input speech is clean speech convolved with room acoustics.Source: Blind Unsupervised Dereverberation… · p. 13

Weighted Prediction Error (WPE)

• WPE models late reverberation as a delayed linear autoregressive process in the STFT domain and estimates its tail from past frames without prior room-acoustic knowledge.The method assumes each frame separates into desired early speech and undesired late reverberation; the stated setup uses 20 taps, delay 3, and 5 iterations.Source: Weighted Prediction Error (WPE) · p. 13

Score-based Generative Models for Speech Enhancement (SGMSE)

• SGMSE is trained for paired dereverberation on clean WSJ0 speech convolved with simulated room impulse responses, using a pretrained NCSN++ checkpoint with approximately 65 million parameters [6, 43, 44].Source: Score-based Generative Models for… · pp. 13–14
• A2ASB is trained for paired dereverberation with SE-MSB’s architecture, data, and parameters, and trains only the backward process.With constant β_t, its intermediate formulation simplifies and is described as mathematically equivalent to the DSB-matching intermediate distribution; its backward posterior is likewise equivalent under that assumption.Source: Paired SE-MSB / Audio-to-Audio… · p. 14

Sepformer (Separation Transformer)

• SepFormer learns short- and long-term dependencies through encoded representations and iterative masking with IntraTransformer and InterTransformer attention blocks.It is trained on WHAM! for denoising, rather than mixed speech enhancement tasks.Source: Sepformer (Separation Transformer) · p. 14

D Computational efficiency

• Mamba2 selective scan costs FLOPscan = 2 · B · T · N (Q · H + 3D), with D = H · P; inference timing uses 100 passes on 2.048-second, 16 kHz samples at batch size 1.Measurements are reported for a consumer-grade GPU and CPU, alongside TFLOPS and parameter counts in Figure 3.Source: D Computational efficiency · pp. 14–15

WPE Flops

• WPE FLOP accounting covers covariance and cross-correlation calculations, solving a linear system, and applying that system to the reverberation tail.The covariance calculation is reported as requiring 8TF(DcKt)^2 FLOPs.Source: WPE Flops · p. 15