Source-linked AI summary
Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations
Yoto Fujita, Simon Leglaive, Laurent Girin
TL;DR
Masked-generative speech enhancement largely used discrete codec tokens, while decoding-policy trade-offs between enhancement performance and computational cost remained unexplored. MARSE compares iterative decoding policies over continuous codec representations, with results indicating a flexible performance–cost trade-off.
Problem
Masked-generative speech enhancement largely used discrete codec tokens, while decoding-policy trade-offs between enhancement performance and computational cost had not been explored.
Source: §1 · p. 1Method
MARSE formulates speech enhancement as iterative autoregressive decoding of masked clean-speech blocks in continuous NAC space, encompassing autoregressive, non-autoregressive, and masked-autoregressive policies.
Source: §1 · p. 1Results
MARSE-causal performed best among MARSE variants in-domain, whereas non-practical MARSE-NC-oracle achieved the best out-of-domain scores.
Source: §3.2 · p. 4Takeaways & Limitations
MARSE’s decoding policies offer a flexible trade-off between speech-enhancement performance and computational cost.
Source: §4 · p. 4The study replaces the original MAR framework’s diffusion model with a simpler, likely less effective Gaussian model that may affect overall performance.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Most previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained using neural audio codecs (NACs). However, a recent study has shown that continuous latent representations of NACs can be advantageous for SE in terms of speech quality and intelligibility. In this work, we propose masked autoregressive SE (MARSE), a method for SE based on iterative decoding of masked clean speech frames using continuous NAC representations of speech. In particular, we investigate a set of different decoding policies, ceteris paribus, that is, using the same DNN (a Conformer model), the same NAC (the DAC codec) and the same training setup. The results show that MARSE enables a flexible trade-off between SE performance and computational cost. Audio examples and code are available online.
1. INTRODUCTION
MARSE frames speech enhancement as iterative autoregressive decoding of masked continuous neural audio codec representations, and compares decoding policies to examine their performance–cost trade-off. The study is motivated by the potential of continuous representations to preserve speech quality and intelligibility better than token-based methods.
1. INTRODUCTION
- Prior token-based SE methods model codec tokens with Transformers and use non-autoregressive, causal autoregressive, or non-causal masked generative decoding.Earlier SE approaches include conventional signal processing and DNN methods.Source: §1 · p. 1
- Continuous NAC representations can substantially improve speech intelligibility and quality, while token-based SE methods may preserve acoustic details poorly, reducing performance on downstream tasks such as ASR.Most token-based methods focus on a specific decoding policy with a specific experimental setup; the performance and computational-cost trade-off regarding decoding policy has not been explored.Source: §1 · p. 1
- MARSE formulates iterative decoding over blocks of continuous NAC representations and fairly compares decoding policies using the same Conformer, DAC codec, and training setup.The policies vary the number of iterations and/or block definition; experiments report a flexible trade-off between speech-enhancement performance and computational cost.Source: §1 · p. 1
2. METHOD
MARSE formulates speech enhancement as estimating clean speech from noisy speech in a pretrained neural audio codec’s continuous latent space. It iteratively decodes masked clean-speech frames, with decoding policies controlling temporal dependencies and computational cost.
2.3. Training
- Training minimizes a supervised MSE-based negative log-likelihood using paired noisy-clean speech and randomly selected masked frames.The masked-frame count follows the same cosine schedule as inference; one iteration sample approximates the expectation, and masked indices are selected uniformly at random.Source: §2 · p. 2, §2.3 · p. 3
2.1. Masked autoregressive speech enhancement
- MARSE applies the masked autoregressive framework to speech enhancement, iteratively predicting masked frames from noisy speech and previously decoded clean frames.A Conformer processes the noisy and partially masked clean representations, with a learnable mask vector replacing unseen frames. Blocks form an autoregressive partition, with frames within each block conditionally independent; the model uses a simpler Gaussian in place of the original diffusion model, which may affect performance.Source: §2.1 · p. 2
2.2. Decoding policies for inference
- MARSE’s decoding policy specifies how many frames to predict per iteration and which frames to select, allowing causal or non-causal dependencies.Causal decoding ranges from all frames at once, assuming mutual independence, to frame-wise autoregression, which requires T forward passes. Non-causal variants select frames by ground-truth prediction error as an unrealistic oracle, or uniformly at random; block sizes follow a cosine schedule.Source: §2.2 · p. 2
2.4. Mitigating exposure bias with quantization
- To mitigate exposure bias, MARSE quantizes visible latent frames with the pretrained codec quantizer during both training and inference.At inference, each iteration’s predicted frames are fed back as visible frames, unlike training, where visible frames come from clean ground truth; this mismatch can accumulate errors.Source: §2.4 · p. 3
3. EXPERIMENTS
Experiments compare MARSE decoding policies under a shared DAC, dataset, Conformer, training, and evaluation setup. Results show a trade-off between enhancement quality and computational cost, with intelligibility gains and performance that varies by decoding policy and iteration count.
3.1. Experimental setup
- Experiments used DAC continuous latents of dimension D = 1024, Libri1Mix for training, validation, and in-domain testing, and LibriDEMAND for out-of-domain evaluation.Libri1Mix comprised 212-hour train-360, 11-hour dev, and 11-hour test subsets; LibriDEMAND was a 4-hour synthetic dataset mixed with DEMAND noise. Training used random 1-second crops with T = 50 embedding vectors; longer inference utterances were segmented into 1-second segments and processed independently.Source: §3.1 · p. 3
- The model used a 16-block Conformer, and evaluation measured DNSMOS SIG, BAK, and OVRL, dWER, and GFLOPs including DAC encoding and decoding.DAC and Conformer dimensions were aligned using learnable linear layers; training used AdamW for 300 epochs across four NVIDIA RTX A100 GPUs. Baselines included pretrained ConvTasNet and DPTNet and retrained C-NAR and C-AR NAC-based models.Source: §3.1 · p. 3
3.2. Results
- MARSE-causal achieved intermediate in-domain quality and cost: SIG = 3.62, BAK = 4.10, OVRL = 3.34, and GFLOPs = 1912, between C-NAR and C-AR.C-AR reported SIG = 3.64, BAK = 4.11, OVRL = 3.37, and GFLOPs = 3856; C-NAR reported SIG = 3.60, BAK = 4.08, OVRL = 3.32, and GFLOPs = 1235. ConvTasNet and DPTNet had globally lower SIG and OVRL than the other models but reasonable computational cost.Source: Table 1 · p. 4, §3.2 · p. 4
- As decoding iterations increased, MARSE-causal OVRL improved between the C-NAR-like N = 1 and C-AR-like N = 50 extremes, with performance starting to stagnate around 20–40 iterations.The iteration analysis used a random subset of 300 Libri1Mix test samples, so its absolute scores may differ from Table 1; 20–40 iterations were proposed as a performance-cost trade-off.Source: Figure 2 · p. 4, §3.2 · p. 4
- MARSE intelligibility was better than C-AR and competitive with C-NAR on both datasets, with dWER = 12.68% versus 20.89% and 12.84% in-domain, and 9.35% versus 15.91% and 9.61% out-of-domain.The values are reported for MARSE-causal, C-AR, and C-NAR, respectively.Source: §3.2 · p. 4
- MARSE-causal performed best among MARSE policies in-domain, while MARSE-NC-oracle scored best out-of-domain, plausibly reflecting temporal modeling benefits in-domain and accumulated decoding errors out-of-domain.The authors suggest that more reliable non-causal frame-selection strategies merit exploration.Source: §3.2 · p. 4
4. CONCLUSION
MARSE uses continuous NAC speech representations with iterative decoding policies and offers a flexible trade-off between enhancement performance and computational cost.
4. CONCLUSION
- MARSE applies iterative decoding policies to speech enhancement using continuous NAC representations.The work proposes the method and explores different policies.Source: §4 · p. 4
- MARSE enables a flexible trade-off between speech enhancement performance and computational cost.Source: §4 · p. 4
- Future work could develop a nonoracle confidence measure to choose a more effective non-causal decoding order than random index selection.This direction may improve generalization, as suggested by MARSE-NC-oracle results on out-of-domain data.Source: §4 · p. 4