Source-linked AI summary
Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations
Yoto Fujita, Simon Leglaive, Laurent Girin
TL;DR
Speech enhancement research has largely used discrete NAC tokens, while continuous NAC representations may better preserve speech quality and intelligibility. MARSE applies iterative masked decoding to continuous NAC representations and compares decoding policies under a common setup. The results show a flexible trade-off between enhancement performance and computational cost, with intelligibility advantages over C-AR and competitive performance with C-NAR.
Problem
Speech enhancement methods based on discrete NAC tokens may limit acoustic-detail preservation, while decoding-policy trade-offs between performance and computational cost remain underexplored.
Method
MARSE performs iterative masked decoding of clean-speech frames represented by continuous NAC encoder embeddings before quantization.
Results
MARSE provides a flexible trade-off between speech-enhancement performance and computational cost, with dWER of 12.68% for MARSE-causal versus 20.89% for C-AR and 12.84% for C-NAR on Libri1Mix.
Takeaways & Limitations
Different decoding policies allow MARSE to adjust enhancement quality and computational cost, while improved non-causal frame-selection strategies remain a future direction.
Takeaways & Limitations
The Gaussian model replacing diffusion may reduce overall performance, although the simplification does not affect the study's focus on comparing decoding policies.
Abstract
from arXiv · showhide
Most previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained using neural audio codecs (NACs). However, a recent study has shown that continuous latent representations of NACs can be advantageous for SE in terms of speech quality and intelligibility. In this work, we propose masked autoregressive SE (MARSE), a method for SE based on iterative decoding of masked clean speech frames using continuous NAC representations of speech. In particular, we investigate a set of different decoding policies, ceteris paribus, that is, using the same DNN (a Conformer model), the same NAC (the DAC codec) and the same training setup. The results show that MARSE enables a flexible trade-off between SE performance and computational cost. Audio examples and code are available online.
1 CentraleSup´elec, IETR (UMR CNRS 6164), France 2 Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France
The paper concerns speech enhancement, masked generative modeling, neural audio codecs, autoregressive modeling, and speech representations.
- The paper addresses speech enhancement as its central application area.
- Its modeling framework combines masked generative modeling with autoregressive modeling.
- Neural audio codecs and speech representations are core technical themes.
1. INTRODUCTION
The introduction motivates speech enhancement with continuous neural audio codec representations and presents MARSE as a unified framework for comparing decoding policies under a fair setup.
- Motivation: Token-based speech enhancement can limit preservation of acoustic details relevant to speech quality, intelligibility, and some downstream tasks such as ASR.
- Motivation: Continuous NAC representations use encoder embeddings before quantization and have been reported to improve speech intelligibility and quality.
- Motivation: Prior token-based methods generally focus on specific decoding policies and setups, leaving performance–computational-cost trade-offs underexplored.
- Contribution: MARSE formulates iterative decoding over blocks of continuous NAC representations as an autoregressive probabilistic process.
- Contribution: The study unifies AR, non-AR, and masked AR policies and compares their effects on quality, intelligibility, and computational cost.
- Contribution: The comparison uses the same Conformer-based network architecture, NAC, and experimental setup across decoding-policy variants.
2. METHOD
MARSE models speech enhancement in continuous neural audio codec latent space and performs iterative masked-frame decoding. Its decoding policy controls temporal dependencies, parallelism, and the computational cost of inference.
- Masked autoregressive speech enhancement: MARSE estimates the conditional distribution of clean speech given noisy speech using continuous latent representations from a pretrained neural audio codec.The representations are encoder outputs before quantization, with T time frames and latent dimension D.
- Masked autoregressive speech enhancement: The model partitions clean-speech frames into disjoint masked sets M(i), predicts each set from noisy speech and previously decoded visible frames, and reinjects predictions at the next iteration.At inference, a Conformer processes noisy and partially masked clean-speech representations, replacing masked positions with a learnable mask vector.
- Masked autoregressive speech enhancement: Within each decoding block, clean-speech latent vectors are assumed conditionally independent, while dependencies across blocks form an autoregressive model.The block factorization remains a valid probability density when each conditional factor is valid.
- Masked autoregressive speech enhancement: The framework replaces the original MAR diffusion model with a simpler, likely less effective Gaussian model to focus on formalizing and comparing decoding policies.The authors state that this simplification may affect overall performance without affecting the study’s focus.
- Decoding policies for inference: Decoding policies determine the number of iterations and the frames decoded in parallel, thereby specifying temporal dependencies and inference computation.MARSE permits arbitrary iteration counts and different strategies for selecting M(i).
- Causal decoding: Causal policies range from one-pass non-AR decoding, which assumes frame independence, to frame-wise causal AR decoding, which uses T forward passes and makes no conditional-independence assumptions.Intermediate block-wise causal policies trade parallel decoding against temporal modeling expressiveness.
- Non-causal decoding: Non-causal policies can decode future-conditioned frames, using either oracle clean-speech index selection or uniformly random selection of remaining masked-frame indices.Oracle selection prioritizes frames according to their signal-to-noise characteristics, whereas random selection partitions frames into randomly chosen blocks.
- Training: MARSE training uses paired noisy-clean speech data and an MSE-equivalent objective, with masked-frame counts sampled according to a cosine schedule.Training visible frames come from ground-truth clean speech, unlike inference, where previously predicted frames are reinjected.
3. EXPERIMENTS
Experiments evaluate MARSE with continuous DAC representations under matched configurations and compare its decoding policies with conventional and NAC-based baselines. Results show a controllable trade-off between enhancement quality, intelligibility, and computational cost across in-domain, out-of-domain, and iterative-decoding settings.
- Experimental setup: MARSE was evaluated with continuous DAC latent representations, a Conformer model, matched training settings, and DNSMOS, dWER, and GFLOPs metrics.The DAC representation has dimension D = 1024; DNSMOS measures SIG, BAK, and OVRL, while dWER measures phonetic preservation.
- Experimental setup: The comparison included ConvTasNet, DPTNet, C-NAR all-at-once decoding, and C-AR frame-wise causal decoding.C-NAR uses N = 1, whereas C-AR uses N = T; conventional non-AR baselines had reasonable computational cost but lower SIG and OVRL quality.
- Results: C-AR achieved higher in-domain quality than C-NAR but required 3856 GFLOPs versus 1235 GFLOPs for C-NAR.C-AR scored SIG = 3.64, BAK = 4.11, OVRL = 3.37; C-NAR scored SIG = 3.60, BAK = 4.08, OVRL = 3.32.
- Results: Among MARSE variants, MARSE-causal performed best in-domain, while MARSE-NC-oracle achieved the best out-of-domain scores.The paper attributes this pattern to temporal modeling in-domain and accumulated iterative-decoding errors out-of-domain, where reliable frame selection is more important.
- Results: As decoding iterations increased from N = 1 to N = 50, MARSE-causal moved from C-NAR-like OVRL toward C-AR-like OVRL.At N = 1, MARSE and C-NAR both decode frames all at once; at N = 50, MARSE-causal and C-AR both use frame-wise causal decoding.
4. CONCLUSION
The paper proposes MARSE for speech enhancement with continuous neural audio codec representations and evaluates iterative decoding policies. Its results support a flexible trade-off between enhancement performance and computational cost, while motivating improved nonoracle frame-selection strategies.
- Conclusion: MARSE applies iterative decoding policies to speech enhancement using continuous neural audio codec representations.The method is evaluated across different decoding policies under a common experimental framework.
- Conclusion: The experiments show that MARSE can trade speech-enhancement performance against computational cost through its decoding policy.The conclusion identifies this flexibility as a central result of the study.
- Conclusion: Future work may use a nonoracle confidence measure to select a more effective non-causal decoding order than random index selection.This direction is motivated by the out-of-domain results of MARSE-NC-oracle.