Masked Audio Generation using a Single Non-Autoregressive Transformer

1 INTRODUCTION

• Autoregressive audio language models have high inference latency, while diffusion approaches may require hundreds of decoding steps and have difficulty with long-form sequences.Some reported diffusion generations are 10 seconds; others use cascaded super-resolution models to reach 30 seconds.Source: §1 · p. 2
• Unlike SoundStorm, which conditions on semantic tokens from an autoregressive model, MAGNET is a single non-autoregressive model evaluated for music and audio generation.The passage distinguishes music and audio from speech by their use of the full frequency spectrum.Source: §1 · p. 2
• Across text-to-music and text-to-audio generation, MAGNET achieves comparable results to evaluated baselines while being x7 faster than the autoregressive-based method.The evaluation includes objective metrics and a human study.Source: §1 · p. 2
• MAGNET generates sequences up to 30 seconds long using one model, and also introduces rescoring and a hybrid autoregressive/non-autoregressive approach.The hybrid generates the sequence beginning autoregressively while decoding the rest in parallel.Source: §1 · p. 2

2 BACKGROUND

• EnCodec encodes audio into a lower-frame-rate representation and quantizes it with RVQ into K discrete token sequences, each of length T.In RVQ, each quantizer encodes the error left by the previous quantizer.Source: §2 · p. 3
• Autoregressive modeling factors the conditional joint probability of an audio-token sequence into token probabilities conditioned on preceding tokens.The condition y is a semantic representation, and the chain rule expresses the sequence probability as a product of conditional probabilities.Source: §2 · p. 3
• In the autoregressive masking formulation, future tokens are masked, and the masking strategy must preserve the probability chain rule.The mask marks token positions at or after the current position.Source: §2 · p. 3
• Non-autoregressive modeling modifies the masking strategy to predict arbitrary subsets from unmasked tokens over several decoding steps.During training, sampled masked tokens are replaced with a special mask token and predicted from unmasked tokens; a related RVQ approach predicts each codebook from preceding codebooks.Source: §2 · p. 3

3 METHOD

• MAGNET combines masked audio-token generation with span masking, restricted context, sampling, and rescoring as core modeling modifications.Its masked language model is non-autoregressive, conditioned on a semantic representation, and operates on discrete streams obtained from EnCodec.Source: §3 · p. 4

3.1 MASKING STRATEGY

• A 60 ms span length gave the best overall performance; spans are used because adjacent audio tokens share information, and inference re-masks spans rather than individual tokens.Inference uses non-overlapping spans for computational efficiency and re-masks the least probable spans.Source: §3.1 · p. 4

3.2 RESTRICTED CONTEXT

• For codebooks above the first, MAGNET restricts self-attention to tokens within approximately 200 ms, using the measured local temporal context of the audio encoder.The encoder’s convolutional network has an approximately 160 ms receptive field, while the effective field including its LSTM is approximately 180 ms.Source: §3.2 · pp. 4–5

3.3 MODEL INFERENCE

• At each inference iteration, MAGNET ranks candidate spans using model confidence, while an external model can rescore spans and combine its probabilities with MAGNET’s for sampling.The paper uses MUSICGEN and AudioGen as rescorers and reports improved audio quality and more stable inference; applying rescoring slows inference in the reported ablation.Source: Figure 1 · p. 4, §3.3 · p. 5
• Classifier-free guidance is annealed as the masking rate decreases, shifting generation from text guidance toward contextual infilling.The guidance coefficient interpolates between initial and final values according to the masking schedule.Source: §3.3 · p. 5

• MAGNeT uses the training data of Copet et al. (2023) for music and Kreuk et al. (2022a) for audio; MusicGen and Mousai were retrained on the same dataset.For Table 1, MusicLM uses a public API for human studies, while AudioLDM2 and MusicLM use original FAD results.Source: §4 · p. 5, Table 1 · p. 6
• Audio is encoded at 50 Hz into four EnCodec codebooks of size 2048, and pretrained T5 extracts semantic text representations for conditioning.Source: §4 · p. 5
• The 300M- and 1.5B-parameter models train on random 30-second crops for 1M steps, using 32 and 64 GPUs respectively.Ablations use 10-second generation models; inference uses nucleus sampling and classifier-free guidance.Source: §4 · p. 6
• The study reports FAD, KL divergence, and CLAP for objective evaluation, plus human ratings of overall quality and text relevance on 1–100 scales.Each sampled file received at least five ratings; CrowdMOS filters noisy annotations and outliers.Source: §4 · p. 6

5.1 TEXT-TO-MUSIC GENERATION

• On MusicCaps, MAGNeT matches MusicGen with lower latency and fewer decoding steps; against AudioLDM2 it has worse FAD and CLAP but better KL subjective scores.Text-to-audio results show similar trends: comparable performance to the autoregressive baseline with substantially faster generation.Source: §5.1 · p. 7

5.2 ANALYSIS

• At small batches, MAGNeT reaches 600 ms latency for one sample and is over 10 times faster; above batch size 64, autoregressive throughput scales linearly while MAGNeT is bounded near 2.8 samples/second.Latency and throughput were measured for 10-second samples on an A100; the figure describes MAGNeT as particularly suited to small batches and MusicGen to higher-throughput large batches.Source: §5.2 · p. 7, Figure 2 · p. 7
• With a 1-second autoregressive prompt, Hybrid-MAGNeT beats the full autoregressive baseline on FAD at 3.2s versus 17.6s latency; longer prompts further lower FAD.The comparison uses 30-second generations without rescoring.Source: §5.2 · pp. 7–8

5.3 ABLATION

• Restricted temporal context improves performance across settings, and span length 3, corresponding to 60 ms, yields the best FAD.Table 2 reports in-domain FAD across span lengths with and without restricted context.Source: §5.3 · p. 8, Table 2 · p. 8
• The CFG schedule λ0 = 10 and λ1 = 1 yields the best FAD among evaluated setups, aligning stronger text adherence early with greater focus on decoded tokens later.Source: §5.3 · p. 8
• Rescoring improves performance on almost all metrics, but increases inference time.Table 3 reports means and 95% confidence intervals.Source: §5.3 · p. 8, Table 3 · p. 8
• Using one decoding step per higher codebook level reduces latency to 370 ms, at the cost of an 8% FAD increase versus 10 steps per higher level.Reducing steps at higher levels affects quality less than reducing steps at the first level.Source: §5.3 · p. 8
• Across 20 iterations, MAGNeT first predicts sparse spans at disconnected times, then fills gaps until the token sequence is complete.Source: §5.3 · p. 8

• Autoregressive audio generation includes transformer language models over discrete audio representations, including approaches for environmental sounds, music, and singing accompaniment.Examples include Kreuk et al. (2022a), Dhariwal et al. (2020), Copet et al. (2023), and Donahue et al. (2023).Source: §6 · pp. 8–9
• Diffusion approaches generate audio from continuous or discrete representations, including latent-diffusion methods for text-to-audio and text-to-music.Examples include Yang et al. (2022), Huang et al. (2023b), Liu et al. (2023a;b), and Schneider et al. (2023).Source: §6 · p. 9
• Masked generation spans machine translation and image synthesis, while SoundStorm applies it to speech and dialogue using autoregressively obtained semantic tokens.Ghazvininejad et al. (2019), Chang et al. (2022), and Borsos et al. (2023b) are cited as examples.Source: §6 · p. 9
• MAGNeT differs from SoundStorm by using a single non-autoregressive model, and from VampNet by using one model rather than separate coarse- and fine-token models.The passage also distinguishes MAGNeT from VampNet by its exploration of text-to-music generation without audio prompting.Source: §6 · p. 9

7 DISCUSSION

• At each decoding step, the non-autoregressive model re-encodes the whole sequence, unlike autoregressive models that can cache past keys and values.Adopting caching for unchanged time steps in non-autoregressive models is proposed as possible but requires further research.Source: §7 · p. 9
• MAGNET achieves competitive performance with autoregressive methods while being approximately 7 times faster.The conclusion describes the evaluation as showing comparable or minor quality degradation.Source: §7 · p. 9
• The paper also explores a hybrid approach combining autoregressive and non-autoregressive models.The conclusion identifies this as an explored approach but does not specify its results here.Source: §7 · p. 9

A.1 IMPLEMENTATION DETAILS

• The setup uses the official EnCodec model, which converts audio into a 50 Hz discrete representation with four codebooks of size 2048.Text preprocessing follows the cited prior work.Source: §A.1 · p. 14
• MAGNeT-small has 300M parameters and MAGNeT-large has 1.5B; both train on random 30-second crops for 1M steps.The models use AdamW, batch size 192, and 32 GPUs for small or 64 GPUs for large training.Source: §A.1 · p. 14
• Inference uses nucleus sampling with top-p 0.9 and temperature annealed from 3.0 to zero, plus classifier-free guidance annealed from 10.0 to 1.0.The classifier-free guidance condition-dropout rate during training is 0.3.Source: §A.1 · p. 14

A.2 DATASETS

• Training data follows the cited setup of 20K hours of licensed music, including 10K high-quality tracks and ShutterStock and Pond5 collections.The latter collections contain 25K and 365K instrument-only tracks, respectively.Source: §A.2 · p. 14
• Main evaluation uses MusicCaps, with 5.5K ten-second samples and a 1K genre-balanced subset, plus the cited in-domain test set.Objective metrics use the unbalanced MusicCaps set; qualitative examples use the genre-balanced subset, and ablations use the in-domain set.Source: §A.2 · p. 14

A.3 EVALUATION

• Music-generation baselines include Mousai, MusicGen, AudioLDM2, and MusicLM; Mousai is trained on the authors’ dataset for comparison.Source: §A.3 · p. 15
• Objective evaluation uses FAD, KL divergence, and CLAP; CLAP is reported for music generation to measure audio-text alignment.The passage states that low FAD indicates plausible generated audio.Source: §A.3 · p. 15
• Human raters score overall audio quality and relevance to text on 1-to-100 scales, with at least five raters per sampled file.Raters are recruited through Amazon Mechanical Turk, and noisy annotations and outliers are filtered using CrowdMOS.Source: §A.3 · p. 15

B RECEPTIVE FIELD ANALYSIS

• EnCodec’s effective receptive field is bounded by 100 ms in each direction, motivating MAGNeT’s restricted-transformer neighborhood for codebooks beyond the first.The analysis measures the middle latent vector’s response to an impulse, for both the full encoder and a version without its LSTM block.Source: B RECEPTIVE FIELD ANALYSIS · p. 15, Figure 3 · p. 16

C SPAN MASKING

• Uniformly sampled span starts define random masks; Proposition C.1 derives the expected masking rate by counting placements that leave a token unmasked.Each sampled start is extended to a span of length l.Source: C SPAN MASKING · pp. 15–16

D MODEL INFERENCE

• During inference, MAGNeT remasks the least-probable non-overlapping spans, using the maximum token probability as each span’s probability.The figure caption describes span probabilities as a convex combination of restricted-transformer confidence and pretrained-rescorer probability.Source: Figure 4 · p. 17
• After each decoding step, unmasked spans receive infinite probability so they are not remasked and remain anchors for later iterations.The inference figure presents the process for one codebook for clarity, while practice runs it for every codebook.Source: D MODEL INFERENCE · p. 16, Figure 4 · p. 17

E HYBRID-MAGNET TRAINING

• Hybrid-MAGNeT samples a timestep, trains preceding positions causally across codebooks, and applies MAGNeT’s objective to succeeding positions, using Copet et al. (2023)’s delay pattern.Figure 5 illustrates the two objectives; the shifted pattern adapts temporal context to codebook-level shifts.Source: E HYBRID-MAGNET TRAINING · pp. 17–18, Figure 5 · p. 18
• Table 5 reports text-to-audio FAD and KL scores across methods.Source: Table 5 · p. 19
• Figure 7 varies first-level and higher-level decoding steps around a (20, 10, 10, 10) schedule to show the quality/latency trade-off.Source: Figure 7 · p. 19

F ITERATIVE DECODING DYNAMICS

• With s = 20, decoding begins at sparse, disconnected time spans and iteratively fills gaps to a complete sequence non-causally.Figure 6 plots the chosen masks over time and decoding steps for a 10-second sample.Source: Figure 6 · p. 18, F ITERATIVE DECODING DYNAMICS · p. 19

• MAGNET is comparable or slightly worse than AudioGen on AudioCaps text-to-audio results, with significantly lower latency.Source: G ADDITIONAL RESULTS · p. 19
• For codebook levels above 1, attention is locally restricted; in practice, five time-steps per side yield 11 tokens.Source: Figure 8 · p. 20
• The (20, 5, 5, 5) schedule achieves 0.61 FAD at latency close to the (10, 10, 10, 10) schedule’s 0.65 FAD, despite a smaller total step count.The passage says reducing decoding steps for higher codebook levels affects quality less than reducing steps for the first level.Source: G ADDITIONAL RESULTS · p. 20
• λ0 = 10, λ1 = 1 yields the best FAD among evaluated CFG setups; Table 6 also reports KL and CLAP scores.Source: G ADDITIONAL RESULTS · p. 20, Table 6 · p. 20