PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioLM: a Language Modeling Approach to Audio Generation

Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, Neil Zeghidour

arXiv:2209.03143v2cs.SDcs.LGeess.AS

TL;DR

Audio generation needs both high-fidelity local detail and coherent long-term structure, which individual audio tokenizers do not provide simultaneously. AudioLM combines semantic tokens from self-supervised audio representations with acoustic codec tokens in a hierarchical language-modeling framework. It generates coherent speech and piano continuations, including text-free speech from unseen speakers while preserving voice and prosody.

  • Problem

    Audio synthesis must capture multiple abstraction scales, but high audio quality and long-term consistency remain difficult to achieve without strong supervision.

  • Method

    AudioLM hierarchically models semantic tokens for long-term structure and acoustic SoundStream tokens for high-quality waveform synthesis.

  • Results

    AudioLM generates coherent speech without text and coherent piano continuations, while preserving unseen speakers’ voice, prosody, and recording conditions from short prompts.

  • Takeaways & Limitations

    The framework extends language-model-based generation beyond text and symbolic music by producing natural, structurally coherent audio continuations from raw waveforms.

Abstract

from arXiv · show

We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.

I. INTRODUCTION

AudioLM addresses the tension between high-fidelity audio synthesis and long-term structure by combining semantic and acoustic tokens. Experiments show coherent speech and piano continuations without textual or symbolic supervision, while preserving speech characteristics.

  • Motivation: Audio synthesis must represent both local acoustic detail and higher-level structure, but achieving both remains difficult without strong supervision.Speech spans acoustic, phonetic, prosodic, syntactic, and semantic scales, while music combines long-term organization with non-stationary signals.
  • Motivation: Existing high-quality synthesis models can generate unstructured audio without strong conditioning, while prior textless speech models have limited diversity and quality.The prior textless model was trained only on clean speech and produced a single-speaker system.
  • Approach: AudioLM combines semantic and acoustic tokens hierarchically to achieve long-term consistency and high-quality audio synthesis.Semantic tokens capture structure, while SoundStream acoustic tokens capture waveform details for reconstruction.
  • Speech: AudioLM generates coherent speech without textual annotations and preserves voice, prosody, and recording conditions from 3seconds prompts of unseen speakers.The reported continuations remain coherent in phonetics, syntax, and semantics.
  • Music: When trained on piano recordings, AudioLM generates continuations coherent with prompts in melody, harmony, tone, and rhythm.This extends the framework beyond speech without relying on symbolic music representations.
  • Risks: The authors acknowledge risks from speech continuation and train a classifier that detects AudioLM-generated synthetic speech with very high accuracy.The classifier is presented as a mitigation for risks associated with realistic generated speech.

II. RELATED WORK

Related work establishes the value of neural audio synthesis, discrete representations, and language modeling for long-range structure. AudioLM combines these directions through hybrid tokenization and multi-stage Transformer modeling.

  • High-fidelity neural audio synthesis: Neural synthesis methods such as autoregressive, adversarial, and diffusion models have substantially improved audio signal quality.These approaches provide the high-fidelity synthesis foundation motivating AudioLM.
  • Discrete audio representations: Audio discretization enables language-model pretraining, but the tokenization strategy strongly affects downstream performance.Masked language modeling predicts contiguous spans of discrete audio tokens from broad context.
  • Discrete audio representations: Self-supervised audio representations encode coarse information such as phonemes and word identity but are poorly invertible for reconstructing fine audio details.Their long-term dependency modeling benefits discriminative tasks while limiting direct waveform reconstruction.
  • Language modeling: Self-attention supports rich long-range dependency modeling, although standard attention has quadratic computational cost in sequence length.This motivates compact token sequences for audio language modeling.
  • Audio generation with language models: Prior hierarchical and textless systems demonstrate temporal coherence in music or speech, but AudioLM targets both coherence and high-quality synthesis.The related approaches include Jukebox and speech-domain autoregressive models without transcription.
  • AudioLM: AudioLM addresses these representation and modeling challenges with hybrid tokenization and a multi-stage Transformer-based language model.The framework combines discrete audio representations with hierarchical language modeling.

A. Components

AudioLM encodes waveforms into discrete representations, models them autoregressively, and decodes predicted tokens back into audio. Its hybrid design assigns long-term structure to semantic tokens and reconstruction quality to acoustic tokens.

  • Components: AudioLM comprises a tokenizer, a decoder-only Transformer language model, and a detokenizer that reconstructs waveforms from predicted discrete tokens.The tokenizer compresses the waveform into a shorter token sequence before language modeling.
  • Components: Using 2-3 orders of magnitude fewer tokens than waveform samples expands the language model’s temporal context while frozen tokenizers decouple representation learning from language-model training.The reduction is important because standard self-attention scales quadratically with sequence length.
  • Representation trade-offs: AudioLM must balance a sufficiently long token sequence for high-quality waveform reconstruction against a compact sequence that captures long-term dependencies.This conflicting requirement motivates combining acoustic and semantic tokens.
  • Acoustic tokens: SoundStream produces acoustic tokens for high-quality synthesis, using 50 Hz embeddings and residual vector quantization to represent 16 kHz waveforms.For N = 1024 and Q = 4, the configured representation has a bitrate of 2000 bps.
  • Token trade-offs: Semantic tokens from w2v-BERT provide self-supervised linguistic representations, while acoustic tokens and semantic tokens offer complementary reconstruction and phonetic properties.The comparison evaluates reconstruction with ViSQOL and phonetic discriminability with ABX error rate.
  • Token trade-offs: Modeling acoustic tokens alone preserves speaker identity and recording conditions but produces linguistically inconsistent, often babbling, speech continuations.This experiment illustrates why acoustic detail alone does not ensure linguistic coherence.

C. Hierarchical modeling of semantic and acoustic tokens

AudioLM hierarchically models semantic tokens first, then coarse and fine acoustic tokens, combining long-term structure with high-fidelity synthesis. Separate stages reduce sequence length and progressively refine the generated audio.

  • Semantic and coarse acoustic modeling: AudioLM first models semantic tokens to capture long-term temporal structure, then models coarse acoustic tokens conditioned on them.The semantic stage targets structural coherence, while the coarse acoustic stage predicts SoundStream tokens from the coarse quantizers.
  • Token hierarchy: With the default configuration, each semantic token corresponds to 2Q′ coarse acoustic tokens and 2(Q −Q′) fine acoustic tokens.The factor of 2 reflects SoundStream embeddings being sampled twice as frequently as w2v-BERT embeddings.
  • Fine acoustic modeling: The fine acoustic stage predicts remaining quantizers using coarse tokens and prior fine tokens, removing residual lossy-compression artifacts.Fine tokens at the current time step are conditioned on decoded coarser tokens and previously generated fine tokens.
  • Design rationale: AudioLM separates coarse and fine acoustic modeling because fine details can ignore semantic tokens after conditioning on coarse tokens, shortening the processed sequence.The design also assumes fine acoustic details are determined locally.

D. Inference

During inference, AudioLM autoregressively extends semantic tokens, generates corresponding coarse and fine acoustic continuations, and decodes them into a waveform. This procedure supports prompted continuation from short audio contexts.

  • Semantic continuation: For continuation, the first stage autoregressively generates semantic tokens after the prompt’s semantic context.The generated semantic continuation is conditioned on the prompt semantic tokens.
  • Acoustic continuation: The coarse acoustic model receives the full semantic sequence and prompt coarse tokens before sampling acoustic continuations.The semantic prompt and generated continuation are concatenated with the prompt’s coarse acoustic tokens.
  • Waveform reconstruction: The fine acoustic model processes the sampled coarse tokens, after which SoundStream reconstructs the prompt and generated tokens as a waveform.The final waveform combines the prompt with sampled acoustic tokens.
  • Inference result: With only 3 seconds of speech from an unseen speaker, AudioLM generates continuations that are hardly distinguishable from real speech.The result is reported for prompted speech continuation in the inference evaluation.

IV. EXPERIMENTS

The experiments evaluate AudioLM across speech and piano continuation, requiring preservation of prompt characteristics and coherent generation beyond the training examples. Speech and piano tests therefore probe different aspects of generalization.

  • Speech continuation: Speech continuation evaluates whether generated content preserves the prompt’s speaker identity, prosody, recording conditions, syntax, and semantic consistency.The task expects new content while retaining properties of the speech prompt.
  • Piano continuation: Piano continuation evaluates coherence with the prompt in melody, harmony, and rhythm.The task uses piano music rather than symbolic musical representations.
  • Generalization: Speech prompts come from unseen speakers and piano prompts from unseen performances, so both tasks test generalization beyond training data.The evaluation settings require consistent continuation for unfamiliar speakers or performances.

A. Datasets

The speech system is trained on the diverse, noisy unlab-60k Libri-Light split, while token-representation choices are examined through linguistic and clustering evaluations. AudioLM performs strongly despite the noisier training data.

  • Speech dataset: All speech components are trained on Libri-Light’s unlab-60k split, containing 60k hours of English speech.The training includes SoundStream, w2v-BERT, the w2v-BERT k-means quantizer, and decoder-only Transformers.
  • Representation evaluation: Figure 3 evaluates w2v-BERT layers with ABX and evaluates layer-7 k-means sizes using sWUGGY and sBLIMP development scores.These measurements support selecting the intermediate layer and cluster count for semantic-token quantization.
  • Training-data robustness: AudioLM shows strong performance on the more diverse and noisy unlab-60k subset, reducing the data-preparation effort required to apply the framework.The passage characterizes the model as robust to training-data quality.

B. Model selection, training and inference

AudioLM selects semantic and acoustic tokenizers to balance linguistic structure with reconstruction quality, then models these representations hierarchically. Inference uses temperature sampling and short speech prompts for continuation.

  • Model selection: Semantic tokens are selected from the 7th w2v-BERT XL MLM layer using K = 1024 k-means clusters.Selection uses ABX, sWUGGY, sBLIMP, and subjective listening tests.
  • Model selection: The SoundStream acoustic tokenizer uses 12 residual quantizer layers, producing 600 tokens per second at 6000 bps for 16 kHz audio.Its reconstruction quality is evaluated with ViSQOL.
  • Training: AudioLM uses identical 0.3B-parameter decoder-only Transformers across three stages, trained with 30-, 10-, and 3-second crops.Each stage uses 12 layers, 16 attention heads, and 1024-dimensional embeddings.
  • Inference: Temperature sampling uses 0.6, 0.8, and 0.6 across the three stages to trade off diversity and semantic consistency.Speech continuation uses 3-second prompts conditioned on semantic and acoustic tokens.

C. Information represented by the semantic tokens

The experiments separate the information carried by semantic and acoustic tokens. Semantic tokens preserve linguistic content, while acoustic tokens primarily determine speaker identity, recording conditions, and related acoustic variation.

  • Experimental hypothesis: The experiments test whether semantic tokens mainly capture linguistic content while acoustic tokens capture speaker identity and recording conditions.This hypothesis motivates AudioLM’s hierarchical separation of token types.
  • Semantic content: Low WER and CER show that acoustic generation conditioned on ground-truth semantic tokens closely follows the original transcripts.Errors mainly arise from proper nouns, sentence-ending tokens, and background noise.
  • Acoustic variation: Repeated acoustic sampling conditioned on identical semantic tokens produces varied speakers and recording conditions.The qualitative variation supports a limited role for semantic tokens in speaker identity.
  • Speaker identity: Speaker classification accuracy remains low for acoustic generation from fixed semantic tokens, indicating that speaker identity is mostly determined by acoustic tokens.The reported accuracy is 3.2%, compared with 0.3% chance performance.
  • Prosody: Rhythm and intonation vary only slightly across samples sharing semantic tokens, suggesting that prosodic features are captured mostly by acoustic tokens.The evidence is based on subjective comparison of synthesized samples.

E. Probing the linguistic knowledge of AudioLM

AudioLM’s semantic-token language model is evaluated for lexical and syntactic knowledge without text supervision. It achieves the strongest reported performance among comparable unsupervised systems and improves over prior work on both probing metrics.

  • Probing tasks: sWUGGY tests whether the model prefers real words over similar-sounding non-words, while sBLIMP evaluates syntactic preferences.Both metrics are zero-shot probes of linguistic knowledge.
  • Evaluation: The evaluation uses 10,000 sWUGGY pairs and 6,300 sBLIMP pairs, synthesized using four voices.The sWUGGY results also distinguish an in-vocabulary subset.
  • Baselines: AudioLM is compared with text-based toplines, CPC-based systems, HuBERT-only models, and causal baselines such as GSLM.The comparisons include both supervised and non-text-supervised systems.
  • Results: AudioLM achieves the highest sWUGGY scores across both splits among systems without text supervision.The comparison covers the reported development-set splits.
  • Results: AudioLM attains the highest sBLIMP score and improves by 8% relative over CPC-BERT, outperforming a supervised forced-alignment phonetic-transcription topline.The paper interprets this as improved lexical and syntactic judgment quality without textual supervision.

F. Generating coherent continuations

AudioLM generates short-prompt speech continuations that preserve speaker identity and are difficult for human listeners to distinguish from real speech. However, a classifier detects the synthetic continuations with high accuracy.

  • Prompted continuation: AudioLM continues speech coherently from prompts of only 3 seconds, with experiments generating 7-second continuations.The evaluation examines acoustic consistency in the continuation rather than only linguistic content.
  • Speaker consistency: Speaker classification accuracy exceeds 92% when checking whether the prompt and generated continuation use the same speaker.Conditioning on both semantic and acoustic tokens preserves speaker identity.
  • Subjective evaluation: Human raters identify original versus synthesized 10-second samples with 51.2% accuracy, not significantly different from chance at p = 0.23.The task jointly tests linguistic correctness, acoustic coherence, and generation artifacts.
  • Synthetic-speech detection: A binary audio classifier reaches 98.6% accuracy in distinguishing AudioLM continuations from SoundStream-compressed original samples.The result contrasts classifier detectability with the near-chance human discrimination rate.

I. Piano continuation

AudioLM extends coherent continuation beyond speech to piano music while preserving high audio quality. Its broader use also raises risks around bias, representation gaps, and speaker impersonation.

  • Piano continuation: AudioLM generates high-quality piano continuations with coherent melody and temporal structure.The piano experiments retrain all components on 40k hours of varied piano recordings and use four-second prompts.
  • Piano continuation: The full AudioLM framework preserves melody and temporal structure, whereas acoustic tokens alone provide equally high audio quality without consistent structure.This comparison isolates the contribution of the full hierarchical framework to musical coherence.
  • Piano continuation: AudioLM hierarchically predicts semantic and acoustic tokens through three language-modeling stages, from coarse structure to fine acoustic detail.The hybrid tokenization scheme combines long-term structure with high-fidelity synthesis.
  • Broader impact: The framework's realistic speech continuation can reflect societal biases, mismatch accents or dialects for underrepresented groups, and enable spoofing or impersonation.The authors identify generated-speech detection as an important safeguard against misuse.
Loading 2209.03143v2…