Variable-rate discrete representation learning

1. Introduction

• Salient information varies across signals: speech includes silences and varied pronunciation rates, while images contain both background textures and salient features.Source: §1 · p. 1
• Fixed-rate digital representations encode the same amount per unit regardless of perceptual salience, and fixed-size learned representations can likewise waste downstream computation.Source: §1 · pp. 1–2

1.1. Event-based representations

• Event-based representations encode salient changes at irregular intervals, adapting representation size to the signal's perceptually relevant information.In speech, fewer events can represent silence than speech, motivating computation and memory that scale with salient content.Source: §1.1 · p. 2
• Manually defining meaningful events is impractical because speech changes may involve phonemes as well as volume or pitch, motivating unsupervised event discovery.Source: §1.1 · p. 2

1.2. Contributions

• SlowAEs learn slowly varying discrete representations using an adaptive group-sparse slowness penalty and scalar quantisation, producing event-based codes amenable to run-length encoding.The approach is designed to identify meaningful changes automatically from data rather than require events to be specified manually.Source: §1.2 · p. 2, §1.1 · p. 2
• Run-length Transformers (RLTs) autoregressively model discrete event-based representations by exploiting their underlying structure.Source: §1.2 · p. 2
• The authors combine SlowAEs and RLTs in an unsupervised speech-domain language model trained on audiobook recordings, producing grammatical and sometimes meaningful new utterances.Source: §1.2 · p. 2

2. Discrete representation learning

• Discrete representations have limited capacity that can help generative models focus on salient statistical dependencies rather than perceptually irrelevant detail.Source: §2 · p. 2

2.1. Vector quantisation

• VQ-VAE uses a learned codebook to map encoder outputs to nearest entries, producing integer sequences that can be modelled with categorical distributions.Straight-through estimation passes gradients through the non-differentiable quantisation step, though encoder estimates are biased.Source: §2.1 · p. 3

2.2. Scalar quantisation

• Scalar quantisation avoids learning a codebook, but rounding imposes ordinal relationships between codes and may require more bits than VQ for comparable fidelity.Source: §2.2 · p. 3

2.3. Learning representations

• The paper uses an autoencoder to map generated signals back to the input domain and an autoregressive decoder to model local structure, letting the encoder focus on higher-level structure.Source: §2.3 · p. 3
• Fixed-size representations allocate the same number of codes to equal-sized input regions, limiting capacity where salient content is concentrated and leaving other codes to represent irrelevant information.This can make downstream models less able to use their capacity efficiently.Source: §2.3 · p. 3

3.1. Slow autoencoders (SlowAEs)

• SlowAEs learn slowly varying discrete bottlenecks that can be run-length encoded as compact (value, length) events when changes are infrequent.The encoder output is quantized, and the decoder is trained to reconstruct the input signal.Source: §3.1 · pp. 3–4

3.1.1. SLOWNESS PENALTY

• Group-sparse slowness combines temporal differences across channels and encourages sparse, synchronized changes; it is the default penalty used in the work.The penalty acts as L1 across sequence time and L2 across other dimensions.Source: §3.1.1 · pp. 4–5

3.1.2. SCHMITT TRIGGER QUANTISATION (STQ)

• STQ uses a margin to make quantization levels sticky, suppress small fluctuations, and reduce inefficient event sequences; models without STQ learn less efficient representations.The increased margin also raises average quantization error, while straight-through estimation still works in practice.Source: §3.1.2 · p. 5

3.1.3. AUTOMATIC PENALTY WEIGHTING

• An adaptive update adjusts λ toward a target average event rate, avoiding manual tuning and enabling model variants to be compared at approximately constant representation capacity.The update increases λ when estimated AER is too high and decreases it when AER is too low.Source: §3.1.3 · p. 5

3.1.4. SLOWAES FOR SPEECH

• The speech SlowAE uses four channels, 15 quantization levels, group-sparse slowness, and STQ; it can target 75 Hz while retaining mostly intelligible reconstructions.The decoder is conditioned on speaker identity, which encourages speaker-independent representations.Source: §3.1.4 · p. 6

3.2. Run-length Transformers (RLTs)

• RLTs model event sequences autoregressively with Transformers, whose lack of fixed-topology assumptions suits irregularly sampled events.Input embeddings can provide information about the structure of the slowly varying representations.Source: §3.2 · p. 6

3.2.1. MULTI-CHANNEL RUN-LENGTH ENCODING

• Multi-channel run-length encoding interleaves events by time and then channel, making each event’s channel and offset inferable from preceding lengths.The model therefore predicts only each event’s value and length; runs longer than 256 are split.Source: §3.2.1 · pp. 6–7

3.2.2. ARCHITECTURE

• The RLT predicts the next event value with a causally masked Transformer, then uses an MLP conditioned on that value to predict its length.Explicit embeddings of deterministic channel and offset information accelerate convergence.Source: §3.2.2 · p. 7

• Speech language modelling is explored to capture language beyond grammar and vocabulary and to test whether models can learn both from speech alone.The authors also note potential application to languages with limited or no written resources.Source: §4 · p. 8
• Because speech signals contain far more information than their textual transcriptions, the authors use SlowAEs to retain high-level information while abstracting irrelevant details.The representations support reconstruction of intelligible speech, making speech-level modelling tractable, though more computationally expensive than text modelling.Source: §4 · p. 8
• The method is tested only on English speech, though the authors state that it is not specific to English or European languages.They also report no incompatibility issues of the kind textual language models may face with other writing systems and tokenisation strategies.Source: §4 · p. 8
• The audiobook dataset represents a restricted mode of speech: written language read aloud, relatively clean compared with other modes.The authors identify conversational-speech datasets as an important direction for future work.Source: §4 · p. 8

Slowness and predictability priors

• Slowness priors have been used with VAEs for posterior-collapse prevention and disentangled representations, while predictability is a relaxed form of slowness used in CPC.The cited work includes Razavi et al. (2019a), Klindt et al. (2021), and van den Oord et al. (2018).Source: §5 · p. 8

Structured discrete representation learning

• Discrete representation work combines vector-quantised autoencoders with latent Markov models, CPC, regularisation, or BPE for interpretability, music, speech, and text-to-speech.The passage also mentions spiking neural networks for representation learning.Source: §5 · p. 8

Structure discovery

• Unsupervised structure discovery includes speech-boundary analysis, compressibility-based sequence chunking, mutual-information approaches, learned segmentation, and object-centric visual models.Source: §5 · p. 8

Variable-sized latents

• Variable-sized latent capacity has been studied in models including the Indian buffet process, Stick-breaking VAEs, restricted Boltzmann machines, and nested dropout.These approaches vary latent capacity along axes other than the sequence dimension considered here.Source: §5 · pp. 8–9

‘Language-ification’

• Language-ification casts non-text tasks as token-sequence problems, with examples spanning speech-to-speech translation, video representation learning, music, and image generation.Source: §5 · p. 9

Event-based representations

• Event-based representations are established in symbolic music and event-camera vision, with recent work advancing their use in music modelling.Source: §5 · p. 9

Asynchronous models

• Asynchronous models forgo fixed-quantum inputs, while other approaches adaptively allocate capacity; run-length Transformers implicitly use more computation where speech requires more events.Some models operate directly on compressed representations, usually while preserving some topological structure.Source: §5 · p. 9

Speech language models

• Concurrent work develops speech-domain language models for resynthesis and novel-utterance generation using three strategies for continuous intermediate representations.Source: §5 · p. 9

6.1. Datasets

• Libri-light supplies roughly 60,000 hours of audiobook speech for generative modelling; separate, higher-quality LibriTTS recordings train SlowAEs, and proprietary single-speaker recordings support decoders and evaluation.The authors avoid rendering samples in voices from either public audiobook dataset because the models may imitate speakers.Source: §6.1 · pp. 9–10

6.2.1. EVALUATION THROUGH SPEECH RECOGNITION

• The authors use phoneme accuracy from a convolutional ASR model on 16 held-out four-second utterances as a proxy for reconstruction intelligibility, but caution that it can disagree with subjective assessments.The model predicts aligned phoneme labels from spectrogram frames.Source: §6.2.1 · p. 10

6.2.2. AVERAGE EVENT RATES

• Automatic penalty weighting controls average event rate, whereas fixed λ is sensitive; accuracy falls sharply at λ ≥30, and the 75Hz reference target trades peak accuracy for shorter sequences that ease generative modelling.Across 1,024 two-second excerpts, event and phoneme counts are strongly correlated (Pearson’s ρ = 0.866; Spearman’s ρ = 0.753).Source: §6.2.2 · p. 10, Figure 7 · p. 11

6.2.3. NUMBER OF CHANNELS, QUANTISATION LEVELS

• At a fixed 75Hz target, one channel is insufficient and four outperform two, while eight versus four is less clear; 21 quantisation levels show no clear benefit over 15.The authors describe four channels and 15 levels as a balance between expressivity and efficiency.Source: §6.2.3 · p. 10

6.2.4. SLOWNESS PENALTIES; 6.2.5. SCHMITT TRIGGER QUANTISATION, NOISE

• Slowness penalties yield comparable phoneme accuracy, though subjective assessment slightly favors group-sparse; sparse and group-sparse penalties allow larger jumps, and ablations show Schmitt trigger quantisation improves bin use while decoder noise reduces events in silence.Group sparsity encourages synchronous changes across channels.Source: §6.2.4 · pp. 10–11, §6.2.5 · p. 11

6.3.1. INPUT EMBEDDINGS

• In RLT training, removing channel or offset embeddings worsens convergence, output embeddings matter more than input embeddings, and absolute positional embeddings slightly worsen results.The embedding ablations compare holdout negative log-likelihood during training.Source: §6.3.1 · pp. 11–12, Figure 9 · p. 12

6.3.2. SAMPLING AND RERANKING

• Prompted completions are mostly intelligible and often grammatical, with some semantic relevance; unconditional reranked samples include meaningful phrases but rarely fully coherent sentences, while book conditioning influences topic, vocabulary, and style.Nucleus sampling with p = 0.8 noticeably improves overall sample quality.Source: §6.3.2 · p. 13

6.4. Variable-rate and fixed-rate representations

• SlowAE and VQ-VAE representations have similar intelligibility, while the authors conclude variable-rate representations maintain sample quality and adapt representation size.Variable-rate completions are generally shorter, but their prompts occupy a larger share of the fixed-length event sequence and are reconstructed more phonemically accurately.Source: §6.4 · pp. 13–14

Capabilities

• RLT samples show local grammatical and semantic structure and rudimentary prompt understanding, but lack global coherence.The authors compare unconditional samples to char-rnn outputs.Source: §7 · p. 14
• The authors provide no quantitative analysis of RLT sample quality, and transcription-based metrics capture only part of speech quality.They note that human-in-the-loop evaluation may be needed until convincing computational metrics are developed.Source: §7 · p. 14

Interpretability

• SlowAE events are defined implicitly by learned state changes, potentially hampering interpretability, though this is less consequential when events serve as model inputs.The structure of event sequences beyond their correlation with semantically meaningful changes remains for future study.Source: §7 · p. 14

End-to-end learning

• SlowAEs and RLTs are trained sequentially because their objectives differ and may conflict; jointly optimizing predictability could collapse representations and minimize their information content.The authors leave integrated end-to-end learning as an open question.Source: §7 · p. 14

Scale and efficiency

• Speech models are two orders of magnitude smaller than the largest textual language models, and the authors presume larger models may be needed for comparable semantic coherence.They also propose improving representations to reduce the event rate needed for a given intelligibility level.Source: §7 · pp. 14–15

Other considerations

• Event sequences preserve timing exactly, while slowly changing SlowAE values could be encoded as ternary differences resembling spike trains.Abstracting away length variation could reduce information content without significantly affecting reconstruction quality.Source: §7 · p. 15

Applications and other domains

• Variable-rate speech representations may support speech tasks beyond language modelling, and the approach could extend to music, images, video, and event-camera models.Generalizing from sequences to grid- or graph-structured topologies raises open questions, including a 2D analogue of run-length encoding.Source: §7 · p. 15

Other considerations

• The work assumes discrete representations for generative modelling, while continuous representations and hierarchical structure remain unexplored options.Hierarchical representations are suggested as one possible way to abstract length variations.Source: §7 · p. 15

A.1. Slow autoencoders

• The reference SlowAE uses an encoder, conditioning stack, decoder, and quantiser; its variants change slowness penalty, quantisation, or target AER while holding other aspects fixed.The reference target AER is 75Hz.Source: §A.1 · p. 20
• The encoder downsamples 16kHz audio by 32× into four channels, Schmitt-trigger quantisation uses 15 levels, and the decoder is a conditional WaveNet.The conditioning stack upsamples to the audio resolution; the encoder and conditioning stack use anti-causal dilated convolutions.Source: §A.1 · p. 20
• SlowAEs use Adam for 200,000 updates on randomly cropped 360ms LibriTTS clips, with decoder-side Gaussian noise and Polyak averaging.Training uses batch size 64 and a learning-rate schedule of 2·10^-4, 6·10^-5, then 2·10^-5.Source: §A.1 · p. 20

A.2. Post-hoc decoders

• Post-hoc decoders keep checkpointed encoder parameters fixed, use clean input without speaker embeddings, and upsample conditioning to 24kHz.They are trained on the proprietary single-speaker dataset at its original 24kHz resolution.Source: §A.2 · p. 20

A.3. Run-length transformers

• Run-length Transformers use learnable relative positional embeddings and input-stream layer normalisation; model variants have about 1 billion or 2.4 billion parameters.All are trained for 1,000,000 steps; larger models use 128 TPUv4 cores and optimizer state partitioning to fit accelerator memory.Source: §A.3 · pp. 20–21

A.4. ASR model for intelligibility evaluation

• The ASR model predicts 41 phoneme classes from log-scaled mel spectrograms using a 60-block residual network and cross-entropy training.Inputs are 24kHz audio converted to 80-bin spectrograms with frames aligned to labels at 200Hz; accuracy is averaged across frames on a holdout set.Source: §A.4 · p. 21

B. Implementations

• The JAX implementations cover run-length encoding and decoding and Schmitt trigger quantisation, with batch dimensions added via jax.vmap.The provided implementations themselves do not account for batch dimensions.Source: B. Implementations · p. 21

C. Negative results

• The appendix’s negative results are largely anecdotal and are not supported by thorough ablations.The authors caution that failed attempts should not be taken to mean the ideas can never work.Source: C. Negative results · p. 21

C.1. Alternative representation learning strategies

• VQ-VAE code sequences were only weakly compressible with BPE, which grouped code pairs into near-fixed-rate sequences.The difficult-to-control trade-off between compressibility and reconstruction quality led the authors to use integer quantisation with run-length encoding instead.Source: §C.1 · p. 21
• Restricting SlowAE jumps to {−1, 0, +1} hampered early learning; gradually imposing the restriction later was proposed but not tested.Source: §C.1 · p. 21

C.2. Reducing event rates

• Attempts to reduce event rates while maintaining intelligibility included decoder augmentations, side channels, lower-rate encoder inputs, and alternative slowness penalties.Pitch shifts, equalizers, volume changes, and temporal jitter were not helpful; side channels did not meaningfully change the representations, and lower-rate or mel-spectrogram inputs reduced intelligibility.Source: §C.2 · pp. 21–22
• Group-sparse slowness worked better than logarithmic variants, while L1/2 group sparsity produced burstier outputs and larger jumps but hurt intelligibility.Combining L1 and L2 slowness penalties proved difficult to control and behaved mostly like one penalty or the other.Source: §C.2 · p. 22

C.3. Alternative RLT architectures

• Explicit relative attention and multi-stream Transformers were too computationally inefficient on the authors’ hardware, while vanilla Transformers scaled more easily to billions of parameters.Source: §C.3 · p. 22
• Equal weighting of value and length losses gave the best likelihood results.The authors report that reweighting the components made both worse, contrary to expectations.Source: §C.3 · p. 22

D. Sample transcriptions

• Table 2 gives best-effort manual transcriptions of completion samples from the 2.4-billion-parameter RLT; “??” marks unintelligible spans and ellipses indicate expected continuations.Source: D. Sample transcriptions · p. 22, Table 2 · p. 23
• Figure 10 contrasts SlowAE variants: L1 can collapse channels, L2 and L4 change more gradually, and omitting quantisation or decoder noise has stated drawbacks.Without Schmitt-trigger quantisation, quantisation levels are used inefficiently; without decoder-side noise, capacity is wasted in silent input regions.Source: Figure 10 · p. 23
• Figure 11 shows event density varying over time across fixed slowness weights, with event rates changing significantly even when weights differ by less than an order of magnitude.Figure 12 shows similar temporal event-density patterns across target average event rates, with higher targets using quantisation levels better.Source: Figure 11 · p. 24, Figure 12 · p. 24
• Figure 13 shows that more channels increase expressiveness but require slower channel changes to meet the fixed 75Hz target; k = 3 has large quantisation error, while k = 10 uses bins inefficiently.Source: Figure 13 · p. 25
• Figure 14 specifies causal decoder and anti-causal encoder and conditioning dilated convolutions; Figure 15 depicts JAX implementations of run-length encoding, decoding, and Schmitt-trigger quantisation.Source: Figure 14 · p. 25, Figure 15 · p. 26