Source-linked AI summary
textless-lib: a Library for Textless Spoken Language Processing
Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, Yossi Adi
TL;DR
Textless spoken-language modeling lacks textual supervision, and its existing implementations can be complicated to run. textless-lib packages speech-processing building blocks and example tasks; the examples demonstrate probing, compression, and continuation, with reported findings on speaker information and compression potential.
Problem
Textless spoken-language modeling can support languages with few textual resources and capture spoken features absent from text, but existing implementations require domain expertise and complicated multi-script setups.
Source: §1 · pp. 1–2Method
Results
The examples show continuous CPC and HuBERT representations identify held-out speakers, quantization adds speaker-invariance, and discretized HuBERT representations have strong potential for extreme speech compression.
Source: §4.1 · p. 6, §4.2 · p. 6Takeaways & Limitations
The authors conclude that textless-lib simplifies textless spoken-language research and may be useful to speech researchers and the broader NLP community.
Source: §5 · p. 6The current vocoder setup lacks the unit-duration prediction model needed to feed discrete units directly to the vocoder; adding such models and other components remains future work.
Source: §3.2 · p. 5, §5 · p. 6
Abstract
from arXiv · showhide
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large. The code, documentation, and pre-trained models are available at https://github.com/facebookresearch/textlesslib/ .
1 Introduction
Textless spoken language modeling learns acoustic and linguistic characteristics directly from raw audio, without textual supervision. textless-lib addresses the complexity of this research setting with a PyTorch library and example pipelines for spoken-language tasks.
1 Introduction
- Textless spoken language modeling learns acoustic and linguistic characteristics from raw audio without textual supervision, building on progress in speech representation learning, language modeling, and synthesis.A GSLM pipeline encodes speech into discrete units, models those units, and decodes them to generate speech.Source: §1 · p. 1
- The figure contrasts language modeling for speech continuation with direct speech resynthesis.The examples are attributed to Lakhotia et al. (2021) and Polyak et al. (2021), respectively.Source: Figure 1 · p. 1
- Textless modeling can serve languages with few textual resources and represent prosody, non-verbal vocalizations, and speaker identity that text does not capture.The motivation also includes high-resource languages whose oral and written forms differ in lexicon and syntax.Source: §1 · pp. 1–2
- Prior implementations required domain expertise and multiple scripts and checkpoints, making textless spoken-language modeling a complicated setup.The cited GSLM implementation had roughly four launching scripts and dozens of checkpoints; another system used four scripts across two repositories.Source: §1 · p. 2
- Textless-lib provides building blocks and example pipelines for processing, encoding, modeling, and generating speech, with examples including probing, compression, and continuation.The authors aim to make these tasks possible with a few lines of code.Source: §1 · p. 2
2 Background
The background presents textless speech processing as a pipeline that converts audio to units, optionally models those units, and reconstructs speech. It describes the representations and synthesis and sequence-modeling options used in that pipeline.
2 Background
- A common pipeline encodes speech into discrete pseudo-text units, models units for tasks such as continuation or conversion, then reconstructs speech; direct resynthesis can omit unit modeling.The pipeline uses speech-to-units, units-to-units, and units-to-speech components.Source: §2 · p. 2
2.1 Speech to Units
- The speech-to-units encoder maps a waveform to lower-rate dense vectors, which quantization converts into discrete tokens; collapsing repeated tokens simplifies long sequences but loses tempo information.The available-models table summarizes pretrained models and notes that quantizers were trained on LibriSpeech dev-clean.Source: §2.1 · p. 2, Table 1 · p. 2
2.2 Units to Speech
- Units-to-speech reconstruction can compose a mel-spectrogram estimator with a vocoder, while aligned streams may represent discrete units, duration, and pitch.For duped units, synthesis can omit the mel-spectrogram estimator and feed units directly to a vocoder; another option predicts durations to expand the sequence first.Source: §2.2 · p. 3, Figure 2 · p. 3
2.3 Unit to Units
- A units-to-units model maps one discrete sequence to another; language models support continuation, while sequence-to-sequence models can frame emotion conversion as translation.The pipeline can also use common NLP architectures to model discrete unit sequences.Source: §2.3 · p. 3
3 Library Overview
textless-lib aims to simplify textless spoken-language research with loosely coupled building blocks, interfaces, preprocessing support, and pre-trained models. Its examples and components are designed to support flexible research use.
3 Library Overview
- The library supplies core building blocks and pre-trained models with minimal coupling, while keeping U2U models and evaluation metrics outside its core functionality.This separation is intended to keep the library simpler and easier to use.Source: §3 · p. 3
3.1 Interfaces
- SpeechEncoder and vocoder interfaces separate speech encoding from decoding, enabling researchers to mix components and study them independently.SpeechEncoder produces discrete pseudo-text units and can also provide duration and pitch streams.Source: §3.1 · pp. 3–4
- Dataset wrappers expose quantized representations, while preprocessing tools can distribute dense-model work across GPUs or precompute audio-token streams.The tools support multi-GPU and multi-node parallelism; on-the-fly preprocessing may be unnecessary when data augmentation is not randomized.Source: §3.1 · pp. 4–5
3.2 Pre-trained Models
- The library includes HuBERT and CPC representations, YAAPT pitch extraction, and k-means quantizers for vocabulary sizes 50, 100, 200, and 500.Per-speaker pitch normalization assumes the speaker mean is known; prefix-based normalization trades precision for flexibility.Source: §3.2 · p. 5
4 Examples
The examples demonstrate textless-lib through speaker probing, discrete speech resynthesis and compression, and speech continuation. They illustrate component use and provide starter code for these application types.
4 Examples
- The examples illustrate library components and offer starter code for applications such as probing representations and building unit-based language models or high-compression codecs.Discrete resynthesis is the speech-audio-to-deduplicated-units-to-speech-audio pipeline.Source: §4 · p. 5, §4.2 · p. 6
4.1 Speaker Probing
- Continuous CPC and HuBERT representations identify speakers in held-out utterances, whereas quantization adds speaker-invariance.The experiment trains a two-layer Transformer to predict anonymized speaker identifiers.Source: §4.1 · p. 6
4.2 Speech Resynthesis
- Discretized HuBERT representations show strong potential for extreme speech compression, with bitrate and ASR word error rate evaluated across vocabulary sizes.WER is calculated using wav2vec 2.0-based ASR against ground-truth transcripts; unigram token probabilities are fitted on LibriLight 6K transcripts.Source: §4.2 · p. 6
4.3 Speech Continuation
- The speech-continuation example reimplements the GSLM pipeline and presents three continuations of the same prompt generated with different random seeds.It uses a LARGE wav2vec 2.0 model trained on LibriSpeech-960h with CTC loss and a KenLM 4-gram decoder.Source: §4.3 · p. 6
5 Discussion and Future Work
textless-lib is a PyTorch library for textless spoken-language research, providing building blocks for speech processing and synthesis. Its examples demonstrate representation probing, speech compression, and speech continuation; future work includes improving components and expanding tasks, models, and training support.
5 Discussion and Future Work
- textless-lib is a PyTorch library for textless spoken-language research that simplifies speech processing and synthesis.The authors describe it as a resource for both speech researchers and the NLP community.Source: §5 · p. 6
- The library demonstrates its usability through representation probing, speech compression, and speech continuation examples.Source: §5 · p. 6
- Future work includes improving existing building blocks, adding example tasks, expanding pretrained models, and enabling training of the components.Source: §5 · p. 6
- Polyak et al. (2021) used non-deduplicated streams, unlike this setup, and therefore obtained different bitrates.Source: §5 · p. 6