Source-linked AI summary
Using the Mimi codec for metalinguistic representations
Artem Saloev, Erin Pacquetet, Nicolas Ballier
TL;DR
It is unclear whether Mimi’s discretized audio tokens align with phonetic categories. Aligning its semantic codebook with TIMIT transcriptions, the paper finds that word-aligned mappings best capture the representations’ structure.
Problem
The paper asks whether Mimi’s discretized audio tokens align with phonological and phonetic categories, a question central to interpreting neural audio representations.
Method
The authors align Mimi’s semantic codebook tokens with TIMIT’s time-aligned phonetic transcriptions at phone, word, and utterance levels.
Results
Word-aligned transcriptions best capture the gist of Mimi’s semantic token representations, revealing word allophone networks.
Takeaways & Limitations
Mimi’s semantic tokens provide preliminary evidence for context-dependent, allophonic representations rather than fixed phonological categories.
Takeaways & Limitations
The relationship between phonetic units and Mimi tokens is not strictly one-to-one, with phonetic entities mapped onto multiple context-dependent tokens.
Abstract
from arXiv · showhide
In this paper, we focus on the dictionary of 2048 tokens used in Mimi semantic token codebook, the neural codec of the Moshi language model. We show that the ABX experiment carried out with Mimi fails to capture the mapping of the semantic tokens to phone realisations. By realigning Mimi representations to the TIMIT corpus transcriptions, we show that the 2048 tokens IDs of the semantic codebook map to quadphone, triphone, biphone, phone and subphone realisations.
1 Introduction
Codec-based audio language models use discrete neural codec tokens for speech modelling, but it remains unclear how these tokens encode voices and speech features. This paper investigates Mimi’s semantic token representations by aligning them with time-aligned TIMIT phonetic transcriptions.
- Motivation: Codec-based audio language models convert waveforms into discrete tokens that Transformers predict and decoders reconstruct into speech.Neural codecs typically use multiple residual-vector-quantisation codebooks, producing few tokens per frame.
- Motivation: Discrete audio tokens enable stable long-context modelling, controllable bit-rate and latency, and potentially faster coarse-to-fine generation.These systems increasingly support conversational agents and speech-to-speech translation.
- Research question: Limited-frame-rate codecs raise the fundamental question of whether learned tokens partition continuous acoustic signals in linguistically meaningful ways.The case study uses 80 ms frames, motivated partly by real-time interaction and latency constraints.
- Research question: The study tests Mimi’s multilevel representations against time-aligned TIMIT phonetic transcriptions to examine how far codec and phonetic representations coincide.Mimi is a streaming neural codec whose semantic information is distilled into part of its quantization process using WavLM-derived audio tokens.
- Contributions: Word-aligned TIMIT transcriptions were found most relevant for capturing the gist of Mimi’s semantic token representations and representing word allophone networks.The analysis also matched semantic tokens with phonetic data at phone, word, and utterance levels.
- Contributions: The paper contributes a replicable preliminary inventory of Mimi’s semantic tokens in TIMIT and discusses theoretical and phonetic implications of semantic speech tokenisation.The method and code are intended to support analysis across the other 23 languages used to train Mimi’s neural audio codec.
2 Previous Research
Previous research has evaluated neural codec interpretability mainly through text-to-sound generation and decoding outputs, while Mimi’s semantic tokens were assessed for phonetic discriminability using ABX tasks. The paper argues that ABX inadequately represents the phonetic content of Mimi’s 80 ms speech slices, which may encode subphonemic details or complex co-articulation.
- Previous neural-codec research: Prior neural-codec studies tested text-to-sound generation, including speech synthesis evaluated with UTMOS, and primarily examined interpretability at the decoding end.These studies compared neural audio codec outputs with human judgment.
- Mimi evaluation: Mimi’s semantic tokens were evaluated with ABX discrimination tasks using ABX error rate to characterize phonetic discriminability.The ABX metric compares distances between embeddings of same-triphone instances and minimally different contrastive triphones; lower error indicates better discriminability.
- Limitations of ABX: The paper argues that ABX assumes phonemic representations, whereas Mimi’s 80 ms speech slices can fall below phoneme level in elongated vowels or span consonant clusters.The authors therefore consider Mimi tokens unlikely to have a consistent phonemic representation.
- Limitations of ABX: Mimi’s token time frame is instead argued to capture complex co-articulation phenomena involving mutual influences between speech elements.This motivates questioning whether ABX captures what an individual Mimi token represents phonetically.
3 Material and Methods
The study extracted Mimi’s eight codebooks but analyzed only codebook_0, the semantic-token codebook, using TIMIT audio and time-aligned transcriptions. It realigned token sequences with phone, word, and utterance transcriptions despite mismatched 80 ms token intervals, evaluating phone associations with PNMI rather than PER.
- Mimi representations: The analysis extracted Mimi’s eight codebooks and focused on codebook_0, the codebook intended to represent semantic tokens.The semantic tokens are produced by a single vector quantizer distilled from WavLM embeddings and combined with separate acoustic tokens.
- Corpus: The researchers used TIMIT’s official training and test partitions, containing 4,620 training utterances and 1,344 test utterances.The recordings cover 630 speakers across eight American English dialect regions, with phonetically balanced sentences.
- Corpus: Each TIMIT utterance consisted of a 16kHz, 16-bit linear PCM waveform with sentence-, word-, and phoneme-level time-aligned transcriptions.These alignments supported matching audio-derived Mimi tokens to linguistic transcription units.
- Alignment: The audio was encoded into Mimi codebooks, and semantic codebook_0 sequences were matched to corresponding TIMIT words and phones.Because each token represents an 80 ms audio interval, token boundaries rarely coincide with phone or word boundaries; tokens may span multiple phones, and words may span multiple intervals.
- Analysis levels: The study reported three analysis levels—phone, word, and utterance—to account for time-dependent tokens that do not align with phonetic transcriptions or word pronunciations.Utterance-level analysis aimed to predict different TIMIT regions from transcriptions encoded into tokens.
- Evaluation: At phone level, the researchers used Phone-Normalized Mutual Information instead of Phone Error Rate to compare Mimi token associations with TIMIT phonetic alignments.The procedure replicated the SpeechTokenizer analysis method.
4 Results
Mimi semantic tokens show strong but context-sensitive correspondences with TIMIT phones: they are dominated by particular phonemes while each phoneme maps to many competing tokens. Across results, the codebook exhibits broad utilization, effective matching purity, strong word-level consistency, and limited dialect-region predictability.
- Token–phone correspondences: 1793 of 2048 token IDs were identified in TIMIT, yielding an 87.54% Mimi semantic codebook utilization rate.The passage states this utilization is more optimal than the rate reported for EnCodec NAC.
- Token–phone correspondences: PNMI was 0.66, above HuBERT (0.43) and EnCodec (0.28) but below SpeechTokenizer (0.71).The measure captures matching purity between majority votes and alternative candidates within token receptive fields.
- Word-level consistency: Among 2,378 words shared across training and testing, only 85 had different test transcriptions, and 3.57% of token sequences were not predicted.Dialectal features may explain the observed variation in token sequences.
- Dialect-region prediction: A random forest classified seven TIMIT dialect regions with 32.63% accuracy, significantly above the no-information rate (p < .001).The 80–20 split produced a 95% confidence interval of [26.69%, 39.01%].
- Entropy-based analysis: Tokens had low entropy conditional on phonemes, whereas each phoneme corresponded to a broad distribution of competing tokens.This asymmetry indicates dominant phoneme associations for tokens but many-to-one mappings from phonemes to tokens.
- Token stability and allophony: Many phonemes showed Jensen–Shannon divergence often exceeding 0.8 between token distributions in shared sentences, supporting context-dependent allophonic encoding.The results indicate sensitivity to phonetic context and coarticulation rather than phonemic distinctions alone.
5 Discussion
Mimi’s semantic codebook appears to encode fine-grained acoustic-phonetic patterns rather than phonemes as discrete symbolic units. Its context-dependent tokens reflect variation shaped by coarticulation, prosody, and speaker-specific factors, suggesting an emergent inventory of subphonetic units.
- Discussion: Mimi’s semantic codebook does not encode phonemes as discrete symbolic units, but captures finer-grained acoustic-phonetic patterns.This aligns with bypassing transcription, where linguistic structure is inferred directly from speech without predefined phonemic categories.
- Discussion: Mimi tokens function as context-dependent acoustic units, with one phoneme corresponding to multiple token realizations across contexts.The observed entropy asymmetry and high cross-context divergence support this interpretation.
- Discussion: The token variation depends on coarticulation, prosody, and speaker-specific factors, suggesting an emergent inventory of subphonetic units.This parallels context-dependent states in HMM-based speech recognition, which represent clustered acoustic realizations conditioned on phonetic environment rather than phonemes themselves.
6 Potential Applications for Phonetic Modeling
Mimi’s semantic tokens resemble context-dependent subphonetic units rather than discrete phonemes, encoding acoustic-phonetic variation through distributed, many-to-many mappings. Across languages and codebooks, token usage shows structured, increasingly distributed representations, while language-specific differences remain visible.
- Subphonetic modeling: Mimi tokens capture context-dependent acoustic-phonetic realizations shaped by coarticulation and phonetic environment rather than discrete phoneme-like symbols.This parallels senones, which model variation in phoneme realizations across acoustic environments, while Mimi encodes finer-grained variation than phoneme categories alone.
- Cross-linguistic structure: Across 27 European Union languages, token distributions cluster by language family, suggesting that the first codebook captures phonetic variation shared across related languages.The clustering supports an allophonic interpretation because the tokens appear to encode articulatory or acoustic properties that generalize across languages.
- Distributed encoding: Mimi does not map phonetic units one-to-one onto tokens; instead, phonetic entities are represented through multiple context-dependent tokens in a distributed codebook.The entropy analysis shows that each phoneme corresponds to a broad set of competing tokens, and combinations across codebooks represent phonetic units.
- Codebook behavior: Entropy increases from codebook_0 to higher codebooks across all language families, indicating progressively more uniform and largely language-independent token usage.The shared trend suggests that representational structure is not specific to one language family.
- Codebook behavior: Missing-token counts decrease sharply and maximum token probability declines across codebooks, indicating improved acoustic coverage and less dominance by a few units.Together, these patterns support capacity distributed across multiple codebooks and increasingly fine-grained encoding at deeper levels.
- Probability distributions: CB0–CB2 have highly peaked token distributions, whereas CB4–CB7 are flatter and more uniform, indicating a shift from selective to broader token use.The earliest codebooks may capture lower-level acoustic or phonetic features, while later codebooks use tokens less selectively.
7 Conclusion
The paper aligns Mimi’s semantic-token codebook with TIMIT phones and measures the consistency of this mapping, while identifying acoustic-code analysis and broader metalinguistic representation as future directions. It also notes limitations concerning gender effects, token duplicates, and unexamined acoustic correlates.
- The study compares TIMIT phone alignments with Mimi’s first semantic-token codebook and measures the consistency of the resulting mapping.
- The seven acoustic codebooks remain to be investigated for links to suprasegmental speech properties and other acoustic correlates.Suggested analyses include energy, entropy, frequency content, and mid-temporal formant extractions for assigning prototypical vowel values to token IDs.
- The work does not fully investigate gender effects or potential duplicate tokens associated with different male and female formant values.
- Mimi tokens appear to function as indexical units whose distributions reflect shared acoustic structure and context-dependent variation rather than fixed phonological categories.This perspective motivates computational creolistics and a computational metalinguistic representation of allophony.