Probing the phon* representations of the Mimi codebooks

1 Introduction

• Neural audio codecs provide discrete speech interfaces for large language models and are increasingly used in speech generation and conversational systems, yet their linguistic content remains poorly understood.The paper situates this question across codecs including SoundStream (Zeghidour et al., 2021), EnCodec (Défossez et al., 2022), and Mimi (Défossez et al., 2024).Source: §1 · p. 1
• Mimi’s 80 ms tokens can span multiple phonetic events, making segment-local ABX assumptions difficult to reconcile with its fixed tokenization.The study instead examines statistical relationships between tokens and phonetic categories while accounting for token–phoneme overlap.Source: §1 · p. 1
• The study analyzes CB0 with entropy-based probing, token–phoneme association measures, cross-sentence stability tests, and controlled interventions.The framework draws on posteriorgram analyses (Churchwell et al., 2024) and bidirectional token–phoneme methods for SpeechTokenizer.Source: §1 · p. 1
• Individual Mimi tokens show preferences for specific phonetic categories, while phonemes are realized through distributed, context-sensitive token families.The authors conclude that the first codebook is not a simple phoneme inventory and captures richer acoustic structure relevant to speech reconstruction.Source: §1 · p. 2

2.1 Neural Audio Codecs

• Neural audio codecs have expanded from compression systems into speech tokenizers for spoken language models, but the linguistic interpretation of their tokens remains unresolved.The open possibilities include phonemes, finer subphonemic realizations, prosodic events, or combinations of representational levels.Source: §2.1 · p. 2

2.2 Previous Research on Mimi

• Mimi’s first codebook is described as carrying semantically relevant information, while the remaining codebooks contribute acoustic information; systematic distributional probing of CB0 had not been reported.Prior work discussed prosodic features in the last seven codebooks.Source: §2.2 · p. 2

2.3 The SpeechTokenizer Method

• SpeechTokenizer’s Codebook Analysis uses forced phonetic alignments and token sequences to estimate token–phoneme associations, and this study extends that perspective to additional CB0 analyses.The extensions include entropy asymmetries, token-family structure, cross-sentence stability, and controlled interventions.Source: §2.3 · p. 3

2.4 Limitations of ABX Evaluation for Fixed-Frame Codecs

• Mimi’s fixed 80 ms tokens routinely span multiple phonemes, so ABX scores mix target-category information with neighboring-segment context; distributional probing instead analyzes token–phoneme associations across occurrences.The authors argue that this avoids the confound of treating each token as a segment-sized event.Source: §2.4 · p. 3

3.1 Data

• TIMIT analysis is restricted to the two sentences shared by all speakers, with only modal-length utterances retained for strict positional comparisons.The retained modal counts are 42 frames for SA1 and 35 for SA2; the authors acknowledge that this excludes speakers whose speech rates differ from the modal duration.Source: §3.1 · pp. 3–4

3.2 Token Manipulation and Resynthesis Validation

• Controlled interventions replace individual tokens or all tokens in a selected codebook before decoding, and ASR-based WER measures linguistic changes in the reconstructed waveforms.Single-frame substitutions are exploratory; global constant-token replacement provides the quantitative intervention condition.Source: §3.2 · p. 4

3.3 Forced Alignment

• MFA supplies phone-level intervals, and each 80 ms Mimi token is assigned the phoneme occupying the largest proportion of its duration.The overlap procedure is used because phoneme durations vary and token and phoneme boundaries do not consistently coincide.Source: §3.3 · p. 4

4.1 Duration and Token Distribution Across Speakers

• Across speakers at corresponding positions, mean token entropy is 3.79 bits for SA1 and 4.30 for SA2, below the 11-bit maximum for a 2,048-token codebook.This indicates structured rather than uniform token assignments, but entropy alone cannot establish genuine phonetic organization.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 Token–Phoneme Alignment under Controlled Temporal Conditions

• Token-to-phoneme entropy is 0.31 bits for SA1 and 0.33 for SA2, with mean dominant-phoneme probability approximately 0.89 in both.Most tokens are dominated by a single phonetic category, although the association is not strictly one-to-one.Source: §4.2 · p. 5

4.3 Phoneme–Token Alignment

• Phoneme-to-token entropy is 2.58 bits for SA1 and 2.63 for SA2, corresponding to approximately six competing tokens per phoneme on average rather than 82 under uniform assignment.The concentrated competition is consistent with contextual phoneme variants being represented by distinct related tokens.Source: §4.3 · pp. 5–6, Figure 3 · p. 6

4.4 Cross-sentence stability of token families

• Token-family distributions differ across sentences beyond the permutation baseline: mean JSD is 0.61 versus 0.14, with 14 of 15 shared phonemes significant after correction.The instability suggests that CB0 captures fine-grained speech realization varying with sentence context, not only broad phonemic identity.Source: §4.4 · p. 6

4.5 PNMI Analysis

• PNMI is 0.84 for both SA1 and SA2, and Mimi CB0 outperforms EnCodec and SpeechTokenizer RVQ-1 despite lacking an explicit phoneme-level training objective.The comparison is indicative because Mimi uses modal-length subsets while the reported comparison systems use the full TIMIT training set.Source: §4.5 · pp. 6–7, Table 2 · p. 7

4.6 Phonological Information Across the Mimi Codebooks

• CB0 has the strongest phoneme correspondence across Mimi’s eight codebooks, while the remaining codebooks retain PNMI values of approximately 0.55–0.67.Phonological information is therefore not exclusive to CB0 and remains accessible, at reduced organization, throughout the RVQ hierarchy.Source: §4.6 · p. 7, Table 3 · p. 7

4.7 Controlled Interventions

• Global replacement of CB0 tokens raises WER to 0.701, a +0.208 increase over reconstruction, while replacing one frame adds no WER increase beyond reconstruction.Results use SI and SX sentences from one speaker; single-frame perturbations are absorbed by the remaining seven codebooks.Source: §4.7 · pp. 7–8, Table 4 · p. 8

• ARPAbet labels are mostly categorical and omit many contextual processes, so after vowel-stress variants are collapsed, aligned labels show little surface alternation.Source: §5.1 · p. 8
• Phonemes draw on structured competing token sets: the mean effective competitor set is approximately 6 tokens (22.58 for SA1), far below 82 expected under uniform random assignment.Source: §5.1 · p. 8
• Same-phoneme token families vary across sentences: mean JSD = 0.61 (p < 0.05) for 14 of 15 shared phonemes, with /L/ entropy rising from 1.70 bits in SA1 to 3.93 bits in SA2.Source: §5.1 · p. 8
• This instability is consistent with tokens reflecting coarticulation and reduction, particularly in intervocalic environments, and the surface acoustic realization of segments in context.Source: §5.1 · p. 8

• Across the two utterances pronounced by all TIMIT speakers, CB0 tokens correlate with allophonic rather than phonemic representations.Source: §6 · p. 9
• The first codebook appears to yield phonetic categories, while Mimi’s wider 80 ms token window may also capture speaker-related acoustic detail.Source: §6 · p. 9

• The study focuses on two TIMIT utterances shared by all speakers; the authors say the allophonic token distribution should be tested on more diverse English varieties.Source: Limitations · p. 9