PaperlayerView exact revision on arXiv

Source-linked AI summary

[b]=[d]-[t]+[p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic

Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho, David Harwath, David R. Mortensen

arXiv:2602.18899v3eess.AScs.CLcs.LGcs.SD

TL;DR

How phonetic information is structured in self-supervised speech representations remains underexplored. This study tests phonological vector analogies and examines how vector scale relates to acoustic realization, finding compositional directions and continuous feature variation.

  • Problem

    Although prior work examined information encoded in self-supervised speech models, how that information is structured remains underexplored.

    Source: §1 · p. 1

  • Method

    The study tests symmetric phonological analogies in phone representations and examines scaled phonological vectors against acoustic measurements.

    Source: §1 · p. 2, §3.1.1 · p. 3, §3.1.4 · p. 4

  • Results

    Analogies based on all 19 tested features consistently hold; for eight acoustically measured features, measurements strongly correlate with vector scale during interpolation and extrapolation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings provide evidence that phonological features can emerge from acoustic regularities and may be continuous rather than strictly binary.

    Source: §6 · p. 9

    The study examines few S3Ms and only the PanPhon feature system, cannot isolate causes of model differences, and uses one vocoder whose characteristics may influence synthesis results.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Self-supervised speech models (S3Ms) are known to encode rich phonetic information, yet how this information is structured remains underexplored. We conduct a comprehensive study across 96 languages to analyze the underlying structure of S3M representations, with particular attention to phonological vectors. We first show that there exist linear directions within the model's representation space that correspond to phonological features. We further demonstrate that the scale of these phonological vectors correlate to the degree of acoustic realization of their corresponding phonological features in a continuous manner. For example, the difference between [d] and [t] yields a voicing vector: adding this vector to [p] produces [b], while scaling it results in a continuum of voicing. Together, these findings indicate that S3Ms encode speech using phonologically interpretable and compositional vectors, demonstrating phonological vector arithmetic. All code and interactive demos are available at https://github.com/juice500ml/phonetic-arithmetic .

1 Introduction

The paper asks whether self-supervised speech models represent phonology compositionally, addressing an underexplored aspect of how their information is structured. It reports phonological vector analogies and a continuous relationship between vector scale and acoustic realization.

1 Introduction

  • The study asks whether S3Ms represent phonology in a compositional manner analogous to the linguistic relations found in word representations.The structure of information in S3M representations remains underexplored despite prior analyses of encoded information and acoustic organization.Source: §1 · p. 1
  • Subtracting r[t] from r[d] gives a voicing vector, while subtracting r[t] from r[p] gives a change-of-place-of-articulation vector.The paper describes these as symmetric compositional phonological vectors.Source: §1 · p. 2
  • Analogies based on all 19 tested PanPhon features consistently hold in S3M representations.The tested features include voicing, place of articulation, and manner of articulation.Source: §1 · p. 2
  • For eight acoustically measured features, measurements strongly correlate with vector scale during both interpolation and extrapolation.The findings suggest that S3Ms encode these features as continua through vector directions and scales, rather than only as binary distinctions.Source: §1 · p. 2
  • The paper identifies vector direction with phonological features and vector scale with their degree, enabling interpretable control of speech synthesis.The authors present these as the paper’s two contributions to understanding S3M representations.Source: §1 · p. 2

2 Datasets

The study uses two phonetically transcribed and manually segmented datasets to assess English-specific and cross-linguistic generalizability. TIMIT provides English speech, while VoxAngeles broadens language coverage and excludes English.

2 Datasets

  • TIMIT and VoxAngeles provide the manually segmented speech data for evaluating English-specific and cross-linguistic generalizability.Both datasets are phonetically transcribed.Source: §2 · p. 2
  • TIMIT contains utterances and phonetic segmentations from 630 English speakers with balanced phonetic and dialectal coverage.The analysis excludes diphthongs and merges plosive closures with their subsequent releases.Source: §2 · p. 2
  • VoxAngeles covers audio from the UCLA Phonetics Archive in 95 languages across 21 language families, and excludes English.The study uses its full set because it has no predefined train/test splits; its non-English phones support testing generalization beyond English phones.Source: §2 · p. 2

3 Experiment 1: Direction of Phonological Vectors

This section tests whether S3M representations support phonological analogies, comparing them with spectral baselines and examining generalization to unseen phones and layerwise patterns.

3 Experiment 1: Direction of Phonological Vectors

  • The experiment tests whether linear phonological vectors satisfying phonological analogies exist in S3M representations by examining their directions.Phone features are drawn from PanPhon; its discrete values are expanded for the analysis.Source: §3 · p. 3, §3.1.1 · p. 3

3.1.1 Creating phonological analogies

  • Each phone quadruplet encodes two symmetric analogies, and the set excludes phones without PanPhon mappings or with fewer than 50 dataset occurrences.The study uses 236 TIMIT and 468 VoxAngeles quadruplets; no quadruplets were obtained for constricted-glottis and consonantal features in either dataset.Source: §3.1.1 · p. 3

3.1.2 Measuring cosine similarities / 3.1.3 Comparing cosine similarities

  • The evaluation averages cosine similarities across randomly sampled phone representations, bootstraps uncertainty, and counts a quadruplet as successful when the specified similarity ordering holds.Same-phone and different-phone comparisons provide reference baselines for assessing the analogy approximation.Source: §3.1.2 · p. 3, §3.1.3 · pp. 3–4

3.1.4 Speech representations

  • The study compares log MelSpec and MFCC with three English-trained monolingual S3Ms, extracting layerwise representations and average-pooling sliced features into phone vectors.The S3Ms are wav2vec 2.0, HuBERT, and WavLM in LARGE configuration, with 25 layerwise representations per model.Source: §3.1.4 · p. 4

3.2.1 S3Ms vs. Spectral representations

  • On TIMIT, HuBERT’s last layer scored 94% and WavLM’s 92%, versus 19% for MFCC and 0% for MelSpec; wav2vec 2.0’s middle layer scored 61%.Figure 2 compares S3Ms with spectral representations on TIMIT and VoxAngeles.Source: Figure 2 · p. 4, §3.2.1 · p. 4

3.2.2 Do S3Ms generalize to unseen phones?

  • On VoxAngeles, 316 of 468 analogies (68%) included non-English phones; WavLM, HuBERT, and wav2vec 2.0 scored 93%, 45%, and 39%, versus 19% for MFCC and 0% for MelSpec.The authors interpret this as evidence that English-only S3Ms capture phonological structure beyond English-specific phones.Source: §3.2.2 · p. 4, Figure 2 · p. 4

3.2.3 Vowels vs. Consonants

  • WavLM success rates peak at the final layer; vowels align with the first intermediate peak, while consonant peaks vary by dataset.Vowels peak earlier and consonants later in the authors’ proposed account of differing temporal-context needs; they suggest the final layer unifies these patterns.Source: §3.2.3 · p. 5
  • The authors found no substantial layerwise-pattern effect from individual phonological features or analogy phonological distances, leaving other causes for future investigation.They also report that deeper layers may leverage broader context and that different temporal complexities are preferentially contextualized at different layers.Source: §3.2.3 · p. 5

4 Experiment 2: Scale of Phonological Vectors

The study tests whether scaling phonological vectors produces graded acoustic changes, using vocoder-resynthesized speech and acoustic measurements. The observed changes follow expected feature trajectories, remain interpretable beyond interpolation in most cases, and include coordinated spectral and temporal cues.

4.1.1 Modifying representations through phonological vectors

  • Phonological vectors are differences between mean representations of phones with and without a feature, computed separately for consonants and vowels.The voicing vector, for example, contrasts averaged voiced and unvoiced consonant representations.Source: §4.1.1 · p. 5

4.1.2 Analyzing modified representations through acoustic measurements

  • The analysis scales vectors on target-phone frames, resynthesizes modified representations with a vocoder, and measures acoustic changes against λ using Spearman’s rank correlation.The vocoders were trained on LibriTTS for English and FLEURS-R for multilingual speech.Source: §4.1.1 · p. 5, §4.1.2 · p. 6, §4.1.3 · p. 6

4.1.4 Phonological vectors

  • Eight vectors were tested against established acoustic measurements: four vowel features and four consonant features.The tested features were high, low, back, round, nasal, sonorant, strident, and voice.Source: §4.1.4 · p. 6

4.2.1 Quantitative analyses

  • Across TIMIT features, acoustic measurements changed monotonically with vector scale and had the theoretically expected correlation signs; VoxAngeles showed similar trends on unseen phones.The changes were monotonic but not necessarily linear.Source: §4.2.1 · p. 7
  • Scaled voicing vectors produced smooth, graded changes rather than binary toggles, and most features remained acoustically interpretable beyond |λ| ≤ 1.Extrapolation saturated for already-sonorant segments, voiced consonants, and non-stridents.Source: §4.2.1 · pp. 7–8

4.2.2 Qualitative analyses

  • Applying the rounding vector to [i] lowered all formants for positive λ, consistent with lip rounding and extending the effect to a vowel absent from English.For [b], positive voicing-vector scales lowered voice onset time and could extend voicing into closure; strident scales introduced high-frequency frication while removing the burst.Source: §4.2.2 · p. 8
  • Nasalization weakened [b]’s burst and introduced a low-frequency murmur, while the broader analyses showed vectors changing temporal cues as well as spectral envelopes.The changes were interpreted as coherent modifications along phonological dimensions.Source: §4.2.2 · p. 8

5 Related works

Prior work examined what speech-model representations encode and how information is structured, while related studies identified linear geometry and phonological axes in other settings. This paper situates its contribution as a cross-lingual S3M analysis that also studies vector scale.

Word analogies

  • Word-embedding analogy research established linear syntactic and semantic relations and continuous adjective scales, while evaluation concerns prompted alternatives based on relational directions.The paper draws an analogy between continuous semantic scales and scaling phonological vectors.Source: §5 · p. 8

Linear representation hypothesis

  • Linear representation and steering-vector research motivates looking for interpretable feature directions and controlling behavior by adding vectors to model representations.The cited context connects those ideas to phonological vectors and controllable speech synthesis.Source: §5 · p. 8

Speech model interpretability

  • Earlier speech-model interpretability work studied encoded spectral, phonetic, articulatory, syllabic, lexical, syntactic, and semantic information, as well as layerwise and similarity-based structure.The cited studies include Pasad et al. (2021, 2023), Choi and Yeo (2022), and Abdullah et al. (2023).Source: §5 · p. 8

Phonological analogies

  • Prior work found linear geometry for morphological inflection and identified phonological axes in S3Ms, while other studies used phonological analogies for speech and text representations.The paper distinguishes its work through large-scale cross-lingual analysis of S3Ms and exploration of phonological-vector scale.Source: §5 · p. 9

Controllable speech synthesis

  • Interpretable-feature synthesis methods use explicitly designed phonetic, articulatory, or phonological representations, unlike the paper’s self-supervised vectors.The authors do not directly compare synthesis performance and describe such vectors as a possible basis for future applications.Source: §5 · p. 9

6 Conclusion

The paper concludes that self-supervised speech models learn composable, scalable phonological vectors without phonological supervision. These findings clarify representation structure and support fine-grained synthesis control and a continuous view of phonological features.

6 Conclusion

  • S3Ms trained only on speech, without phonological supervision, learn linearly composable and scalable phonological vectors.The authors connect this finding to a clearer account of S3M representation structure.Source: §6 · p. 9
  • The vectors offer intuitive interpretations of S3M representations and fine-grained control of speech synthesis along phonological dimensions.The conclusion frames this as a speech-processing contribution.Source: §6 · p. 9
  • The findings provide empirical evidence that phonological features can emerge from acoustic regularities and motivate treating them as continuous rather than strictly binary.The conclusion presents this as a contribution to linguistics.Source: §6 · p. 9

Limitations

The study covers a limited portion of the model and feature space, and synthesis findings may depend partly on the vocoder. The authors caution that these constraints limit what can be concluded about model differences and feature systems.

Limitations

  • Only a small number of S3Ms were studied, and their differing behavior could not be explained from the reported results.The authors characterize the investigated space of possibilities as limited.Source: Limitations · p. 9
  • The study uses PanPhon’s feature system, leaving unresolved whether vector extraction depends on coherent natural-class features or consistent phonetic differences.The authors note that multiple articulatory and acoustic feature systems are possible.Source: Limitations · p. 9
  • Because only one vocoder was evaluated, some synthesis behaviors may reflect vocoder-specific characteristics rather than the S3Ms alone.Source: Limitations · p. 9

Ethics statement

The work uses existing research-licensed datasets and collects no new data. Its speech-synthesis experiments are intended for scientific analysis, while possible misuse and downstream safeguards are acknowledged.

Ethics statement

  • The experiments use publicly available datasets and do not collect new data; deceptive-content applicability is not evaluated or claimed.The authors encourage safeguards for similar techniques in downstream or user-facing systems.Source: Ethics statement · p. 9

Use of AI assistants

The appendix describes analogy evaluation, model representations, and acoustic measurements used to investigate phonological structure. It also reports the manuscript’s use of AI assistants.

Use of AI assistants

  • AI assistants supported code completion, minor text editing, and grammar polishing; authors conceived, verified, and finalized the scientific content, code, results, and analyses.The assistance was primarily for code auto-completion, minor text editing, and grammar polishing.Source: Use of AI assistants · p. 9

A.1 Item- and offset-based analogy tests

  • The study primarily evaluates item-based analogies and uses offset-based pairing consistency as a secondary test.Offset-based tests discard many analogies—36 versus 236 in TIMIT and 112 versus 468 in VoxAngeles—limiting analyses such as vowel/consonant comparisons; success-rate estimates account for distributions across utterances.Source: §A.1 · p. 12

A.2 Details on S3Ms

  • The three LARGE S3Ms provide 25 layerwise representations each, spanning CNN outputs and all 24 transformer layers.The models are wav2vec 2.0, HuBERT, and WavLM, with approximately 300M parameters.Source: §A.2 · pp. 12–13

A.3 Acoustic measurements

  • Acoustic correlates for the eight features are measured using formants, F1 bandwidth, center of gravity, and harmonics-to-noise ratio.F1 measures vowel height, F2 backness and roundness, F1 bandwidth nasality, center of gravity voicing and stridency, and HNR sonorance.Source: §A.3 · p. 13

A.4 Details on vocoder training

The appendix details vocoder training and supplementary tests of phonological relations. It also compares representation-slicing strategies for extracting phone vectors.

A.4 Details on vocoder training

  • The vocoder follows Vocos with modified architecture and training settings, and is trained on LibriTTS and FLEURS-R speech.Batch size and learning rate are each increased eightfold for faster convergence; FLEURS-R contains 1.3k hours of enhanced read speech in 102 languages.Source: §A.4 · p. 13

B.1.1 Settings

  • The offset-based test uses PCS, an ROC-AUC measure contrasting offsets from matching phonological relations with shuffled mismatches.A random baseline is PCS 0.5.Source: §B.1.1 · p. 13

B.1.2 Results

  • PCS confirms S3Ms outperform spectral representations and generally score higher in later layers, but unlike success rates, PCS typically peaks in intermediate layers.PCS also shows non-negligible performance on unseen phones, below that on seen phones; the authors leave the metric discrepancy for future work.Source: §B.1.2 · pp. 13–14, Figure 9 · p. 13

B.2.1 Settings

  • Phone vectors are extracted by averaging representations after either feature slicing the representation matrix or audio slicing the waveform.Feature slicing retains contextualized representations, whereas audio slicing restricts the temporal receptive field to the phone segment and may add margins to meet a minimum window size.Source: §B.2.1 · p. 14

B.2.2 Results

The supplementary results compare slicing strategies, model similarities, and phonological-vector analyses across TIMIT and VoxAngeles. They report layer, feature, distance, and sample-size patterns, as well as limits of constructing vectors from a single phone pair.

B.2.2 Results

  • For S3Ms, feature slicing is more effective than audio slicing, but the reverse holds for spectral representations; feature-sliced WavLM last-layer representations show strongest overall performance.With audio slicing, MFCC achieves 67% success on TIMIT and 50% on VoxAngeles; the authors suspect per-phone magnitude normalization contributes to this improvement.Source: §B.2.2 · p. 14

B.3.2 Results

  • Later wav2vec 2.0 similarities approach 1.0, while WavLM does not show this collapse and has larger margins than MFCC.The authors associate wav2vec 2.0’s collapse with reduced success rates and suggest WavLM’s non-collapsing similarities may support more usable vectors for synthesis.Source: Figure 12 · p. 15, §B.3.1 · p. 15, §B.3.2 · p. 15

B.4.2 Results

  • Fine-tuning XLSR-53 for phone recognition improves success rates over the base model across nearly all layers, with different layerwise patterns for Wav2vec2Phoneme and MultIPA.Wav2vec2Phoneme rises sharply in deeper layers; MultIPA’s gains concentrate earlier, a contrast the authors suggest may relate to language coverage.Source: Figure 13 · p. 15, §B.4.1 · p. 15, §B.4.2 · p. 15

B.5.2 Results

  • Feature-wise layer trends are largely consistent across TIMIT and VoxAngeles, without clear feature-specific differences apart from a small syllabic/lateral TIMIT case.The exception involves four permutations of syllabic and non-syllabic [l] and [n] phones.Source: Figure 14 · p. 16, Figure 15 · p. 16, Figure 16 · p. 16

B.6.2 Results

  • Layerwise trends remain largely consistent across phonological distances in both datasets, with no increase-related variation apart from bins containing four quadruplets.Those bins arise from permutations of a single phone set.Source: §B.6.1 · pp. 16–17, Figure 18 · p. 17, Figure 19 · p. 17

B.7.2 Results

  • For both datasets, vector-approximation cosine similarity improves as sample size increases and approaches 1.0 at N = 256.The authors conclude that a few hundred samples suffice for accurate approximation.Source: §B.7.1 · p. 17, §B.7.2 · p. 17, Figure 20 · p. 18

B.8.2 Results

  • A fixed phone pair yields cosine similarity around 0.5 even with hundreds to over a thousand TIMIT samples per phone, unlike the full-data vector.The authors report that pair-derived vectors differ from the full-data vectors and suggest that contrasts across multiple features may explain the difference.Source: §B.8.1 · p. 17, §B.8.2 · pp. 17–18, Figure 21 · p. 18

B.9.1 Settings

  • The appendix evaluates pairwise cosine similarities among eight phonological vectors for TIMIT and VoxAngeles.The supplied passage introduces the comparison but does not state its findings.Source: §B.9.1 · p. 18

B.9.2 Results

Across datasets, extracted phonological vectors show interpretable relationships, while WavLM vector scales track acoustic realization; MFCC-derived vectors show little to no controllability in synthesis experiments.

B.9.2 Results

  • Across TIMIT and VoxAngeles, cosine similarities reveal near-orthogonal vowel and consonant vectors, with interpretable relationships among vowel height, backness, and rounding.The figures show pairwise cosine similarities; the reported vowel relationships include high versus low and rounding with high, back, and low vectors on VoxAngeles.Source: §B.9.2 · p. 18, Figure 22 · p. 19, Figure 23 · p. 19
  • Nasal, sonorant, and voice vectors have positive similarities, while strident and sonorant vectors have negative similarities.The text relates these patterns to nasals being sonorants, sonorants almost always being voiced, and stridents not being sonorant.Source: §B.9.2 · p. 18

B.10.2 Results

  • VoxAngeles with a FLEURS-R-trained vocoder shows trends nearly identical to TIMIT with a LibriTTS-trained vocoder, including monotonic acoustic relationships and smooth interpolation and extrapolation.The close correspondence between seen and unseen language settings indicates effective generalization of WavLM representations to unseen languages through phonological vectors.Source: §B.10.2 · p. 19, §B.10.1 · p. 19, Figure 26 · p. 21

B.11.2 Results

  • The round-vector scale reflects dataset-specific formant patterns: the weakest correlation is in F1 for TIMIT and F3 for VoxAngeles.Formant distributions differ by dataset as well: TIMIT shows more prominent F2 and F3 differences, while VoxAngeles shows more prominent F1 and F2 differences.Source: §B.11.2 · pp. 19–20, Figure 24 · p. 19, §B.11.1 · p. 19

B.12.2 Results

  • With λ = 0, acoustic-measurement differences between original and resynthesized speech are tightly concentrated around zero across tested consonants and vowels.This identity-resynthesis comparison checks the stability of the vocoder and acoustic measurements.Source: §B.12.2 · p. 20, §B.12.1 · p. 20, Figure 27 · p. 21

B.13.2 Results

  • MFCC-derived vectors show little to no correlation with acoustic measurements across slicing strategies, unlike WavLM, and are ineffective for synthesis experiments.Weak, inconsistent exceptions occur for back vowels with audio-sliced MFCCs and stridents with feature-sliced MFCCs; the tests use both slicing strategies.Source: §B.13.2 · p. 20
Loading 2602.18899v3…