PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters

Abdul Rehman, Jian-Jun Zhang, Xiaosong Yang

arXiv:2610.05575v1cs.CLcs.AI

TL;DR

Expressive TTS research often assumes text can guide prosodic style, but prior evaluations do not directly measure that premise against acoustically derived labels. This paper tests it with controlled cross-modal cluster prediction and finds a weak, mainly word-identity association rather than compact text-space style regions.

  • Problem

    Prior text-to-style work and expressive-appropriateness evaluation presuppose that text constrains style, but do not measure that constraint against an explicit null while separating length, label priors, and word identity.

    Source: §I · p. 1

  • Method

    The study predicts acoustically derived utterance clusters from text embeddings for six speakers, using length erasure, a majority-class floor, and bag-of-words as controls.

    Source: A. Task Definition · p. 4, §X · p. 11

  • Results

    Text predicts acoustic clusters above the majority-class floor for all six speakers (+0.111 top-3), but bag-of-words achieves three quarters of this result and clusters are not compact in text space in any of 360 configurations.

    Source: §I · p. 2, §X · p. 12

  • Takeaways & Limitations

    Within this design, text informs utterance-level delivery mainly through word choice, which may cue delivery or mark topic and recording situation; reference-free style selection cannot assume more.

    Source: §X · p. 12

    The labels are unsupervised acoustic partitions, not perceptually validated speaking styles, and the study uses one K = 30 clustering for each speaker–speech-model pair.

    Source: D. Limitations · p. 11

Abstract

from arXiv · show

Much of expressive text-to-speech research rests on an untested assumption that written text carries enough information to select an appropriate prosodic style for its delivery. Text-predicted style models improve listener preference, and expressive-appropriateness evaluation presupposes that context constrains style, yet neither measures the assumption itself. This paper tests it as a falsifiable hypothesis against style labels derived from acoustics alone. For each of six speakers in a 1,200-hour conversational corpus, utterances are clustered in the spaces of five speech models, including a prosody-only control, and the cluster of held-out utterances is predicted from twelve text embedding models. Three controls are applied: utterance length is erased from the speech embeddings; accuracy is scored against the majority-class floor of unbalanced clusters rather than uniform chance; and a bag-of-words baseline measures word identity alone. Text predicts the cluster above that floor for all six speakers (+0.111 top-3 accuracy), but bag-of-words achieves three quarters of this. Sentence embeddings add only +0.026, largest for encoders not trained for sentence semantics and reversed by tree-based probes for all others. Acoustic clusters are not compact in text embedding space in any of 360 configurations. The prosody-only space weakens the association for five speakers, but not for the speaker showing it most strongly. Text thus informs these delivery clusters mainly through word choice, whether as a cue to prosody or as a marker of topic and recording situation, and reference-free style selection cannot assume more.

I. INTRODUCTION

The paper tests whether text predicts utterance-level prosodic style beyond length and label frequency, using style labels derived independently from acoustics. It finds a weak association carried mainly by word identity, while text embeddings do not form compact regions for the acoustic clusters.

I. INTRODUCTION

  • Text-predicted style models and expressive-appropriateness evaluations suggest text is informative, but neither directly measures how strongly it constrains style against an explicit null.Prior work uses co-trained latent targets or synthesis and preference outcomes, without separating length, label priors, and word identity as possible carriers.Source: §I · p. 1
  • The test asks whether text predicts acoustically derived style categories beyond length and label frequency, and whether those categories occupy distinguishable regions in text space.The paper frames the claim probabilistically because the same sentence can have multiple contextually valid renderings.Source: §I · p. 1
  • The evaluation predicts held-out acoustic cluster labels from text embeddings and compares performance with length-only and bag-of-words arms using the same labels and utterances.A prosody-only speech representation is included to test whether alignment persists when lexical information is substantially reduced.Source: §I · p. 2, Figure 1 · p. 2
  • +0.111 top-3 accuracy: text predicts acoustic clusters above the majority-class floor for all six speakers, while bag-of-words achieves three quarters of that result.Sentence embeddings add only +0.026 over bag-of-words; the association is weaker in the prosody-only space for five speakers, but not for the speaker with the strongest association.Source: §I · p. 2
  • Acoustic clusters are not compact in text embedding space in any of 360 configurations, so the geometric prediction fails despite above-floor classification.Word identity may cue delivery, but it may also track topic and recording situation.Source: §I · pp. 1–2

II. BACKGROUND

The background motivates within-speaker clustering, distinguishes the paper's direct test from prior text-to-style work, and explains why frozen sentence embeddings isolate information availability from exploitation.

A. Self-Supervised Speech Representations

  • Self-supervised speech encoders carry prosodic structure, but strong downstream classification does not ensure coherent unsupervised clusters, motivating within-speaker clustering and a prosody-specific control.Clustering across speakers could otherwise reflect dominant speaker identity.Source: A. Self-Supervised Speech… · pp. 2–3

B. Prosody and Style in TTS

  • Prior TTS work predicts or aligns style from text, but its co-trained targets and synthesis or preference outcomes do not measure whether text alone suffices or isolate lexical, length, and channel information.Expressive-appropriateness work treats the text-to-style link as a premise rather than its object of measurement.Source: B. Prosody and Style in TTS · p. 3

C. Sentence Embeddings

  • Frozen text embeddings and shallow classifiers test whether text information is present and accessible, while plain masked-language-model encoders help distinguish semantic from surface explanations.The setup separates information availability from how a deployment might best exploit it.Source: C. Sentence Embeddings · p. 3

III. A MINIMAL PROSODY-ONLY REPRESENTATION

The paper introduces TIMBRESPACE as a reduced-lexical-information control, then defines the within-speaker prediction and geometry tests using shared acoustic cluster labels.

III. A MINIMAL PROSODY-ONLY REPRESENTATION

  • TIMBRESPACE is a prosody-focused speech representation intended to retain prosody and timbre while minimizing lexical content, enabling a test of lexical leakage.Its six acoustic features exclude phoneme-discriminative representations such as MFCCs, though F1 and F2 retain some vowel information.Source: §III · p. 3

A. Task Definition

  • For each speaker and speech model, K-means assigns 30 acoustic clusters as style labels; classifiers predict them from text embeddings and a separate test measures cluster geometry in text space.The clustering is shared across text models, so their comparisons use identical labels.Source: A. Task Definition · pp. 3–4

B. Speech Representation Models

  • Five frozen speech models are compared at relative encoder depths, with final-layer results primary and indices 1 and 3 included for depth comparisons.The set includes four self-supervised baselines, a multilingual XLSR-53 reference, and the TIMBRESPACE prosody-only control.Source: B. Speech Representation Models · p. 4, Table I · p. 4

C. Text Embedding Models

The text and classifier setup compares sentence embeddings with length and word-identity references, while controls address length structure and probe-selection optimism.

C. Text Embedding Models

  • Twelve text embedding models span similarity-trained transformers, dense retrievers, an NLI encoder, instruction-tuned embedders, and two plain masked-language-model controls.The plain encoders test whether effects require training for sentence similarity.Source: C. Text Embedding Models · p. 4

D. Classifiers and Reference Baselines

  • Six classifier families cover different geometric assumptions, and the reported weighted F1 uses the best-performing classifier per configuration.Because best-probe selection is made on held-out data, the arm comparisons are also repeated with a single classifier fixed across arms.Source: D. Classifiers and Reference Baselines · p. 4
  • The bag-of-words TF-IDF arm measures what word identity alone predicts, while the length-only arm is the floor text representations must clear.Both reference arms use the same labels and held-out utterances as the text embedding models.Source: D. Classifiers and Reference Baselines · p. 4

1) Utterance Length:

  • Speech embeddings encode utterance length strongly, so the study removes length-conditional means before clustering and retains length as a diagnostic because residual association remains.Erasure cuts the length-only baseline's lift by 40 to 70%, but residual association remains at η2 = 0.19 to 0.52.Source: 1) Utterance Length · pp. 4–5

2) Sequence Length Beyond the Trained Range:

Long sequences are windowed to reduce length sensitivity, and canonical ordering makes clustering comparisons reproducible. The analysis uses balanced per-speaker data and within-speaker, source-video-separated evaluation.

2) Sequence Length Beyond the Trained Range:

  • 43.5% of sequences exceed TIMBRESPACE’s 622-token training range, so overlapping windows with stride 311 are averaged per token to reduce length sensitivity to WavLM’s level.Short sequences remain bit-identical, and waveform models are stable over the same range (cosine ≥0.96).Source: 2) Sequence Length Beyond the Trained… · p. 5

3) Clustering Instability:

  • Row shuffling alone yields K-means partitions with adjusted Rand index 0.47–0.85, so canonical sorting, float64, ten initializations, and fixed cached partitions ensure reproducible comparisons.All twelve text models for a speaker and speech model score the same partition.Source: 3) Clustering Instability · p. 5

V. DATASETS

  • The corpus contains about 1,200 hours from six speakers, subsampled to 200 hours per speaker across diverse topics and conversational contexts.Audio is denoised before embedding, and transcript-boundary segments are filtered and given neighboring text context when needed.Source: §V · p. 5
  • Source videos are split deterministically 80/20, preventing a video from appearing on both sides while leaving recurring topics and settings in both sets.Random rather than chronological selection also avoids systematic era differences between training and test data.Source: §V · p. 5
  • Clustering, training, and evaluation are performed independently within each speaker, so the measured association reflects individual delivery variation rather than between-speaker differences.The six speakers are treated as complete experimental replications, not as a population sample.Source: §V · p. 5

VI. EVALUATION METRICS

The evaluation compares text-based cluster prediction with an imbalance-aware majority floor, measures cluster geometry in text space, and tests effects across speakers. A separate IEMOCAP validation applies the same encoding and clustering pipeline to labeled speech.

A. Classification Metrics and the Chance Level

  • The majority-top-3 rate—the share of a speaker’s three largest clusters—is the constant-predictor floor, averaging 0.255 and reaching 0.448 across configurations.Uniform top-3 chance is 0.10, but unbalanced clusters make it an inappropriate reference; top-3 accuracy and weighted F1 are reported.Source: A. Classification Metrics and the… · p. 5

B. Geometric Coherence Metrics

  • Cosine silhouette and the ratio of mean inter- to intra-cluster distance assess whether acoustic clusters are compact in text embedding space; comparisons are only within each text model.Raw coherence values are not comparable across models because their cosine similarity scales differ.Source: B. Geometric Coherence Metrics · p. 5

C. Statistical Testing

  • Speakers are the replication units, with cells averaged within speaker before across-speaker tests; utterance-level binomial and two-proportion tests are nominal because within-video utterances are correlated.The protocol also reports the mean configuration and the worst configuration per speaker.Source: C. Statistical Testing · p. 6

VII. VALIDATION ON LABELED SPEECH

  • IEMOCAP validation repeats encoding and clustering with the same pipeline on speaker-disjoint training and test sessions, including two unseen test speakers.The reported probe performance is best weighted F1 over the classifier set on held-out Session 5; its identity axis combines the two unseen speakers, who differ in gender.Source: §VII · p. 6, B. Geometric Coherence Metrics · p. 6

A. Emotion Is Decodable from All Four SSL Encoders

Frozen speech representations encode emotion above the majority floor, but identity structure is substantially stronger at the first layer. TIMBRESPACE is less effective at supervised categorical probing yet shows the strongest unsupervised organization along paralinguistic axes.

A. Emotion Is Decodable from All Four SSL Encoders

  • 0.53–0.59 weighted F1 on four-way emotion exceeds the 0.197 majority-class floor for all four SSL encoders.The results are a conservative lower bound on style information because they use frozen single layers with utterance length erased; commonly reported IEMOCAP accuracy above 70% uses different conditions.Source: A. Emotion Is Decodable from All Four… · p. 6

B. Identity Dominates the Embedding Geometry

  • At the first layer, identity η2 is 0.71–0.95 versus at most 0.12 for emotion, and unsupervised K-means purity against identity reaches 0.81–0.91.This supports the within-speaker protocol: pooling speakers before clustering yields identity partitions rather than style partitions.Source: B. Identity Dominates the Embedding… · p. 6

C. The Non-Lexical Control Behaves as Required

  • At the final layer, TIMBRESPACE leads all label-free metrics on both axes, with mean η2 at 69 times the null and maximum η2 of 0.248; it also separates unseen speakers at F1 = 0.83.Despite being weakest under supervised probing, its non-lexical acoustic space is the most strongly organized under unsupervised measurement; cluster–emotion χ2 = 506 (p < 10^-40).Source: C. The Non-Lexical Control Behaves as… · pp. 6–7

D. Layer Choice Changes Geometry More Than Decodability

Text embeddings predict acoustically derived clusters above the majority-top-3 floor for all six speakers, but most of the association is already captured by word identity. Layer choice changes representation geometry much more than supervised decodability.

D. Layer Choice Changes Geometry More Than Decodability

  • Between first and final layers, SSL probe performance stays within 0.05 F1 while identity η̄² falls 6–25 times and cluster–identity ARI falls 4–20 times.The final layer is less dominated by voice identity, but labeled-speech results do not settle which depth to use for clustering.Source: D. Layer Choice Changes Geometry More… · p. 7

VIII. RESULTS

  • The final-layer XLSR-53 partition is degenerate, and its imbalance-adjusted floor prevents an apparent margin over uniform chance from implying text predictability.XLSR-53 clears its floor significantly in only 10 of 72 cells, versus 50–72 of 72 for the other four speech models.Source: §VIII · p. 7, A. Does Text Predict the Acoustic… · p. 7

A. Does Text Predict the Acoustic Cluster at All?

  • +0.111 top-3 accuracy above the majority-top-3 floor: text embeddings clear the floor for all six speakers and in 293 of 360 configurations.The embedding arm exceeds the length arm in 359 of 360 cells.Source: A. Does Text Predict the Acoustic… · p. 7

B. What Carries the Association?

  • TF-IDF achieves 0.340 top-3 accuracy, a +0.085 lift over the floor, while sentence embeddings add only +0.026 on average.Embedding accuracy tracks bag-of-words accuracy closely (Spearman ρ = 0.92); for speaker F3, TF-IDF reaches 0.484 versus 0.503 for embeddings.Source: B. What Carries the Association? · pp. 7–8
  • Under tree ensembles, nine similarity-trained and instruction-tuned encoders lose to TF-IDF for every speaker; only the two MLM encoders retain an advantage.Their advantages are +0.014 and +0.010, respectively; the text-model comparison uses fixed classifiers across embedding and TF-IDF arms.Source: B. What Carries the Association? · p. 8, Table V · p. 8

C. Does Better Semantics Help?

Sentence embeddings add little over word identity, and the strongest increment comes from encoders not trained for sentence semantics. A prosody-only representation weakens text predictability for most speakers, while the strongest-associated speaker remains an exception.

C. Does Better Semantics Help?

  • roberta-base (+0.063) and distilbert-base-uncased (+0.051) lead all other encoders, while the largest instruction-tuned embedders rank last (+0.008 and +0.004).The encoder ordering is significant; the proposed residual signal lies in surface form rather than sentence semantics.Source: C. Does Better Semantics Help? · p. 8

D. Speech Models and the Prosody-Only Control

  • TIMBRESPACE reaches 0.274 top-3, a +0.070 lift over its floor, below each working waveform encoder for every speaker.Its increment over TF-IDF is +0.010, versus +0.026 to +0.035 for waveform encoders; differences in those increments are not significant across speech models.Source: D. Speech Models and the Prosody-Only… · p. 8
  • XLSR-53’s final-layer clusters are tightly separated but text-unpredictable, showing that cluster compactness and text predictability are distinct properties.Its final layer is nearly low-rank, with ten principal components carrying 92% of variance.Source: D. Speech Models and the Prosody-Only… · pp. 8–9
  • For five speakers, TIMBRESPACE retains at most about a third of waveform-encoder lift, but F3 retains 81% (+0.273 versus +0.338).F3’s bag-of-words prediction is +0.262 above the floor; its prosody-only association therefore cannot be explained by lexical leakage through the speech encoder alone.Source: D. Speech Models and the Prosody-Only… · p. 9
  • Embedding width correlates with lift over the floor, but the compactness account is incomplete because TIMBRESPACE aligns best with emotion and speaker on labeled speech.The correlation is ρ = 0.38 speaker-centered (0.45 without XLSR-53), compared with 0.77 at the first layer.Source: D. Speech Models and the Prosody-Only… · p. 9

E. Cluster Geometry

Acoustic clusters are not compact in text embedding space, even under a favorable subsample. Separation ratio, unlike silhouette, tracks text predictability across configurations.

E. Cluster Geometry

  • Across all 360 configurations, text-space cosine silhouette is negative (−0.230 to −0.025), and the mean separation ratio is 0.253.The ratio exceeds one in only 6 of 360 configurations, even when measured on up to 100 utterances nearest each cluster centroid.Source: E. Cluster Geometry · p. 9
  • Separation ratio correlates with predictability (ρ = 0.61; 0.58 after within-speaker centering), whereas silhouette is inversely related (ρ = −0.34).The four non-degenerate encoders also have weak acoustic-space mean silhouettes, −0.08 to −0.05.Source: E. Cluster Geometry · p. 9

F. Sensitivity to Encoder Depth

Across encoder depths, text predicts acoustic delivery clusters above the majority floor, but the association is small and largely lexical. The results do not distinguish whether vocabulary cues delivery or marks topic and recording situation.

F. Sensitivity to Encoder Depth

  • Excluding collapsed XLSR-53, the final layer yields the largest lift over the unchanged floor (Friedman p=.030) and the largest weighted F1 for all six speakers (p=.031 against either earlier depth).Per encoder, the final layer has the largest lift for five of six speakers in each of HuBERT-L, HuBERT-XL, WavLM-L and TIMBRESPACE; XLSR-53 peaks at index 3 and collapses at the final layer. Speaker share of variance in top-3 accuracy falls from 0.87 to 0.52, while weakest-speaker lift rises from +0.016 to +0.074.Source: F. Sensitivity to Encoder Depth · p. 10

G. Verdict on the Null

  • All six speakers clear the majority floor (p = .016), but three quarters of the association comes from word identity and sentence-level meaning adds little.The embedding arm’s advantage over bag-of-words is positive for all six speakers, but holds for similarity-trained encoders only under linear probes.Source: G. Verdict on the Null · p. 10

A. A Lexically Mediated Association

  • F3’s clusters are almost as text-predictable from prosody-only features as from waveform encoders, consistent with recording-context effects; genuine lexical delivery cues remain an alternative.The proposed tests distinguish content words that track topic from function words and punctuation that may cue delivery.Source: A. A Lexically Mediated Association · pp. 10–11
  • The findings do not establish that prosody is unpredictable from text: they concern unsupervised 30-cluster partitions of mean-pooled, within-speaker utterance embeddings.Whether the lexical association reflects delivery cues or topic and recording situation remains unresolved.Source: A. A Lexically Mediated Association · p. 11

B. The Controls Are the Finding

With length erasure, a majority-class floor, and a bag-of-words reference, the paper finds a weak, mostly lexical association between text and acoustic clusters. Its scope and possible practical use remain bounded by the data and by uncertainty over whether vocabulary signals delivery or topic.

B. The Controls Are the Finding

  • Length erasure, a majority-class floor, and bag-of-words separate duration and imbalance artifacts from lexical association; without these controls, the 0.111 lift would be attributed to semantics.Erasing duration is conservative because duration can be both a confound and a genuine paralinguistic cue.Source: B. The Controls Are the Finding · p. 11

C. Implications for Reference-Free TTS

  • A text-matching retrieval would reach roughly 0.37 top-3 against a 0.26 constant-predictor floor, mostly through vocabulary that may track topic rather than intended style.This bounds rather than contradicts text-to-style systems whose style representations are co-trained with synthesis objectives.Source: C. Implications for Reference-Free TTS · p. 11

D. Limitations

  • Evidence covers English conversational YouTube speech from six speakers; read speech, other languages, and population-level inference remain outside the study’s scope.Labels are unsupervised acoustic partitions rather than perceptually validated speaking styles.Source: D. Limitations · p. 11

X. CONCLUSION

  • Text predicts acoustic clusters for all six speakers (+0.111 top-3), but sentence embeddings add only +0.026 and clusters are never compact in text space.The conclusions hold across encoder depths; speaker identity accounts for over half the variance in the headline metric.Source: §X · pp. 11–12
  • Reference-free style selection cannot assume more than a mostly word-choice link, whose words may cue delivery or mark topic and recording situation.Separating content words from delivery cues, relating clusters to source video, and perceptually validating the clusters are proposed next steps.Source: §X · p. 12
Loading 2610.05575v1…