PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Does Timbre Hide? Localizing Speaker Information in Speech Signals

Yiquan Zhou, Yixuan Wang, Xiaozhe Qi, Xin Gao, Shihao Li

arXiv:2610.29150v1cs.SDcs.AI

TL;DR

The paper asks where speaker-related timbre information resides and how it remains coupled to content, a question relevant to systems that must control target voice while preserving linguistic content. It probes speech signals and learned representations using speaker identity as a measurable proxy. The results locate strong perceived information in low/mid-frequency magnitude structure while also finding accessible information in shallow SSL layers and some token streams.

  • Problem

    The paper investigates where speaker-related timbre information resides and how it is coupled with content, since generative speech systems must preserve linguistic content while controlling target voice.

    Source: §I · p. 1

  • Method

    The study uses speaker identity as a measurable proxy and combines signal manipulations, representation probes, and human similarity ratings.

    Source: §I · p. 1, A. Datasets and Common Setup · p. 2

  • Results

    Speaker-related timbre is strongest in low/mid-frequency magnitude structure, while shallow SSL layers and some token streams retain accessible speaker information.

    Source: §I · p. 1, §IV · p. 6, §V · p. 6

  • Takeaways & Limitations

    Content, codec, and token pipelines should be checked for residual speaker leakage, and evaluations should distinguish machine discriminability from perceived speaker similarity.

    Source: §IV · p. 6

    Many probes are closed-set, and WavLM-base-sv’s 16 kHz input limits SV-based frequency analysis above 8 kHz.

    Source: §V · p. 6

Abstract

from arXiv · show

Timbre similarity is a key evaluation target for generative speech systems, including voice conversion, speaker-cloning text-to-speech, and zero-shot speech models. When the generated voice does not match the target speaker, the problem often reflects imperfect separation of speaker-related timbre from content, making it important to identify where timbre information resides and which representations still carry it. Using speaker identity as a measurable proxy, we probe frequency bands, phase/magnitude structure, WORLD source/filter decomposition, voiced/unvoiced regions, self-supervised learning (SSL) layers, EnCodec/HuBERT tokens, and a 45-listener test across Mandarin, English, and Japanese speech/singing data. Speaker information is distributed: 1--8 kHz magnitude and vocal-tract spectral-envelope structure dominate perception, above-8 kHz bands remain machine-discriminative, shallow SSL layers expose compact speaker information, and token leakage depends on tokenizer/layer choice. These findings indicate where generation systems should condition on or transform speaker information, and where content, codec, and token pipelines should audit residual speaker leakage.

I. INTRODUCTION

The paper uses speaker identity as a measurable proxy to locate speaker-related timbre information across speech signals and learned representations. Its probes distinguish machine-detectable speaker information from perceived similarity and show that signal regions and representations differ in what they preserve.

I. INTRODUCTION

  • The motivating generation systems include voice conversion, speaker-cloning TTS, and zero-shot speech models, which must preserve linguistic content while controlling target voice.Speaker-related information remaining in content, codec, or SSL representations can leak into generation.Source: §I · p. 1
  • Speaker identity serves as a measurable proxy because timbre is perceptual and difficult to measure directly at scale.The paper keeps machine discriminability separate from human similarity judgments.Source: §I · p. 1
  • The study combines frequency filtering, signal-structure edits, representation probes, and a 45-listener similarity test to examine speaker information from multiple perspectives.The probes include phase/magnitude and WORLD source/filter manipulations, SSL-layer and token analysis, and robustness checks.Source: §I · p. 1
  • Speaker information is concentrated in 1–8 kHz magnitude structure, especially the vocal-tract spectral envelope, while phase, F0, and high-frequency-only information are weaker under the probes.Classifier-usable high-frequency information does not yield strong same-speaker similarity for listeners.Source: §I · p. 1
  • Shallow SSL layers and some token streams retain accessible speaker information, so content and codec representations should be checked for leakage.The amount of token leakage depends on tokenizer and layer choice.Source: §I · p. 1

II. EXPERIMENTAL SETUP

The experimental setup uses multiple datasets to meet the differing needs of its probes. Speaker identity is the common target, while measurement depends on the probe and its input constraints.

A. Datasets and Common Setup

  • Datasets span Mandarin, English, emotion, singing, and controlled Chinese–English speech because probes require different sampling rates, languages, and style labels.AISHELL, VCTK, ESD, CREMA-D, NUS-48E, JVS/JVS-MuSiC, and a self-recorded corpus serve different experiment groups.Source: A. Datasets and Common Setup · p. 2
  • All experiments measure speaker identity, using probe-specific metrics including SV scores, closed-set classification accuracy, and listener similarity ratings.Listener ratings use a 1–5 scale; discriminability is the same-speaker score minus the different-speaker score.Source: A. Datasets and Common Setup · p. 2
  • WavLM-base-sv is the main SV backend, with ECAPA-TDNN used to repeat phase/magnitude and source/filter tests for backend robustness.WavLM-base-sv requires 16 kHz input, so spectral content above 8 kHz is discarded before embedding extraction.Source: A. Datasets and Common Setup · p. 2

B. Experiment Design

The experiments manipulate signals and probe learned representations to test where speaker information remains accessible. They compare signal structure, frequency regions, speech features, discrete tokens, and robustness to style and phonation changes.

B. Experiment Design

  • Frequency filtering tests which spectral regions preserve speaker information, using SV curves below 8 kHz and high-sample-rate mel probes and listening trials for broader bands.Mel-cosine separability provides an additional classifier-free check.Source: B. Experiment Design · p. 2
  • Phase/magnitude edits compare randomized-phase signals retaining magnitude with randomized-magnitude signals retaining phase against unmodified speech.This manipulation tests whether speaker information is carried mainly by spectral magnitude or phase.Source: B. Experiment Design · p. 2
  • WORLD source/filter swaps retain one speaker’s F0 or spectral envelope while replacing the other component, then compare similarity retention.The experiment tests whether speaker information is more closely linked to vocal-tract filtering or pitch alone.Source: B. Experiment Design · p. 2
  • Layer-wise features from WavLM, HuBERT, Wav2Vec 2.0, and Whisper are classified with and without PCA to test speaker information and its compactness.The main metric is closed-set speaker-classification accuracy.Source: B. Experiment Design · pp. 2–3
  • EnCodec RVQ and HuBERT token histograms, including HuBERT bigrams, test whether discrete-unit speaker leakage varies with tokenizer, layer, codebook size, and RVQ depth.The main metric is closed-set speaker-classification accuracy.Source: B. Experiment Design · p. 3
  • Emotion, speech–singing transfer, and voiced/unvoiced tests examine whether speaker information survives changes in style and phonation.The checks use cross-condition similarity gaps and speaker-classification accuracy.Source: B. Experiment Design · p. 3

III. RESULTS

Frequency-band probes find strongest speaker information in low/mid frequencies, while measurable information persists at higher frequencies. Listener ratings distinguish that classifier-usable information from perceived same-speaker similarity.

A. Frequency-Band Experiment Results

  • WavLM lowpass discriminability nearly reaches its full-band score by 7.5 kHz, while performance drops sharply when highpass cutoffs exceed 1–2 kHz.Together, the curves indicate information across a broader low/mid-frequency range rather than sub-1 kHz content alone.Source: A. Frequency-Band Experiment Results · p. 3
  • 92.1% VCTK speaker-classification accuracy at 1–4 kHz and 82.5% AISHELL-3 accuracy at 1–4 kHz show strong low/mid-band information; above-8 kHz bands remain measurable but weaker.The mel MLP checks show accuracy decreasing with frequency, with scores above shuffled-label and duration/energy baselines.Source: A. Frequency-Band Experiment Results · p. 3, Table II · p. 4
  • d′ = 1.86 at 4–8 kHz for AISHELL-3 mel-cosine separability declines toward d′ = 0.97 in the highest tested band.This classifier-free check follows the same frequency trend as the mel MLP probes.Source: A. Frequency-Band Experiment Results · p. 4
  • 2.57 points is the mean paired listener-rating difference between low/mid-frequency same-speaker conditions and high-frequency-only conditions.The < 8 kHz versus > 8 kHz contrast is 4.45 vs. 1.24 (p = 1.3 × 10−39); high-band classifier information does not provide strong listener-perceived similarity under these manipulations.Source: Table III · p. 4, A. Frequency-Band Experiment Results · p. 4, Figure 3 · p. 4

B. Phase/Magnitude Experiment Results

Magnitude retains substantially more speaker information than phase in the phase/magnitude experiment.

B. Phase/Magnitude Experiment Results

  • 81.6% of original discriminability is preserved by keeping magnitude and randomizing phase, while phase-kept discriminability falls to -0.199 with WavLM and 0.003 with ECAPA.Both phase-kept values are effectively at or below chance.Source: B. Phase/Magnitude Experiment Results · p. 4

C. Source/Filter Experiment Results

Across the source/filter tests, the vocal-tract spectral envelope preserves more speaker information than F0 alone.

C. Source/Filter Experiment Results

  • WavLM and ECAPA both favor magnitude/filter information over phase/F0-dominant information in the second-backend validation.Source: Figure 4 · p. 5
  • 92.4% of full-signal same-speaker similarity is retained with WavLM when keeping the spectral envelope, versus 82.0% with only F0; ECAPA retains 79.3% versus 26.2%.The table summarizes phase/magnitude and source/filter results across two speaker-verification backends.Source: C. Source/Filter Experiment Results · p. 5, Table IV · p. 5

D. SSL Representation Experiment Results

Speaker information is strongest in early-to-middle SSL layers and declines with depth or reduced dimensionality; token results vary by tokenizer and layer.

D. SSL Representation Experiment Results

E. Token Experiment Results

  • HuBERT k-means tokens preserve more speaker information than the tested 24 kHz EnCodec RVQ histograms under the order-free histogram probe, but this is not a universal codec ranking.The amount of retained information depends strongly on tokenizer and layer choice.Source: E. Token Experiment Results · p. 5, Table VI · p. 5

F. Robustness Experiment Results

Speaker information remains detectable across emotion, singing, and unvoiced speech, although it weakens with style or phonation changes.

F. Robustness Experiment Results

  • 23.7% accuracy transfers from 10,000 read-speech utterances by 100 speakers to 600 singing segments, above 1.0% chance but below within-mode accuracy.Other checks found 89.2% mean accuracy across held-out emotions and 70.9% accuracy using unvoiced-only active frames.Source: F. Robustness Experiment Results · p. 5

IV. DISCUSSION

Perceived speaker timbre is concentrated in low- and mid-frequency magnitude structure, particularly the vocal-tract spectral envelope, while machine-discriminative high bands do not yield strong listener similarity ratings. Speaker information also persists in learned representations and across style and phonation changes, motivating targeted audits and speaker-aware control.

IV. DISCUSSION

  • Low- and mid-frequency magnitude structure, especially the 1–8 kHz vocal-tract spectral envelope, carries the strongest perceived speaker-related timbre, rather than phase or F0 alone.WORLD swaps validated with a second speaker-verification backend also point to the vocal-tract spectral envelope; its filtering applies to unvoiced segments as well.Source: §IV · pp. 5–6
  • Above-8 kHz bands remain useful to closed-set classifiers and mel-cosine checks, but high-frequency-only same-speaker pairs receive low listener ratings.High-band reconstruction may contribute acoustic detail such as frication or airiness, but is unlikely to restore speaker similarity under these conditions.Source: §IV · p. 6
  • Shallow SSL layers retain more compact speaker information than deeper layers, while HuBERT and EnCodec tokens are not automatically speaker-free.Token leakage varies with source layer, codebook size, and residual vector quantization layer.Source: §IV · p. 6
  • Speaker information remains measurable across emotion, speech–singing changes, and unvoiced segments, although larger domain changes—especially speech-to-singing—reduce its strength.The measured information remains above chance.Source: §IV · p. 6
  • Voice conversion and speaker-cloning TTS should control low/mid spectral-envelope information, while anonymization and tokenized-speech pipelines should audit leakage in shallow SSL features, codec layers, and token streams.Evaluation should distinguish machine speaker discriminability, perceived similarity, and representation-level leakage.Source: §IV · p. 6

V. LIMITATIONS AND CONCLUSION

The paper concludes that perceived speaker-related timbre is strongest in low/mid-frequency spectral-magnitude structure, while learned representations can retain recoverable speaker information. Its conclusions are bounded by closed-set probes, backend and sampling-rate constraints, and less natural narrowband listening stimuli.

V. LIMITATIONS AND CONCLUSION

  • The study's probes are often closed-set, so speaker-classification accuracy and SV-style scores should not be read as open-set speaker-verification performance.WavLM-base-sv operates at 16 kHz, limiting SV-based frequency analysis above 8 kHz; other high-sample-rate probes and listening tests reduce but do not replace this limitation.Source: §V · p. 6
  • High-frequency listening stimuli were narrowband and less natural than full-band speech, so their low ratings describe perceived voice-timbre similarity under the study's filtering conditions.Emotion, singing, and token findings would benefit from more speaker backends, richer listener metadata, better-controlled playback, and sequence-aware token probes.Source: §V · p. 6
  • The strongest perceived speaker-related information is concentrated in low/mid-frequency spectral-magnitude structure, specifically the 1–8 kHz vocal-tract spectral envelope, while phase, F0, and high-frequency-only information is weaker under the probes.Shallow SSL layers and some token streams also expose recoverable speaker information, and classifier-discriminative high-frequency bands do not produce strong same-speaker listening ratings.Source: §V · p. 6
Loading 2610.29150v1…