Source-linked AI summary
Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry
Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee
TL;DR
Speaker-verification accuracy is commonly used to estimate human voice similarity, although whether it captures fine-grained human judgments remains unclear. The paper compares verification and perceptual alignment, studies embedding geometry, and manipulates embedding dimensions; EER does not track alignment, while effective dimensionality does.
Problem
Whether speaker-verification performance reflects fine-grained human perceptual similarity remains largely unexplored, despite its use to evaluate voice similarity in speech generation.
Source: §1 · p. 1Method
The study compares EER and human perceptual alignment across model conditions and training objectives, analyzes embedding geometry, and manipulates embedding dimensions.
Source: §1 · p. 1Results
EER’s Spearman correlation with perceptual alignment is only +0.07, while effective dimensionality’s correlation is −0.95; bottlenecking raises alignment correlation from 0.08 to 0.74.
Source: §1 · p. 1Takeaways & Limitations
Effective dimensionality is identified as an alignment indicator that requires no human annotation.
Source: §6 · p. 4Alignment evaluation uses different-speaker pairs only, since same-speaker pairs can add a strong identity signal and inflate correlation.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
Speaker verification (SV) models are commonly assumed to better capture nuances among speaker characteristics as verification accuracy improves, leading to their widespread use as automated proxies for human voice similarity in speech generation tasks. However, by establishing a human perceptual alignment metric and conducting systematic analysis, we demonstrate that perceptual alignment is governed far more by how a model is trained (its learning objective) than by how well it performs (EER). Notably, standard margin-based classification losses (e.g., AAM-Softmax) yield substantially lower perceptual alignment than prototypical metric losses, while EER itself fails to track human judgment, directly challenging the community's implicit assumption. We trace this divergence to embedding geometry, where a model's effective dimensionality ($d_{\mathrm{eff}}$) tracks perceptual alignment with a $-0.95$ rank correlation, revealing that the dimensional spread favored by classification losses fundamentally clashes with the low-dimensional nature of human voice perception. Imposing a dimensionality bottleneck compresses $d_{\mathrm{eff}}$ and raises perceptual alignment ($ρ_{\mathrm{align}}$) from 0.08 to 0.74, establishing a principled geometric criterion for evaluating voice similarity.
1. INTRODUCTION
The paper asks whether speaker-verification performance reflects fine-grained human voice similarity and what embedding property tracks that alignment. It systematically compares model conditions and training objectives, analyzes embedding geometry, and manipulates embedding dimensions.
1. INTRODUCTION
- The study evaluates alignment across model conditions and objectives, examines embedding geometry, and manipulates that geometry to test its relationship with human perception.The questions are whether better EER predicts higher perceptual alignment and which embedding property tracks alignment.Source: §1 · p. 1
- EER’s Spearman correlation with perceptual alignment is only +0.07, so EER is not a reliable proxy for human judgment.Source: §1 · p. 1
- The best and worst training objectives differ by more than a factor of three in human perceptual alignment in every model condition, without an EER cost.This makes training objective a stronger differentiator of alignment than EER in the reported comparisons.Source: §1 · p. 1
- Effective dimensionality has a −0.95 Spearman correlation with perceptual alignment.The study tests effective dimensionality as a measure of how many directions a model spreads its speakers over.Source: §1 · p. 1
- Narrowing AM-Softmax embeddings on ECAPA-TDNN lowers effective dimensionality and raises perceptual alignment correlation from 0.08 to 0.74.AM-Softmax was the worst-aligned objective in the experiment.Source: §1 · p. 1
2. RELATED WORK
Prior work established speaker embeddings as tools for verification and similarity scoring, but whether verification performance predicts human perceptual similarity remains empirically unexamined. Training objectives shape embedding similarities, and dimensionality is a potential geometric link to human judgments whose direct relationship remains insufficiently understood.
2.1. Speaker Embedding Models, Speaker Verification and Human Perceptual Similarity
- The assumption that the best-performing verification model is the best proxy for human perception remains empirically unexamined.Earlier work compared model scores with listener ratings, while later work focused on selecting the best model proxy.Source: §2.1 · p. 2
2.2. Speaker Embedding Training Objectives
- Classification objectives compare normalized embeddings with margin-constrained class centers, whereas metric objectives compare embeddings with one another or batch centroids.How these objective families affect alignment with human perceptual similarity remains unexplored.Source: §2.2 · p. 2
2.3. Representation Geometry and Dimensionality
- Effective dimensionality may link speaker embeddings to human similarity judgments, which prior perceptual studies show can be captured in low-dimensional spaces.The direct connection between representation dimensionality and perceptual alignment remains insufficiently understood.Source: §2.3 · p. 2
3. METHOD
The method measures perceptual alignment by correlating embedding cosine similarities with listener ratings for different-speaker VoxSim pairs. It also quantifies the spread of speaker embeddings using effective dimensionality.
3.1. Scoring Perceptual Alignment
- Perceptual alignment is the Spearman rank correlation between embedding cosine similarities and mean listener ratings across different-speaker VoxSim pairs.The evaluation excludes same-speaker pairs because their binary identity signal could inflate correlation without measuring graded similarity among different speakers.Source: §3.1 · p. 2
3.2. Effective dimensionality
- Effective dimensionality measures the intrinsic geometric spread of the embedding space and is computed as the participation ratio of the PCA eigenvalue spectrum of speaker centroids.It is the equivalent number of equal-variance orthogonal directions producing the same covariation pattern; larger values indicate a more uniform speaker space.Source: §3.2 · p. 2
4. EXPERIMENTAL SETUP
The experiments vary speaker-embedding models and training objectives, then evaluate verification and human perceptual alignment on separate speaker-disjoint test sets.
4.1. Models and training
- Five model conditions vary architecture family, input representation, and embedding width; objective comparisons use seven losses and three seeds per condition.The conditions include three log-mel models and two WavLM-based stages; dimension bottlenecks are separately tested on ECAPA-TDNN at nout values 512, 32, 10, and 3.Source: §4.1 · pp. 2–3
4.3. Evaluation data
- Restricting VoxSim to 16,905 different-speaker pairs removes an identity-only shortcut that already reaches SRCC 0.715 across all pairs.Centroids suppress utterance noise to isolate between-speaker geometry, and the PR metric is less sensitive to long-tail noise eigenvalues.Source: §4.1 · p. 2
4.2. Training objectives
- Seven objectives span classification losses (NSL, AM-Softmax, AAM-Softmax) and metric-learning losses (Prototypical, Angular Prototypical, GE2E, SupCon).Classification uses 200-utterance batches, while metric learning uses batches up to 400 speakers × 2 utterances.Source: §4.2 · p. 3
4.3. Evaluation data
- Evaluation uses VoxCeleb1-O for EER and VoxSim listener ratings for perceptual alignment, with no test speakers shared with training.VoxSim includes 27,697 unique speaker pairs rated by 13 listeners on a 1–6 similarity scale; EER uses raw cosine similarity on full-recording embeddings.Source: §4.3 · p. 3
5. RESULTS AND ANALYSIS
The results show that verification error does not reliably track human perceptual alignment, while training objectives and embedding geometry do. Constraining embedding dimension raises alignment, with a verification-performance trade-off at the narrowest widths.
5.1. Lower EER Does Not Imply Higher Perceptual Alignment
- Spearman(EER, ρalign) is only +0.07 across 35 training conditions, with unstable within-condition correlations and public checkpoints showing low EER but poor alignment.Fig. 2 compares ECAPA runs and four public checkpoints using EER on VoxSim pairs.Source: §5.1 · p. 3, Figure 2 · p. 4
- Prototypical losses reach ρalign 0.395–0.499 versus 0.080–0.298 for classification losses; replacing AAM-Softmax with Angular Prototypical raises ECAPA ρalign from 0.111 ± 0.013 to 0.406 ± 0.007 at tied-best EER 1.47%.Across every model condition, the best and worst objectives differ in alignment by more than a factor of three without an EER cost.Source: §5.1 · p. 3
5.2. Effective Dimensionality tracks Perceptual Alignment
- Across 35 training conditions, deff and ρalign correlate at −0.95; within model conditions, correlations range from −0.71 to −1.00, significant in four of five.For ReDimNet-B2, the permutation-test p-value is .088; Fig. 3 shows the ECAPA runs.Source: §5.2 · p. 4, Figure 3 · p. 4
5.3. Dimension Bottlenecks Can Further Improve Perceptual Alignment
- Widening ECAPA from 192 to 512 dimensions changes EER, ρalign, and deff little, but constraining dimension below intrinsic dimensionality compresses deff and raises alignment for all four objectives.The nominal embedding dimension is only an upper bound; unused directions do not alter geometry.Source: §5.3 · p. 4
- At nout=3, AM-Softmax reaches ρalign 0.738 with deff=1.5, but EER rises to 13–37%; AAM-Softmax fails to converge in two of three seeds.Table 3 reports the dimension bottleneck results, including collapsed AAM-Softmax seeds with deff ≈1.1 and EER > 40%.Source: §5.3 · p. 4, Table 3 · p. 4
6. CONCLUSION
The conclusion argues that embedding geometry, especially effective dimensionality, is more informative about human perceptual similarity than verification performance. Bottlenecking embedding dimension improves perceptual alignment but reduces verification accuracy.
6. CONCLUSION
- Effective dimensionality is identified as an indicator of perceptual alignment, with embeddings spread across fewer directions mirroring the low-dimensional structure of human voice perception.The paper presents deff as a geometric criterion that requires no human annotation.Source: §6 · p. 4
- Bottlenecking embedding dimension compresses deff and improves perceptual alignment, but directly reduces verification accuracy.Source: §6 · p. 4
- 20.5% EER is reported as comparable to human verification rates of 17.7% on VoxSim and 15.8–26.5% on VoxCeleb1.Source: §6 · p. 4