PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpreting hierarchical organisation of speaker embeddings

Yanze Xu, Wenwu Wang, Mark D. Plumbley

arXiv:2609.15203v2eess.AScs.AI

TL;DR

Speaker-embedding organisation studies are rarely framed within XAI, and whether flat clusters contain deeper hierarchical relationships remains an open question. The paper analyses and evaluates hierarchical clusters, then interprets them with matched semantic classes; the clusters align with classes related to speaker identity, gender, and nationality.

  • Problem

    Prior studies of representation organisation are rarely framed within XAI, and whether flat clusters exhibit deeper hierarchical relationships remains open.

    Source: §1 · p. 1

  • Method

    The paper applies SLINK to analyse hierarchical clusters, evaluates their alignment with predefined classes using CCM, and uses HCCM to match clusters with individual and conjunctive classes.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    Clusters align strongly with identity-, gender-, and nationality-related classes; nationality&gender classes achieve a CCM score of 0.8316 versus 0.7710 for nationality-related individual classes.

    Source: §5 · p. 4, §4.2 · p. 4

  • Takeaways & Limitations

    HCCM interpretations associate gender-related individual classes with higher-level clusters, and identity-related and nationality&gender classes with clusters at remaining positions.

    Source: §5 · p. 4

    The evaluation assumes that a higher CCM matching degree corresponds to a better hierarchical representation organisation because clusters align better with predefined class groupings.

    Source: §3.1 · p. 2

Abstract

from arXiv · show

Speaker recognition neural networks recognise speaker identities from input utterances by learning latent representations (i.e. speaker embeddings). However, these networks' internal mechanisms remain largely opaque, motivating research in explainable artificial intelligence (XAI) to understand them. Existing studies have analysed how speaker embeddings are organised, but rarely frame these analyses within XAI. This work proposes to explain and interpret the organisation of speaker embeddings from an XAI perspective. To this end, we apply a hierarchical clustering algorithm, Single-Linkage Clustering (SLINK), to analyse whether our prepared speaker embeddings naturally form clusters with hierarchical relationships. The resulting hierarchical organisation (i.e. hierarchical clusters) is evaluated using the Cluster-Class Matching (CCM) method. Moreover, we propose a new method, termed Hierarchical Cluster-Class Matching (HCCM), to identify which hierarchical clusters best match individual semantic classes like male and conjunctive semantic classes like UK & male, thereby interpreting the clusters using their matched classes. We quantify the matching degree with a new metric called the L-score, which makes imperfect matches diagnosable. HCCM's results reveal that the hierarchical clusters analysed by SLINK are well interpreted using classes related to speaker identity, gender, and nationality, showing the semantics inside the hierarchical organisation of the examined speaker embeddings.

1. INTRODUCTION

Speaker-recognition networks map utterances to speaker embeddings, but analyses of representation organisation are rarely framed within XAI. This paper examines hierarchical relationships among embeddings and interprets the resulting clusters using semantic classes.

  • In speaker recognition, a trained network maps an utterance to a speaker embedding, a latent representation used to distinguish speaker identities; XAI seeks to make neural-network decisions more understandable.Source: §1 · p. 1
  • Prior dimensionality-reduction and flat-clustering studies leave open whether flat representation clusters contain deeper hierarchical relationships.Source: §1 · p. 1
  • The paper uses SLINK to analyse whether prepared speaker embeddings form hierarchically related clusters, then evaluates those clusters with CCM against predefined semantic classes.Source: §1 · p. 1
  • HCCM finds one-to-one matches between hierarchical clusters and individual or conjunctive semantic classes, such as male or UK & male, to interpret the clusters.Source: §1 · p. 1
  • The L-score measures each matched cluster–class pair by the weaker of precision and recall, making an imperfect match’s limiting side diagnosable.Source: §1 · p. 2
  • SLINK’s hierarchical clusters achieve a good CCM evaluation score and are interpreted using classes related to speaker identity, gender, and nationality.Source: §1 · p. 2

2. RELATED WORK

This section contrasts prior interpretability approaches and hierarchical clustering uses with the paper’s focus on semantically interpreting hierarchical speaker-embedding clusters.

  • For generation networks, latent-representation changes can be interpreted through semantic changes in outputs, but speaker-recognition representations lack directly observable generated outputs.Source: §2 · p. 2
  • In speaker diarisation, hierarchical organisations are intermediate results, with non-hierarchical cluster subsets selected for final output; they are rarely visualised or semantically interpreted.Source: §2 · p. 2
  • CCM calculates an overall matching score, whereas HCCM seeks one-to-one cluster–class matches and explicitly uses individual and conjunction-based semantic classes.Source: §2 · p. 2

3. METHODOLOGY

The methodology clusters speaker embeddings hierarchically and evaluates their alignment with predefined semantic classes. HCCM then matches clusters to individual or conjunctive classes, using the L-score to make match quality diagnosable.

3.1. Analysing and Evaluating Hierarchical Representation Organisation

  • SLINK analyses embeddings from utterances in a dataset disjoint from training, defining cluster distance by the closest representation pair and using Euclidean distance.Source: §3.1 · p. 2
  • CCM evaluates cluster alignment by aggregating the highest F-score for each predefined class; higher overall scores are assumed to indicate better alignment.Source: §3.1 · p. 2

3.2. Semantically Interpreting Hierarchical Representation Organisation

  • HCCM uses individual classes for single attributes and conjunctive classes formed by combining individual classes, such as UK & male.Source: §3.2 · p. 3
  • HCCM iteratively selects the globally best cluster–class pair from remaining classes and all clusters, then removes the matched class, yielding up to |C(0)| distinct cluster interpretations.Source: §3.2 · p. 3
  • The L-score takes the weaker of precision and recall as the match degree, making it easier to diagnose which measure limits an imperfect match.For P(c, h) = 0.9 and R(c, h) = 0.6, the L-score is 0.6 and is limited by recall.Source: §3.2 · p. 3

4. EXPERIMENTS

Experiments evaluate SLINK’s hierarchy on VoxCeleb1 embeddings using CCM, then interpret selected clusters with HCCM. The hierarchy aligns with several semantic-class types, and individual clusters receive high-scoring class matches.

4.1. Experimental Setup

  • ResNetSE34L, trained on VoxCeleb2 development-set spectrograms, processes 4-second VoxCeleb1 test utterances; evaluation uses 51 individual and 13 nonempty conjunctive classes.Source: §4.1 · p. 3

4.2. Evaluation Results of Hierarchical Representation Organisation

  • 0.8944 average CCM; identity, gender, nationality, and nationality&gender score 0.9754, 0.9997, 0.7710, and 0.8316, respectively, with conjunctions exceeding nationality.Source: §4.2 · pp. 3–4, Table 1 · p. 4

4.3. Interpretation Results of Hierarchical Representation Organisation

  • Fig. 1 shows only clusters containing more than 2,000 representations; Table 2 reports only the highest-L-score class when a cluster has multiple matches.Source: §4.3 · p. 4, Figure 1 · p. 4, Table 2 · p. 4
  • Clusters 47151/47152 match female/male at L-scores 0.9993/0.9994; 47156 matches India & male at 0.9962, while 47164 matches Eddie Izzard at 0.9704.Source: Table 2 · p. 4, §4.3 · p. 4
  • Among clusters shown in Fig. 1, none is interpreted by a nationality-only class; six are interpreted by nationality&gender conjunctions.Source: §4.3 · p. 4, Figure 1 · p. 4

5. CONCLUSION

The examined network’s SLINK hierarchy aligns strongly with identity-, gender-, and nationality-related classes. HCCM places gender matches at higher levels and identity and nationality&gender matches at remaining positions.

  • The hierarchy aligns strongly with identity-, gender-, and nationality-related classes, and nationality&gender conjunctions are more suitable than nationality-only classes for evaluating it.Source: §5 · p. 4
  • HCCM matches gender-related classes to higher-level clusters, while identity-related and nationality&gender classes match clusters at remaining positions.Source: §5 · p. 4
Loading 2609.15203v2…