PaperlayerView exact revision on arXiv

Source-linked AI summary

Serialized Multi-Layer Multi-Head Attention for Neural Speaker Embedding

Hongning Zhu, Kong Aik Lee, Haizhou Li

arXiv:2107.06493v1cs.SDcs.CLeess.AS

TL;DR

Prior speaker-embedding methods aggregate frame features through a single pooling layer, motivating a deeper attention architecture. The paper proposes serialized multi-layer multi-head attention with an input-aware query and reports improvements over baseline methods on VoxCeleb1 and SITW.

  • Problem

    Prior attention-based speaker-embedding methods use only one pooling layer to aggregate frame-level features into a fixed-length vector.

    Source: §1 · p. 1

  • Method

    The method serially stacks self-attention modules that propagate attentive statistics between layers and uses an input-aware query for each utterance.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3

  • Results

    9.7% in EER and 8.1% in DCF10^-2 improvement over other baseline methods is reported on VoxCeleb1 and SITW.

    Source: §6 · p. 4

  • Takeaways & Limitations

    Increasing the number of stacked layers yields consistent improvement in the evaluated speaker-embedding approach.

    Source: §6 · p. 4

Abstract

from arXiv · show

This paper proposes a serialized multi-layer multi-head attention for neural speaker embedding in text-independent speaker verification. In prior works, frame-level features from one layer are aggregated to form an utterance-level representation. Inspired by the Transformer network, our proposed method utilizes the hierarchical architecture of stacked self-attention mechanisms to derive refined features that are more correlated with speakers. Serialized attention mechanism contains a stack of self-attention modules to create fixed-dimensional representations of speakers. Instead of utilizing multi-head attention in parallel, the proposed serialized multi-layer multi-head attention is designed to aggregate and propagate attentive statistics from one layer to the next in a serialized manner. In addition, we employ an input-aware query for each utterance with the statistics pooling. With more layers stacked, the neural network can learn more discriminative speaker embeddings. Experiment results on VoxCeleb1 dataset and SITW dataset show that our proposed method outperforms other baseline methods, including x-vectors and other x-vectors + conventional attentive pooling approaches by 9.7% in EER and 8.1% in DCF0.01.

1. Introduction

The paper addresses fixed-dimensional speaker representations for text-independent verification, where prior attention methods still aggregate frame features through a single pooling layer. It proposes serialized multi-layer multi-head attention and reports effectiveness over conventional attentive pooling on VoxCeleb1 and SITW.

1. Introduction

  • DNN speaker embeddings provide fixed-dimensional speaker representations and outperform i-vectors in speaker verification.The task compares a test utterance's speaker identity with reference speech.Source: §1 · p. 1
  • Prior attention methods address unequal frame importance, but still use one pooling layer to form a fixed-length utterance vector.Plain statistics pooling assigns equal weights to frame features, overlooking critical frames.Source: §1 · p. 1
  • The proposed serialized multi-layer multi-head attention propagates attentive statistics across layers, using an input-aware query and temporal context to form speaker embeddings.The weighted statistics from different layers are fused into the next layer's features and aggregated into the utterance-level embedding.Source: §1 · p. 1
  • Experiments on VoxCeleb1 and SITW show effectiveness over conventional attentive pooling approaches.Source: §1 · p. 1

2. Attention in Neural Speaker Embedding

The section reviews frame aggregation for speaker embeddings, from statistics pooling to attentive statistics pooling and self-attentive pooling.

2. Attention in Neural Speaker Embedding

  • Neural speaker embeddings are fixed-dimensional DNN representations; x-vectors aggregate frame features, while attention weights express frame importance.Source: §2 · pp. 1–2

2.1. Statistics pooling

  • Statistics pooling computes temporal mean and standard deviation of frame vectors, concatenates them, then maps the representation to a speaker embedding.Source: §2.1 · p. 2

2.2. Attentive statistics pooling

  • Attentive statistics pooling uses normalized frame scores as weights to compute weighted mean and standard deviation, rather than uniformly weighting time steps.Source: §2.2 · p. 2

2.3. Self-attentive pooling

  • Self-attentive pooling uses frame value/key vectors with a trainable, time-invariant query to produce temporal attention weights and weighted statistics.Source: §2.3 · p. 2

3. Serialized Multi-head Attention

The proposed embedding network places serialized multi-head attention between TDNN frame processing and speaker classification, accumulating utterance-level representations across layers. It derives an utterance-specific query from statistics pooling.

3. Serialized Multi-head Attention

  • The network combines a TDNN frame-level feature processor, serialized attention for fixed-dimensional aggregation, and feed-forward speaker-classification layers.Figure 1 depicts these stages in the embedding network.Source: §3 · p. 2, Figure 1 · p. 2

3.1. Serialized attention

  • Self-attention layers pass attentive information forward serially, and each layer’s weighted mean and standard deviation form a head summed into the final utterance embedding.Residual connections and layer normalization are applied around the self-attention and feed-forward sublayers.Source: §3.1 · pp. 2–3

3.2. Input-aware self-attention

  • For each utterance, statistics pooling generates an input-aware query; the input frames provide values and are linearly projected into keys for attention.The weighted mean is added to all frames after an affine transformation in the residual connection.Source: §3.2 · p. 3

3.3. Serialized multi-head embedding

  • Each self-attention layer feeds a two-transformation, ReLU feed-forward module that processes its output for the next layer.After training as a speaker classifier, the utterance-level embedding is used for speaker verification.Source: §3.3 · p. 3

4. Experiment

Experiments train on augmented VoxCeleb2 and evaluate on SITW and VoxCeleb1 test sets spanning short and long utterances. The setup compares x-vector pooling baselines with the proposed serialized-attention system.

4.1. Dataset

  • Training uses the augmented VoxCeleb2 development set, comprising 5,994 speakers and 1,092,009 utterances.Source: §4.1 · p. 3
  • Evaluation covers two SITW core-core sets and VoxCeleb1-E and VoxCeleb1-H, with utterance lengths from 4 to 180 seconds.Tables 1 and 2 report performance on SITW-dev/eval and VoxCeleb1-H/E, respectively.Source: §4.1 · p. 3, Table 1 · p. 4, Table 2 · p. 4

4.2. Experiments setup

  • The x-vector baselines use statistics pooling, attentive statistics pooling, or self-attentive pooling; acoustic inputs combine 23-dimensional MFCCs and 3-dimensional pitch.MFCCs use 25 ms frames, cepstral mean-normalization, and voice-activity detection; the input features are shared across experiments.Source: §4.2 · p. 3
  • Training chunks utterances into 200-frame sequences and uses AdamW; embeddings are reduced to 128 dimensions with LDA before length normalization and PLDA scoring.The proposed attention uses d=256, dk=128, and dff=512; its fourth TDNN layer projects frame features from 512 to 256 dimensions.Source: §4.2 · pp. 3–4

5. Results

Serialized attention improves on the reported baselines across SITW and VoxCeleb1 results, with the six-layer system strongest on VoxCeleb1. SITW results also include a DCF10−3 exception for self-attentive pooling.

5. Results

  • Performance is reported using EER and minimum DCF with target priors of 0.01 and 0.001.Source: §5 · p. 4

5.1. Results on SITW

  • Nearly 14.96% and 8.73% EER improvements over baselines are reported on SITW-dev and SITW-eval, respectively.Self-attentive pooling has the best SITW-eval DCF10−3, while serialized attention performs better in EER and DCF10−2.Source: §5.1 · p. 4

5.2. Results on VoxCeleb1

  • 11.33%, 11.76%, and 6.67% relative improvements over statistics pooling in VoxCeleb1-H, and 8.17%, 14.49%, and 13.28% in VoxCeleb1-E, correspond to EER, DCF10−2, and DCF10−3.The six-layer serialized-attention system has the best results among evaluated approaches, with fewer or comparable parameters.Source: §5.2 · p. 4
  • The four-layer serialized model outperforms single-layer attention-based pooling with 13% fewer parameters, while stacking more layers further improves VoxCeleb1 results.On SITW, N=6 gives the best performance with 1.11M more parameters than N=4.Source: §5.2 · p. 4, §5.1 · p. 4

6. Conclusion

The proposed serialized multi-layer multi-head attention aggregates weighted statistics across layers to create speaker embeddings, with consistent improvement as layers are added. On VoxCeleb1 and SITW, it outperforms other baseline methods in EER and DCF10−2.

6. Conclusion

  • 9.7% in EER and 8.1% in DCF10−2: the method outperforms other baseline methods on VoxCeleb1 and SITW.Source: §6 · p. 4
  • The method stacks self-attention modules and aggregates weighted statistics from different layers in a serialized manner for speaker verification.Source: §6 · p. 4
  • Increasing the number of stacked layers yields consistent improvement, and the input-aware query is used to capture more discriminative features.Source: §6 · p. 4
Loading 2107.06493v1…