PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Speaker Verification as a Threat to Speaker Anonymization

Ashi Garg, Cristina Aggazzotti, Leibny Paola García-Perera, Nicholas Andrews

arXiv:2607.19636v1eess.AS

TL;DR

Speaker anonymization is typically evaluated on isolated utterances and mainly suppresses acoustic identity, leaving open whether accumulated multimodal cues enable speaker verification. This paper evaluates multi-utterance audio, prosodic, and linguistic aggregation, finding that frame-level Audio+Text fusion lowers A–A EER from 37.59% to 22.63% at N = 15.

  • Problem

    Existing anonymization methods and evaluations focus mainly on isolated utterances and acoustic identity, despite multi-utterance communication exposing additional speaker cues.

  • Method

    The study compares utterance- and frame-level aggregation for audio-only and multimodal speaker verification using acoustic, linguistic, and prosodic information.

  • Results

    Multimodal systems outperform audio-only baselines, with frame-level Audio+Text reducing A–A EER from 37.59% to 22.63% at N = 15.

  • Takeaways & Limitations

    Anonymized speech retains exploitable linguistic, prosodic, and cross-modal speaker cues, enabling stronger privacy attacks when complementary modalities are combined.

Abstract

from arXiv · show

Most automatic speaker verification (ASV) systems operate on individual utterances, despite real-world interactions typically consisting of multiple utterances. As speech accumulates, increasingly rich speaker information becomes available through acoustic, prosodic, and linguistic cues, potentially challenging speaker anonymization methods that primarily target vocal characteristics. We investigate ASV in a multi-utterance, multimodal setting and examine whether aggregating information across anonymized speech impacts privacy. We first study audio-only aggregation across multiple anonymized utterances and observe consistent performance improvements as more speech becomes available. We then incorporate prosodic and linguistic information, showing that multimodal systems outperform unimodal approaches. Finally, we compare aggregation strategies and find that frame-level aggregation yields the lowest EERs. Even with only five anonymized utterances, combining audio and text reduces EER by over 15% relative to audio-only aggregation, demonstrating that substantial speaker-discriminative information remains accessible despite anonymization.

I. INTRODUCTION · II. RELATED WORK

The paper argues that anonymized speech may retain speaker-identifying information across multiple utterances and modalities. It therefore investigates multi-utterance acoustic, prosodic, and linguistic aggregation, comparing utterance- and frame-level strategies.

  • I. INTRODUCTION: Speech encodes speaker identity through vocal characteristics, prosodic habits, and linguistic choices, motivating anonymization that preserves linguistic content while obscuring recognition.The VoicePrivacy Challenge provides standardized evaluation for speaker anonymization.
  • I. INTRODUCTION: Multiple utterances accumulate vocal, prosodic, and linguistic information that attackers can aggregate, whereas ASV systems commonly process isolated utterances.This mismatch motivates evaluating verification under more realistic multi-utterance access.
  • I. INTRODUCTION: The study constructs robust speaker representations by aggregating several utterances with strategies operating at utterance and fine-grained frame levels.These temporal resolutions are intended to capture speaker characteristics before or after utterance-level pooling.
  • I. INTRODUCTION: The proposed multimodal representations combine acoustic embeddings with linguistic content from ASR transcripts and prosodic features across multiple utterances.The work examines how these complementary speaker-discriminative cues can be effectively aggregated.
  • I. INTRODUCTION: Aggregating audio across multiple anonymized utterances consistently improves ASV performance, with gains increasing as more utterances become available.The experiments use conversational telephone speech and address whether multi-utterance aggregation improves verification of anonymized speech.
  • I. INTRODUCTION: Multimodal systems combining acoustic, prosodic, and linguistic information outperform unimodal systems, while frame-level aggregation is consistently the most effective strategy.These findings indicate that residual speaker-discriminative information remains distributed across temporal regions after anonymization.
  • II. RELATED WORK: Prior multi-utterance aggregation methods were evaluated mainly on original speech, leaving their effectiveness on anonymized speech and privacy impact largely unexplored.Existing work has also focused primarily on recognition and verification in unanonymized speech.
  • II. RELATED WORK: Speaker-discriminative information can occur in acoustic, linguistic, prosodic, fundamental-frequency, rhythm, facial, and speech cues, but it remains unclear which survive anonymization.This uncertainty reflects that voice anonymization does not perfectly disentangle speaker identity from other speech attributes.

III. PROPOSED METHOD · A. Utterance-level Aggregation · 1) Audio Query Attention:

The method aggregates multiple utterances at either the utterance or frame level, using audio-only and multimodal strategies. For utterance-level audio aggregation, learnable query attention adaptively combines independently encoded utterance embeddings into a speaker representation.

  • III. PROPOSED METHOD: The proposed method studies aggregation at both utterance and frame levels to capture speaker information at different temporal granularities.Utterance-level methods aggregate independently encoded utterance embeddings, whereas frame-level methods combine frame representations before speaker pooling.
  • III. PROPOSED METHOD: Both aggregation levels support audio-only and multimodal strategies incorporating linguistic and prosodic information.
  • A. Utterance-level Aggregation: Utterance-level aggregation independently extracts methods from each utterance’s embedding before combining information across utterances.
  • 1) Audio Query Attention:: Because utterances vary in speaker-discriminative content across phonetic content, speaking style, and recording conditions, the method uses learnable query attention to weight them adaptively.
  • 1) Audio Query Attention:: Given N utterance embeddings of dimension D, a learnable query vector q ∈ R^D attends over the embedding set through multi-head attention.
  • 1) Audio Query Attention:: The attention operation uses temperature τ with K = V = X and produces an aggregated speaker embedding z ∈ R^D.

2) Utterance-level Audio-Text Fusion: · 3) Utterance-level Audio-Prosody Fusion:

The paper fuses utterance-level audio with complementary text or prosodic information to preserve speaker-discriminative cues beyond acoustic characteristics. Audio-text fusion uses projected embeddings, while audio-prosody fusion incorporates three extracted utterance-level features before projection into speaker representations.

  • 2) Utterance-level Audio-Text Fusion:: Audio-text fusion targets speaker-discriminative linguistic information that may remain after anonymization by combining complementary audio and text embeddings at the utterance level.
  • 2) Utterance-level Audio-Text Fusion:: For each utterance, WavLM-ECAPA-TDNN produces the audio embedding and LUAR produces the text embedding for multimodal representation learning.
  • 2) Utterance-level Audio-Text Fusion:: The audio and text embeddings are projected into a shared 256-dimensional space, layer normalized, concatenated, and mapped to a 192-dimensional speaker representation.
  • 3) Utterance-level Audio-Prosody Fusion:: Audio-prosody fusion augments conventional speaker embeddings with mean fundamental frequency, voiced ratio, and speaking rate as complementary speaker information.
  • 3) Utterance-level Audio-Prosody Fusion:: Mean fundamental frequency is extracted with Praat via Parselmouth, while speaking rate is estimated from Whisper-medium word-level timestamps and syllable counts.
  • 3) Utterance-level Audio-Prosody Fusion:: The voiced ratio uses voiced and total pitch-frame counts, denoted Nvoiced and Ntotal, respectively.
  • 3) Utterance-level Audio-Prosody Fusion:: The prosodic feature vector is p = [F0mean, rspkrate, rvoiced], combining mean fundamental frequency, speaking rate, and voiced ratio.
  • 3) Utterance-level Audio-Prosody Fusion:: The prosodic vector is projected into a learned embedding space, concatenated with utterance-level audio, and linearly projected into the final speaker representation.

B. Frame-level Aggregation · 1) Frame Concatenation: · 2) Frame-level Audio-Text Fusion:

The section aggregates anonymized speech at the frame level before speaker pooling, preserving frame-level information across utterances. It also fuses frame-level audio representations with aggregated linguistic representations, including equal and text-dominant weighting schemes.

  • B. Frame-level Aggregation: Frame representations are extracted before attentive statistical pooling and aggregated across multiple utterances before producing the final speaker embedding.This approach investigates whether frame-level aggregation preserves speaker-discriminative cues potentially lost during utterance-level pooling.
  • 1) Frame Concatenation:: Frame sequences from N utterances are concatenated along the temporal dimension before attentive statistical pooling.The resulting unified sequence contains the combined frames from the utterances.
  • 1) Frame Concatenation:: Speaker pooling is applied over the aggregated sequence containing the sum of frames across the N utterances.The supplied passage specifies the aggregated sequence as containing PN i=1 Ti frames.
  • 2) Frame-level Audio-Text Fusion:: Audio representations use frame-level aggregation, while LUAR aggregates textual information across the same utterances into a speaker-level linguistic representation.The two modalities are therefore aggregated over matching utterance sets.
  • 2) Frame-level Audio-Text Fusion:: Audio and text embeddings are fused by concatenation followed by a linear projection.This is the primary fusion procedure described for the frame-level audio-text system.
  • 2) Frame-level Audio-Text Fusion:: Weighted fusion evaluates equal audio-text weights of (0.5, 0.5) and a text-dominant configuration of (0.2, 0.8).The weights are assigned to audio and text embeddings, respectively, to examine their relative contributions to speaker verification performance.

3) Frame-level Audio Aggregation with Prosodic Fusion: · 4) Hybrid Acoustic-Textual Speaker Verification:

Frame-level acoustic aggregation is fused with utterance-level prosodic statistics to capture both speaker characteristics and variation across utterances. The hybrid acoustic-textual system aligns lexical tokens with acoustic frames, contextualizes the fused sequence, and combines it with a transcript-level LUAR representation for speaker embedding generation.

  • 3) Frame-level Audio Aggregation with Prosodic Fusion:: Prosodic fusion aggregates F0mean, rspkrate, and rvoiced across N utterances using means and standard deviations, producing a six-dimensional summary.The summary captures overall prosodic characteristics and their variation across utterances.
  • 3) Frame-level Audio Aggregation with Prosodic Fusion:: The prosodic summary is projected into a learned embedding space and fused with the speaker-level acoustic embedding from frame concatenation and attentive statistical pooling.
  • 4) Hybrid Acoustic-Textual Speaker Verification:: The hybrid architecture models interactions between lexical tokens and acoustic realizations because speakers exhibit characteristic pronunciation and articulation patterns.This extends fusion beyond independently derived utterance-level modality representations.
  • 4) Hybrid Acoustic-Textual Speaker Verification:: For each utterance, WavLM-ECAPA-TDNN provides a frame-level acoustic feature map H ∈RT ×C, while the ASR transcript yields LUAR-tokenizer embeddings E ∈ RL×D.T denotes acoustic-frame count, C acoustic feature dimension, L token-sequence length after tokenization and padding, and D LUAR embedding dimension.
  • 4) Hybrid Acoustic-Textual Speaker Verification:: Text tokens serve as queries and acoustic frames as keys and values to align lexical and acoustic information through token-to-frame cross-attention.The resulting alignment matrix A ∈RL×T produces token-conditioned acoustic representations.
  • 4) Hybrid Acoustic-Textual Speaker Verification:: Original token embeddings are concatenated with aligned acoustic representations, projected into a shared hybrid space, and contextualized by a single-layer Transformer encoder.Padding and special tokens are masked during processing.
  • 4) Hybrid Acoustic-Textual Speaker Verification:: Masked mean pooling over non-padding and non-special tokens produces the utterance-level hybrid representation zhyb ∈Rdh.
  • 4) Hybrid Acoustic-Textual Speaker Verification:: The final speaker embedding concatenates the pooled hybrid representation with a full-transcript LUAR representation and projects the result using learnable matrices.The LUAR branch produces an utterance-level textual representation zluar ∈Rdh before concatenation and projection.

5) Recursive Joint Cross-Attention (RJCA):

RJCA models intra-modal and cross-modal relationships through recursive attention rather than simple feature concatenation. Its aggregated-text variant incorporates a speaker-level LUAR representation computed across all available utterances.

  • RJCA explicitly models intra-modal and cross-modal relationships through recursive attention operations, unlike simple feature concatenation.
  • Utterance-level multimodal representations: Audio and textual embedding sequences are processed with modality-specific BiLSTMs before concatenation into a joint multimodal representation.The embeddings are extracted from N utterances using WavLM-ECAPA-TDNN for audio and corresponding textual embeddings.
  • Recursive joint cross-attention: Cross-attention between the joint representation and each modality captures modality-specific dependencies and interactions between acoustic and textual speaker cues.Two recursive joint cross-attention layers are applied sequentially, with each layer receiving the attended representations produced by the previous layer.
  • Aggregated textual representation: The aggregated-text RJCA variant computes a speaker-level LUAR representation from all available utterances and repeats it across utterance positions.Because this textual representation already aggregates speaker-level information, the text branch omits its modality-specific BiLSTM.

6) WavLM-Whisper cross-attention:

This section introduces cross-attention fusion to combine complementary frame-level representations from WavLM-ECAPA-TDNN and Whisper. Utterance features are independently encoded, positionally embedded, concatenated, contextualized with self-attention, and used in the fusion process.

  • WavLM-Whisper cross-attention:: Cross-attention fuses complementary frame-level information from WavLM-ECAPA-TDNN and Whisper.The strategy operates on frame-level representations.
  • WavLM-Whisper cross-attention:: Utterances are independently encoded with utterance-level positional embeddings, concatenated temporally, and contextualized using self-attention across the aggregated sequence.Utterances are randomly sampled rather than taken from a continuous recording, so only utterance-level positional embeddings are added.

IV. EXPERIMENTS · A. Data

The experiments use conversational Fisher telephone calls and speaker-disjoint splits to construct multi-utterance verification trials. Models are evaluated under original–anonymized and anonymized–anonymized conditions using end-to-end or two-stage training.

  • A. Data: The study uses Fisher English Training Speech Corpus conversational telephone calls between strangers, lasting up to 10 minutes.The authors distinguish Fisher from read-speech and short-clip ASV datasets because those alternatives provide less evidence about linguistic choices or multiple utterances per speaker.
  • A. Data: Speaker-disjoint splits contain 5,712 training speakers, 250 validation speakers, and 1,753 evaluation speakers.Model selection uses a fixed validation list with 100 target and 100 non-target trials per speaker.
  • A. Data: A topic-controlled evaluation produced similar results, so the experiments use the simpler setting without topic control.Fisher additionally permits control over the topic under discussion.
  • A. Data: Evaluation speakers appear in at least two telephone calls, enabling disjoint enrollment and target utterance pools.Multi-utterance trials are sampled from separate enrollment and target datasets constructed at the utterance level.
  • A. Data: The experiments compare end-to-end training with two-stage training, which freezes a separately trained audio encoder before training aggregation or fusion.The two-stage training speakers are partitioned into base-training and additional subsets, although the supplied passage truncates before giving the remaining details.
  • A. Data: The evaluation follows VoicePrivacy with original–anonymized and anonymized–anonymized conditions.Original–anonymized trials use original enrollment and anonymized target speech from another call, whereas anonymized–anonymized trials anonymize both sides using the same method.
  • A. Data: Multimodal systems use Whisper-medium transcripts generated from the corresponding audio in each evaluation condition.This aligns the text input with the audio used for original–anonymized and anonymized–anonymized trials.
  • A. Data: Utterances requiring anonymization are processed with Stream-Voice-Anon, using randomly selected LibriSpeech target speakers whose utterances exceed 4 seconds.In the anonymized–anonymized condition, enrollment and target utterances are anonymized independently with this procedure.

B. Baselines · C. Setup

The paper establishes audio-only and textual speaker-verification baselines, then defines end-to-end and two-stage training setups for multi-utterance and multimodal evaluation. Performance is measured by cosine-similarity verification scores and reported as Equal Error Rate (EER).

  • B. Baselines: Two audio-only baselines use an x-vector SpeechBrain model and a WavLM-ECAPA-TDNN system trained on Fisher data.The WavLM-ECAPA-TDNN system combines WavLM front-end features with an ECAPA-TDNN backbone.
  • B. Baselines: Multi-utterance baseline evaluation aggregates enrollment and target embeddings with mean pooling.The aggregation produces a pooled representation from multiple utterances on each side of a verification trial.
  • B. Baselines: LUAR provides textual representations from transcribed speech by modeling speaker-identifying linguistic style and content.LUAR is fine-tuned on Fisher transcripts generated with Whisper-medium, using speakers disjoint from the evaluation sets.
  • C. Setup: End-to-end models use a fixed WavLM-Large frontend, while ECAPA-TDNN and aggregation or fusion modules are optimized jointly.Multi-utterance training uses batches of 64 speakers with N = 5 sampled utterances per speaker.
  • C. Setup: The hybrid acoustic-textual method trains for 30 epochs with learning rate 5 × 10−5, weight decay 0.01, gradient clipping 1.0, and four-head cross-attention.LUAR fine-tuning uses learning rate 2× 10−5, batch size 32, temperature 0.01, maximum token length 30, and 512-dimensional embeddings for 40 epochs.
  • C. Setup: Two-stage lazy-informed models freeze pretrained audio and LUAR encoders and optimize only aggregation or fusion modules on fusion-training speakers.This setting is used for all utterance-level aggregation and multimodal models.
  • C. Setup: The semi-informed setting anonymizes the Fisher training split at the utterance level with the evaluation configuration and initializes each variant from its lazy-informed counterpart.This setting models an attacker with access to anonymized training data while retaining initialization from representations learned on original utterances.
  • C. Setup: Evaluation aggregates enrollment and target utterances into speaker embeddings, scores them with cosine similarity, and reports Equal Error Rate (EER).The aggregation follows each model architecture for every verification trial.

V. RESULTS AND DISCUSSION

Across anonymized-speech evaluations, multi-utterance aggregation improves speaker verification, with frame-level aggregation strongest. Multimodal systems further exploit residual acoustic, prosodic, linguistic, and cross-modal speaker cues, producing stronger privacy attacks than unimodal systems.

  • RQ1: Multi-utterance audio aggregation improves ASV performance on anonymized speech: Multi-utterance audio aggregation consistently outperforms WavLM-ECAPA-TDNN and x-vector baselines, with frame-level aggregation providing larger gains across O–A and A–A settings.Gains increase steadily as more utterances become available, indicating that residual speaker information accumulates across anonymized recordings.
  • RQ1: Multi-utterance audio aggregation improves ASV performance on anonymized speech: Frame-level aggregation achieves the best performance across evaluated enrollment conditions, and semi-informed training further benefits both aggregation strategies.Aggregation gains also persist in O–O evaluation, showing that the effect is not specific to anonymized speech.
  • RQ2: Multimodal systems outperform unimodal systems: Text-only systems outperform the audio-only baseline, while Audio+Prosody and Audio+Text further improve speaker verification under anonymized-speech evaluation.These results show that lexical, linguistic, and prosodic patterns remain informative after anonymization.
  • RQ2: Multimodal systems outperform unimodal systems: 37.59% to 22.63%: equal-weighted Audio+Text reduces A–A EER at N = 15, while frame-level Audio+Prosody reaches 25.20%.The Audio+Text system with equal weighting achieves the lowest A–A EER across all values of N.
  • RQ3: Frame-level aggregation is significantly better: Frame-level Audio+Text fusion achieves the lowest EERs across most evaluation settings and provides the strongest privacy-leakage attack among multimodal approaches.At N = 15, utterance-level Audio+Prosody and Audio+Text relatively reduce A–A EER by 20.7% and 17.3%, respectively.

VI. CONCLUSIONS AND FUTURE WORK

The study shows that multi-utterance, multimodal verification remains effective against speaker anonymization: audio aggregation improves verification, multimodal and frame-level strategies perform best, and linguistic cues remain robust. Future work should examine consistent speaker-level pseudonyms across utterances and ASR transcription errors.

  • Anonymization does not completely remove speaker-discriminative information when verification aggregates multiple anonymized utterances.
  • Multimodal systems outperform unimodal counterparts, while frame-level aggregation is the most effective strategy across settings.
  • Linguistic cues remain particularly robust to voice anonymization and become increasingly informative as more utterances become available.
  • A semi-informed attacker with access to anonymized training data further amplifies the verification gains.
  • Future work: Future work should investigate speaker-level anonymization using consistent pseudo-speakers across utterances as a complementary threat model.
  • Future work: Future work should characterize transcription errors introduced by ASR transcripts of anonymized audio.
Loading 2607.19636v1…