PaperlayerView exact revision on arXiv

Source-linked AI summary

Who is Real Bob? Adversarial Attacks on Speaker Recognition Systems

Guangke Chen, Sen Chen, Lingling Fan, Xiaoning Du, Zhe Zhao, Fu Song, Yang Liu

arXiv:1911.01840v2eess.AScs.CRcs.LGcs.MMcs.SD

TL;DR

The paper addresses the limited evidence on adversarial attacks against speaker recognition systems in practical black-box settings. It proposes FAKEBOB, an optimization-based attack incorporating confidence, distortion, and score-threshold estimation, and reports high success across open-source and commercial systems, including physical-world tests. The findings expose security weaknesses and show that several speech-recognition defenses have limited effects against FAKEBOB.

  • Problem

    Existing adversarial attacks on speaker recognition systems considered white-box settings, leaving practical black-box security implications insufficiently understood.

  • Method

    FAKEBOB formulates adversarial voice generation as an optimization problem balancing confidence and maximal distortion, incorporates score-threshold estimation, and uses gradient estimation.

  • Results

    99% targeted attack success rate was achieved on open-source and commercial systems, with effectiveness also demonstrated under over-the-air physical-world playback.

  • Takeaways & Limitations

    The findings reveal security implications for speaker recognition systems and indicate a need for more robust systems and effective domain-specific defenses.

  • Takeaways & Limitations

    Four evaluated defense methods had limited effects against FAKEBOB, and the attack fails when dynamically generated phrases come from a very large or infinite set.

Abstract

from arXiv · show

Speaker recognition (SR) is widely used in our daily life as a biometric authentication or identification mechanism. The popularity of SR brings in serious security concerns, as demonstrated by recent adversarial attacks. However, the impacts of such threats in the practical black-box setting are still open, since current attacks consider the white-box setting only. In this paper, we conduct the first comprehensive and systematic study of the adversarial attacks on SR systems (SRSs) to understand their security weakness in the practical blackbox setting. For this purpose, we propose an adversarial attack, named FAKEBOB, to craft adversarial samples. Specifically, we formulate the adversarial sample generation as an optimization problem, incorporated with the confidence of adversarial samples and maximal distortion to balance between the strength and imperceptibility of adversarial voices. One key contribution is to propose a novel algorithm to estimate the score threshold, a feature in SRSs, and use it in the optimization problem to solve the optimization problem. We demonstrate that FAKEBOB achieves 99% targeted attack success rate on both open-source and commercial systems. We further demonstrate that FAKEBOB is also effective on both open-source and commercial systems when playing over the air in the physical world. Moreover, we have conducted a human study which reveals that it is hard for human to differentiate the speakers of the original and adversarial voices. Last but not least, we show that four promising defense methods for adversarial attack from the speech recognition domain become ineffective on SRSs against FAKEBOB, which calls for more effective defense methods. We highlight that our study peeks into the security implications of adversarial attacks on SRSs, and realistically fosters to improve the security robustness of SRSs.

I. INTRODUCTION

Speaker recognition systems face an open security question in practical black-box settings, where existing attacks were limited to white-box access. FAKEBOB addresses this gap and achieves strong, transferable, physical-world attacks while evading several defenses.

  • Speaker recognition systems are widely deployed, but their machine-learning implementations are vulnerable to adversarial attacks, motivating security analysis.Applications include biometric authentication, forensic testing, and personalized smart-device services.
  • Existing adversarial attacks on speaker recognition systems were limited to white-box settings, leaving practical black-box security insufficiently studied.Black-box attackers obtain at most decisions and enrolled-speaker scores, making attacks more practical but challenging.
  • FAKEBOB formulates adversarial voice generation as an optimization problem balancing confidence and maximal L∞ distortion, while incorporating an estimated speaker-recognition score threshold.It uses gradient estimation to solve the optimization problem and targets both conventional tasks and open-set identification.
  • 99% targeted attack success rate was achieved across ivector-PLDA, GMM-UBM, and xvector-PLDA systems, while Talentedsoft reached 100% within 2,500 queries on average.FAKEBOB also achieved 34%-68% transferability except for Microsoft Azure speaker verification.
  • FAKEBOB remained effective over the air against open-source systems and Microsoft Azure open-set identification, while human listeners struggled to distinguish original from adversarial voices.The study evaluated physical-world practicability and perceptual imperceptibility separately.
  • Four speech-recognition defenses—audio squeezing, local smoothing, quantization, and temporal-dependency detection—had limited effects against FAKEBOB.The authors therefore call for more effective, domain-specific defenses for speaker recognition systems.
  • The study links black-box speaker-recognition weakness to risks including bypassed biometric authentication and subsequent hidden voice-command attacks.Examples include financial transactions, smart devices, and voice-enabled cars.

A. Speaker Recognition System (SRS)

Speaker recognition systems identify people from vocal characteristics through offline modeling and online enrollment and recognition. They support open-set identification, close-set identification, and speaker verification, with distinct task settings and threat-model parameters.

  • Speaker recognition identifies a person from utterances using audio characteristics of the speaker and supports biometric, forensic, and commercial applications.The field has been studied actively for four decades and is implemented in open-source and commercial platforms.
  • A typical speaker recognition system contains feature extraction, UBM construction, speaker-model construction, scoring, and decision modules.Offline processing builds background and speaker models; online processing handles enrollment and recognition.
  • Feature extraction converts raw speech into acoustic vectors, with MFCC identified as the most popular practical algorithm.Other listed methods include SSC and PLP.
  • Speaker recognition tasks: Open-set identification compares scores for multiple enrolled speakers against a threshold and rejects inputs whose largest score is below that threshold.The highest-scoring enrolled speaker is selected only when its score reaches θ.
  • Speaker recognition tasks: Close-set identification always assigns an input to an enrolled speaker, whereas speaker verification accepts or rejects against exactly one enrolled speaker.The tasks differ in whether rejection is possible and how many speakers are enrolled.
  • System settings: Text-dependent systems require predefined sentences and are used for speaker verification, while text-independent systems allow arbitrary speech and support all tasks.Text-dependent systems offer high accuracy on short utterances but require repeated training utterances; text-independent systems may need longer utterances.
  • SRS implementations: The paper evaluates ivector-PLDA, GMM-UBM, and xvector-PLDA systems, with ivector-based methods described as state of the art and GMM-UBM as competitive on short utterances.xvector denotes a DNN-based speaker-recognition method.
  • Threat model: The threat model varies attack type, speaker gender relation, channel, recognition task, and whether the target system exposes scores or only decisions.The study defines 40 scenarios but evaluates 16 representative ones.

III. METHODOLOGY

The methodology targets practical black-box attacks across speaker-recognition tasks, balancing adversarial strength, transferability, and human imperceptibility. FAKEBOB combines threshold-aware optimization with gradient estimation and evaluates physical-world robustness and defenses.

  • Research Questions: The study asks whether adversarial attacks can work across speaker-recognition tasks in practical black-box settings.It also considers transferability to unknown and commercial systems, over-the-air operation, human imperceptibility, and defenses.
  • Design Philosophy: FAKEBOB uses gradient estimation and a novel score-threshold estimation algorithm because speaker-recognition systems require threshold-aware attacks.Existing gradient-based methods cannot directly transfer when a resulting score remains below the system threshold.
  • Design Philosophy: The attack increases adversarial strength and noise-amplitude range to support transferability and over-the-air operation on potentially different systems.This design addresses practical attacks against unknown systems in the physical world.
  • Design Philosophy: FAKEBOB constrains per-sample perturbations with the L∞ norm and includes a human study to assess whether adversarial voices retain speaker identity perceptually.The methodology treats both naturalness and perceived speaker identity as aspects of human imperceptibility.
  • Defense Evaluation: The evaluation includes four speech-recognition defenses: audio squeezing, local smoothing, quantization, and temporal dependency detection.These methods are used because domain-specific defenses for adversarial attacks on speaker-recognition systems were unavailable.

C. Overview of Our Attack: FAKEBOB

FAKEBOB formulates black-box adversarial voice generation as constrained optimization for targeted and untargeted attacks across OSI, CSI, and SV systems. It combines threshold estimation, NES gradient estimation, and BIM updates while controlling confidence and distortion.

  • Overview of Our Attack: FAKEBOB: FAKEBOB addresses two challenges: crafting imperceptible black-box adversarial samples and making them effective on unknown systems over the air.The overview defines task-specific losses for targeted and untargeted attacks across OSI, CSI, and SV systems.
  • Problem Formulation: The attack creates a valid adversarial voice x+δ by constraining amplitudes to [−1, 1] and limiting per-sample distortion with an L∞ bound.The resulting audio is transformed back into a file format before being supplied to the target system.
  • Problem Formulation: The optimization minimizes a loss under validity and distortion constraints, prioritizing rapid loss reduction rather than directly minimizing perturbation size.When minimized, the adversarial voice is recognized as the target speaker in targeted attacks or an enrolled speaker in untargeted attacks.
  • Attack on OSI Systems: Targeted OSI loss requires the target score to exceed both the score threshold and every other enrolled-speaker score.The confidence parameter κ controls how confidently the target speaker is recognized.
  • Attack on OSI Systems: NES estimates gradients from system queries, and BIM uses those estimates to iteratively update and clip adversarial voices.Per-sample clipping keeps each iterate within the source voice’s L∞ ε-neighborhood and preserves valid amplitudes.
  • Attack on OSI Systems: The threshold-estimation algorithm increases a candidate threshold until a crafted voice is accepted, then returns its maximal enrolled-speaker score.The algorithm may not terminate if every crafted voice remains rejected, although this did not occur in the experiments.

C. Attack on CSI Systems

For closed-set identification systems, FAKEBOB adapts its loss by omitting the rejection threshold. Targeted attacks seek a dominant target score with a confidence margin, while untargeted attacks suppress the true speaker.

  • Attack on CSI Systems: CSI systems always classify an input as one enrolled speaker, so FAKEBOB’s CSI loss omits the OSI rejection threshold.The targeted loss is defined for a target speaker within the enrolled speaker group.
  • Attack on CSI Systems: A targeted CSI attack seeks a small perturbation making the target score largest and at least κ greater than the second-largest enrolled-speaker score.The confidence margin separates the target from competing enrolled speakers.
  • Attack on CSI Systems: An untargeted CSI attack seeks a small perturbation making the largest non-true-speaker score at least κ greater than the true speaker’s score.Here, m denotes the true speaker of the original voice.

D. Attack on SV Systems

FAKEBOB adapts the OSI attack to speaker-verification systems by treating the enrolled speaker group as a singleton. Its loss seeks acceptance with a confidence margin above the threshold.

  • Attack on SV Systems: SV systems have one enrolled speaker and decide whether the input was uttered by that speaker, so FAKEBOB models the speaker group as a singleton.This directly adapts the OSI formulation to verification.
  • Attack on SV Systems: The SV attack seeks a small perturbation whose enrolled-speaker score exceeds the threshold by at least κ.Threshold estimation must replace the OSI loss with f′ = max{θ′ − S(x), −κ}.

V. ATTACK EVALUATION

FAKEBOB is evaluated across multiple speaker-recognition systems, tasks, and attack settings, including threshold estimation, effectiveness, efficiency, and gender variation. It achieves high targeted attack success while producing relatively imperceptible perturbations, though inter-gender attacks are more difficult.

  • Dataset and Experiment Design: The study evaluates effectiveness, efficiency, transferability, practicability, imperceptibility, and robustness across 16 representative attack scenarios.The scenarios are selected from 40 combinations of attack types, channels, genders, tasks, and available system information.
  • Dataset and Experiment Design: FAKEBOB is evaluated on ivector-PLDA, GMM-UBM, and xvector-PLDA systems, using datasets including VoxCeleb1, VoxCeleb2, and LibriSpeech.The evaluation covers open-source Kaldi systems and commercial systems, with representative attack scenarios spanning multiple task and channel settings.
  • Effectiveness and Efficiency: 99.0% ASR is achieved on both ivector and GMM OSI systems, with average SNRs of 31.5 dB and 31.4 dB, respectively.The corresponding perturbations are reported as less than 0.071% and 0.072%.
  • Effectiveness and Efficiency: Inter-gender attacks require more iterations and execution time and achieve lower ASR than intra-gender attacks.The paper attributes this difference to the sounds of male and female speakers and reports that known target gender makes intra-gender attacks easier.
  • Effectiveness and Efficiency: Threshold estimation error is less than 0.03 for ivector and less than 0.003 for GMM, with estimation completed in less than 13.4 minutes.The actual thresholds are accessed only for evaluation; the attack itself operates in the black-box setting.
  • Effectiveness and Efficiency: FAKEBOB achieves 100% ASR on the commercial Talentedsoft system within 50 iterations, or 2,500 queries, on average.The system is proprietary and completely black-box, and the iterative attack can space queries over time.

C. Transferability

The paper evaluates FAKEBOB's transferability across architectures, datasets, parameters, devices, distances, and acoustic environments. Transferability declines as source and target systems differ more, while over-the-air attacks remain effective under several practical conditions.

  • Transferability: Transferability attacks are evaluated across architectures, training datasets, and key parameters using GMM and ivector source systems with additional target systems.The experiments include cross-architecture, cross-dataset, and cross-parameter settings.
  • Transferability: 34%-68% ASR and 40%-100% UTR are achieved for all OSI transferability attacks except B →A.Attacks between ivector systems differing in one key parameter achieve 100% ASR and UTR, while larger system differences reduce transferability.
  • Transferability: ASR and UTR reach 100% at κ = 4.5 in the B →F OSI experiment.Both measures increase quickly as κ rises from 0.5 to 5.0 under fixed ϵ = 0.05.
  • Transferability: SV-Azure yields 10% ASR from 200 adversarial samples, while 190 samples return “error, too noisy” instead of acceptance or rejection.The authors suspect a proprietary input-noise check, based on average SNRs of 8.8 dB for the 190 voices and 11.5 dB for the other 10.
  • Practicability for Over-the-Air Attack: At least 75% ASR and UTR are achieved for every tested loudspeaker-microphone pair, with 100% ASR for JBL clip3 or DELL paired with iPhone 6 Plus.iPhone 6 Plus attacks achieve at least 14% higher ASR and 16% higher UTR than OPPO attacks with the loudspeaker fixed.
  • Practicability for Over-the-Air Attack: 100% ASR and UTR are achieved at distances up to 1 meter, falling to 10% at 8 meters while FRR rises to 32%.At 2 meters, ASR and UTR are 70%; at 4 meters, they are 40% and 50%, respectively.
  • Practicability for Over-the-Air Attack: FAKEBOB achieves at least 48% ASR and UTR when background-noise volume is no more than 60 dB, across the tested noise types.ASR and UTR decrease as white-noise volume increases, while FRR rises quickly.

E. Human-Imperceptibility via Human Study

A human study tests whether listeners can detect noise or distinguish speakers in original and adversarial voices. Participants generally struggled to distinguish normal from adversarial speakers, especially for ineffective over-the-air adversarial samples.

  • Study Design: The human study uses MTurk questionnaires to assess whether voices sound clean or noisy and whether paired voices come from the same speaker.The study was approved by the institutions' Institutional Review Board.
  • Study Design: The study filters questionnaires using silent-voice and gender-mismatch concentration tests before analyzing responses.Only questionnaires with all concentration questions answered correctly are retained.
  • Results: 54.6% of participants judged adversarial pairs to come from the same speaker, close to the 53.7% baseline for normal pairs.This indicates that participants could not differentiate speakers of normal and adversarial voices reliably.
  • Results: 64.9% of participants judged pairs containing ineffective adversarial voices to come from the same speaker, exceeding the 53.7% normal-pair baseline.The paper characterizes these ineffective adversarial voices as more imperceptible to human listeners.
  • Results: FAKEBOB causes systems to accept adversarial voices as target-speaker utterances while most ineffective samples remain difficult for ordinary users to distinguish.The effective adversarial samples produce human-study results comparable to prior related work.
  • Defense Evaluation: Four evaluated defenses do not reliably stop FAKEBOB: local smoothing raises attack cost, audio squeezing does not affect attack cost or ASR, and two others are unsuitable.The defenses are local smoothing, quantization, audio squeezing, and temporal dependency detection.

VI. DISCUSSION OF THE POSSIBLE ARM RACE

The discussion examines defenses and attack extensions around FAKEBOB, emphasizing practical countermeasures and remaining imperceptibility limitations. It concludes that existing defenses have limited effects and that stronger protection is needed.

  • Mitigation of FAKEBOB: Four defenses reported as promising for speech recognition have limited effects against FAKEBOB, motivating more effective mitigation methods.
  • Mitigation of FAKEBOB: Liveness detection could detect FAKEBOB because adversarial voices must be played through loudspeakers with physical characteristics differing from human-generated voices.
  • Mitigation of FAKEBOB: Text-dependent systems with dynamically generated sentences increase attack costs by requiring simultaneous attacks on speaker and speech recognition.
  • Possible advanced attacks: Decision-only systems constrain direct interaction, so transferability or boundary attacks may be needed, with larger source-target gaps limiting transferability.
  • Possible advanced attacks: Although human evaluation finds FAKEBOB reasonably imperceptible, noisier voices and some speaker-identification differences leave room for psychoacoustic improvements.
  • Conclusion: The study reports 99% targeted attack success across open-source and commercial systems, including effectiveness over the air, underscoring SRS security implications.

APPENDIX

The appendix compares FAKEBOB with a PSO-based attack on an ivector system. FAKEBOB is reported as substantially more effective and efficient, especially across different initial-score ranges.

  • Comparison setup: FAKEBOB and a PSO-based method are compared using an optimization formulation and PSO with particles iteratively updating candidate solutions.
  • Effectiveness: 33% targeted attack success rate is achieved by the PSO-based method, only one-third of FAKEBOB’s rate.
  • Effectiveness: When [S(x0)]t ≤−0.5, the PSO-based method fails for all voices, while FAKEBOB remains effective across initial input-score ranges.
  • Effectiveness: 5.3% and 17.6% ASR are obtained by PSO in two intermediate initial-score ranges, whereas FAKEBOB is more effective regardless of initial scores.
  • Efficiency: FAKEBOB requires fewer iterations and less execution time than PSO in most cases, with its efficiency advantage increasing for higher initial scores.
  • Summary: The experiments summarize FAKEBOB as more effective and efficient than the PSO-based method.

C. Results of Tuning the Parameter ϵ

The parameter ϵ controls a trade-off between adversarial-voice imperceptibility and attack cost. The study selects ϵ = 0.002 as a suitable balance for the CSI task.

  • ϵ = 0.002 is selected because it provides higher average SNR while losing only 1% success rate compared with other tested values.
  • ϵ = 0.001 reduces perturbation further but lowers success rate to 41% for ivector and 87% for GMM.
  • Reducing ϵ increases attack cost sharply: ϵ = 0.002 requires 1.6 times the iterations and 1.4 times the execution time of ϵ = 0.003.
  • The table reports tuning results for ϵ on ivector and GMM systems in the CSI task.

D. Experiment results of FAKEBOB on xvector system

The experiments evaluate FAKEBOB on xvector systems and against several defenses. Results show strong attack performance, while temporal-dependency detection and some preprocessing defenses are ineffective or impose substantial costs.

  • xvector evaluation: FAKEBOB is evaluated against a DNN-based xvector SRS using a pre-trained xvector model and a speech-to-text system.
  • Median filter: Median filtering reduces low-confidence UTR from 99% to nearly 0% only by increasing normal-voice FRR to 35%, versus a 4.2% baseline.
  • Median filter: With median filtering, FAKEBOB still reaches 90% ASR within 250 maximum iterations, while solving the remaining 9% requires 15,000 iterations.
  • Audio squeezing: At τ = 0.9, audio squeezing lowers low-confidence UTR to 17% while keeping normal-voice FRR near baseline at 6% versus 4.2%; high-confidence UTR remains unchanged.
  • Audio squeezing: At τ = 0.5, audio squeezing increases attack performance, with FAKEBOB achieving 100% ASR in 200 iterations under defense.
  • Quantization: Quantization is unsuitable because all processed voice frames are classified as unvoiced by the VAD component, producing no system output.
  • Temporal dependency detection: Temporal-dependency detection fails to distinguish adversarial voices, with AUC values of 46.7% and 50.5%, close to random guessing.
Loading 1911.01840v2…