PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systems

Hadi Abdullah, Kevin Warren, Vincent Bindschaedler, Nicolas Papernot, Patrick Traynor

arXiv:2007.06622v3cs.CRcs.LGcs.SD

TL;DR

Speech and speaker recognition systems are widely deployed but remain vulnerable to manipulated inputs, and their end-to-end audio pipelines differ substantially from image models. The paper systematizes attacks and defenses with a threat-model taxonomy, then tests transferability experimentally. It finds that optimization-based attacks almost universally fail to transfer, leaving substantial mitigation work unresolved.

  • Problem

    Speech and speaker recognition systems are widely deployed and vulnerable to manipulated inputs, but attacks and defenses cannot be assumed to work like those in image models.

  • Method

    The paper develops a threat-model taxonomy, categorizes existing attacks and defenses, and experimentally tests transferability across voice processing systems.

  • Results

    Optimization-based adversarial attacks almost universally fail to transfer across voice processing systems, although selected high-confidence scenarios achieve limited transferability.

  • Takeaways & Limitations

    Voice processing systems require attack analysis and mitigations tailored to their pipelines, because image-domain methods do not readily extend and existing defenses remain limited.

Abstract

from arXiv · show

Speech and speaker recognition systems are employed in a variety of applications, from personal assistants to telephony surveillance and biometric authentication. The wide deployment of these systems has been made possible by the improved accuracy in neural networks. Like other systems based on neural networks, recent research has demonstrated that speech and speaker recognition systems are vulnerable to attacks using manipulated inputs. However, as we demonstrate in this paper, the end-to-end architecture of speech and speaker systems and the nature of their inputs make attacks and defenses against them substantially different than those in the image space. We demonstrate this first by systematizing existing research in this space and providing a taxonomy through which the community can evaluate future work. We then demonstrate experimentally that attacks against these models almost universally fail to transfer. In so doing, we argue that substantial additional work is required to provide adequate mitigations in this space.

I. INTRODUCTION

Voice processing systems improve accessibility and authentication, but their neural-network architectures and audio pipelines make attacks unlike those in images. This paper develops a threat-model taxonomy, categorizes prior work, and experimentally investigates transferability.

  • Motivation: Voice processing systems support accessibility, screen-free interaction, and nearly invisible authentication across applications such as assistants, surveillance, and conferencing.They include ASR systems that convert speech to text and speaker-identification systems that identify people from speech.
  • Motivation: Neural networks improved speech transcription and identification accuracy while also exposing voice processing systems to adversarial attacks.The paper cautions that image-domain attack results cannot simply be assumed to apply to these systems.
  • Contributions: The paper provides a threat-model taxonomy for reasoning about differing attacker assumptions and capabilities against voice processing systems.It also categorizes existing attacks and defenses using this framework.
  • Contributions: Gradient-based adversarial attacks do not currently achieve transferability across voice processing systems, even across separately trained instances with matched configurations.The paper reports this conclusion as an experimental contribution and examines why optimization attacks fail to provide transferability.
  • System Background: ASR pipelines preprocess speech, extract features such as MFCCs, infer a model output, and decode that output into human-readable transcription.MFCC processing includes frequency analysis, Mel filtering, logarithmic scaling, and cosine-transform components that retain salient information.

3) Inference:

VPS inference uses neural models after feature extraction, with RNNs supporting variable-length speech and contextual dependencies. ASR decoding converts frame-level character distributions into readable transcriptions through sequence search.

  • 3) Inference:: The inference stage passes extracted features to a probabilistic model, with this discussion focusing on neural networks as the dominant VPS choice.
  • 3) Inference:: Beam search uses RNN output distributions over characters to calculate sequence probabilities and select likely CTC outputs.The example’s character set is {l,o,c,k,-}.
  • 3) Inference:: CNNs can be constrained for speech because fixed input and output sizes may discard contextual information from arbitrary-length utterances.
  • 3) Inference:: RNNs address CNNs’ fixed-size limitation by processing variable-length speech while carrying contextual information through a hidden state.This suits speech with varying pronunciation speeds and sentence lengths.
  • 3) Inference:: ASR inference produces a character-by-time probability matrix, so decoding must evaluate multiple possible transcriptions rather than selecting each frame’s most likely character.For “lock,” frame outputs can form strings such as “llo-ockk” or “ll-ok-kk” before decoding.

C. Speaker Identification (SI)

Speaker identification systems determine or verify speaker identity through pipelines resembling ASR, but they stop before ASR decoding. The section frames SI attacks as evasion attacks that manipulate audio or exploit model optimization to alter speaker decisions.

  • C. Speaker Identification (SI): SI systems either identify the speaker in a voice sample or verify whether a claimed identity matches it.
  • C. Speaker Identification (SI): SI and ASR pipelines share preprocessing, feature extraction, and inference, but SI directly outputs speaker probabilities instead of performing decoding.
  • C. Speaker Identification (SI): Evasion attacks craft malicious audio so a model produces an incorrect output while the altered sample may remain benign to human listeners.The paper focuses on evasion attacks among adversarial-machine-learning threats.
  • C. Speaker Identification (SI): Gradient-based attacks optimize perturbations using model gradients or approximations, balancing a desired label change against excessive input modification.
  • C. Speaker Identification (SI): Transferability would let an adversarial sample crafted for one model fool another, but the paper reports it is extremely unlikely across separately trained VPS instances.The reported comparison includes models sharing architecture, hyperparameters, seeds, and train-test splits.

III. ATTACKS AGAINST VPSES

Attacks against VPSes cannot be treated as straightforward extensions of image attacks because speech systems preprocess sequential, discrete-domain inputs. These differences motivate a VPS-specific threat taxonomy and expose optimization and component-specific limitations.

  • III. ATTACKS AGAINST VPSES: Speech-recognition pipelines rely on hard-coded signal-processing feature extraction before neural inference, unlike image classifiers that generally operate directly on pixels.
  • III. ATTACKS AGAINST VPSES: Signal-processing attacks exploit differences between human hearing and feature extraction, making them less model-dependent than inference-targeting attacks.
  • III. ATTACKS AGAINST VPSES: Sequential audio requires attackers to optimize perturbations across all time steps, making unrolling difficult and vulnerable to exploding or vanishing gradients.
  • III. ATTACKS AGAINST VPSES: Speech models must handle discrete language tokens and sequence-to-sequence inputs and outputs, adding constraints absent from continuous image inputs.
  • III. ATTACKS AGAINST VPSES: Non-differentiable decoding such as beam search can constrain targeted attacks, while attacks on remaining statistical components are insufficiently understood.
  • III. ATTACKS AGAINST VPSES: The paper introduces a threat model for audio evasion attacks and uses it to organize existing work and identify open problems.

1) Untargeted:

VPS attacks differ by the attacker’s desired output and available model information. The taxonomy distinguishes targeted and untargeted goals, direct and indirect gradient access, signal-processing methods, and miscellaneous attacks outside the pipeline-focused scope.

  • 1) Untargeted:: Untargeted attacks succeed when SI misidentifies a speaker or ASR produces any transcription different from the correct one.
  • 1) Untargeted:: Targeted attacks seek a specific speaker identity or transcription, and can be harder when the desired sequence has less similar semantics to the original.The example contrasts changing LOCK to CLOCK with changing LOCK to OPEN.
  • 1) Untargeted:: Direct attacks use model weights to compute gradients, whereas indirect attacks estimate gradients by repeatedly querying the target model.
  • 1) Untargeted:: Signal-processing attacks target discrepancies between human hearing and feature extraction rather than directly attacking inference, enabling faster and more query-efficient attacks.
  • 1) Untargeted:: Miscellaneous attacks exploit hardware limitations, random noise, or replayed recordings without targeting a specific VPS pipeline component.Replay attacks capture a victim’s voice and replay the audio.
  • 1) Untargeted:: Attack strength can be characterized by how little knowledge the adversary has about VPS components and operation.

1) Component Categories:

The taxonomy divides VPSes into five component categories and separately classifies attacker knowledge, input/output granularity, and adversarial-audio types.

  • Component categories: VPS pipelines comprise task, preprocessing, feature extraction, inference, and decoding components.
  • Component categories: Task knowledge covers the model’s problem and training data, while preprocessing knowledge covers operations such as down-sampling, noise reduction, and filtering.
  • Component categories: Feature-extraction knowledge concerns signal-processing algorithms, whereas inference knowledge includes model weights, layers, and architecture.
  • Component categories: Decoding converts label probabilities into human-readable transcription and includes the decoding algorithm and its parameters.
  • Attacker knowledge and attack forms: White-box, grey-box, black-box, and no-box access represent decreasing knowledge of VPS components, while attacks also vary by input/output granularity and audio type.

2) Access to the Model:

VPS attacks differ by available queries and outputs, transmission medium, and exposure to distortion; attack strength increases with the distance an exploit survives.

  • Model access: Attackers may query a model repeatedly, but query limits matter because excessive queries can reveal attacks and incur costs on proprietary systems.
  • Model access: Model outputs may be a single transcription label or a probability distribution over labels, with some attackers receiving both forms.
  • Acoustic equipment and mediums: VPS inputs can travel over-line, over-air, over-telephony-network, or other mediums, each introducing different distortion sources.
  • Acoustic equipment and mediums: Over-air attacks face interference, background noise, and imperfect equipment, while telephony attacks face static, compression, packet loss, and jitter.
  • Acoustic equipment and mediums: Attack strength can be measured by how far an adversarial sample travels through lossy mediums without losing its exploitative nature.

1) Targeted White-box Attacks:

Existing attack research is dominated by white-box and over-line methods, while black-box, transferable, and cross-medium attacks remain constrained by model knowledge, distortion, and query demands.

  • Targeted white-box attacks: Most targeted attacks use optimization and embed hidden commands in clean or noisy audio samples.
  • Targeted white-box attacks: White-box attacks require complete target knowledge, making them impractical against proprietary systems unless adversarial samples transfer across models.
  • Clean attacks and mediums: Success over-line does not translate to other mediums because transmission factors interfere with adversarial perturbations.
  • Black-box attacks: Signal-processing attacks show greater promise in black-box settings, requiring fewer than 15 queries and a few seconds to generate an attack sample.
  • Indirect optimization attacks: Indirect black-box optimization attacks require hundreds of thousands of queries, work only over-line, and have not reached sentence-level black-box targeting.
  • Model dependence: Only a small subset of attacks is black-box and model agnostic, while many others exploit components specific to individual VPS architectures.

VI. DEFENSE AND DETECTION TAXONOMY

The defense taxonomy distinguishes attacker adaptivity, resource demands, distortion, model dependence, and deployment cost, while existing coverage remains incomplete across attack mediums.

  • Defense and detection taxonomy: The paper provides a taxonomy for categorizing defenses and detection methods.
  • Attacker adaptivity: Non-adaptive attackers lack knowledge of defense parameters, whereas adaptive attackers know the defense type and parameters and modify their strategy accordingly.
  • Defense objectives: A defense can be stronger by increasing the queries or other resources required to produce a viable adversarial sample.
  • Defense objectives: Defenses may increase the audible distortion needed for successful attacks, but ML-based detectors remain vulnerable to exploitation.
  • Deployment constraints: Additional sensors raise manufacturing costs, so strong defenses should avoid additional deployment costs.
  • Deployment constraints: Existing defenses may depend on attacker distance in over-air settings, with longer distances potentially strengthening the defense.

F. Attack Type:

The paper reviews defenses against adversarial audio, emphasizing that adversarial training is ineffective in important settings and that detection remains constrained by adaptive attackers and deployment conditions.

  • A. Adversarial training: Adversarial training may reduce legitimate accuracy without improving robustness when signal-processing attacks are removed during feature extraction.Both benign and adversarial samples can map to similar feature vectors, undermining the defense.
  • B. Detection strategies: Detection mechanisms face limitations similar to adversarial training, particularly against adaptive adversaries that know the detection method.
  • B. Liveness detection: Liveness detection can address replayed Over-Air attacks, but many methods require short or undisclosed source-target distances and remain vulnerable to adaptive attackers.Blue et al. is described as effective at a larger distance and against adaptive attackers.
  • C. Future direction: Redesigning feature extraction may address signal-processing vulnerabilities, but preserving ASR accuracy while improving robustness is difficult and not guaranteed.The paper specifically identifies established techniques such as the DFT as difficult to replace safely.
  • The paper frames its discussion as identifying key findings, implications, and recommendations for future research.

1) Lack of Transferability for Optimization Attacks:

Optimization attacks rarely transfer across independently trained VPS models, even under matched training conditions, while limited exceptions and the pipeline’s complexity define important scope boundaries.

  • 1) Lack of Transferability for Optimization Attacks: Transferability of optimization attacks is unlikely even when surrogate and target VPSes share architecture, hyperparameters, random seed, and training data.The paper attributes this to GPU-induced nondeterminism that can produce different decision boundaries.
  • 1) Lack of Transferability for Optimization Attacks: The limited Commander Song transfer succeeded from Kaldi to iFlytek but failed for DeepSpeech, possibly because iFlytek used a fine-tuned Kaldi model.The paper cautions that fine-tuned and pretrained counterparts may retain sufficiently similar decision boundaries.
  • 2) Defenses for VPSes: Because transferability is difficult and target VPSes are rarely accessible as white boxes, defenses should prioritize demonstrated black-box attacks such as signal-processing attacks.
  • IX. CONCLUSION: VPS pipelines include processing, feature extraction, and decoding stages that expand the attack surface beyond image models, leaving vulnerabilities and attacks insufficiently characterized.The paper notes that some attack classes, including clean targeted attacks, remain undemonstrated.
  • IX. CONCLUSION: The paper focuses on evasion attacks because poisoning and privacy attacks had not been proposed or evaluated for speech, leaving them as future research directions.
  • IX. CONCLUSION: Telephony attacks lack a defense mechanism, unlike some Over-Air and Over-Line attacks for which liveness or temporal methods show partial success.

XI. APPENDIX

The appendix evaluates whether gradient-based adversarial audio samples transfer across separately trained ASRs, using controlled DeepSpeech models and carefully selected attack targets.

  • A. Transferability: The study frames transferability as the key question because it underpins black-box attacks in images but has shown limited success for gradient-based attacks on audio.The experiments therefore test whether optimization-generated adversarial samples transfer between ASRs.
  • 1) Training:: Nine DeepSpeech ASRs were trained on LibriSpeech with identical architecture, hyperparameters, and data splits, varying only the initial random seed.The models achieved an 8% Word Error Rate and were trained for approximately two days each.
  • 2) Adversarial Sample Generation:: The experiment selected 98 test-set samples that every ASR transcribed correctly, maximizing consistency and the opportunity for transfer.These samples were perturbed with 1000 attack iterations toward 11 target transcriptions.
  • 2) Adversarial Sample Generation:: Attacks produced both high-distortion, high-confidence and low-distortion, low-confidence adversarial audio to test whether distortion affects transferability.The study used a representative optimization attack following a generic gradient-based template.
  • 2) Adversarial Sample Generation:: Targets were chosen as the second- and third-most likely transcriptions because nearby alternatives provide the most favorable case for transfer.Failure even for these close targets would make transfer in more realistic cases unlikely.

3) Transferring Samples:

Across ASRs, optimization-generated adversarial audio almost never transferred, including under matched training conditions; only especially favorable nearby targets produced limited transfer.

  • C. Results: For the second-most-likely target, transferability was 6.5% with low distortion and 40% with high distortion; for the third, it fell to 3.0% and 31%.The corresponding top-10 inclusion rates were 68% and 63%, respectively.
  • C. Results: Even nearly identical target transcriptions transferred infrequently, while moving from the second- to third-most-likely target reduced success from 6.5% to 3.0%.High-confidence samples transferred more often because they were farther from decision boundaries.
  • C. Results: The results indicate that optimization-based attacks generally do not transfer between instances of the same ASR, including models with matching training parameters.The authors attribute this difficulty to substantial GPU-induced training nondeterminism.
  • C. Results: 0 of 239,968 PGD-generated attack files transferred successfully across ASRs, regardless of whether models used the same or different seeds.The PGD results match the Carlini experiments, indicating that transfer difficulty is not specific to one attack.
Loading 2007.06622v3…