PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Hidden Voice Attacks against Speech and Speaker Recognition Systems

Hadi Abdullah, Washington Garcia, Christian Peeters, Patrick Traynor, Kevin R. B. Butler, Joseph Wilson

arXiv:1904.05734v1cs.CRcs.LGcs.SDeess.AS

TL;DR

Hidden voice-command attacks can fool VPSes but often depend on specific models and hardware. This paper instead uses signal-processing knowledge to create black-box perturbations, demonstrating successful attacks across speech and speaker systems and multiple hardware configurations. The authors identify noisy environments as a practical boundary for attack effectiveness.

  • Problem

    Existing hidden-command attacks often require white-box knowledge of a specific model and depend on limited acoustic hardware configurations.

  • Method

    The paper generates unintelligible attack audio by modifying signals to preserve feature vectors produced by common VPS signal-processing algorithms.

  • Results

    The attacks succeed against 12 black-box models, including speech and speaker-recognition systems, and work in both direct-input and speaker-playback settings.

  • Takeaways & Limitations

    Hidden command attacks can be practical across models and real systems because signal-processing knowledge can preserve model-relevant information while reducing human intelligibility.

  • Takeaways & Limitations

    Attack audio has reduced effectiveness in noisy environments because noise interferes with VPS interpretation.

Abstract

from arXiv · show

Voice Processing Systems (VPSes), now widely deployed, have been made significantly more accurate through the application of recent advances in machine learning. However, adversarial machine learning has similarly advanced and has been used to demonstrate that VPSes are vulnerable to the injection of hidden commands - audio obscured by noise that is correctly recognized by a VPS but not by human beings. Such attacks, though, are often highly dependent on white-box knowledge of a specific machine learning model and limited to specific microphones and speakers, making their use across different acoustic hardware platforms (and thus their practicality) limited. In this paper, we break these dependencies and make hidden command attacks more practical through model-agnostic (blackbox) attacks, which exploit knowledge of the signal processing algorithms commonly used by VPSes to generate the data fed into machine learning systems. Specifically, we exploit the fact that multiple source audio samples have similar feature vectors when transformed by acoustic feature extraction algorithms (e.g., FFTs). We develop four classes of perturbations that create unintelligible audio and test them against 12 machine learning models, including 7 proprietary models (e.g., Google Speech API, Bing Speech API, IBM Speech API, Azure Speaker API, etc), and demonstrate successful attacks against all targets. Moreover, we successfully use our maliciously generated audio samples in multiple hardware configurations, demonstrating effectiveness across both models and real systems. In so doing, we demonstrate that domain-specific knowledge of audio signal processing represents a practical means of generating successful hidden voice command attacks.

I. INTRODUCTION

Voice Processing Systems are widely deployed and accurate, but hidden-command attacks expose security risks. This work targets those risks through black-box attacks based on signal-processing features rather than model internals.

  • Motivation: Voice Processing Systems support speech transcription and speaker identification across increasingly common voice interfaces.ASRs convert audio to text, while speaker-identification models determine whether audio matches a trained speaker.
  • Motivation: Prior hidden-command attacks produced audio recognized by VPSes but difficult for humans to understand, often requiring white-box model knowledge.Earlier attacks were also evaluated on limited model and hardware configurations.
  • Approach: The paper modifies audio to create similar feature vectors, enabling powerful model-agnostic attacks against speech and speaker-recognition systems.The approach targets signal-processing features such as FFT frequencies and MFC coefficients rather than model weights.
  • Contributions: The authors generate over 20,000 meaningful-looking audio samples that sound like unintelligible noise while remaining effective in black-box attacks.They evaluate 12 black-box models, including seven proprietary and five local models, without using model architecture or weights.
  • Implications: The paper argues that hidden voice commands become practical because attacks can work across models and real systems, while robustness alone may not detect them.The authors state that attacked and normal inputs can produce virtually identical feature vectors.

B. Signal Processing

VPSes transform audio into lower-dimensional feature vectors before machine-learning inference. Common deterministic signal-processing pipelines retain selected acoustic characteristics while discarding much of the original input.

  • Signal-processing role: Signal processing captures selected data characteristics and outputs a feature vector for machine-learning inference.The quality of these extracted features is linked to the quality of model training.
  • MFCC pipeline: MFCC extraction divides audio into 20 ms windows before transforming each window through several stages.The stages include FFT magnitude computation, Mel filtering, logarithmic powers, and discrete cosine transformation.
  • MFCC pipeline: The FFT magnitude represents each signal frequency and its corresponding intensity in the frequency domain.The resulting magnitude spectrum is one intermediate representation used in MFCC extraction.
  • Alternative methods: Other VPSes use MFSC, linear predictive coding, PLP, or probabilistic approaches such as transfer learning for feature extraction.These methods differ in implementation but likewise transform input before inference.
  • Model inference: Machine-learning models map extracted inputs to continuous or categorical outputs and evaluate errors on unseen test data.The paper describes inference as extrapolating learned mappings from training inputs to new inputs.

D. Psychoacoustics

Psychoacoustics explains why audio can preserve machine-recognized features while becoming difficult for humans to interpret. The paper uses perceptual weaknesses and signal-processing behavior to design such perturbations.

  • Human perception: Psychoacoustics studies how humans perceive and interpret acoustic signals, including speech under competing or altered sound conditions.Successful hidden commands must account for these perceptual effects.
  • Human perception: Humans can focus on one sound source and remain relatively robust to background noise, so simply adding noise does not necessarily reduce speech intelligibility.The Cocktail Party effect describes selective attention among multiple sound sources.
  • Human perception: Humans poorly discern high-frequency differences and are less able to interpret discontinuous or random sounds than smooth audio.These perceptual weaknesses motivate perturbations involving high frequencies and discontinuities.
  • Attack design: The attack exploits psychoacoustics and audio characteristics to add artifacts that signal processing removes while preserving model-relevant information.This supports audio that is uninterpretable to users but correctly inferred by VPSes.
  • Perturbation examples: Figure 2 compares the original signal with outputs produced by four perturbation schemes.Signals (b) through (e) show the perturbed versions of signal (a).

A. Attack Scenario

The attack assumes no knowledge of the victim model or acoustic hardware, but familiarity with audio processing and correctly recognized source samples. Four perturbation techniques alter perception while preserving properties used by VPSes.

  • Threat model: The attacker operates in a black-box setting without knowing the victim model, architecture, weights, or acoustic hardware.The attacker is assumed to possess audio and speech-processing expertise.
  • Threat model: The attacker starts from correctly transcribed audio or a voice sample that the target speaker model identifies as the victim.Perturbations preserve important acoustic properties while changing audible perception.
  • Perturbation techniques: Four perturbation techniques are proposed: Time Domain Inversion, Random Phase Generation, High Frequency Addition, and Time Scaling.Each attack sample can combine one or more perturbations applied successively.
  • Perturbation techniques: Time Domain Inversion preserves the spectrum while inverting small signal windows, creating discontinuities that humans perceive as noise.The method exploits the FFT’s many-to-one mapping from time-domain signals to spectra.
  • Perturbation techniques: Random Phase Generation changes phase while retaining the original magnitude spectrum, producing a new signal that is harder for humans to interpret.The method selects alternative complex-spectrum values with the same magnitude.

D. Attack Audio Generation

The Perturbation Engine generates hidden attack audio by applying selected signal perturbations to source audio, querying the target VPS, and balancing unintelligibility against transcription success.

  • The Perturbation Engine accepts perturbation parameters and audio, then generates attack samples for any model under the black-box assumption.Each parameter set produces a unique attack audio sample.
  • Attack audio generation consists of parameter selection, sample generation, model inference, and audio selection.The workflow is summarized in Figure 3.
  • The four perturbation inputs include speed, high-frequency intensity or frequency, and window size, depending on the perturbation scheme.TDI and RPG use window size, HFA adds a frequency, and TS changes tempo.
  • Attack success requires trading off human decipherability against VPS recognition, because excessive distortion harms transcription while insufficient distortion remains understandable.The attacker queries the VPS to identify acceptable samples.
  • Perturbing words individually rather than an entire phrase enables concatenation of combinations, producing a much larger attack-sample set.The paper illustrates this strategy with “pay money” and Figure 4.

E. Over-the-Line and Over-the-Air

The paper evaluates attacks both through direct file submission and through speakers, emphasizing that practical over-the-air attacks must survive acoustic hardware and environmental degradation without model-specific knowledge.

  • Over-the-Line attacks submit attack audio directly as a .wav file, whereas Over-the-Air attacks play it through a speaker toward the target VPS.The two scenarios distinguish direct model input from physical acoustic transmission.
  • Over-the-Air transmission can degrade fine perturbations through frequency response, static interference, and environmental noise.An attack successful Over-the-Line therefore may fail Over-the-Air.
  • The proposed attacks avoid limitations of prior methods that required the generation speaker or knowledge of the victim’s acoustic hardware.The paper states that its attacks have no such limitations.
  • The evaluation targets a broad set of architectures, black-box APIs, and feature-extraction techniques using only acoustic properties rather than underlying model information.The set includes traditional HMM-GMM, neural, and online API systems.

V. EXPERIMENTAL SETUP

The experiments test perturbation attacks across speech-recognition models, proprietary APIs, direct submission, physical playback, and altered acoustic hardware and noise conditions.

  • The study tests attacks against ten ASRs to assess whether they work across model types and feature-extraction algorithms.The stated hypothesis predicts ASR-independent attack success.
  • The evaluation includes seven proprietary online models, including Google Speech, Bing Speech, IBM Speech, and Azure APIs.These models permitted only limited query counts.
  • Locally hosted models were evaluated with the improved attack method because they could be queried without limits.Proprietary models instead used the generic attack method.
  • For Over-the-Line testing, multiple parameterized samples were submitted as .wav files and the worst-sounding accepted sample was selected for each phrase and model.This process produced one attack file per phrase for each of the ten models.
  • Over-the-Air testing used a speaker and microphone arrangement, then repeated Google Client Speech API trials with an iMac, Motorola Nexus 6, and 55 dB white noise.The second setup was conducted in a lab cubicle to represent harsh realistic conditions.

C. Speaker Verification and Attestation

The paper extends hidden-audio attacks to speaker authentication by preserving voice information while perturbing audio, and evaluates separate speaker-identification and speaker-verification settings.

  • Voice biometrics authenticate commands by comparing audio against a voice blueprint of the presumed user.The paper presents this as an obstacle for prior audio-obfuscation attacks.
  • Speaker Identification models classify recordings among multiple trained speakers and are text-independent.They infer the speaker from arbitrary text read by the speaker.
  • Speaker Verification models decide whether audio belongs to one enrolled speaker and are text-dependent on a requested phrase.They check transcription before comparing the voice blueprint.
  • The speaker-identification experiment trained Azure Identification on eight male and eight female voices, then attacked three speakers from each group.Ten attack samples were generated per selected speaker.
  • The speaker-verification experiment trained Azure Verification with recordings from three men and three women before perturbing test audio.Each phrase had three training samples and one test sample per person.
  • The Perturbation Engine was tested for both direct .wav input and playback through a commodity speaker.These correspond to Over-the-Line and Over-the-Air evaluation modes.

A. Over-the-Line

The attacks successfully fooled speech recognition and speaker identification systems without requiring knowledge of their underlying models. Most generated samples remained effective after acoustic playback, although Kaldi-DNN initially failed on the most distorted samples.

  • Model Evaluation: Ten ASRs and speaker identification models were tested, with attack audio correctly transcribed or classified as the original speaker.The ASR evaluation ignored minor spelling mistakes, while speaker attacks required correct victim-speaker classification.
  • Model Evaluation: The attack required no information about an ASR’s underlying model or preprocessing steps.For each tested ASR, the authors produced correctly identified attack audio despite lacking model details.
  • Model Evaluation: Approximately 80% of 20 generated samples successfully fooled the model for each of four tested phrases.The most obfuscated samples used TDI and TS, and they tricked the model in nearly every Over-the-Air case.
  • Over-the-Air Robustness: Over-the-Air evaluation reused generated samples, and almost all phrases remained correctly transcribed after playback.The attack preserves acoustic properties important to recognition; small accuracy drops were attributed to environmental factors such as background noise.
  • Over-the-Air Robustness: Kaldi-DNN initially failed on all tested Over-the-Air attack samples, but less-perturbed samples generated with larger windows were transcribed correctly.The authors attribute the initial failure to insufficient noisy audio in the Fisher English training data and note that the model may be less effective in noisy environments.

C. General Insights

The results identify parameter settings that balance audible distortion and VPS recognition, while showing that black-box attacks can be generated with few queries and generalized across many models. TDI was more successful than RPG under model-agnostic conditions.

  • Parameter Effects: Window sizes below 1.00 ms produced maximum audible distortion but no correctly transcribed attack samples.If one window size succeeds, samples generated with larger windows also succeed, allowing attackers to search upward from 1.00 ms.
  • Parameter Effects: Attack audio can be tuned by combining perturbation parameters, but excessive distortion pushes samples beyond VPS recognition.Preliminary experiments found TS at 150%, TDI or RPG windows near 1.00 ms, and HFA sine waves above 8000 Hz produced ideal audio.
  • Attack Selection: Attackers can narrow approximately 20,000 generated samples to fewer than ten using parameter-based distortion criteria.For TDI and RPG, smaller windows indicated greater distortion; for HFA and TS, larger values indicated worse-sounding audio.
  • Perturbation Comparison: RPG was less successful than TDI because black-box attackers lack the target model’s FFT type and parameters.The authors argue that time-domain TDI avoids this dependence while still producing successful attacks.
  • Query Efficiency: The attack found working audio for proprietary models in fewer than ten queries.A suggested search generates ten samples from 1.00 ms with 0.50 ms increments, and at least one usually succeeds.
  • Generalization: The evaluation covered the largest number and variety of attacked models, using a feature-extraction-focused design intended to generalize across models.The authors tested real-world phrase varieties because evaluating every ASR phrase was intractable.

A. Improvements on Previous Work

The paper improves hidden voice attacks by making them harder for humans to interpret while avoiding the model-, hardware-, and domain-specific constraints of prior work. Its perturbations exploit psychoacoustics and signal-processing behavior to remain recognizable to VPSes.

  • Our perturbations use psychoacoustic principles to make attack audio difficult for humans to interpret.
  • Figure 6 compares STFTs of speech, Carlini’s perturbation, our perturbation, and white noise by showing frequency intensity over time.Lighter colors indicate higher intensity at a given frequency and time.
  • Our attack audio emphasizes higher-frequency components, whereas Carlini’s perturbed audio has greater intensity in lower frequencies.The paper connects this distribution to differences in human intelligibility.
  • Higher frequencies can be perceived as louder and draw attention away from frequencies normally associated with speech.The paper also discusses filtering and selective-attention effects during VPS preprocessing and human listening.
  • The paper argues that its audio better exhibits characteristics associated with unintelligibility than previous work.This comparison is supported by psychoacoustic literature and Figure 6 rather than a small-scale intelligibility study.
  • Unlike prior white-box attacks, our attack is black-box, model-agnostic, transferable across speaker systems, and takes seconds to generate.Carlini’s attack requires white-box knowledge, is limited to HMM-GMM ASRs and a particular speaker system, and can take upwards of 32 hours to generate.

B. Defenses

The paper evaluates several possible defenses, finding that VAD does not remove its attack audio, while preprocessing-based detection and VPS modifications may help but involve practical trade-offs.

  • 36 attack audio files were processed by the ITU-T G.729 VAD algorithm, and speech was accurately located in all cases.
  • The attack audio continues to work with or without VAD because it introduces no additional noise.By contrast, Carlini’s noisy audio may become harder to transcribe as VAD deployment increases.
  • Classifier-based attack detection is described as impractical and brittle because false positives could cause legitimate commands to be ignored.
  • Modifying or removing VPS preprocessing could hinder the attack, but may decrease VPS accuracy and must occur at or before preprocessing.The paper also reports effectiveness against Mozilla DeepSpeech, which does not use predefined signal processing.
  • A loudspeaker-detection system could identify low sub-bass frequencies and prevent attack audio from reaching the VPS.The passage notes that liveness detection may also be relatively robust, but these defenses could generate false positives.

C. Limitations

The paper identifies practical and evidentiary boundaries for hidden voice attacks, including reduced effectiveness in noisy environments and limitations in comparing intelligibility across studies.

  • Attack audio has reduced effectiveness in noisy environments because noise interferes with VPS interpretation.The paper notes that legitimate commands also degrade in noisy environments.
  • Prior attacks were effective only against particular models, speakers, and microphones, limiting their hardware independence and practical deployment.
  • The paper’s black-box approach works across speech detection and speaker recognition systems over the wire and over the air.It uses signal-processing feature vectors and psychoacoustics rather than attacking a specific underlying model.
Loading 1904.05734v1…