PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Authentication for Voice Assistants

Huan Feng, Kassem Fawaz, Kang G. Shin

arXiv:1701.04507v1cs.CR

TL;DR

Voice assistants need protection because open audio channels permit replay, impersonation, and other command-injection attacks. VAuth provides continuous authentication by matching wearable-collected body vibrations with microphone speech, achieving high accuracy and resisting practical attacks with low overhead.

  • Problem

    Open voice channels expose assistants to replay, impersonation, and mangled-voice attacks, while existing voice biometrics can collide on reduced-dimensional features.

  • Method

    VAuth continuously authenticates speech by matching body-surface vibrations collected by a wearable accelerometer with the voice assistant’s microphone signal.

  • Results

    More than 97% detection accuracy and close to 0 false positives were reported, with robustness across accents, mobility, and languages and successful blocking of replay, mangled-voice, and impersonation attacks.

  • Takeaways & Limitations

    VAuth provides usable continuous authentication without user-specific training and can be integrated into common wearable devices and existing voice-assistant platforms.

Abstract

from arXiv · show

Voice has become an increasingly popular User Interaction (UI) channel, mainly contributing to the ongoing trend of wearables, smart vehicles, and home automation systems. Voice assistants such as Siri, Google Now and Cortana, have become our everyday fixtures, especially in scenarios where touch interfaces are inconvenient or even dangerous to use, such as driving or exercising. Nevertheless, the open nature of the voice channel makes voice assistants difficult to secure and exposed to various attacks as demonstrated by security researchers. In this paper, we present VAuth, the first system that provides continuous and usable authentication for voice assistants. We design VAuth to fit in various widely-adopted wearable devices, such as eyeglasses, earphones/buds and necklaces, where it collects the body-surface vibrations of the user and matches it with the speech signal received by the voice assistant's microphone. VAuth guarantees that the voice assistant executes only the commands that originate from the voice of the owner. We have evaluated VAuth with 18 users and 30 voice commands and find it to achieve an almost perfect matching accuracy with less than 0.1% false positive rate, regardless of VAuth's position on the body and the user's language, accent or mobility. VAuth successfully thwarts different practical attacks, such as replayed attacks, mangled voice attacks, or impersonation attacks. It also has low energy and latency overheads and is compatible with most existing voice assistants.

I. INTRODUCTION

Voice assistants are useful but exposed to replay, impersonation, and mangled-voice attacks because voice is an open channel. VAuth addresses this gap with continuous wearable-based authentication and achieves high accuracy, broad robustness, attack resistance, and low overhead.

  • Motivation: Voice assistants support many convenient or essential tasks, but their open audio channel exposes them to replay, impersonation, and stealthy command-injection attacks.Potential consequences include information theft, financial loss, and unauthorized control of appliances or vehicles.
  • VAuth: VAuth continuously matches body-surface vibrations from a wearable accelerometer with microphone speech so the assistant executes only the owner’s commands.The design uses an instantaneous, segment-by-segment time-domain comparison and filters nonmatching microphone segments.
  • VAuth: VAuth provides ongoing authentication throughout a voice session, addressing the insufficiency of presession authentication when third parties can speak during access.Unlike passwords, PINs, patterns, and fingerprints, it does not assume exclusive device control after session establishment.
  • Usability: VAuth requires no user-specific training and remains usable across wearable positions, accents, mobility patterns, and languages.It can be incorporated into eyeglasses, earbuds, and necklaces touching facial, throat, or sternum areas.
  • Evaluation: More than 97% detection accuracy and close to 0 false positives were achieved across the evaluation, while attacks were blocked and overhead remained low.The reported average latency was 300ms, with recharging required only once a week.

III. BACKGROUND

The background describes speech as a source–filter process and motivates VAuth’s use of correlated microphone and body-vibration signals. It also defines the system architecture, threat scenarios, and deployment assumptions.

  • Speech production: Human speech is modeled as a vocal-fold source shaped by the vocal-tract filter into spectra associated with phonemes.The source–filter model separates excitation from resonant modulation by the tongue, lips, and vocal tract.
  • Speech production: Glottal-pulse spacing represents instantaneous fundamental frequency, whose human range is 80 to 333Hz with cycle lengths of 0.003sec to 0.0125sec.Pitch changes as speakers produce different phonemes.
  • Voice authentication: MFCC-based recognition captures invariant speech features but can be attacked with mangled segments that share the same feature representation as commands.Reduced-dimensional projections allow different signals to collide in the feature space.
  • System model: VAuth combines a wearable accelerometer on the chest, neck, or face with an extended assistant that verifies accelerometer and microphone signals before issuing commands.The system is intended to work with smartphone assistants, Alexa, and phone-based banking systems.
  • Threat model: The threat model includes stealthy wireless or mangled-voice injection, replay or impersonation, and acoustic injection targeting the accelerometer.The wearable is assumed to be a secure possession token, communications are encrypted, and attacks on that channel are outside scope.

A. High-Level Overview

VAuth combines a wearable accelerometer with a voice-assistant extension to continuously authenticate commands by correlating body vibrations with microphone speech. Its prototype supports common wearables and users generally accept its configurations, particularly after learning about voice-assistant security risks.

  • System design: VAuth uses a wearable component to collect accelerometer data and a voice-assistant extension to authenticate and launch commands.The extension correlates wearable and microphone signals and issues a command only when they match.
  • System design: The matching runs server-side, while Bluetooth connects the wearable component and voice-assistant extension.Server-side matching reduces processing burden on the user’s device.
  • Wearable scenarios: VAuth supports earbuds, eyeglasses, and necklaces, with the accelerometer positioned on the facial, throat, or sternum areas.The prototype supports three widely adopted wearable scenarios.
  • Security awareness: 71% (51%) of respondents considered voice-assistant attacks dangerous after priming, and 75%(52%) would take steps to mitigate them.The survey measured perceptions before and after respondents learned about voice-assistant attacks.
  • Usability: 70%(47%) of participants were willing to wear at least one VAuth configuration for security protection.These respondents were the majority of those strongly concerned about security threats.
  • Usability: 73% (81%) of respondents could accommodate weekly charging, while 60% (75%) could accommodate charging every five days.The authors report that VAuth’s energy consumption matches respondents’ requirements.

VI. MATCHING ALGORITHM

VAuth preprocesses accelerometer and microphone signals, identifies speech-related segments, and filters microphone content using body-vibration evidence. It then analyzes segments and applies correlation-based matching to retain only signals supported by the accelerometer.

  • Pipeline overview: VAuth’s matching algorithm takes speech and vibration signals with their sampling frequencies and outputs a match decision plus cleaned speech when matched.The pipeline has three stages: pre-processing, speech-segment analysis, and matching decision.
  • Pre-processing: VAuth high-pass filters accelerometer data at 100 Hz, resamples both signals to 8 kHz, and low-pass filters at 4 kHz to prevent aliasing.The high-pass filter removes low-frequency movement artifacts such as walking or breathing.
  • Pre-processing: VAuth normalizes signal magnitudes, removes spikes using an energy threshold, aligns signals by maximum cross-correlation, and truncates them to equal length.It avoids Dynamic Time Warping because timing information is critical to pitch and DTW requires higher processing load.
  • Pre-processing: VAuth identifies energy segments from the accelerometer signal by mapping portions exceeding 5% of maximum energy to one and other portions to zero.The threshold depends on the average noise level measured while the user is silent.
  • Pre-processing: The accelerometer energy envelope removes microphone segments without corresponding body vibrations, reducing background sounds that the user could not generate.This filtering supports security when the user is not actively speaking and can improve voice-recognition accuracy.

B. Per-Segment Analysis

VAuth cleans and analyzes speech and vibration signals segment by segment, rejecting segments that fail duration, pulse, frequency, or synchronization checks. It then uses normalized cross-correlation and an SVM to decide which microphone segments match and pass the cleaned signal onward.

  • Segment extraction and filtering: VAuth identifies high-energy accelerometer segments and matches corresponding accelerometer and microphone segments one at a time.The surviving segments form the final signals used for verification.
  • Segment extraction and filtering: Segments shorter than 20ms or lacking at least 20ms of identifiable glottal pulses are removed from both signals.The duration threshold corresponds to the approximate length of a single phoneme.
  • Segment extraction and filtering: VAuth removes segments with accelerometer glottal-cycle averages outside 0.003–0.0125 seconds or with over 25% relative pulse-distance mismatch.These checks target speech outside the 80–333Hz range and interfered speech with inconsistent pitch variation.
  • Matching decision: After filtering, normalized cross-correlation discards segments whose maximum coefficient lies in [-0.25,0.25].A matching example, segment s4, has a maximum coefficient of 0.52.
  • Matching decision: The classifier represents each correlation vector with 1001 uniformly sampled values centered on its maximum and uses a degree-1 polynomial-kernel SVM.The SVM is trained offline once using a user-agnostic phoneme dataset and then performs online classification.
  • Matching decision: VAuth filters non-matching microphone segments and passes only matching, cleaned segments to the voice assistant for speech recognition and execution.The matching algorithm compares complete accelerometer and microphone signals in the time domain before filtering the microphone signal.

VII. PHONETIC-LEVEL ANALYSIS

The phonetic-level analysis tests whether chest vibrations are detectable for English phonemes and whether they can support matching without replacing microphone speech recognition. Vibrations are measurable, but existing ASR generally cannot interpret the accelerometer signal directly.

  • Experimental setup: The evaluation studies true-positive phoneme matching and false-positive matching between accelerometer and microphone samples.Two speakers recorded 44 English phoneme examples for the phonetic-level study.
  • Accelerometer energy: All tested phonemes register chest vibrations, with the minimum relative energy of 14% for the male speaker’s OI phoneme.Relative energy was computed after preprocessing and normalization.
  • Accelerometer energy: Accelerometer vibration energy differs more between speakers than between vowels and consonants for the same speaker.The paper relates this difference to anatomical and body-composition factors affecting chest-skin proximity to the sternum.
  • Recognition analysis: Nuance ASR fails to identify the actual spoken words from cleaned accelerometer samples for about half of the phonemes.For most remaining samples it returns suggestions that do not match the spoken words, with matches in only three consonant cases.
  • Recognition analysis: Existing ASR engines cannot interpret the often low-fidelity accelerometer samples, whereas VAuth requires no changes to the ASR engine or voice assistant.The paper notes that retrofitting ASR could improve recognition but would require significant deployment and training changes.

B. Phonemes Detection Accuracy

VAuth evaluates phoneme-level matching by comparing accelerometer and microphone signals with normalized cross-correlation. Across phoneme classes and speakers, it achieves high true-positive matching and very low false-positive rates.

  • Phoneme results: More than 90% true-positive rate is achieved when matching consonant phonemes across speakers, corresponding to fewer than 10% false negatives.The evaluation matches each accelerometer phoneme sample against collected microphone samples.
  • Phoneme results: The false-positive rate is nearly zero when VAuth matches consonant phonemes across speakers.This indicates that VAuth rarely matches an accelerometer sample to an inappropriate microphone sample.
  • Phoneme results: VAuth distinguishes different phonemes from the same speaker and the same phoneme across speakers despite shared pitch or speaker-independent phonetic features.Comparable results are also reported for vowel-phoneme matching.
  • Overall matching: 93% true-positive rate and 0.6% false-positive rate are obtained when matching all accelerometer samples to all microphone samples.The true-positive count is 82 correctly matched accelerometer samples.
  • Silent-user evaluation: Zero false matches are observed when idle accelerometer segments are matched against recorded phonemes, including segments containing abrupt motion.The authors treat the recorded phonemes as representative of possible English sounds.
  • Matching model: VAuth matches accelerometer and microphone signals by segmenting them and evaluating normalized cross-correlation between corresponding signal segments.The binary decision is positive when the maximum normalized cross-correlation exceeds threshold th.

B. Per-segment Analysis

VAuth’s per-segment analysis filters mismatching or unsupported signal portions before forwarding audio, strengthening authentication against command injection.

  • Security mechanism: Per-segment analysis prevents attackers from injecting commands by requiring matching signal segments rather than relying on aggregate signal similarity.Without this analysis, matching segments could mask injected non-matching segments.
  • Security mechanism: VAuth removes portions of the microphone signal without corresponding accelerometer evidence and normalizes detected signal magnitude after removing spikes.These mechanisms address injection while the user is silent or while signal energy contains disturbances.

C. False Positive Rate

The analysis predicts that command-level false positives decrease as commands contain more non-matching phonemes. Under the stated boundedness and threshold assumptions, the decrease is exponential in command length.

  • Command length: Command-level false positives decrease as more phonemes are added to a command.The paper states that longer commands contain more sounds and therefore have lower false-positive rates.
  • False-positive model: A false positive occurs when non-matching accelerometer and microphone signals nevertheless produce a matching result above threshold th.At the phonetic level, this corresponds to a non-matching pair with m_i,j > th.
  • False-positive model: When segment energies are roughly uniform, the final matching value is an average of the segment-level matching values.The per-segment analysis is used to ensure this approximate uniformity.
  • Theoretical result: The false-positive rate of a command composed of n non-matching phonemes decays exponentially fast in n under the model’s assumptions.The assumptions are that matching values are bounded in [0, 1] and threshold th exceeds their mean.
  • Empirical confirmation: User-study results further confirm the theoretical prediction that longer commands reduce false-positive rates.The empirical conclusion follows the formal analysis of the command-level false-positive probability.

IX. EVALUATION

The evaluation studies VAuth with 18 users speaking 30 commands across wearable positions, mobility conditions, and languages. It reports near-perfect detection accuracy, low false positives, and robustness while moving and across languages.

  • Evaluation scope: The evaluation examines accuracy across body positions, accents, mobility patterns, and languages, alongside security properties and prototype overheads.The study also reports delay and energy consumption as evaluation dimensions.
  • User study: The user study evaluates 18 users speaking 30 commands with VAuth positioned as eyeglasses, earbuds, or a necklace, both while still and jogging.Commands were issued to Google Now and evaluated using true-positive and false-positive matching.
  • Still condition: More than 97% average detection accuracy and 0.09% average false-positive rate are measured in the still condition.Most commands are authenticated on the first trial, while few signals leak through authentication.
  • Still condition: Two necklace-wearable outliers are associated with significant voice-energy dips that did not provide enough energy for authentication.Eyeglasses and earbuds achieve almost perfect performance in the reported scenarios.
  • Mobility: VAuth’s matching accuracy remains unaffected while jogging because the algorithm filters disturbances from movement and breathing.For two previous outliers, jogging produced higher accelerometer and microphone energy after filtering.
  • Language: VAuth achieves almost perfect detection accuracy across Arabic, Chinese, Korean, and Persian despite being trained on English phonemes, with one reported exception.The evaluation uses translated versions of the 30 commands and native speakers at the three wearable positions.

B. Security Properties

VAuth protects voice assistants against unauthorized commands from silent or speaking users, including mangled-voice and replay attacks. Its protection relies on matching microphone and accelerometer signals, with proximity-based limits for induced vibrations.

  • Security Analysis: VAuth’s multi-stage matching algorithm is evaluated against attack scenarios involving silent and actively speaking users.
  • Silent User: VAuth prevents unauthorized access when the user is silent, reporting a zero false positive rate across English phonemes.
  • Silent User: Beyond 30cm, even the loudest tested sound cannot induce detectable accelerometer vibrations in exposed or cotton-covered VAuth.For the other tested sound levels, the exposed-device cut-off distance was 5cm.
  • Speaking User: 99% of reconstructed mangled-voice comparisons failed to match the corresponding accelerometer signal.Only a handful of matches occurred among 3240 comparisons, and only with 30 Mel filter bands when the reconstruction closely resembled the original.
  • Speaking User: VAuth differentiates the same phoneme or command across different users and rejects replayed clips of the speaking user.

C. Delay and Energy

VAuth adds delay during remote matching and consumes energy through wearable transmission. The prototype supports command-dependent policies and demonstrates robustness across accents, speech speed, mobility, and languages.

  • Delay: A successful VAuth match takes 300–830ms, with an average of 364ms; an unsuccessful match takes 230–760ms.The delay occurs during accelerometer and microphone upload and the wait for a remote response.
  • Energy: A typical 500mAh Li-Ion wearable battery powers the prototype for around a week under the stated usage assumption.The estimate assumes daytime operation, 100 voice commands per day, and 10 seconds per command; average consumption is 6.3mA.
  • Policy: VAuth can apply customized authentication policies, allowing less-sensitive commands to bypass additional authentication while protecting sensitive appliance controls.
  • Usability: Evaluation shows robustness to changes in accents, speech speed, mobility, and language without user-specific, scenario-dependent training.
  • Conclusion: The prototype evaluation reports high accuracy and a very low false positive rate under practical settings.

APPENDIX A

The appendix provides reference materials for the evaluation: English phonetics, evaluated voice commands, and their source references.

  • Reference Materials: The appendix includes an IPA chart of English phonetics with examples.
  • Reference Materials: The appendix lists the voice commands used to evaluate VAuth.
  • Reference Materials: The phonetics chart and command list are attributed to external reference sources.
Loading 1701.04507v1…