Source-linked AI summary
Your Microphone Array Retains Your Identity: A Robust Voice Liveness Detection System for Smart Speakers
Yan Meng, Jiachun Li, Matthew Pillari, Arjun Deopujari, Liam Brennan, Hafsah Shamsie, Haojin Zhu, Yuan Tian
TL;DR
Smart speakers remain vulnerable to voice spoofing, while passive liveness detection can degrade under environmental changes and user movement. The paper proposes array fingerprint and the ARRAYID system, which use the smart speaker’s microphone array for device-free detection. ARRAYID achieves 99.84% authentication accuracy on the authors’ dataset and outperforms existing passive schemes across evaluated datasets and conditions.
Problem
Existing passive liveness detection schemes can degrade under environmental changes and require fixed user gestures, while alternative defenses may impose additional device or interaction burdens.
Method
ARRAYID uses a circular microphone array, array fingerprint, theoretical sound-propagation analysis, and complementary audio features for passive voice-liveness detection.
Results
99.84% authentication accuracy was achieved on the authors’ dataset, exceeding the best existing-scheme result of 98.81%; ARRAYID also outperformed existing schemes on a third-party dataset.
Takeaways & Limitations
The array fingerprint broadens passive liveness detection to changing environments and user movement without requiring extra devices or actions.
Takeaways & Limitations
Automatically continuous retraining can introduce potential risks, including poisoning attacks that reduce speech-recognition and speaker-verification performance.
Abstract
from arXiv · showhide
Though playing an essential role in smart home systems, smart speakers are vulnerable to voice spoofing attacks. Passive liveness detection, which utilizes only the collected audio rather than the deployed sensors to distinguish between live-human and replayed voices, has drawn increasing attention. However, it faces the challenge of performance degradation under the different environmental factors as well as the strict requirement of the fixed user gestures. In this study, we propose a novel liveness feature, array fingerprint, which utilizes the microphone array inherently adopted by the smart speaker to determine the identity of collected audios. Our theoretical analysis demonstrates that by leveraging the circular layout of microphones, compared with existing schemes, array fingerprint achieves a more robust performance under the environmental change and user's movement. Then, to leverage such a fingerprint, we propose ARRAYID, a lightweight passive detection scheme, and elaborate a series of features working together with array fingerprint. Our evaluation on the dataset containing 32,780 audio samples and 14 spoofing devices shows that ARRAYID achieves an accuracy of 99.84%, which is superior to existing passive liveness detection schemes.
1 Introduction
Smart speakers face voice-spoofing risks, while existing liveness defenses trade off usability, environmental robustness, or user freedom of movement. ARRAYID addresses these challenges with microphone-array fingerprints and achieves high accuracy across datasets and conditions.
- Motivation: Voice spoofing can inject malicious commands into smart speakers, creating safety and privacy risks.Attacks include replay, ultrasound-based, adversarial, and user-impersonation methods.
- Motivation: Existing defenses either burden users with specialized sensors or probe signals, or rely on passive audio features vulnerable to propagation changes and fixed gestures.Mono-audio schemes can be affected by sound-channel changes and spectrum-modulation attacks, while fieldprint methods require stable microphone geometry.
- Results: 99.84% authentication accuracy was achieved on the authors’ dataset, compared with 98.81% for the best existing schemes.On the third-party ReMasc Core dataset, ARRAYID reached 97.78% versus 84.37% for existing schemes.
- Approach: ARRAYID uses microphone-array audio and requires neither an additional user device nor an extra user action.The approach leverages the multiple microphone locations and mutual distances already present in mainstream smart speakers.
- Approach: Array fingerprint is a robust liveness feature designed to address environmental change and user movement through relationships among microphone channels.The paper analyzes sound propagation theoretically and uses multi-channel audio to eliminate distortions from environmental factors.
- Results: ARRAYID was evaluated across distance, direction, spoofing-device, and noise factors, and a 14-device microphone-array dataset was released for further evaluation.The paper reports superiority to existing schemes on both its dataset and a third-party dataset.
2 Preliminaries
Voice spoofing threatens smart speakers, while existing liveness detectors trade off user burden, environmental robustness, or positional freedom. The section introduces attack types, sound-propagation foundations, and limitations of mono-channel and fieldprint-based detection.
- 2.1 Threat Model: Replay attacks reuse legitimate recordings through loudspeakers, while advanced attacks exploit synthesized audio, adversarial noises, or ultrasounds.The threat model assumes attackers possess victim audio and can remotely control an audio device; the study mainly targets replay attacks.
- 2.2 Sound Generation and Propagation: Sound generation and transmission are modeled as source modulation followed by air propagation and microphone-array capture.Human and loudspeaker sources use distinct modulation gains, while each microphone receives the propagated signal according to distance and air-channel gain.
- 2.2 Liveness Detection: Existing liveness detection divides into multi-factor authentication and passive schemes that exploit physical differences between human and machine-generated voices.Multi-factor methods add sensors or probe signals, whereas passive methods use collected audio alone.
- 2.3.1 Mono Channel-based Detection: Mono-channel methods use sub-bass or voice-spectrum features, but propagation changes and spectrum-modulation attacks degrade their reliability.Authentic and spoofed spectra can differ in sub-bass frequencies, yet both change substantially across rooms.
- 2.3.2 Fieldprint-based Detection: Fieldprint detection compares two microphone signals to infer source identity from the surrounding sound field.CAFIELD defines Field = log(y1(f,t) y2(f,t)) from two received audio channels.
- 2.3.2 Fieldprint-based Detection: CAFIELD requires a stable source-to-sensor position and performs poorly at distances greater than 40 cm, limiting use across a room.Its fixed close-to-face smartphone posture conflicts with walking, gesture changes, and typical home-speaker interaction.
3 Array Fingerprint
This section develops array fingerprint as a microphone-array feature for passive liveness detection, addressing environmental and positional distortions that weaken existing approaches. Theoretical analysis, simulation, and case studies show that array fingerprint is source-related, stable under propagation changes, and distinctive across human and spoofed audio.
- 3.1 Theoretical Analysis on Sound Propagation for Smart Speakers: Mono-channel schemes suffer nonlinear, source-dependent distortion when the relative microphone-source distance changes, while fieldprint becomes infeasible when microphone positions vary.Fieldprint assumes a stable microphone-pair distance difference, making it sensitive to user or microphone movement.
- 3.2 Advantage of Array Fingerprint: Definition and Simulation-based Demonstration: Array fingerprint is proposed as a robust liveness feature that uses relationships among microphone-array channels to reduce propagation-factor effects.The feature is motivated by the circular microphone layout in smart speakers.
- 3.2 Advantage of Array Fingerprint: Definition and Simulation-based Demonstration: For N = 6, σd averages 3.38 cm with a range of 7.9×10^-4 cm, whereas N = 2 produces σd values from 0 to 7.07 cm as propagation conditions change.With more than four microphones, σd converges to an approximately constant value in the simulation.
- 3.2 Advantage of Array Fingerprint: Definition and Simulation-based Demonstration: Array fingerprint is mainly related to source audio and is therefore resilient to environmental changes, especially changes in propagation distance.The authors connect this robustness to the near-constant σd obtained with typical circular microphone arrays.
- 3.3 Validation of Array Fingerprint: Across two locations, array channels show distinct audio signals, while replay experiments find similar fingerprints for the same source and different fingerprints for different sources.The replay devices used in the case study were a smartphone and an iPad.
4 The Design of ARRAYID
ARRAYID combines a microphone-array fingerprint with spectrogram-distribution and channel-LPCC features to perform lightweight passive liveness detection. Its pipeline collects multichannel audio, preprocesses spectrograms, extracts complementary features, and detects attacks.
- System Overview: ARRAYID comprises Data Collection, Pre-processing, Feature Extraction, and Attack Detection modules.The system is based on the proposed array fingerprint and auxiliary features.
- Data Pre-processing: The system converts each channel’s audio into a spectrogram, retains frequencies below 5 kHz, and aggregates spectrogram magnitudes into grid matrices.With Fs = 48 kHz and Nfft = 4096, the retained spectrogram has Mspec = 426 frequency elements; grids use MG = 100 and NG = 20.
- Spectrogram Array Feature: ARRAYID computes the spectrogram array fingerprint FG from corresponding grid values across microphones, then averages it across time to retain common components.Time variation is attributed to different phonemes and articulatory gestures, while long-duration averaging produces the FSAP feature.
- Spectrogram Array Feature: The FSAP feature is resampled to NSAP = 40 points, and experiments show similar fingerprints across commands but clear differences between authentic and spoofing audio.The demonstration uses three commands and distances of 0.6 m and 1.2 m.
- Complementary Features: Beyond FSAP, ARRAYID extracts FSDP from channel frequency strengths and FLPC from channel-specific LPCCs, forming X = [FSAP,FSDP,FLPC].FSDP uses averaged channel strengths plus their mean and standard deviation; LPCCs are retained from two selected microphones with order p = 15.
5 Evaluations
ARRAYID is evaluated on multichannel recordings from human participants and 14 replay devices across datasets, users, distances, directions, channel counts, enrollment settings, noise levels, and advanced attacks. It achieves strong overall and cross-dataset performance, while unseen users and strong noise remain important boundaries.
- 5.1 Experiment Setup: The evaluation uses Matrix Creator and Seeed ReSpeaker boards, 14 electrical spoofing devices, and 20 participants producing authentic and replayed commands.The boards sample at 48 kHz; authentic recordings vary across four distances without fixed speaking speed or tone.
- 5.2 Performance of ARRAYID: 99.84% accuracy and 0.17% EER are achieved on 32,780 samples, with 0.05% FAR and 0.39% FRR.Only 13 of 22,539 spoofing samples are falsely accepted, while 40 of 10,241 authentic samples are falsely rejected.
- 5.3 Impact of Various Factors on ARRAYID: 99.78%, 99.82%, and 99.90% accuracy are obtained with 4, 6, and 8 channels, respectively, indicating little performance change with channel count.Cross-distance testing remains above 99.4% in all cases, although performance degrades when the source location changes.
- 5.3 Impact of Various Factors on ARRAYID: 92.97% overall accuracy is obtained for unseen users, declining from 99.84%; 11 users still exceed 95% accuracy, while partial enrollment reaches 99.68%.The paper identifies unseen-user degradation as an open liveness-detection problem; partial enrollment uses authentic samples without spoofing samples.
- 5.4 Robustness Evaluation: Noise increasing from 45 dB to 65 dB reduces accuracy from 98.8% to 86.3%, while modulated-attack accuracy ranges from 92.74% to 100% across devices.ARRAYID detects 100% of the tested hidden voice and VMask attack audios, but strong-noise degradation is left for future work.
6 Discussions
The discussion examines ARRAYID’s enrollment burden, user-classification capability, and robustness across training-data sizes and spoofing devices. It also identifies poisoning risks and degradation from strong noise or other speakers.
- User Enrollment Time in Training: 99.14% to 99.84%: ARRAYID’s detection performance increases as the training proportion rises from 10% to 50%.With 10% training data, ARRAYID achieves 99.14% accuracy and 0.96% EER.
- User Enrollment Time in Training: Less than 3 minutes: enrollment requires an average of 51 samples per user when using 10% of the training data.The estimate is based on 10,241 authentic samples from 20 users and voice commands shorter than 3 seconds.
- Distinguish between Different Users: 99.88%: ARRAYID achieves overall speaker recognition accuracy across authentic samples from 20 users.Accuracy across individual users ranges from 98.5% to 100%, and t-SNE shows visually clustered feature vectors for five users.
- Limitations and Countermeasures: Automatic continuous retraining may expose ARRAYID to poisoning attacks, while strong noise or another speaker’s voice inevitably degrades performance.The enrollment evaluation assumes a silent environment without another talking user.
7 Related Works
Related work addresses smart-speaker spoofing through replay, covert physical-media, adversarial-example, Doppler, gesture, biometric, wireless-signal, and sonar-based approaches.
- Attacks on smart speakers: Replay attacks remain a major threat, while ultrasonic and laser media can spoof voice assistants without the user’s perception.Subtle noises can also generate adversarial-example attacks against smart speakers.
- Countermeasures: Prior defenses use Doppler effects, sound-based gestures, biometric characteristics, wireless signals, or sonar-based angle-of-arrival estimation.These approaches target voice-spoofing security through additional physical or signal-based cues.
8 Conclusion
The paper concludes that ARRAYID enables passive voice-spoofing detection without extra devices by using a microphone-array fingerprint and complementary features. Experiments on the MALD and public datasets show superiority to existing passive schemes and robustness across evaluated factors.
- Conclusion: ARRAYID performs passive voice-spoofing detection without requiring extra devices.The system uses the proposed array fingerprint together with a series of complementary features.
- Conclusion: Array fingerprint is proposed as a liveness feature for improving effectiveness and broadening passive liveness-detection application scenarios.The paper theoretically analyzes existing popular passive liveness-detection schemes before introducing the feature.
- Conclusion: ARRAYID is evaluated on the MALD dataset and another public dataset, with results reported as superior to existing passive liveness-detection schemes.The evaluation also examines multiple factors to demonstrate robustness.
A LPCC Generation Process
The LPCC generation process first obtains linear-prediction coefficients for each microphone-channel signal, then derives LPCC coefficients of order 15. Opposite microphone channels are selected as FLPC for a later feature stage.
- LPC calculation: For each microphone signal y_k(t), the process calculates Linear Prediction Coding with order p = 15 before deriving LPCC features.The LPC coefficients are represented as a = [a0,a1,...,ap].
- LPCC calculation: The LPCC vector c = [c0,c1,...,cp] uses c0 = ln(p), while the remaining coefficients are calculated from the LPC representation.The passage introduces the recurrence for the other LPCC elements.
- FLPC selection: In a six-channel array, the microphone opposite the closest microphone M1 is M4, and their LPCCs are selected as FLPC.The selected opposite-channel LPCCs are used in Section 4.3.3.
B Dataset Descriptions
The dataset uses Matrix Creator and Seeed ReSpeaker core V2 microphone arrays, with spoofing-device, collection-condition, and voice-command details summarized across the experiments.
- Collection setup: The dataset was collected using Matrix Creator and Seeed ReSpeaker core V2 microphone arrays.These devices are shown in Figure 17.
- Dataset organization: Dataset descriptions summarize spoofing devices, collection distances, and audio samples for each user.These details are organized in Tables 6 and 7.
- Voice commands: The experiments use 20 voice commands spanning device control, information queries, communication, and translation.The listed commands include turning on Bluetooth, setting alarms, checking schedules, making calls, and translating phrases.
C Experimental Details of Comparison with Existing Schemes
ARRAYID is compared with VOID under multiple classifier models and on a third-party dataset, with multi-channel features outperforming the mono-channel baseline in the reported evaluations.
- Classifier comparison: ARRAYID outperforms VOID under every tested classifier model.The models are neural network, SVM-RBF, kNN, and decision tree; the results are reported in Table 8.
- Classifier comparison: 98.81% is VOID’s best accuracy on the MALD dataset when using SVM-RBF.This matches the accuracy reported for VOID in its original paper.
- Third-party evaluation: 97.78% versus 84.37% is the reported accuracy comparison between ARRAYID and VOID on the third-party ReMasc Core dataset.ARRAYID achieves the higher value in this comparison.
- Overall comparison: The evaluation concludes that exploiting multi-channel features achieves superior liveness-detection performance compared with the mono-channel scheme.The comparison follows the procedures described for VOID and CAFIELD.
D Details of Modulated Attacks
The modulated-attack evaluation applies device-specific inverse filters to audio collected by a Respeaker microphone array and generates attack samples for three spoofing devices.
- Attack setup: The modulated-attack reimplementation uses iPad 9, Mi phone 9, and Amazon Echo Plus as spoofing devices.The target device is a Respeaker microphone array.
- Inverse filtering: Device-specific inverse filters are calculated following the procedure described for modulated attacks.The frequency responses and inverse filters are shown in Figure 18.
- Attack sample generation: 1,990, 1,791, and 1,994 modulated-attack samples are generated for Echo, iPad, and Mi, respectively.The inverse filters are applied to audio collected by the target device before sample generation.