PaperlayerView exact revision on arXiv

Source-linked AI summary

FeatureSense: Protecting Speaker Attributes in Always-On Audio Sensing System

Bhawana Chhaglani, Sarmistha Sarna Gomasta, Yuvraj Agarwal, Jeremy Gummeson, Prashant Shenoy

arXiv:2505.24115v1cs.SDcs.HCeess.AS

TL;DR

Always-on audio can expose sensitive information beyond speech, including speaker attributes. FeatureSense uses curated audio features and adaptive selection to balance privacy, utility, and cost; it reports 60.6% less speaker leakage while maintaining classification effectiveness.

  • Problem

    Always-on audio can expose sensitive personal information beyond speech, while existing privacy metrics such as WER focus on speech recognition.

    Source: §1 · pp. 2–3

  • Method

    FeatureSense exposes curated privacy-aware audio features instead of raw audio and adaptively selects task-specific subsets to balance privacy, utility, and cost.

    Source: §1 · pp. 2–3, §6.2 · p. 18

  • Results

    60.6% less speaker leakage than existing privacy techniques is reported while maintaining audio-classification effectiveness.

    Source: §1 · p. 3, §11 · p. 31

  • Takeaways & Limitations

    FeatureSense provides a feature-layer approach for privacy-aware audio classification and adaptive feature lists for context-aware applications.

    Source: §11 · pp. 30–31

    Speech leakage evaluation uses formant correlation rather than human intelligibility or ASR tests because granular features do not provide access to audio or spectrograms.

    Source: §9 · p. 28

Abstract

from arXiv · show

Audio is a rich sensing modality that is useful for a variety of human activity recognition tasks. However, the ubiquitous nature of smartphones and smart speakers with always-on microphones has led to numerous privacy concerns and a lack of trust in deploying these audio-based sensing systems. This paper addresses this critical challenge of preserving user privacy when using audio for sensing applications while maintaining utility. While prior work focuses primarily on protecting recoverable speech content, we show that sensitive speaker-specific attributes such as age and gender can still be inferred after masking speech and propose a comprehensive privacy evaluation framework to assess this speaker attribute leakage. We design and implement FeatureSense, an open-source library that provides a set of generalizable privacy-aware audio features that can be used for wide range of sensing applications. We present an adaptive task-specific feature selection algorithm that optimizes the privacy-utility-cost trade-off based on the application requirements. Through our extensive evaluation, we demonstrate the high utility of FeatureSense across a diverse set of sensing tasks. Our system outperforms existing privacy techniques by 60.6% in preserving user-specific privacy. This work provides a foundational framework for ensuring trust in audio sensing by enabling effective privacy-aware audio classification systems.

1 INTRODUCTION

FeatureSense broadens audio privacy beyond speech content to include speaker-attribute leakage. It offers curated audio features and adaptive selection to balance privacy, utility, and computational cost.

1 INTRODUCTION

  • SILI evaluates speaker leakage, including age, gender, and ethnicity, addressing privacy risks that can persist beyond speech suppression.The approach uses derived audio features rather than raw audio for non-speech sensing tasks.Source: §1 · pp. 2–3
  • FeatureSense exposes curated audio features instead of raw audio for non-speech, speaker-invariant classification tasks.The library includes time, spectral, perceptual, voice-specific, statistical, and derived features.Source: §1 · pp. 2, 4
  • 60.6% less speaker leakage is reported alongside 81.2% ESC-50, 85.3% AudioSet, and 97.1% UrbanSounds8k accuracy.The comparison is against existing privacy techniques; the accuracy figures correspond to the named datasets.Source: §1 · pp. 3–4
  • 10 ms of feature processing for 500 ms of audio is reported, supporting real-time use with minimal computational overhead.The paper also describes the system as suitable for always-on sensing on edge devices.Source: §1 · p. 3
  • Adaptive task-specific feature selection balances privacy, utility, and computational cost according to application requirements and device constraints.The algorithm aims to reduce manual feature engineering and domain-knowledge requirements.Source: §1 · pp. 3–4

2 BACKGROUND

Always-on audio sensing can expose private information and speaker attributes, while many privacy techniques focus on suppressing speech. The section reviews audio representations and identifies speaker-attribute leakage as a distinct evaluation gap.

2.1 Privacy Implications of Audio Sensing

  • Always-on audio can capture private conversations, health-related sounds, and background cues about location or activities, while voice analysis can reveal identity, emotion, stress, and health conditions.These risks motivate preventing sensitive information from leaking through audio data.Source: §2.1 · p. 5

2.2 Privacy-Preserving Techniques For Audio

  • Existing audio privacy methods commonly suppress or remove speech, but this does not address speaker-related information that may remain in the signal.The section describes techniques including subsampling, filtering, speech obfuscation, and speech-segment removal; speech intelligibility and ASR measure speech leakage, not remaining speaker attributes.Source: §2.2 · p. 5

2.3 Common Audio Feature Representations

  • High-dimensional audio representations can retain speech, prosody, and speaker information, whereas task-relevant low-dimensional features aim to limit privacy risk and computational overhead.Prior task-specific privacy-aware features require manual engineering; the authors propose generalizable features for varied audio classes.Source: §2.3 · pp. 5–6

2.4 Speaker Leakage

  • Speaker leakage includes demographic information such as age, gender, and ethnicity, but no standard metrics evaluate attribute leakage.Speech intelligibility and ASR assess speech leakage, while speaker-identification measures such as EER assess speaker distinction rather than attribute disclosure.Source: §2.4 · p. 6

3 MOTIVATION: WHY PREVENTING SPEAKER DEMOGRAPHIC LEAKAGE MATTERS

Audio sensing can expose speaker traits even when applications need only event detection, creating risks of profiling, tracking, and privacy violations. The section outlines example scenarios, a passive-feature-access threat model, and intended applications for FeatureSense.

3 MOTIVATION: WHY PREVENTING SPEAKER DEMOGRAPHIC LEAKAGE MATTERS

  • For ambient sound detection, pet activity tracking, and human activity recognition, detecting events such as sirens or coughing is useful, while inferring age, gender, or identity is unnecessary.FeatureSense is described as supporting on-device sensing while minimizing speaker-related privacy risks.Source: §3 · p. 6

3.1 Example Attack Scenarios

  • Cough monitoring, smart speakers, and indoor microphone sensors illustrate risks ranging from demographic profiling to cross-sensor tracking of speaker movement.The indoor-sensor scenario describes how aggregating leaked attributes across sensors and time could theoretically disambiguate speakers and reconstruct movement patterns.Source: §3.1 · p. 7

3.2 Real-World Incidents of Audio-based Speech and Speaker Leakage

  • An accidental disclosure of more than 1,700 household recordings, regulatory actions over children’s voice recordings and location data, and multi-million-dollar settlements illustrate real-world audio privacy incidents.The section presents these incidents as involving speech content or speaker attributes.Source: §3.2 · p. 7

3.3 Attack Overview and Threat Model

  • The threat model gives a passive adversary access only to extracted features, while assuming secure local feature selection, encrypted transmission, and no interference with the sensing pipeline.The adversary may use pretrained models, external datasets, and auxiliary knowledge to infer demographics or reconstruct identity across sessions or locations.Source: §3.3 · p. 8

3.4 FeatureSense Intended Use-cases

  • Intended use cases include always-on systems that detect non-verbal events such as coughs, yawns, or glass breaking without needing raw speech or speaker demographics.FeatureSense is proposed for integration at the device-driver layer to preserve sensing utility while protecting against speaker and demographic leakage.Source: §3.4 · p. 8

4 PRIVACY VULNERABILITIES IN EXISTING PRIVACY TECHNIQUES

The evaluation finds that speech leakage and speaker leakage are distinct, and that existing privacy filters can retain speaker-specific acoustic and demographic information. It assesses both leakage types and introduces separate indices for them.

4 PRIVACY VULNERABILITIES IN EXISTING PRIVACY TECHNIQUES

  • The assessment measures speech and speaker leakage separately because audio can reveal sensitive attributes beyond intelligible speech, including age, gender, and ethnicity.The evaluation uses speech-recognition measures for words and phonemes and assesses demographic-attribute leakage after privacy processing.Source: §4 · pp. 8–9
  • Filtered spectrograms retain spectral and temporal information, including formants, harmonics, pitch, and prosodic patterns that can expose speaker traits.The passage identifies retained pitch in PrivacyMic and partly in Synthetic Sensors, and prosodic patterns in SAMoSA and Synthetic Sensors.Source: §4 · p. 10

4.1 Speech Leakage

  • 9.5% eSTOI marks Kirigami’s lowest speech leakage, but some speech structures remain perceptible; SAMoSA and Synthetic Sensors still leak phonemes despite mostly near-100% WER.PrivacyMic has high PER and low intelligibility; lower WER and PER indicate more speech leakage, while higher eSTOI indicates more leakage.Source: §4.1 · pp. 10–11

4.2 Feasibility of Retaining Speaker-Specific Features

  • 0.969 spectral-envelope, 0.938 formant, and 0.971 MFCC similarity persist under Synthetic Sensors, and no tested filter fully eliminates speaker-identifying features.Kirigami has the lowest pitch similarity (0.360) but retains spectral-envelope, formant, and MFCC similarity.Source: §4.2 · p. 11

4.3 Speaker Demographic Leakage

  • Raw audio yields the highest demographic-prediction accuracy; Kirigami has maximum leakage at the tested threshold, while PrivacyMic has relatively low leakage.SAMoSA and Synthetic Sensors also retain high speaker-attribute leakage; the results cover gender, accent, and age.Source: §4.3 · pp. 11–12

4.4 Assessing Speech and Speaker Leakage: SILI and CSLI

  • SILI averages normalized speaker-attribute accuracy, while CSLI combines WER, PER, and eSTOI to quantify speech leakage relative to raw audio.SILI uses equal attribute weights by default; higher SILI and CSLI values indicate greater leakage.Source: §4.4 · pp. 12–13
  • Kirigami has the lowest speech leakage (CSLI) but the highest speaker leakage (SILI), showing that the two measures capture different risks.PrivacyMic also has lower CSLI but high SILI; SAMoSA and Synthetic Sensors leak substantial speech and show similar speaker leakage to PrivacyMic.Source: §4.4 · p. 13

5 FEATURESENSE OVERVIEW

FeatureSense replaces raw audio with privacy-aware features for downstream non-speech sensing and offers tunable privacy-utility-cost trade-offs. Its overview distinguishes feature groups by their utility and potential to expose speech or speaker attributes, while identifying leakage, latency, and domain expertise as design challenges.

5.1 Overall Approach

  • FeatureSense exposes extracted features rather than raw audio and allows feature subsets to be tuned for application-specific privacy, utility, and cost trade-offs.The approach targets non-speech, speaker-invariant classification tasks such as environmental sound classification.Source: §5.1 · pp. 13–14

5.2 Overview of Audio Feature Groups and Their Privacy Properties

  • Audio features span eight groups, with phonetic and linguistic features often privacy-invasive, voice-specific features in a gray zone, and many statistical or high-level features generally less invasive.The taxonomy also flags timbre as potentially invasive and chroma features as a gray zone.Source: Table 4 · p. 14, §5.2 · pp. 14–15
  • Derived features capture sound properties such as frequency-band energy and transient-to-sustained ratios to help distinguish environmental sound types.Examples include low-, mid-, and high-band energy, low-to-high energy ratio, spectral roughness, and spectral texture.Source: §5.2 · pp. 15–16

5.3 Challenges in Feature-Based Audio Classification

  • Feature-based sensing still faces speaker-attribute leakage, feature-extraction latency, and the need for domain expertise; FeatureSense uses adaptive selection to address task-specific trade-offs.Different applications may require different balances of privacy, utility, and latency.Source: §5.3 · p. 16

6 FEATURESENSE DESIGN

FeatureSense removes features associated with speaker leakage and provides a configurable library of privacy-aware features for non-speech sensing. Its task-specific optimization selects features by balancing application utility and privacy leakage under a latency constraint.

6.1 Feature Selection to Prevent Speaker Leakage

  • FeatureSense assesses individual features for speaker leakage and removes Chroma and Timbre after they show high values across leakage metrics.The analysis uses correlation, mutual information, random-forest importance, and permutation importance; the authors also assess age and accent.Source: §6.1 · p. 17, Figure 11 · p. 17

6.2 FeatureSense Library: Privacy-Aware Audio Features for Non-speech Speaker-invariant Sensing

  • The library exposes a curated set of granular, interpretable features instead of raw audio, excluding phonetic and linguistic features known to leak speech or speaker information.Its features cover time-domain, spectral, perceptual, voice-specific, statistical, time-frequency, and derived groups, with APIs for feature selection and privacy, utility, and cost metrics.Source: §6.2 · pp. 17–18

6.3 Task Specific Feature Selection

  • The feature-selection algorithm uses application-specific utility scores, privacy-leakage scores, and extraction latency to optimize a feature subset under a latency limit.Utility is based on ESC-50 feature importance; privacy scores use age, gender, and accent classifiers trained on Common Voice.Source: §6.3 · pp. 18–19
  • A single parameter α controls the utility-versus-privacy objective, while the latency constraint limits total feature-extraction time.Increasing α prioritizes utility; decreasing α prioritizes privacy leakage reduction.Source: §6.3 · p. 19
  • The framework is intended to let practitioners select features systematically for utility-driven, privacy-sensitive, or latency-constrained applications without requiring domain knowledge.The selected subset is tailored to the application's requirements.Source: §6.3 · p. 19

7 FEATURESENSE IMPLEMENTATION

FeatureSense implements separate feature-extraction and task-specific selection modules, with APIs for obtaining features, querying their metrics, and optimizing subsets. The optimizer accepts application and constraint settings and returns a feature subset for real-time extraction.

Feature extraction

  • The feature-extraction module accepts a microphone device ID and returns audio features from overlapping windows, with configurable window size and sampling rate.Defaults are 500 ms and 16 kHz; users can request selected features, and the implementation uses SciPy, NumPy, and custom methods for advanced features.Source: §7 · p. 20

Task-specific feature selection

  • The optimizer uses linear programming to select features from precomputed rankings according to application category, latency limit, and privacy-utility trade-off.The default settings are α=0.5, T=0.1, and interior sounds; FeatureSense is currently a wrapper, with OS- or hardware-level deployment planned.Source: §7 · pp. 20–21

Library APIs

  • FeatureSense APIs expose real-time feature extraction, per-feature utility, privacy, and cost metrics, and task-specific optimization.The optimization API accepts sound category, latency constraint, and trade-off parameter α, then returns a selected feature subset.Source: §7 · pp. 20–21

8 EXPERIMENTAL EVALUATION

On ESC-50, FeatureSense combines strong classification utility with lower speaker-attribute leakage than the compared privacy techniques. The evaluation also reports low speech leakage under its feature set, while documenting the datasets, metrics, and window-size trade-offs used to assess the system.

8.1 Experimental Setup

  • The evaluation uses Common Voice for speaker attributes, ESC-50 for utility, and MI with the first three formants for speech leakage.It also reports latency in milliseconds and evaluates real-world performance using a Raspberry Pi 5 prototype with USB microphones.Source: §8.1 · pp. 21–22

8.2 Utility Evaluation

  • FeatureSense achieved 81.18% average accuracy, 80% average precision, and 80% average recall on ESC-50 using a Random Forest classifier.The reported category metrics were above 75% for most categories, with slightly lower performance on Exterior/Urban sounds.Source: Figure 13 · p. 22, §8.2 · p. 22
  • FeatureSense had the highest ESC-50 utility among the compared privacy techniques, while other techniques ranged from 71.4% to 74.9% aggregate accuracy.PrivacyMic had the least utility; FeatureSense was slightly above SAMoSA.Source: Figure 14 · p. 22, §8.2 · p. 23
  • A 500 ms window balanced ESC-50 accuracy and gender leakage, yielding 81.2% task accuracy; 200 ms reached 86.9% accuracy but leaked more, while 1 second reduced leakage by 15.7% and yielded 77.39% accuracy.The authors used 500 ms throughout the paper and identify joint window-size optimization as future work.Source: §8.2 · p. 23

8.3.1 Speaker Privacy Leakage.

  • 60.6% average improvement in speaker privacy over other techniques, with FeatureSense reporting the lowest speaker leakage and high utility.The comparison reports that other techniques can retain speaker leakage despite suppressing or removing audio information.Source: Figure 17 · p. 24, Figure 18 · p. 24, §8.3.1 · p. 24
  • After removing Chroma and Timbre, speaker-attribute accuracy was 30% for gender, 40% for age, and 31% for ethnicity, near 33% random guessing for three-class classifiers.Gender accuracy fell from 48.6% to 30% after the correlated features were removed.Source: §8.3.1 · pp. 23–24, Figure 16 · p. 23

8.3.2 Speech Privacy Leakage.

  • Mutual information between extracted features and the first three formants remained below 0.22, which the authors interpret as indicating no significant speech-information leakage in the analyzed feature set.Spectral Spread, Tonality Index, and Spectral Contrast had somewhat higher values than other features.Source: §8.3.2 · p. 24

8.4 Computational Cost Evaluation

Feature extraction was optimized with lower-overhead implementations, and the paper profiles its cost on both a laptop and a Raspberry Pi. On the Pi, all features together take under 40 ms per 500 ms window.

Computational Latency of Feature Extraction

  • 1.2 ms per window for librosa RMS extraction fell to 0.031 ms with numpy, a 97.4% reduction.The optimized implementations use direct numpy and scipy operations in place of librosa overhead.Source: §8.4 · p. 25, Figure 20 · p. 25
  • 81.4% lower computational cost was achieved for some features using approximations, including Hilbert-transform-based reverberation and HNR calculations.Other optimizations vectorized operations that had previously used Python loops.Source: §8.4 · p. 25

Raspberry Pi Evaluation

  • 38.56 ms on Raspberry Pi 5 versus 9.02 ms on the laptop, while total extraction remains below 40 ms per 500 ms window.The Pi measurement is 4.27x slower; the authors note that parallel processing could reduce latency by 50%–75%, depending on workload and core count.Source: §8.4 · pp. 25–26, Figure 21 · p. 26

8.5 Privacy-Utility-Cost Trade-offs across Feature Groups

Feature groups differ in privacy leakage, task utility, and computational cost, so the preferred selection depends on application priorities. Two case studies report configurable privacy-utility trade-offs for cough detection and smart-speaker sound classification.

8.5 Privacy-Utility-Cost Trade-offs across Feature Groups

  • Derived features show high utility with the least privacy leakage, while high-utility feature groups can carry greater leakage or computational cost.Statistical and time-domain features are described as more privacy-preserving; statistical features also offer a reasonable cost-utility trade-off for low-power systems.Source: §8.5 · p. 26

8.6 Case Study 1: Privacy-Aware Cough Detection for Respiratory Health Apps

  • At α=1.0, cough classification retains 98.96% utility and ethnicity leakage falls by 27.2% versus raw audio.At α=0.3, utility retention is 92.17% and ethnicity leakage is reduced by 41.6%; total latency remains under 15 ms across settings.Source: §8.6 · p. 27, Table 7 · p. 27

8.7 Case Study 2: Target Sound Detection with Smart Speaker Privacy Mode

  • At α=0.3, smart-speaker sound classification retains 93.07% utility while reducing age leakage by over 51.4%.Across α=1.0 to 0.3, feature-extraction latency falls from 8.9 ms to 1.48 ms, with total latency under 15 ms across settings.Source: Table 8 · p. 27, §8.7 · p. 28

9 DISCUSSION

The discussion identifies limits to privacy measurement and feature coverage, alongside residual speaker leakage and task-scope constraints. It also describes possible extensions to the framework and deployment approach.

Contextual Privacy

  • Audio context may still reveal location, surroundings, or activities, so the authors describe comprehensive privacy as an ongoing challenge.Examples include background sounds and temporal or sequential audio patterns.Source: §9 · p. 28

Feature List Expansion

  • FeatureSense’s feature list is non-exhaustive, and the authors identify additional derived features and automatic feature generation as future directions.The proposed library is modular and intended to accommodate additional features.Source: §9 · p. 28

Assessing Speech Leakage of Granular Features

  • Granular-feature speech leakage is difficult to evaluate without the audio signal or spectrogram; the study uses formant correlation rather than human intelligibility or ASR-based tests.The authors propose GAN-based speech synthesis as a direction for evaluating candidate audio signals.Source: §9 · p. 28

Extensive Utility Assessment

  • The authors report high utility across several sensing tasks but say the approach might not work for speech recognition, speaker recognition, or emotion detection.They identify further evaluation in domains such as speech/non-speech detection and voice pathology detection as future work.Source: §9 · p. 28

Residual Speaker Leakage and Adversarial Risks

  • FeatureSense may reduce but not eliminate identifiable traits, leaving potential for sophisticated adversarial de-anonymization attacks.The authors describe adversarial training and differential privacy as possible additions to strengthen protections.Source: §9 · pp. 28–29

Adaptive Feature Granularity

  • The current system uses fixed-length 500 ms windows; the authors suggest window duration could become an optimization parameter.Larger windows may suit some persistent or periodic environmental sound tasks, according to the discussion.Source: §9 · p. 29

10 RELATED WORK

Related work spans audio privacy methods and feature-based sensing, but the discussion identifies limited attention to demographic speaker leakage and privacy-aware feature selection. FeatureSense is positioned as a software feature-extraction-stage alternative.

Audio Privacy Techniques

  • Prior audio privacy methods suppress speech through techniques such as subsampling, filtering, segment removal, obfuscation, and replacement.The cited work includes voice activity detection and classifiers for identifying likely speech or voiced segments.Source: §10 · p. 29

Preventing Speaker Attributes Leakage

  • The related-work discussion identifies limited research on preserving and assessing demographic speaker-attribute leakage, distinct from speech-content or identity protection.It notes prior work on emotion masking [9], speaker biometrics, and gender inference from speech emotion recognition systems [? ].Source: §10 · p. 29

Different levels of privacy

  • FeatureSense operates at feature extraction as a software-based alternative to hardware- or model-layer privacy approaches, without requiring new hardware or complex deep-learning models.The authors describe it as modular for integration into sensing pipelines.Source: §10 · pp. 29–30, Table 9 · p. 30

Feature-Based approaches

  • The paper says many feature-based sensing systems do not explicitly assess speaker demographic or speech-content leakage, and feature design often requires domain expertise.FeatureSense is presented as an adaptive framework for balancing privacy, utility, and computational cost.Source: §10 · p. 30

11 CONCLUSION

FeatureSense combines privacy evaluation with privacy-aware audio features and adaptive feature selection. Its approach reduces speaker-related privacy leakage while maintaining audio-sensing effectiveness.

11 CONCLUSION

  • FeatureSense provides a privacy evaluation metric and privacy-aware audio features intended to improve audio-classification accuracy without requiring domain knowledge.Source: §11 · p. 30
  • The library extracts its features in real time and includes adaptive, task-specific feature selection based on application requirements.The resulting adaptive feature list can be used for context-aware applications.Source: §11 · pp. 30–31
  • 60.6%: FeatureSense reduces speaker-related privacy leakage while maintaining the effectiveness of audio-based applications.Its optimization algorithm penalizes privacy-invasive features for extremely sensitive applications.Source: §11 · p. 31

A APPENDIX

The appendix examines how feature pairs contribute to sensitive-attribute leakage, finding that combinations can be more predictive than individual features. It also describes making feature-importance rankings for age, gender, and ethnicity available through FeatureSense.

Privacy Leakage of Individual Features and their Combinations

  • 6.6x: Low Band Enegy + Wavelet Features exhibited higher permutation importance than individual features, highlighting leakage from feature combinations.Timbre and Chroma Features and Spectral Contrast had relatively high predictive power individually.Source: A APPENDIX · p. 34
  • Feature pairs can expose sensitive-attribute patterns even when their individual features do not leak much information.Timbre and Chroma features had high individual permutation-importance scores, but their combination was not the best-performing pair.Source: A APPENDIX · p. 34
  • FeatureSense provides feature-importance rankings from Random Forest classification models for age, gender, and ethnicity.Source: A APPENDIX · p. 34
Loading 2505.24115v1…