PaperlayerView exact revision on arXiv

Source-linked AI summary

Removing Speech, Keeping Activities: A Privacy Firewall for Acoustic Sensing in Assisted Living

Pavlos Nicolaou, Christos Efstratiou

arXiv:2609.02376v1cs.SDcs.CRcs.HC

TL;DR

Speech privacy is a central barrier to deploying acoustic sensing for assisted living, especially because in-home labelled speech data are difficult to collect ethically. The paper uses a synthetic-data-trained U-Net privacy firewall before VGGish/SVM activity recognition. Across benchmark and limited real-world evaluations, it reports complete VAD-detectable speech removal while largely preserving or improving activity recognition.

  • Problem

    Speech privacy concerns and the difficulty of collecting labelled in-home recordings constrain acoustic sensing for assisted living.

  • Method

    A U-Net encoder-decoder trained on synthetic speech-background mixtures removes speech from ambient audio before VGGish transfer learning with an SVM performs activity recognition.

  • Results

    0% VAD-detectable speech remained under all tested conditions, while activity recognition was preserved or improved across ESC-50, SINS, and AudioHive recordings.

  • Takeaways & Limitations

    The pipeline supports privacy-preserving acoustic sensing while retaining environmental information for downstream activity recognition.

  • Takeaways & Limitations

    Results use a single 70/30 split, rely on Silero VAD as a privacy proxy, and evaluate ESC-50 and SINS in-distribution.

Abstract

from arXiv · show

Acoustic sensing offers a promising non-intrusive approach for monitoring daily activities of older adults, yet speech privacy concerns remain a critical barrier to real-world deployment. We present a privacy firewall pipeline based on a U-Net encoder-decoder, trained entirely on synthetic data, that removes speech from ambient audio while preserving environmental sounds indicative of daily activities. Activity recognition is performed using VGGish transfer learning with an SVM classifier. Evaluated on the ESC-50 and SINS datasets across multiple speech content levels, the proposed model reduced residual speech to 0% VAD-detectable speech (Silero Voice Activity Detection) under all tested conditions, outperforming Facebook Denoiser (6.55% residual), SepFormer (36.34%) and ConvTasNet (47.21%) on ESC-50 at the 100\% speech level. On ESC-50 at 40% speech level, classification performance recovers to 85% precision and 85% recall after speech removal, compared with 81%/75% before removal and an 84%/83% speech-free baseline. Evaluation on real-world participant home recordings collected with the AudioHive app showed 0% VAD-detectable speech after processing while maintaining 76% precision and recall. The pipeline enables privacy-preserving acoustic sensing without sacrificing activity recognition performance, addressing a key obstacle to the adoption of ambient monitoring in elderly care.

1. Introduction

Acoustic sensing supports unobtrusive activity monitoring in elderly care, but recording private conversations threatens deployment. The paper introduces a synthetic-data-trained privacy firewall that removes speech while retaining activity-relevant environmental sounds.

  • Motivation: Acoustic sensing can monitor daily routines unobtrusively and support detection of health issues, accidents, and assistance needs.It is passive, cost-effective, and captures contextual environmental information without wearables or extensive infrastructure.
  • Motivation: Privacy concerns about recording conversations were the most important issue raised by care-home occupants and carers during real-world deployment.The paper also identifies costly dataset collection and labelling as a practical challenge for acoustic sensing.
  • Approach: The proposed privacy firewall removes human speech before activity processing while preserving sound features needed to detect environmental activities.Its speech-removal component is a convolutional U-Net encoder-decoder operating on spectrogram representations.
  • Evaluation: 0% VAD-detectable speech remained across all tested speech sound levels, while activity recognition improved overall and exceeded alternative systems.Evaluation covered curated synthetic datasets and limited real-world datasets, including author-collected recordings.
  • Approach: The U-Net speech-removal model is trained entirely on synthetic mixtures, avoiding privacy-sensitive labelled in-home speech data.Synthetic scenarios combine publicly available activity sounds with publicly available speech datasets.

2. Related Work

Related work establishes acoustic activity recognition and source separation as useful but imperfect foundations for domestic sensing. The paper positions its approach as removing speech while preserving the full environmental signal for standard downstream analysis.

  • Acoustic sensing: Acoustic sensing has supported industrial monitoring, healthcare applications, and smart-city tasks through detection of informative environmental sounds.Examples include machinery anomaly detection, respiratory monitoring, fall detection, sleep assessment, traffic monitoring, and public safety.
  • Acoustic sensing: Real-world domestic activity datasets remain scarce because of privacy concerns, difficult ground-truth labelling, and variability in home environments.Consequently, research often relies on controlled studies or synthetic datasets that may not fully capture real-world complexity.
  • Source separation: Source-separation research primarily targets multiple-speaker mixtures, limiting its direct applicability to domestic audio containing diverse environmental sounds.WSJ0-2mix remains a standard benchmark, while newer datasets add noise and reverberation.
  • Speech enhancement: Speech enhancement conventionally removes background noise to preserve speech, whereas this paper reverses the preservation objective.The related computational structure includes encoder-decoder mask estimation, skip connections, and long-range modelling.
  • Activity recognition: Transfer learning addresses limited labelled data and domain mismatch by adapting robust audio features from large-scale corpora to target environments.The approach supports activity classifiers built with relatively few labelled examples.
  • Speech privacy: Prior privacy approaches restrict audible capture, degrade audio, or constrain downstream features, while this approach removes only speech and preserves the full environmental signal.This preserves compatibility with standard downstream audio analyses on the speech-free output.

3. Motivation

Real-world acoustic sensing is constrained by short deployment windows, scarce labelled data, and privacy concerns about captured conversations. The proposed requirements therefore combine speech removal, activity-recognition accuracy, limited-data operation, and adaptable deployment.

  • 3.1.1. Real Life Deployment: 12-month deployments across three care homes exposed practical barriers to real-world acoustic monitoring.These included constrained data collection windows, labelling costs, and occupant and carer privacy concerns.
  • 3.1.1. Real Life Deployment: Short deployment windows limit the time available to collect data before acoustic sensing becomes operational.Operational care environments may allow only approximately one month of raw data collection.
  • 3.1.1. Real Life Deployment: Uncurated real-world recordings require costly and time-consuming annotation, restricting accurately labelled datasets to small scale.Cameras and human observers are avoided because they raise privacy concerns and increase deployment costs.
  • 3.1.1. Real Life Deployment: Switching off audio capture when speech is detected protects privacy but can miss activity sounds when background radio speech triggers the detector.The paper motivates speech removal that preserves environmental audio instead of disabling capture.
  • 3.2. Requirements: The system is required to remove speech while preserving relevant environmental features for accurate acoustic activity recognition.Additional requirements include transfer learning for limited labels, short data collection periods, minimal retraining, and adaptation to new environments.

4. Datasets

The study combines public environmental-sound datasets, speech data, and deployment recordings to represent household audio and evaluate activity sensing. SINS supplies unrestricted real-world household activities, while AudioHive collects participant-labelled home recordings in two speech conditions.

  • 4.1. Public Datasets: Background audio comes from ESC-50, SINS, and the authors’ collection, while LibriSpeech supplies speech for synthetic mixtures.The background recordings represent environmental home sounds; speech data are combined with them to construct datasets.
  • 4.1. Public Datasets: ESC-50 contains 2,000 labelled environmental recordings across 50 classes, with 5-second clips resampled to 16 kHz for this work.Each class contains 40 examples.
  • 4.1. Public Datasets: LibriSpeech contributes approximately 1,000 hours of 16 kHz speech, from which one 30-minute chapter was selected from each of 20 books.The selection provides speech from 20 speakers for the study’s synthetic data requirement.
  • 4.2.1. Sound INterfacing through the Swarm (SINS) Dataset: SINS records unrestricted daily activities in a household across one week, covering 16 labelled activities in five rooms.The deployment includes absences, visitors, computer work, and transitions between activities.
  • 4.2.2. Data Collection: AudioHive collected participant-labelled home audio in two phases: one prohibiting speech and another allowing speech during activities.The first phase involved 10 participants recording morning routines for five days; the second involved 12 adults recording for seven days.

5. Experimental Methodology

The methodology combines audio capture, a privacy firewall, and performance analysis to remove speech while retaining background sounds for acoustic activity classification. It evaluates a VGGish/SVM activity classifier and Silero VAD using real-world and public datasets, including synthetic speech mixtures.

  • 5.1. Overview: The pipeline captures audio, applies the Privacy Firewall, and analyzes privacy and activity-recognition performance.The three stages are audio capture, speech protection, and performance analysis.
  • 5.1. Overview: The experiments assess activity recognition, speech impact, and feasibility of removing speech while retaining background activity sounds.These tasks establish a baseline, measure degradation after speech insertion, and evaluate speech removal.
  • 5.1. Overview: Speech was mixed with background audio at 100%, 80%, 60%, and 40% relative amplitude to represent different real acoustic conditions.The lower levels attenuate speech relative to the 100% condition by approximately 1.9, 4.4, and 8.0 dB, respectively.
  • 5.1. Overview: The study uses SINS real-deployment data and ESC-50 environmental sounds, with controlled mixture experiments interpreted as in-distribution privacy-utility evaluations.The methodology combines a home-environment dataset with a public benchmark and synthetic speech-background mixtures.
  • 5.2.1. Baseline: VGGish transfer learning extracts audio embeddings, which an SVM classifier uses for acoustic activity recognition when labelled target-environment data are limited.VGGish processes spectrogram patches into 128-dimensional embeddings before downstream classification.
  • 5.2.2. Results: Baseline AAC achieved 84% precision and 90% recall on SINS and 84% precision and 83% recall on ESC-50, while Silero VAD detected 0.2% and 0.6% speech, respectively.These baselines provide reference points for measuring speech impact and speech-removal performance.

5.3. Stage 2: The Impact of Speech on Acoustic Activity Recognition

Stage 2 adds synthetic speech to SINS and ESC-50 background recordings at four sound levels, then measures its effects on acoustic activity classification and voice detection. Speech substantially degrades classification at full volume, with lower speech levels producing better outcomes.

  • 5.3. Stage 2: The Impact of Speech on Acoustic Activity Recognition: Synthetic speech was overlaid on SINS and ESC-50 recordings at 100%, 80%, 60%, and 40% sound levels, producing eight datasets.The same AAC and VAD pipeline was applied to each speech-enriched variant.
  • 5.3. Stage 2: The Impact of Speech on Acoustic Activity Recognition: At 100% speech on SINS, AAC fell to 50% precision and 51% recall, while VAD detected speech in 49% of audio.The SINS speech-free baseline was 84% precision, 90% recall, and 0.2% VAD-detected speech.
  • 5.3. Stage 2: The Impact of Speech on Acoustic Activity Recognition: At 100% speech on ESC-50, AAC fell to 70% precision and 69% recall, while VAD detected speech in 67.5% of audio.The ESC-50 speech-free baseline was 84% precision, 83% recall, and 0.6% VAD-detected speech.
  • 5.3. Stage 2: The Impact of Speech on Acoustic Activity Recognition: Lower speech volume improved acoustic activity classification and reduced the percentage of audio in which voice activity was detected.This pattern was observed across the speech sound levels tested.
  • 5.3. Stage 2: The Impact of Speech on Acoustic Activity Recognition: The results motivate applying speech removal to synthetic mixtures while keeping background audio intact to improve classification and protect conversational privacy.Stage 3 evaluates whether suppressing speech can recover activity-recognition utility.

6. Privacy Firewall: A Speech Removal Framework

The privacy firewall removes speech from mixed ambient audio while preserving environmental sounds for acoustic activity recognition. Its U-Net spectrogram model is evaluated against public baselines using VAD and AAC metrics.

  • Privacy firewall pipeline: The pipeline removes speech before activity analysis while preserving background acoustic features.Speech removal is followed by VGGish-based activity classification and Silero VAD privacy evaluation.
  • Baseline comparison: Public speech enhancement and separation models served as informative baselines rather than direct competitors for speech removal with environmental-sound preservation.The evaluated models included Facebook Denoiser, SepFormer, and ConvTasNet on ESC-50 mixtures with speech at 100% sound level.
  • Baseline comparison: 67.5% to 6.55%: Facebook Denoiser reduced VAD-detected speech but lowered AAC precision and recall to 67% and 66%.The unprocessed speech-mixture dataset achieved 70% precision and 69% recall, while the speech-free baseline achieved 84% and 83%.
  • Model architecture: A U-Net encoder-decoder maps mixed speech-background spectrograms to estimated background-only spectrograms.The model uses symmetric skip connections and operates on 96×64×1 log-mel spectrograms.
  • Proposed model results: 85% precision and 85% recall were reached on ESC-50 at 40% speech after removal, matching the speech-free baseline of 84%/83%.At 100% speech, performance recovered to 73% precision and 81% recall, with 0% VAD-detected speech.
  • Proposed model results: 0% VAD-detectable speech was achieved across ESC-50, SINS, and synthetic alternative datasets after proposed speech removal.The result indicates no residual speech was detected by Silero VAD, but unintelligibility was left for future validation.

7. Evaluation Results and Discussion

Evaluation covers synthetic mixtures and participant home recordings, with synthetic results representing in-distribution performance and real recordings providing evidence under unseen conditions. The proposed pipeline suppresses VAD-detectable speech while generally improving or preserving activity classification, subject to important validation limits.

  • Comparative results: The proposed model enhanced AAC relative to speech-contaminated synthetic datasets, narrowed the gap to speech-free baselines, and outperformed the tested off-the-shelf models.The speech-free baseline was not fully matched in all cases.
  • Evaluation scope: Synthetic ESC-50 and SINS evaluations use the same source pools and mixing procedure as training, making them in-distribution results.A stratified 70/30 split was used.
  • Evaluation scope: Real-world data collections contain recordings outside the synthetic training recipe and therefore provide out-of-distribution evidence of generalisation.The collections were recorded by participants using the AudioHive App.
  • Real-world evaluation: 6.8% VAD-detected speech in the Second Data Collection was mitigated while the pipeline classified acoustic activities.The authors describe the result as robust to minor speech interference.
  • Comparative results: 0% residual VAD-detected speech was achieved compared with 6.55% for Facebook Denoiser, 36.34% for SepFormer, and 47.21% for ConvTasNet on ESC-50 at 100% speech.AAC performance improved over speech-contaminated input, although precision and recall declined as speech interference increased.
  • Limitations: The reported results lack cross-validation and confidence intervals, and VAD alone does not establish residual speech absence or unintelligibility.The real-world test set contained only 6.8% VAD-detected speech, limiting stress tests under heavy speech conditions.

8. Conclusions

The paper presents a U-Net-based privacy firewall trained on synthetic mixtures to remove speech from ambient audio while preserving activity-relevant environmental sounds. Across ESC-50, SINS, and real-world recordings, it reports zero VAD-detectable residual speech and retained or improved activity recognition, while acknowledging evaluation limitations.

  • 8. Conclusions: The pipeline combines a synthetic-data-trained U-Net speech remover with a VGGish-based SVM activity recognizer.The speech-removal stage targets speech in ambient audio while preserving environmental sounds.
  • 8. Conclusions: 0% VAD-detectable speech was achieved across tested ESC-50 and SINS speech-contamination levels, with 85% precision and 85% recall on ESC-50 at 40% speech.The ESC-50 result compares with 81%/75% before removal and an 84%/83% speech-free baseline; SINS improved from 59%/58% to 71%/77%.
  • 8. Conclusions: On ESC-50 at 100% speech, residual speech was 0% for the proposed model versus 6.55% for Facebook Denoiser, 36.34% for SepFormer, and 47.21% for ConvTasNet.These are the three public baselines evaluated in that condition.
  • 8. Conclusions: Real-world AudioHive home recordings showed 0% VAD-detectable speech after processing while maintaining 76% precision and recall.The recordings came from participant homes and were used as a limited real-world evaluation.
  • 8. Conclusions: The study identifies in-distribution synthetic evaluations and VAD detection as an operational privacy proxy among its limitations.Planned extensions include ASR intelligibility and signal-level evaluations, human listening studies, held-out speech sources, and care-home deployment.
  • 8. Conclusions: The modular pipeline is intended to integrate into existing acoustic sensing systems with minimal modification, addressing privacy-related deployment barriers in elderly care.The stated motivation is increasing acceptance and adoption of ambient monitoring.

Ethics and informed consent

The AudioHive participant study received ethics approval and obtained written informed consent. Participants were informed about withdrawal rights and audio-data handling, while SINS was used under its original authors’ ethical approvals and access terms.

  • Ethics and informed consent: The AudioHive study was reviewed and approved by the University of Kent’s Central Research Ethics Advisory Group.The study involved human participants.
  • Ethics and informed consent: All AudioHive participants provided written informed consent before participation.Participants were informed of their right to withdraw.
  • Ethics and informed consent: Participants were informed how their audio data would be stored, processed, and anonymised.The SINS dataset was collected under its original authors’ ethical approvals and used according to its access terms.

Funding and acknowledgements

The research was informed by the ADAPTIVE project, which funded and supported collaboration among academic, healthcare, civic, and industry partners. The authors acknowledge those partners and the study participants.

  • Funding and acknowledgements: The research was informed by experience from the ADAPTIVE project on non-invasive elderly care technology.ADAPTIVE stands for AI-based Dementia Assistive and Passive Technology for non-Invasive Elderly care.
  • Funding and acknowledgements: ADAPTIVE was funded by UK Research and Innovation under grant reference 68239.
  • Funding and acknowledgements: The project involved the University of Kent, MiiCare, East Kent Hospitals University NHS Foundation Trust, and Bristol City Council.The authors thank the project partners and study participants.

Data availability

ESC-50 and LibriSpeech are publicly available from their original sources, while SINS is available under its authors’ access terms. AudioHive home recordings cannot be shared publicly because they may contain privacy-sensitive acoustic information.

  • Data availability: ESC-50 and LibriSpeech are publicly available from their original sources.
  • Data availability: SINS is available from its original authors under their terms of access.
  • Data availability: AudioHive home-environment recordings cannot be shared publicly because they may contain privacy-sensitive acoustic information.This restriction is consistent with the study’s ethics approval and informed-consent conditions.

CRediT authorship contribution statement

Pavlos Nicolaou contributed across the paper’s conceptual, technical, analytical, and writing activities. Christos Efstratiou contributed to conceptualization, resources, supervision, administration, funding, and review.

  • Pavlos Nicolaou handled conceptualization, methodology, software, validation, formal analysis, investigation, data curation, visualization, and writing.
  • Both authors contributed to conceptualization, methodology, and writing activities.
  • Christos Efstratiou contributed conceptualization, methodology, resources, supervision, project administration, funding acquisition, and writing review.
Loading 2609.02376v1…