PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-Based Activities of Daily Living (ADL) Recognition with Large-Scale Acoustic Embeddings from Online Videos

Dawei Liang, Edison Thomaz

arXiv:1810.08691v2cs.HCcs.LGcs.SDeess.AS

TL;DR

Prior work suggests inertial sensors are not effective for recognizing complex, multidimensional activities [p04]. The paper presents a scheme for recognizing activities of daily living in the home [p01], reporting promising performance and robustness to environmental variability [p03].

  • Problem

    Prior work suggests inertial sensors are not effective at recognizing complex, multidimensional activities.

    Source: §1 · p. 1

  • Method

    The paper presents a novel scheme for recognizing activities of daily living in the home.

    Source: §1 · p. 1

  • Results

    The method yielded promising performance for activities and was robust to environmental variability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework achieved promising performance and robustness to environmental variability in different test scenarios.

    Source: §6 · p. 16

    The framework failed to recognize almost all brushing activities.

    Source: §5.2 · p. 15

Abstract

from arXiv · show

Over the years, activity sensing and recognition has been shown to play a key enabling role in a wide range of applications, from sustainability and human-computer interaction to health care. While many recognition tasks have traditionally employed inertial sensors, acoustic-based methods offer the benefit of capturing rich contextual information, which can be useful when discriminating complex activities. Given the emergence of deep learning techniques and leveraging new, large-scaled multi-media datasets, this paper revisits the opportunity of training audio-based classifiers without the onerous and time-consuming task of annotating audio data. We propose a framework for audio-based activity recognition that makes use of millions of embedding features from public online video sound clips. Based on the combination of oversampling and deep learning approaches, our framework does not require further feature processing or outliers filtering as in prior work. We evaluated our approach in the context of Activities of Daily Living (ADL) by recognizing 15 everyday activities with 14 participants in their own homes, achieving 64.2% and 83.6% averaged within-subject accuracy in terms of top-1 and top-3 classification respectively. Individual class performance was also examined in the paper to further study the co-occurrence characteristics of the activities and the robustness of the framework.

1 INTRODUCTION

The paper investigates whether large-scale online video audio embeddings can support home activity recognition without collecting labeled audio from users. It combines oversampling and deep learning and reports promising performance and robustness to environmental variability.

1 INTRODUCTION

  • Audio may help distinguish complex activities because daily activities generate characteristic sounds capturable with an off-the-shelf microphone.The introduction contrasts this potential with prior work suggesting inertial sensors alone are not effective for complex, multidimensional activities.Source: §1 · p. 1
  • The study uses millions of embeddings from general-sourced YouTube videos as its only training set instead of collecting ground-truth audio and labels from users.The paper frames this as a feasibility study for recognizing activities of daily living in the home.Source: §1 · pp. 1–2
  • To address highly unbalanced online data, the method combines both oversampling and deep learning.Source: §1 · p. 2
  • The framework was evaluated with 14 subjects in their homes across 15 activities and yielded promising performance with environmental robustness.The introduction describes its performance as promising and robust to environmental variability.Source: §1 · p. 2

2 RELATED WORK

Prior activity-recognition work spans inertial and audio sensing, but faces limits in generalization, data collection, and scaling. This paper situates online audio as a potential source for activity recognition and reports promising performance without feature augmentation or semi-supervised learning.

2.1 Inertial Sensing

  • Inertial sensing can face training-set and resource constraints, subject- and location-sensitive generalization, and limited activity-class availability.Prior work often focused on simple activities involving limited sensor types; combining modalities may improve class availability.Source: §2.1 · p. 2

2.2 Audio Sensing

  • Audio systems span mobile ambient-sound applications and home sensing, including bathroom activity detection using MFCC features.The cited bathroom work targeted six common activities.Source: §2.2 · pp. 2–3

2.3 Audio-Based Activity Recognition with Online Data

  • Prior audio recognition often required laborious user-specific ground-truth collection, and prior online-data work still relied on user data or manual outlier filtering.The paper describes those requirements as difficult to scale across multiple activity classes.Source: §2.3 · p. 3
  • The authors identify adapting the large scale of online audio samples to real-world activity recognition as an underexplored challenge, compounded by ambiguous video sound sources.The source videos may range from movies and cartoons to crowd-sourced material.Source: §2.3 · p. 4
  • The paper examines individual activity recognition using online video sound clips alone and reports promising performance without feature augmentation or semi-supervised learning.The authors contrast this with prior work focused mainly on environmental-context classification, where individual activity-class statistics remained largely unexplored.Source: §2.3 · p. 4

3 IMPLEMENTATION

The implementation uses Audio Set/VGGish embeddings to train classifiers for 15 home activities, covering label association, oversampling, segmentation, preprocessing, and training. It describes a custom convolutional network built on fixed VGGish features.

3.1 Audio Set

  • The dataset contains audio soundtracks drawn from general YouTube videos, with 527 class labels and 10-second clips; each clip has ten roughly 1-Hz embeddings, released as 128 PCA coefficients.Embeddings were PCA-whitened before release and quantized to 8 bits per embedding element.Source: §3.1 · p. 4

3.2 Label Association

  • The authors selected 15 home activities and associated them with 18 Audio Set labels, using related objects or contexts when exact label matches were unavailable.The authors describe this indirect matching as subjective, with no quantified measure of similarity between labels and activities.Source: §3.2 · pp. 5–6

3.4 Architecture

  • A fixed pretrained VGGish extractor feeds segmented embeddings to a trained classifier with three 1-dimensional convolutional layers and two dense layers.The segmentation step was added between the extraction and classification networks to improve recognition performance.Source: §3.4 · pp. 7–8, Figure 2 · p. 8
  • The implementation used 16-bit waveform input and manually converted other audio formats to WAV, following Audio Set log-mel preprocessing and PCA and element-wise quantization steps.Source: §3.4 · p. 8
  • VGGish produces a vector of 128 embeddings for each second of input audio, and its parameters remain fixed during training and validation.Source: §3.4 · p. 8
  • The classifier uses three 1D convolutional layers followed by dense layers, with a 15-neuron softmax output for the 15 activity labels.The convolutional layers use 19, 20, and 30 channels, kernel size 5, stride 1, and linear activation; the first dense layer has 500 neurons.Source: §3.4 · p. 8
  • The network uses categorical cross-entropy and stochastic gradient descent with Nesterov momentum, taking segmented, normalized embeddings as input and outputting label probabilities.The learning rate is 0.001, decay is 1e-6, and momentum is 0.9.Source: §3.4 · p. 8
  • The system groups embeddings into fixed-size, non-overlapping windows and averages each group to form a segment-level classification instance.The averaged embeddings are standardized using min-max scaling before classification.Source: §3.4 · p. 9
  • Training embeddings were split 90%/10% for training and validation, with batches of 100 and training stopped after validation stopped improving, typically after 15–20 epochs.Training ran on a TACC Maverick server with an NVIDIA K40 GPU, and random seeds were set to 0.Source: §3.4 · p. 9

4 FEASIBILITY STUDY

In a controlled pilot, random oversampling with the classification network achieved the strongest recognition results, while segmentation size materially affected F-score. The discussion highlights a gap between performance on oversampled Audio Set embeddings and ambient home recordings, and cautions that deeper models did not improve real-world performance.

4.1 Pilot Test

  • 81.1% overall accuracy and 80.0% overall F-score were achieved by random oversampling with the classification network, the best-performing tested combination.The random forest without sampling had the lowest reported results: 34.4% accuracy and 24.5% F-score.Source: Table 3 · p. 9, §4.1 · p. 11
  • The pilot used nearby smartphone recordings in a low-variability setting, excluding irrelevant environmental noise and other ongoing activities.Each of the 15 activities was recorded for 60 seconds; the dedicated study was intended to test the framework on ambient recordings.Source: §4.1 · pp. 9–11
  • Raw-embedding input overfit majority classes, whereas random oversampling produced promising results for most classes; the worst top-1 class, flushing toilet, reached 17% class accuracy.The authors suggest the long segmentation length may have captured too much irrelevant information for that short activity.Source: §4.1 · p. 11
  • 65% F-score with one embedding vector per segment rose to over 80% with bigger segments, then stopped improving beyond five vectors.The random-guess level was around 7%.Source: Figure 4 · p. 11, §4.1 · p. 12

4.2 More Discussions towards Domain Shifts

  • The network reached 53% training and validation accuracy on oversampled Audio Set embeddings, yet the top-1 scheme exceeded 80% on ambient recordings.Deeper layers could improve Audio Set validation performance, but did not improve—and might harm—real-world performance.Source: §4.2 · p. 12

5 IN-THE-WILD TESTS

This section presents ADL recognition tests in lab and actual-home settings, including in-the-wild scripted tests and activity-level results showing robust classes and weaknesses linked to real-world variation and overlapping sounds. It also notes that the framework was developed without feature augmentation or semi-supervised learning.

5.1 Test Design

  • Tests used scripted instructions that participants freely performed at home, with chatting, simultaneous activities, normal appliances, and their own tools introducing variability.The 14 tests took place in volunteers’ own homes.Source: §5.1 · pp. 12–13

5.2 Results and Discussions

  • 12,078 seconds (201 minutes) of target-related audio accounted for 37.6% of the 32,105 seconds (535 minutes) collected from 14 subjects.The labeled timestamps were used to manually segment target activities from the raw recordings.Source: §5.2 · p. 13
  • For classification performance within subjects, averaged top-1 accuracy was 64.16% and averaged top-3 accuracy was 83.59%, respectively, for all 14 subjects.Source: §5.2 · pp. 13–14, Figure 5 · p. 14
  • Shavering, chopping food, and microwave use had almost 100% averaged class accuracy and almost zero deviation; floor cleaning also performed satisfactorily.The paper attributes floor-cleaning performance to its clear, unique sound features.Source: §5.2 · p. 15
  • Most flushing activities were misclassified, likely because pumping was too brief for the segmentation length and flushing sounds overlapped with washing or frying.The unit segmentation length was 10 embedding vectors (9.6 seconds).Source: §5.2 · pp. 13, 15
  • Frying, boiling, juice squeezing, and brushing had high deviations; varied tools, styles, and appliance noise contributed, and electric-toothbrush brushing was largely unrecognized.The paper suggests the electric-toothbrush failure may reflect a lack of relevant training samples in Audio Set.Source: §5.2 · p. 15
  • Most activities improved significantly from top-1 to top-3, nearing 100% mean accuracy with smaller deviations, consistent with activity co-occurrence and overlapping acoustic features.The paper gives examples including simultaneous chatting with outdoor strolling and a music show playing on TV.Source: §5.2 · p. 15
  • The framework was developed without feature augmentation or semi-supervised learning processes.Source: §5.2 · p. 16
  • 81.1% overall classification accuracy was achieved for 15 activity classes in the lab study.Source: §5.2 · p. 16
  • 64.2% top-1 and 83.6% top-3 performance were achieved in multi-subject tests with 14 participants in their actual homes.Source: §5.2 · p. 16

6 CONCLUSION

The paper presents an audio activity-recognition framework trained on online YouTube soundtracks, combining oversampling with deep learning to recognize 15 common home activities. In lab and in-home tests, it showed promising performance and robustness to environmental variability.

6 CONCLUSION

  • The framework uses large-scale online YouTube soundtracks as its only training set to recognize 15 common home-related activities.The authors identify collecting ground-truth user data for multi-class audio learning as time-consuming and laborious.Source: §6 · p. 16
  • Oversampling and deep-learning architectures address the highly unbalanced training classes without feature augmentation or semi-supervised learning.The passage also says no further feature processing is needed.Source: §6 · p. 16
  • The framework showed promising performance and robustness to environmental variability in lab pilot tests and in-the-wild scripted tests with multiple subjects at home.The evaluation also considered activity-label associations and the effects of embedding segmentation.Source: §6 · p. 16
Loading 1810.08691v2…