PaperlayerView exact revision on arXiv

Source-linked AI summary

"What's That Sound?": A Versatile, Robust, and Lightweight Convolutional Transformer for Environment Sound Recognition

Julia Huang

arXiv:2609.27762v1cs.SDcs.LG

TL;DR

The paper addresses the cost and limited sound-detection use of hearing aids, alongside computational and size constraints in Transformer-based audio models. It proposes RALCT, combining CNN and Transformer processing, and reports 94.56% accuracy for its optimized UrbanSound8K model while presenting mobile integration as a practical direction.

  • Problem

    The paper frames environmental sound classification as support for hearing-impaired users, noting hearing aids are costly and limited and that large Transformers can be difficult to train or deploy.

    Source: BACKGROUND · p. 1, Transformers · p. 2

  • Method

    RALCT combines CNN feature extraction and Transformer encoder blocks with a concatenated MFCC and spectrogram representation.

    Source: Proposed Process and Model Architecture · p. 3

  • Results

    94.56% accuracy was reached by the most optimized RALCT at epoch 119, and the conclusion reports consistent state-of-the-art performance across over ten versions.

    Source: RESULTS · p. 4, CONCLUSION · p. 5

  • Takeaways & Limitations

    The authors present RALCT and the Audiority app prototype as a mobile-oriented approach to displaying predicted sound categories for hearing-impaired users' safety.

    Source: Summary and Significance · p. 5

    The authors suspect augmentation probabilities above 0.2 were limited by RAM overload and Colab crashes, restricting exposure to a wider variety of modifications.

    Source: Limitations and Improvements · p. 5

Abstract

from arXiv · show

The conventional hearing aid is both costly and lim- ited in usage, as it is not intended to detect non-speech audio. Our objective is to develop a machine learning solution to provide a more accurate and affordable mechanism to identify surrounding sounds to improve the safety of the hearing impaired, i.e., if a car is honking behind pedestrians, or a gunshot is fired, and they need to move away from the source. By adding randomized augmentations to audio, concatenating a Mel-Frequency Cepstral Coefficients (MFCCs) diagram and a log-mel Spectrogram, and including Convolutional Neural Networks (CNNs) in a Trans- former architecture, the Randomized Audiomentational Layered Convolutional Transformers (RALCT) model efficiently extracts features from diversified audio representations. In addition, RALCT is small enough, with only approximately 310,000 parameters, to be deployed into mobile devices. Experimental results on the UrbanSound8K dataset resulted in an accuracy consistently over 93% for all variations of RALCT with the highest at 94.56%, reaching state-of-the-art levels. To leverage the capabilities of this technology, a mobile app is developed to be integrated with the model to provide real-time safety control. RALCT thus represents a robust, lightweight, affordable, and versatile deep learning tool to aid the navigation and safety of the hearing impaired.

BACKGROUND

Environmental sound classification is presented as a way to aid the hearing-impaired community, for whom conventional hearing aids are costly and limited in use. The paper motivates lightweight AI-based sound identification and reviews CNNs as a common approach to environmental sound classification.

BACKGROUND

  • Environmental sound classification is framed as a task for aiding the hearing-impaired community, while hearing aids are described as costly and limited in usage.The passage also says hearing aids are usually designed to magnify voices rather than detect background sounds.Source: BACKGROUND · p. 1
  • The paper proposes lightweight AI-based sound identification as a more reliable and affordable solution for hearing-impaired users.Source: BACKGROUND · p. 1

PREVIOUS WORK

  • Conventional environmental sound classification commonly uses audio augmentations and a single model, with CNNs popular for learning localized features.Source: PREVIOUS WORK · p. 1

CNNs

Prior CNN-based UrbanSound8K systems use augmentation or ensembles, but the paper identifies lost feature-position information as an issue for CNNs. Transformer approaches capture dependencies and feature locations, while their computational and storage demands can constrain audio applications.

CNNs

Transformers

  • AudioCLIP reached 90.07% and MhaNN 92.2% on UrbanSound8K, but Transformer attention can be costly for long inputs and large models may not fit small devices.The paper describes Transformers as recording feature locations and dependencies; it notes AudioCLIP has 30 million parameters.Source: Transformers · p. 2

METHODOLOGY

The paper aims for an affordable, resource-efficient, robust sound-classification system that can run in real time on mobile devices. It also envisages extending the model to indoor or wilderness sounds by training it on another dataset.

Motivation of Approach

  • The proposed solution is intended to require low setup and training resources, remain accurate despite slight parameter changes, and detect outdoor audio in real time on mobile devices.Source: Motivation of Approach · p. 2
  • The authors propose developing the approach for indoor or wilderness navigation by training the model on another dataset, such as one containing doorbell or snake-hissing sounds.Source: Motivation of Approach · p. 2
  • The stated objective is a scalable solution for a variety of outdoor scenarios, potentially usable through a phone app by hearing-impaired users.Source: Motivation of Approach · p. 2

Proposed Process and Model Architecture

RALCT combines randomized audio augmentation and two audio representations with CNN and Transformer components. Its pipeline uses CNN feature extraction and dimensionality reduction before Transformer processing, then classifies the input among ten audio classes.

Proposed Process and Model Architecture

  • The study covers dataset processing, training a lightweight deep-learning architecture, and developing a mobile app for model integration.Source: Proposed Process and Model Architecture · p. 2
  • RALCT combines CNN feature extraction with Transformer modeling of long-range dependencies, using randomized augmentations and MFCCs plus a log-mel spectrogram as diversified audio representations.The authors say CNN reduction of representation dimensions permits transfer to the Transformer without sacrificing RAM or computational cost.Source: Proposed Process and Model Architecture · p. 2
  • Randomized time stretch, pitch shift, and noise are applied during training with adjustable probabilities, exposing the model to varied audio modifications across epochs.Time stretch varies speed from 0.8× to 1.5×; pitch shift changes pitch by five semitones.Source: Proposed Process and Model Architecture · pp. 2–3
  • The model separately normalizes log-mel spectrogram and MFCC representations, passes their combined representation through three CNN layers, then uses positional encoding and two Transformer encoder blocks.Source: Figure 1 · p. 2, Proposed Process and Model Architecture · p. 3
  • Dropout is used after convolutional layers and between attention layers, while max pooling downsamples feature maps to reduce dimensions and computation.The Transformer also uses multi-head attention and a residual connection.Source: Proposed Process and Model Architecture · p. 3
  • RALCT flattens the resulting feature map, feeds it to dense layers, and outputs one of ten audio class names through softmax.Source: Proposed Process and Model Architecture · p. 3

Dataset

The evaluation uses a cleaned UrbanSound8K dataset and compares training configurations using a held-out validation split.

Dataset

  • UrbanSound8K was cleaned to remove or relabel about 11 clips that were multilabel, mislabeled, or outside its classes, leaving 8,721 clips for processing.The dataset contains 10 sound categories; the tracks are mono, 16-bit WAV files sampled at 22,050 or 44,100 Hz.Source: Dataset · p. 3

Training Setup

  • 80% of tracks were randomly assigned to training and 20% to validation; accuracy averages classification results across 10 classes.The study used Adam, cosine decay over 120 epochs, initial learning rates of 10^-4 or 10^-3, and batch sizes of 32 or 64.Source: Training Setup · p. 3
  • At batch sizes 32 and 64, training took 1.1385 hours and 46.428 minutes, respectively, with 34 and 23 seconds per epoch step, respectively.Source: Training Setup · p. 3

RESULTS

RALCT achieved validation accuracy comparable to the strongest listed architecture with fewer training epochs, while its tested variants remained above 93%. The results also report consistent learning curves, category-level differences, and errors in the confusion matrix.

Overall comparison

  • MhaNN-SVM surpassed RALCT by 0.04%, reaching 94.60% after 400 epochs, while RALCT reached comparable accuracy after only 119 epochs; MhaNN alone achieved 92.20%.Table I presents accuracies of different architectures on UrbanSound8K compared with the most optimized RALCT version.Source: RESULTS · p. 4, Table I · p. 4

Ablations

  • Across tested RALCT variants, validation accuracy stayed above 93%, and the 310K-parameter version performed best.Table II reports the accuracies of different RALCT versions and ablation studies.Source: RESULTS · p. 4, Table II · p. 4
  • Increasing convolutional-layer dropout hurt accuracy, while an initial learning rate of 10^-3 improved performance.The authors suggest excessive dropout may remove critical data connections and attribute the learning-rate result to faster learning.Source: RESULTS · p. 4

Confusion matrix

  • The confusion matrix reports 96 incorrectly classified files out of 8,721, described in the passage as about 11%.Table III is the confusion matrix for the most optimized RALCT version.Source: Table III · p. 4, RESULTS · p. 5

Learning curves

  • 94.56% validation accuracy was reached at epoch 119, with later training and validation curves running in parallel and remaining consistent.Figure 2 compares training and validation losses and accuracies.Source: RESULTS · pp. 4–5, Figure 2 · p. 5

Category results

  • RALCT performed best on gunshots and somewhat poorer on children playing and street music, where quiet sounds or loud background noise made clips hard to distinguish.Figure 3 presents precision, recall, and F1-score by category.Source: RESULTS · p. 5, Figure 3 · p. 5

CONCLUSION

The authors present a lightweight machine-learning framework intended to improve hearing-impaired users’ lives; they claim it is more reliable and affordable than hearing aids and suitable for mobile integration.

CONCLUSION

  • The authors describe their framework as a novel, lightweight machine-learning solution intended to improve the lives of hearing-impaired people.Source: CONCLUSION · p. 5
  • RALCT uses fewer resources than existing ESC architectures through its small parameter count and fast training time, supporting mobile integration.The authors also report consistent state-of-the-art performance across more than ten versions.Source: CONCLUSION · p. 5
  • The authors claim their solution is more reliable and affordable than hearing aids.Source: CONCLUSION · p. 5

Limitations and Improvements

The authors suspect augmentation probability settings constrained RALCT’s performance because higher settings caused RAM overload and crashes in Colab.

Limitations and Improvements

Mobile App Integration

RALCT is presented as a compact mobile model, and Audiority is an iOS prototype that detects selected nearby sounds in real time; Android development and real-world testing remain future work.

Mobile App Integration

  • RALCT has 310K parameters and occupies 2.8 MB as a TensorFlow Lite model, supporting mobile deployment.Source: Mobile App Integration · p. 5
  • Audiority listens through a phone’s built-in microphone and detects YouTube clips of car horns, gunshots, and sirens within one second.It displays the sound name and a picture, and can record detection history with time, date, and sound name.Source: Mobile App Integration · p. 5
  • Future work includes an Android version and testing the solution in real-world scenarios with hearing-impaired people.Source: Mobile App Integration · p. 5

Summary and Significance

The authors frame RALCT as a lightweight mobile-deployable algorithm for outdoor safety, paired with a phone app that displays predicted sound categories as alerts.

Summary and Significance

  • RALCT is presented as a lightweight algorithm designed to improve outdoor safety for hearing-impaired people and suitable for mobile deployment.Source: Summary and Significance · p. 5
  • The accompanying phone app displays predicted sound categories to alert people to potential dangers in real-time environments.Source: Summary and Significance · p. 5
  • The authors believe the research could develop into a market solution to assist the hearing-impaired community.Source: Summary and Significance · p. 5
Loading 2609.27762v1…