PaperlayerView exact revision on arXiv

Source-linked AI summary

Acoustic Scene Classification

Daniele Barchiesi, Dimitrios Giannoulis, Dan Stowell, Mark D. Plumbley

arXiv:1411.3715v1cs.SDcs.LG

TL;DR

Acoustic scene classification asks how environments can be identified from the sounds they produce. The paper surveys prior work, formulates a general ASC framework, and evaluates challenge algorithms against a baseline and human listeners. Three algorithms significantly outperform the baseline, while the best algorithm matches the median human accuracy; some scenes remain misclassified by every algorithm, indicating room for improvement.

  • Problem

    ASC concerns classifying environments from sounds, but scene categories are difficult to define and the paper addresses the need for a general, fair benchmark for ASC techniques.

  • Method

    The paper provides a historical survey and general ASC framework, then evaluates challenge algorithms using a baseline, performance metrics, significance tests, and human comparisons.

  • Results

    Three algorithms significantly outperform the baseline, and the best-performing algorithm matches the median accuracy obtained by humans.

  • Takeaways & Limitations

    Human and computer classifications share common misclassified class pairs, while every scene is correctly classified by at least some individuals.

  • Takeaways & Limitations

    Some scenes are misclassified by all algorithms, suggesting scope for improvement before algorithms reach and surpass human ability.

Abstract

from arXiv · show

In this article we present an account of the state-of-the-art in acoustic scene classification (ASC), the task of classifying environments from the sounds they produce. Starting from a historical review of previous research in this area, we define a general framework for ASC and present different imple- mentations of its components. We then describe a range of different algorithms submitted for a data challenge that was held to provide a general and fair benchmark for ASC techniques. The dataset recorded for this purpose is presented, along with the performance metrics that are used to evaluate the algorithms and statistical significance tests to compare the submitted methods. We use a baseline method that employs MFCCS, GMMS and a maximum likelihood criterion as a benchmark, and only find sufficient evidence to conclude that three algorithms significantly outperform it. We also evaluate the human classification accuracy in performing a similar classification task. The best performing algorithm achieves a mean accuracy that matches the median accuracy obtained by humans, and common pairs of classes are misclassified by both computers and humans. However, all acoustic scenes are correctly classified by at least some individuals, while there are scenes that are misclassified by all algorithms.

I. INTRODUCTION

Acoustic scene classification associates semantic labels with environments based on their sounds, supporting machine-listening applications while posing difficult open-ended categorization and recognition problems. The paper surveys ASC, presents a general framework and challenge-based evaluations, and compares algorithmic performance with human listening.

  • ASC associates a semantic label with an audio stream identifying the environment where it was produced.
  • ASC supports applications including context-aware services, wearable devices, robotics navigation, audio archiving, and preprocessing for other audio algorithms.
  • Because no comprehensive taxonomy covers all environments and their acoustic events are unbounded, ASC is typically treated as supervised classification over predefined classes.
  • The paper offers a tutorial and state-of-the-art survey, a framework for describing ASC components, challenge-based evaluation, and comparisons with human performance.
  • Earlier computational systems used features and classifiers such as MFCCs, recurrent neural networks, k-nearest neighbours, and HMMs for scene recognition.

A. Features

ASC systems use diverse time-, frequency-, cepstral-, spatial-, and voicing-related features to represent acoustic scenes. These features capture spectral content, temporal structure, spatial position, and tonal or harmonic properties.

  • Time- and frequency-domain features include zero-crossing rate, spectral centroid, spectral roll-off, and sub-band energy descriptors.
  • MFCCs summarize coarse spectral content by applying a discrete cosine transform to the logarithm of Mel-frequency filter-bank coefficients.
  • Stereo recordings can provide inter-aural time difference and inter-aural level difference features linked to sound-source position.
  • Voicing and tone-fit features characterize fundamental-frequency or tonal properties that may correspond to discriminative events within scenes.
  • Linear predictive coefficients model signals through autoregressive relationships and encode information about their general spectral characteristics.

8) Parametric approximation features:

Parametric approximation features represent audio signals using basis functions or learned components, while event-based methods encode scenes through detected-event sequences and statistics.

  • Parametric approximation expresses a signal as a linear combination of basis functions whose parameters describe contributing time-frequency structures.
  • Gabor-transform representations parameterize basis functions by frequency, time scale, time shift, and frequency phase, locating active scene events.
  • Basis functions may be predefined or learned unsupervised from data, including through sparse restricted Boltzmann machines applied to MFCCs.
  • Matrix factorization represents spectrograms with elementary functions and activation functions that encode salient spectral elements over time.
  • Event-based ASC detects annotated or learned acoustic events and compares their temporal histograms or sequences using scene-specific models.

B. Feature processing

Feature processing transforms or selects representations before statistical modeling and classification. ASC systems combine generative or discriminative models to represent class distributions, temporal evolution, and decision boundaries.

  • Feature transforms such as PCA reduce dimensionality by projecting high-dimensional features onto lower-dimensional subspaces while retaining variance.
  • Fisher scores select feature subsets by measuring how separable features from different classes are likely to be.
  • Discrete time derivatives add information about the temporal evolution of frame-level scene properties.
  • Generative models estimate class-specific feature distributions and classify unlabelled samples according to the most likely generating model.
  • SVMs learn separating hyperplanes for class regions, with one-versus-all and one-versus-one schemes extending binary classification to multiple scenes.
  • HMMs model temporal event order through transition probabilities, while RQA captures recurrence, periodicity, and stationary segments.

5) i-vector:

The i-vector approach models MFCC sequences with Gaussian mixture models, producing a compact scene representation for probabilistic discriminant analysis. The broader ASC pipeline then uses learned models and decision criteria to assign labels, often aggregating frame-level decisions.

  • i-vector: i-vectors derive a low-dimensional representation of an acoustic scene from GMM parameters learned from MFCCs.The representation summarizes scene properties and is supplied to generative probabilistic linear discriminant analysis.
  • Decision criteria: Decision criteria map feature vectors and learned statistical models to categories for unlabelled samples.Their form depends on the statistical learning method used.
  • Decision criteria: Frame-level classifications are commonly combined by assigning a scene the category most frequently assigned to its frames.Weighted majority voting can give greater importance to selected frames, such as higher-energy frames.
  • Decision criteria: Nearest-neighbour criteria use the closest training vector, while k-nearest-neighbour criteria classify from the most common label among the k closest vectors.The distance is often Euclidean.
  • Decision criteria: Generative models assign vectors to the class most likely to have generated them, while MAP additionally incorporates class marginal likelihood information.MAP can use contextual information, such as geographic priors over likely environments.
  • Meta-algorithms: Meta-algorithms combine multiple classifiers or weak learners to reduce classification error, including tree-bagging with majority-vote scene decisions.Tree-bagging trains weak learners on bootstrapped copies of the training data and combines their frame-level outputs.

2) Normalized compression dissimilarity and random forest:

Normalized compression dissimilarity represents relationships between audio scenes through compressed file sizes, after which a random forest performs classification. The section also situates this method within a general ASC framework based on feature extraction, statistical modelling, and decision functions.

  • Normalized compression dissimilarity and random forest: The compression-based method defines normalized compression distance between audio scenes using the sizes of files produced by an audio coder.Pairwise distances are then used for classification with a random forest.
  • Normalized compression dissimilarity and random forest: A random forest classifies scenes from the resulting set of pairwise compression distances.The random forest is described as a classification meta-algorithm.
  • General ASC framework: ASC systems transform short signal frames into feature vectors, often reducing dimensionality to obtain more generalisable and discriminative representations.The transform T maps frames in R^D to features in R^K, typically with K much smaller than D.
  • General ASC framework: Sequences of features are modelled because individual time-localised frames cannot summarise soundscapes containing events at different times.Statistical models abstract category-level properties from empirical recordings.
  • General ASC framework: After training, the feature transform is applied to a new recording and a function G combines its features with the learned model to return a class label.Most algorithms differ primarily in their choices of T, S, and G.
  • General ASC framework: Bag-of-frames methods follow the general ASC structure but ignore the ordering of feature sequences during statistical modelling.They are named by analogy with bag-of-words representations in text classification.

IV. CHALLENGE ON DETECTION AND CLASSIFICATION OF ACOUSTIC SCENES AND EVENTS

The DCASE challenge addresses the lack of coordinated, reproducible, and fair benchmarking for acoustic scene classification. It provides standardised urban recordings, a held-back evaluation set, submitted algorithms, and a conventional baseline for comparison.

  • Motivation: DCASE was organised to test and compare ASC and event-detection algorithms while promoting reproducible research.The challenge was organised with the IEEE AASP Technical Committee.
  • Motivation: Existing ASC studies often relied on non-public datasets, making sustainable and reproducible experiments difficult.Public sound repositories also vary in recording conditions and quality, requiring substantial curation for rigorous evaluation.
  • Dataset: The challenge dataset contains recordings from 10 urban environments in the London area.Scenes include buses, streets, offices, markets, parks, restaurants, supermarkets, tubes, and tube stations.
  • Dataset: Two disjoint datasets were built from the same recordings, with one public for development and one held back for evaluating submitted methods.Each dataset contains ten 30-second clips per scene, totalling 100 recordings.
  • List of submissions: Eleven algorithms were submitted by research institutions worldwide and described in accompanying extended abstracts.The paper also provides a table defining the algorithm acronyms used for comparison.
  • Baseline: The benchmark baseline uses MFCCs, GMMs, and a maximum-likelihood criterion as standard audio-analysis components.These components provide a comparison with more sophisticated ASC techniques.

1) Cross-validation:

The submitted ASC methods were evaluated with stratified 5-fold cross-validation on a private dataset, using accuracy, confusion matrices, confidence intervals, and paired significance comparisons. Results show varied performance, partial complementarity among algorithms, and difficult scene pairs shared across methods.

  • Cross-validation: Stratified 5-fold cross-validation uses five runs with 80 training recordings and 20 test recordings, while preserving class proportions.The five test subsets are disjoint, so all 100 recordings are classified during testing.
  • Results: The grey boxes group methods whose accuracies cannot be judged significantly different at the 95% significance level.Confidence intervals are based on the fold accuracies under a Gaussian assumption.
  • Results: 55% is the baseline mean accuracy, while GSR, RG, LTT, and RNH approach or exceed 70% mean accuracy.Other methods fall between 55% and 65%, whereas OE performs close to chance and significantly worse than the other methods.
  • Results: For almost 80% of soundscapes, some algorithms classify correctly and incorrect algorithms do not all choose the same wrong label.Combining decisions therefore produces a relatively robust meta-classifier, although majority voting remains far from perfect.
  • Results: The pairs park–quiet-street and tube–tube-station are commonly confused by the majority-vote solution.This indicates that some scene distinctions remain difficult across the evaluated methods.
  • Results: OE achieved the highest accuracy on the public dataset but performed near chance on the private dataset, suggesting overfitting to training data.Its learned models did not generalise to the test signals.

C. Ranking of algorithms

The study ranks ASC algorithms using paired sign tests on file-level classification outcomes, while examining difficult soundscapes, algorithm decision similarity, and human accuracy. Three algorithms significantly outperform the baseline, and classification difficulty varies substantially across soundscapes.

  • Statistical ranking: Paired sign tests compare classifiers file by file and provide a basis for ranking their accuracies.Each file is represented by a binary correct-or-incorrect outcome, and pairwise differences are tested for significance.
  • Statistical ranking: Methods grouped in the same grey box have no statistically significant accuracy difference at the chosen significance level.The ranking therefore orders only methods whose pairwise differences can be judged significant.
  • Algorithm ranking: OE differs significantly from every other technique, while ELF through CHR form a cluster not significantly different from the baseline.GSR and RG significantly exceed the baseline but not NR, NHL, or CHR; RNH outperforms most remaining methods without significantly exceeding GSR, RG, or LTT.
  • Statistical ranking: The majority-vote meta-heuristic is excluded from the ranking because its paired variables violate the sign test’s independence assumption.All submitted methods were evaluated on every file of the same held-back dataset.
  • Decision similarity: Two-dimensional scaling places algorithms with similar labelling decisions near one another, but OE lies apart from the others after achieving low scores.The projection is based on pairwise disagreement distances and was judged sufficiently representative by its low stress.
  • Soundscape difficulty: Ten soundscapes were correctly classified by at most 10% of algorithms, while bus, busy-street, quiet-street, and tubestation examples included files classified correctly by none.Accuracy varies greatly among soundscapes in the same category, except for office and restaurant, which show more consistent accuracies.

1) Cumulative accuracy:

The analysis examines human cumulative classification accuracy and broader algorithmic design patterns in acoustic scene classification. Human accuracy did not improve during testing, while temporal modeling and hybrid learning emerged as promising algorithmic directions, despite limited baseline gains.

  • Human cumulative accuracy: People who classified most or all 100 samples tended to achieve higher overall accuracy than those who classified only a few.
  • Human cumulative accuracy: −0.0028 was the average derivative of cumulative human accuracy, and tests found no evidence that accuracy improved during the test.The authors interpret this as evidence against participants learning from exposure to the soundscapes.
  • Human confusion patterns: ‘Supermarket’ and ‘openairmarket’ were the most commonly misclassified human categories, with additional confusion between ‘park’ and ‘quietstreet’.‘Tube’ and ‘tubestation’ were also confused, but to a lesser extent.
  • Algorithmic design patterns: Most significantly stronger methods used SVMs with generative-model parameters, suggesting a tradeoff between discrimination and generalisation.
  • Algorithmic design patterns: Better-scoring algorithms modeled temporal relationships between features, supporting the use of time evolution to represent complex acoustic scenes.RNH modeled MFCC periodicities, RG modeled spectrogram time-frequency structure, and GSR modeled local-feature trends.
  • Algorithmic design patterns: Seven of eleven ASC methods did not significantly outperform the baseline, suggesting a possible performance ceiling and motivating alternative paradigms.The authors identify temporal modeling as one possible direction and note that meta-algorithms may be prone to over-fitting.

B. Comparison of human and algorithmic results

The paper compares algorithmic and human acoustic scene classification using related but non-identical evaluation setups. The best algorithm matched median human accuracy, while both groups shared confusions among similar classes, with humans showing broader aggregate coverage.

  • Evaluation scope: The human and algorithmic experiments do not permit a statistically rigorous comparison because their experimental designs differed.The datasets were disjoint, although recorded with the same equipment and under the same conditions, supporting qualitative comparison only.
  • Performance comparison: RNH achieved mean accuracy on the private DCASE dataset similar to the median human accuracy on the public DCASE dataset.
  • Coverage of scenes: Across participants, every acoustic scene was correctly classified by at least some individuals, whereas some scenes were misclassified by all algorithms.
  • Error distributions: Human errors declined gradually from easier to more challenging soundscapes, unlike the distribution of algorithmic errors.The authors relate this pattern to human experience handling ambiguous or weakly distinctive soundscapes.
  • Shared confusions: Human and algorithmic confusion matrices showed common errors for ‘park’ versus ‘quietstreet’ and ‘tube’ versus ‘tubestation’.
  • Interpretation: Shared class confusions do not establish that algorithms emulate human audition; similar sound events may make some class groups inherently ambiguous.

C. Further research

The paper identifies several directions for extending acoustic scene classification, including resource-aware, adaptive, hierarchical, multimodal, and event-aware systems. Its benchmark also shows that some methods approach human accuracy, while certain scenes remain difficult for all algorithms.

  • Complexity and deployment: ASC methods should accommodate limited smartphone and tablet processing power, including real-time application constraints.Training can be performed offline, but operators applied to unlabelled signals may need to be simple enough for real-time results.
  • Adaptive categories: Progressively trained systems could add user-recorded environments and refine models through user confirmation or rejection of classifications.Such systems require more user intervention but are expected to be more precise and relevant than fully automated systems.
  • Hierarchical and cost-sensitive classification: Hierarchical classification could distinguish broad indoor–outdoor categories before finer scene distinctions, while assigning different costs to different errors.This design could prioritize outdoor-versus-indoor accuracy over distinctions such as park versus busy street.
  • Multimodal and event-aware systems: Multimodal fusion could combine audio with video, geolocation, temperature, or humidity to provide machines with context awareness.The paper also proposes combining event detection with frame-based ASC, such as using a detected car horn to distinguish a park from a quiet street.
  • Conclusions: The DCASE comparison found that some submitted methods significantly outperformed the baseline and approached human accuracy, although some scenes were misclassified by all algorithms.All soundscapes were correctly classified by at least some individuals, indicating a remaining gap between algorithmic and human performance.

APPENDIX

The appendix describes a baseline ASC pipeline that extracts and normalizes MFCC features, models each class with a GMM, and classifies new signals by maximum likelihood. Its design includes parameter trade-offs and treats feature sequences as unordered in a bag-of-frames representation.

  • MFCC feature extraction: MFCC extraction maps Fourier-magnitude bins onto Mel bands, applies a logarithmic scale and DCT, then retains the first K coefficients as features.The parameters θ = {D, R, K} determine frame dimensions, Mel bands, and retained DCT coefficients, balancing generalisation and discrimination.
  • Feature processing: Feature normalization subtracts the dataset-wide mean from each coefficient and divides by its global standard deviation.After normalization, coefficient means and standard deviations are 0 and 1, respectively.
  • GMM modeling: Each acoustic class is modeled as a Gaussian mixture whose components represent different statistical patterns in the feature vectors.The number of components controls a trade-off between model accuracy and over-fitting; too many components can fit random training variation.
  • Maximum-likelihood classification: For an unlabelled signal, the system normalizes extracted features using training statistics and selects the class whose model gives the highest likelihood.The likelihood coefficients are computed from the observed data under each learned class model.
  • Baseline representation: The baseline is a bag-of-frames technique in which the ordering of the feature sequence does not affect classification.Randomly permuting the feature sequence therefore leaves the baseline computation unchanged.
Loading 1411.3715v1…