PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine learning in acoustics: theory and applications

Michael J. Bianco, Peter Gerstoft, James Traer, Emma Ozanich, Marie A. Roch, Sharon Gannot, Charles-Alban Deledalle

arXiv:1905.04418v4eess.SPcs.LGcs.SDeess.ASphysics.app-ph

TL;DR

Acoustics has many specialized challenges, motivating a data-driven alternative to conventional processing. This review introduces machine learning theory and surveys applications, finding substantial advances while noting that representative training data remain necessary.

  • Problem

    Specialized acoustics algorithms face diverse challenges, motivating a general data-driven framework as an alternative to highly specialized methods.

  • Method

    The review introduces machine learning theory, including deep learning, and surveys applications across acoustics research areas.

  • Results

    Machine learning enables advances across acoustics, including competitive source localization and sound-event detection results, while state-of-the-art environmental-sound systems remain below human perception on many inference tasks.

  • Takeaways & Limitations

    Open, publicly available datasets can encourage innovation across acoustics and increase the benefits of machine learning.

  • Takeaways & Limitations

    Machine-learning methods require large amounts of representative training data, limiting practicality when such data are unavailable.

Abstract

from arXiv · show

Acoustic data provide scientific and engineering insights in fields ranging from biology and communications to ocean and Earth science. We survey the recent advances and transformative potential of machine learning (ML), including deep learning, in the field of acoustics. ML is a broad family of techniques, which are often based in statistics, for automatically detecting and utilizing patterns in data. Relative to conventional acoustics and signal processing, ML is data-driven. Given sufficient training data, ML can discover complex relationships between features and desired labels or actions, or between features themselves. With large volumes of training data, ML can discover models describing complex acoustic phenomena such as human speech and reverberation. ML in acoustics is rapidly developing with compelling results and significant future promise. We first introduce ML, then highlight ML developments in four acoustics research areas: source localization in speech processing, source localization in ocean acoustics, bioacoustics, and environmental sounds in everyday scenes.

I. INTRODUCTION

The review presents machine learning as a data-driven complement to physical acoustics, surveying its theory and applications across acoustics. It emphasizes recent progress alongside challenges involving data requirements, interpretability, generalization, and model complexity.

  • ML can model complex acoustic patterns and provide compelling solutions to challenges including corrupted, sparse, reverberant, and high-volume data.
  • ML automatically detects and uses patterns in data to estimate labels or discover useful structure, often using statistical methods for uncertain measurements.
  • ML-based methods can outperform conventional processing, but they require substantial data, may be difficult to interpret, and are typically optimized for particular tasks.
  • Hybrid models combine physical intuition with data-driven insights, while ML can also learn task-specific representations rather than relying solely on engineered domain features.
  • The review introduces ML and deep learning, develops fundamental algorithms, and surveys applications in speaker localization, ocean acoustics, bioacoustics, and environmental sounds.
  • Polynomial regression illustrates that increasing model capacity lowers training error while test error eventually rises, making model selection central to generalization.

D. Cross-validation

The section presents cross-validation and Bayesian modeling as tools for assessing generalization and incorporating uncertainty, while highlighting data sparsity and computational trade-offs. High dimensionality makes local neighborhoods less meaningful and can increase training-data demands.

  • D. Cross-validation: K-fold cross-validation estimates generalization by repeatedly training on K−1 folds and evaluating the excluded fold, producing error as a function of hyperparameters.
  • D. Cross-validation: Cross-validation can require many training runs, while tuning multiple hyperparameters may make the number of runs exponential in the number of parameters.
  • E. Curse of dimensionality: In N dimensions, a neighborhood occupying length fraction l covers volume fraction l^N, so data become increasingly sparse and distance-based methods such as K-means degrade.
  • E. Curse of dimensionality: Prior assumptions and regularization can enable training with smaller datasets, although the manifold assumption is not always correct.
  • F. Bayesian machine learning: Bayesian ML combines prior knowledge with data through parameter posteriors and represents uncertainty in estimated parameters and predictions.
  • F. Bayesian machine learning: Bayesian parameter estimation can prevent overfitting and provide predictive distributions, but estimating posterior distributions increases computational cost.

III. SUPERVISED LEARNING

Supervised learning learns mappings from labeled inputs to outputs, using regression or classification; the section develops MAP and regularized estimators before illustrating acoustic DOA estimation and binary classification.

  • Supervised learning learns predictive mappings from labeled input-output pairs, with regression for real or complex outputs and classification for categorical outputs.
  • MAP estimation selects the most probable output conditioned on observed features, but returns a point estimate rather than the full posterior distribution.
  • In DOA estimation, Fourier-transformed measurements from an acoustic array are represented with discretized array-response basis functions.
  • Gaussian priors produce ℓ2-regularized least squares, whereas Laplacian priors produce sparse ℓ1 estimates that underpin LASSO and compressive sensing.
  • Ridge regression can classify Gaussian classes well, but performs poorly on concentric distributions requiring a nonlinear decision boundary.

B. Support vector machines

Support vector machines classify or regress using maximum-margin optimization and kernelized feature spaces, allowing nonlinear boundaries while retaining a formulation for large transformed spaces.

  • SVMs estimate a separating hyperplane by maximizing the margin subject to correct classification for linearly separable data.The margin is dM = 2/∥w∥2.
  • Slack variables permit some misclassifications when classes overlap, while C controls the trade-off between slack penalties and margin size.
  • Kernel functions represent inner products in transformed feature spaces, enabling nonlinear classification without explicitly computing potentially infinite-dimensional feature mappings.
  • Only points participating in the margin-constrained solution contribute to prediction; these points are called support vectors.
  • With an RBF kernel, SVMs separate circular classes where linear regression fails, but SVMs provide hard labels rather than probabilistic outputs.
  • For large datasets, SVMs can have high computational cost, and generic kernel machines may generalize poorly.

C. Neural networks: multi-layer perceptron

Feed-forward neural networks learn nonlinear mappings through layered transformations and activations, with gradient-based training making them flexible function approximators for acoustic applications.

  • Neural networks overcome linear-model limitations by learning nonlinear input mappings from data rather than prescribing all transformations in advance.
  • Feed-forward networks map features to labels through intermediate hidden layers, with information flowing only from inputs toward outputs.
  • A network’s depth is the number of hidden layers plus the output layer, and deep learning refers to networks with many layers.
  • Hidden and output units apply nonlinear activation functions after learned linear transformations of their inputs.
  • Classification networks commonly use cross-entropy loss, with backpropagation supplying gradients for optimizing weights across layers.
  • Stochastic gradient descent and Adam are identified as popular neural-network training algorithms, while ReLU is used in hidden layers of modern architectures.

IV. UNSUPERVISED LEARNING

Unsupervised learning models feature structure without annotated targets, using dimensionality reduction, clustering, latent-variable models, dictionary learning, and autoencoders for representation and restoration tasks.

  • Unsupervised learning models features without annotated examples or prediction targets, seeking useful structures and latent representations.
  • The section surveys PCA, EM with GMMs, K-means, dictionary learning, and autoencoders as unsupervised methods.
  • Dictionary-learning methods support compression, denoising, inpainting, and inverse problems, including applications to travel-time tomography.
  • PCA learns an orthogonal linear transformation whose leading components capture the largest feature variances, enabling lower-dimensional visualization and modeling.
  • Truncated PCA produces a lossy compressed representation with less information than the original data.
  • GMMs model arbitrary probability distributions as mixtures of Gaussian densities, with EM alternating between latent-responsibility estimation and parameter updates.
  • EM optimization is non-convex, so solutions can become trapped in local minima; multiple initializations can partly mitigate this issue.

C. K-means

K-means discovers clusters by alternating hard nearest-centroid assignments with centroid updates, while dictionary learning extends this idea to sparse representations and learned atoms. K-SVD alternates sparse coding and dictionary updates, using SVD to update atoms and coefficients.

  • K-means assigns each feature vector to its nearest centroid using Euclidean distance, then updates centroids with cluster means.
  • K-means alternates between categorizing features and updating centroids, requiring initialization and a chosen or estimated number of clusters.
  • Dictionary learning represents examples as xm ≈ Dβm using a small number of nonzero coefficients from a learned dictionary.
  • Sparse coding with an ℓ0 penalty is non-convex and often impractical to solve exactly, motivating approximate methods such as OMP and SBL.
  • K-SVD alternates sparse coding and dictionary updates, learning atoms through SVD of restricted reconstruction errors.
  • When T = 1, K-SVD reduces to gain-shape vector quantization, with atoms defining radial partitions based on inner products.

E. Autoencoder networks

Autoencoders learn representations by reconstructing their inputs through an encoder and decoder. Their bottleneck structure supports dimensionality reduction and feature learning, while capacity and regularization determine whether useful features emerge.

  • An autoencoder learns useful data properties by approximating its input, prioritizing which aspects should be copied.
  • The encoder maps inputs to hidden units, while the decoder maps those units back to an output with the input dimension.
  • Undercomplete autoencoders compress inputs into lower-dimensional codes suited to extracting salient features, similar to PCA.
  • Overcomplete autoencoders can learn redundant, shift-invariant features, but require regularization to avoid failing to learn useful representations.
  • Autoencoders support data interpretation, visualization, and feature extraction, with acoustic applications including speech enhancement and novelty detection.
  • Deep learning learns hierarchical feature representations through cascades of nonlinear transformations rather than relying solely on handcrafted features.

A. Activation Functions and Rectifiers

Activation functions strongly affect gradient-based training in deep neural networks. Sigmoid and tanh activations can produce vanishing or exploding gradients, whereas rectifiers and optimization strategies improve convergence and training stability.

  • Sigmoid activations create loss landscapes with plateaus and cliffs, causing gradient updates to vanish or explode in deep networks.
  • Back-propagation with early sigmoid or tanh activations in deep networks is slow, unstable, and leads to poor solutions.
  • ReLU is zero for negative inputs and linear for positive inputs, giving active units derivative one when a > 0.
  • In practice, rectifiers produce tremendous improvement in convergence, while gradient clipping thresholds gradients to address exploding gradients.
  • Unsupervised layer-wise pretraining was introduced to help avoid poor stationary points, although modern approaches often train networks end-to-end.
  • Dropout randomly removes units during training to encourage specialized sparse features, while batch normalization enables higher learning rates and faster optimization.
  • CNNs reduce parameters for correlated signals through local receptive fields and shared weights, which also provide shift invariance.
  • CNN architecture has been applied to broadband direction-of-arrival estimation, where each class corresponds to a time frame.

D. Transfer learning

Transfer learning reuses representations learned from large datasets when labeled data for the target task are scarce. Deep learning applications in acoustics use this broader toolkit for detection, localization, enhancement, and seismic interpretation, with performance often competitive with conventional methods.

  • D. Transfer learning: Training deep classifiers from scratch requires large labeled datasets, motivating transfer learning when such data are unavailable.
  • D. Transfer learning: Transfer learning reuses early network stages as generic feature extractors, then trains a classifier and optionally fine-tunes all layers end-to-end.
  • F. Applications in Acoustics: Convolutional recurrent networks achieved state-of-the-art results in the 2017 DCASE sound event detection challenge.
  • F. Applications in Acoustics: CNNs using only STFT phase achieved competitive broadband DOA estimation results against SRP-PHAT beamforming and generalized from synthetic noise to speech.
  • F. Applications in Acoustics: Two 50-layer ResNets trained on millions of synthetic acoustic fields achieved competitive source range and depth prediction errors against genetic-algorithm inversion methods.
  • F. Applications in Acoustics: U-net and adversarial deep-learning architectures for single-microphone speech dereverberation outperformed competing methods in most cases.
  • F. Applications in Acoustics: Deep learning has been applied to seismic fault, channel, salt-dome, and facies interpretation, including semi-supervised classification for sparsely labeled exploration fields.

VI. SPEAKER LOCALIZATION IN REVERBERANT ENVIRONMENTS

Speaker localization is framed as learning-based classification over candidate positions to address noisy and reverberant scenes, including multiple concurrent speakers. The section reviews GMM/EM and manifold-learning approaches, with batch EM reported to achieve higher resolution than SRP-PHAT.

  • Speaker enhancement, camera steering, teleconferencing, and robot audition motivate acoustic source localization, but background noise and reverberation remain major challenges.
  • The survey examines unsupervised GMM classification and semi-supervised manifold learning for localization and tracking.
  • Localization is cast as classification over a predefined grid of candidate positions, with active speakers inferred from peaked GMM weights.The GMM parameters are estimated using expectation-maximization because the maximum-likelihood problem has no closed-form solution.
  • Features include PRP representations and raw microphone signals modeled with GMMs under assumptions about propagation, reverberation, and stationary known noise.The measurement model uses direct-path transfer functions, speech signals, ambient noise or diffuse reverberation, and sparse activity indicators.
  • Batch EM achieves much higher resolution than classical SRP-PHAT in the comparison shown for a distributed microphone-array setting.The procedure alternates E- and M-steps until a predefined iteration count is reached.
  • Distributed and recursive EM variants target wireless sensor networks and online applications, while related methods address high reverberation and periodic phase modeling.

B. Speaker localization and tracking using manifold learning

Manifold learning treats reverberant room reflections as location fingerprints, embedding high-dimensional RTFs into a low-dimensional space that preserves meaningful geometric structure. Semi-supervised and multi-manifold methods then use limited position labels to estimate and track speaker locations.

  • Motivation: Reflection patterns from room surfaces and objects can uniquely characterize source location, motivating manifold-learning localization.The acoustic response varies with a limited number of intrinsic degrees of freedom.
  • RTF representation: RTFs concatenate relative transfer-function values across a relevant frequency band and remain independent of the source signal.They therefore provide acoustic features for localization.
  • Diffusion mapping: Diffusion mapping constructs a graph of RTFs and embeds them into a lower-dimensional Euclidean space using dominant nontrivial eigenvectors.The resulting diffusion distance approximates the manifold geometry while simplifying distance and ordering measurements.
  • Localization evidence: The low-dimensional representation corresponds one-to-one with physical source location, enabling data-driven localization from reverberant responses.A simulated two-microphone scenario demonstrated correspondence between the embedding and angle of arrival.
  • Semi-supervised localization: Semi-supervised localization combines many unlabeled RTFs with a small set of accurately labeled positions to anchor the manifold to physical coordinates.The mapping is regularized for smoothness and solved in a reproducing kernel Hilbert space.
  • Performance and extensions: At T60 = 600 ms and SNR=5 dB, semi-supervised localization achieved ≈3° RMSE versus 18° for classical GCC.Multi-manifold extensions fuse microphone-pair viewpoints, and simulations demonstrated very good tracking capabilities.

VII. SOURCE LOCALIZATION IN OCEAN ACOUSTICS

Machine learning offers data-driven alternatives to physics-based ocean source localization, while simulations can supplement experimental data when measurements are insufficient. Recent neural models localized ships and targets accurately across simulated and real ocean environments, often addressing model mismatch affecting MFP.

  • Motivation: Ocean localization traditionally relies on physics-based propagation models, whereas ML infers source-location patterns directly from acoustic data.ML is not fully model-free when simulated data are incorporated for training.
  • Conventional methods: Matched-field processing has achieved reasonable success but remains vulnerable to model mismatch and solution ambiguity.Data-replica MFP can alleviate mismatch when closely matched data are available.
  • Recent ML approaches: Feed-forward neural networks localized cargo ships from Noise09 and Santa Barbara Channel experiments after training on sample covariance matrices and simulated data.The NN performed well on simulated data, while MFP encountered solution ambiguity.
  • Recent ML approaches: A deep time-delay neural network trained across simulated environments tracked ship location accurately, whereas MFP overestimated range under depth mismatch.The TDNN used covariance-matrix eigenvalues and training at multiple ocean depths.
  • Recent ML approaches: A deep residual CNN trained on tens of millions of samples achieved lower range error and competitive depth error than SAGA inversion.The model used single-hydrophone acoustic amplitude across numerous environmental configurations.
  • Future directions: Future systems are expected to combine propagation modeling, parallel and cloud computation, and large-scale acoustic storage for real-time localization.This is presented as a future research direction rather than a demonstrated result.

VIII. BIOACOUSTICS

Machine learning in bioacoustics supports detection, identification, classification, and ecological inference from animal sounds. Supervised and deep models dominate current applications, while unsupervised clustering and learned features offer additional approaches but performance can degrade across equipment, sites, and changing calls.

  • Applications: Bioacoustic ML addresses whether animals are present, which individual or species vocalizes, and what call or song was produced.Acoustic measurements also support biological, ecological, and management questions such as estimating animal density.
  • Features: Cepstral representations capture spectral-envelope shape in a low-dimensional feature set, while learned features infer representations from data.Learned features may operate on standard features or relatively unprocessed time-frequency representations.
  • Recognition methods: Template-based methods work well for highly stereotyped calls but may require recalibration as vocal characteristics drift over time.Dynamic time warping supports nonlinear time-frequency matching, while documented blue-whale call changes illustrate temporal drift.
  • Recognition methods: Supervised learning is the primary bioacoustic ML paradigm, spanning linear discriminant analysis, GMMs, hidden Markov models, and SVMs.These methods have classified calls by species and modeled spectral or sequence variation.
  • Deep learning: Deep CNNs and RNNs have been used for bat, whale, sperm-whale-click, and bird identification and classification.Deep models reduced issues associated with overfitting in earlier neural networks through more data, architectural changes, and regularization.
  • Limitations: Equipment and sampling-site mismatch can severely degrade classification performance, causing field results to differ from laboratory expectations.Long-term monitoring also requires storing algorithm scores, decisions, parameters, and descriptions for comparison and trend analysis.

IX. REVERBERATION AND ENVIRONMENTAL SOUNDS IN EVERYDAY SCENES

Everyday acoustic scenes combine variable sources with reverberation, making robust classification and inference difficult. ML approaches address this through reverberant training, signal modeling, spatial features, and environmental-parameter estimation, while large data and realistic scene coverage remain important constraints.

  • Challenges: Natural scenes contain diverse, variable sources whose reverberant mixtures distort waveforms and challenge acoustic classification.Algorithms must handle inter-class variation, intra-class variation, and context-dependent reverberation.
  • Applications: Robust scene analysis could support hearing aids, siren localization in vehicles, material inspection, aircraft classification, and animal-farm monitoring.These applications all require handling the complexities of natural acoustic scenes.
  • Reverberation: Reverberation is both a source of distortion and a potential source of information about the environment.Natural scenes exhibit statistical regularities in impulse-response properties that humans exploit and ML can potentially learn.
  • Reverberation: Separating source signal and impulse response is generally ill-posed because both are unknown.Algorithms must instead tolerate natural variations or estimate the signal and environmental response from their mixture.
  • Methods and data: Training with reverberant speech or simulated rooms can improve robustness and reconstruct dry-speech spectrograms, but broad generalization may require exceptionally large datasets.REVERB, ASpIRE, and ACE challenges provide labeled data and benchmarks for reverberant speech or room-acoustic estimation.
  • Methods and data: Spatial covariance exploits stronger cross-microphone correlation of direct sound than reverberation for dereverberation and related estimation tasks.Reverberation sums signals arriving from many directions and is therefore less correlated across channels.
  • Environmental inference: Reverberation time can be estimated from decay-rate histograms measured over short signal windows.RT is the time required for reverberant energy to decay by a specified amount.

B. Environmental sounds

Environmental-sound recognition is difficult because scenes contain diverse, variable, overlapping sources, but machine learning has enabled substantial progress through varied labels, multimodal data, learned features, and physical sound synthesis. Current systems still fall short of human perception, while open data support continued advances.

  • Challenges: Natural acoustic scenes combine many source types, substantial within-class variability, and simultaneous overlapping sounds.These factors require recognition systems to distinguish both differences between sources and variation within each source class.
  • Training data: Large labelled datasets, weak labels from online videos, and audiovisual annotations provide alternative training resources for environmental-sound classifiers.Online-video scale can compensate partly for noisy, unsynchronized metadata labels.
  • Acoustic representations: Environmental-sound classifiers commonly learn features from transformed acoustic representations such as STFT, mel-spaced, Gammatone, or ERB domains.Some algorithms learn directly from waveforms, but many state-of-the-art systems first map sounds to lower-dimensional representations.
  • Physical synthesis: Physical models can simulate contact sounds and automatically label them with parameters such as mass, material, velocity, and impact force.These synthetic datasets may support classifiers that infer fine-grained physical properties, including material, shape, or size.
  • Audiovisual learning: Audiovisual models use object motion and image information to associate, synthesize, or separate sounds in everyday scenes.Neural networks have synthesized plausible impacts for silent collision videos, while synchronized audio-video structure can support source separation.
  • Human-level interpretation: Machine-learning methods have enabled significant progress in everyday-scene recognition, but current systems do not match human perception in many inference tasks.The review identifies this gap alongside machine learning’s broader potential in acoustics and the value of open data.
Loading 1905.04418v4…