PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Sound Source Localization with Deep Learning Methods

Pierre-Amaury Grumiaux, Srđan Kitić, Laurent Girin, Alexandre Guérin

arXiv:2109.03465v3cs.SDcs.LGeess.AS

TL;DR

Sound source localization remains difficult in noisy, reverberant environments with multiple simultaneous sources. This survey organizes deep-learning approaches by architecture, inputs, outputs, datasets, and training strategies, reporting stronger performance than conventional methods in the reviewed literature.

  • Problem

    Sound source localization remains challenging when noise, reverberation, and multiple simultaneous sources are present.

    Source: §1 · p. 1

  • Method

    The survey categorizes deep-learning localization literature by architecture, input features, output strategy, datasets, and training strategy.

    Source: §9 · p. 32

  • Results

    Reported works generally indicate DNN-based superiority; Chakrabarty and Habets [2017a] showed a two-fold increase in overall DoA classification accuracy over SRP-PHAT in low-SNR conditions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tables II–V enable quick searches for deep-learning localization methods with specified characteristics.

    Source: §9 · p. 32

    Deep-learning localization models generally assume identical microphone-array setups during training and inference, requiring retraining for different arrays.

    Source: §9.2 · p. 33

Abstract

from arXiv · show

This article is a survey on deep learning methods for single and multiple sound source localization. We are particularly interested in sound source localization in indoor/domestic environment, where reverberation and diffuse noise are present. We provide an exhaustive topography of the neural-based localization literature in this context, organized according to several aspects: the neural network architecture, the type of input features, the output strategy (classification or regression), the types of data used for model training and evaluation, and the model training strategy. This way, an interested reader can easily comprehend the vast panorama of the deep learning-based sound source localization methods. Tables summarizing the literature survey are provided at the end of the paper for a quick search of methods with a given set of target characteristics.

1 Introduction

The paper surveys deep-learning approaches to sound source localization, motivated by the difficulty of conventional methods in noisy, reverberant, multisource environments and the growth of data-driven alternatives. It defines the survey’s scope, explains the multichannel and neural-learning principles underlying SSL, and organizes the literature by method characteristics and configurations.

1.3 Outline of the paper

  • The survey focuses on audible sound sources in indoor office or domestic environments, excludes audio-visual systems, includes SELD localization methods, and covers architectures, features, output strategies, datasets, and learning paradigms.SSL generally denotes estimating azimuth and elevation relative to a microphone array without estimating source distance; the paper also discusses conventional methods as baselines and feature sources.Source: §1 · p. 1, §1.1 · p. 2, §1.3 · p. 4

1 Introduction

  • DNN-based SSL methods have often outperformed conventional methods, including two-fold higher DoA classification accuracy than SRP-PHAT, a 25% accuracy increase over ICA, and 50% lower average angular error than MUSIC.These gains were reported respectively by Chakrabarty and Habets [2017a], Perotin et al. [2018b], and Adavanne et al. [2018] in low-SNR or reverberant conditions.Source: §1 · p. 2

1.1 Aim of the paper

  • The survey reviews 156 papers from 2011–2021 and classifies their architectures, features, outputs, datasets, learning strategies, and localization configurations into a taxonomy with four summary tables.The tables cover 2011–2018 collectively and each of 2019, 2020, and 2021, enabling readers to select methods by characteristics.Source: §1.1 · p. 2

1.2 General principle of DL-based SSL

  • DL-based SSL maps multichannel microphone signals, optionally through feature extraction, to source locations or DoAs by learning complex spatial relationships from training data, but models may fail when array geometry or other setup conditions change.Interchannel delays, amplitudes, reverberation, and noise encode spatial information; DNNs exploit these relationships without explicit signal-model assumptions, at the cost of reduced generality.Source: §1.2 · pp. 2–3

2 Acoustic environment and sound source configurations

The survey focuses on indoor SSL under reverberation and noise, covering source types, source counts, overlapping sources, and static or moving sources. It distinguishes assumptions about source activity and number from methods that estimate them jointly with localization.

2.1 Acoustic environments

  • Indoor SSL is studied in moderate-size closed rooms, where room impulse responses capture direct and multipath propagation for each source, array, orientation, and room configuration.Reverberation complicates localization relative to anechoic propagation, while diffuse environmental noise and recording artifacts provide additional perturbations.Source: §2.1 · p. 5

2.2 Source types

  • SSL systems often target speech, while DCASE-oriented systems localize diverse events whose distinct spectra can aid detection and localization even when sources overlap.Examples include alarms, crying babies, crashes, barking dogs, screams, speech, footsteps, door knockings, ringings, phones, and piano sounds.Source: §2.2 · p. 5

2.3 Number of sources

  • Single-source SSL assumes at most one active source, whereas multi-source SSL addresses simultaneously emitting sources and is substantially more difficult.Source activity or count may be known, detected beforehand, or estimated jointly with localization; multi-source systems may use probability thresholds, separate counting networks, or joint output branches.Source: §2.3 · pp. 5–6

2.4 Moving sources

  • The survey excludes standalone source tracking but notes that training with moving sources improves moving-source localization, while static-source training can still retain fair to good performance.Tracking typically uses successive DoA estimates, and static-source training is partly motivated by limited moving-source datasets and cumbersome moving-source simulation.Source: §2.4 · p. 6

3 Conventional SSL methods

Conventional SSL methods estimate source directions using time differences, spatial power or intensity, subspace structure, probabilistic models, sparse representations, or source independence. They remain important as deep-learning baselines and feature sources, but their assumptions create characteristic computational and robustness tradeoffs.

Time-delay and spatial-map methods

  • GCC-PHAT estimates TDoA by maximizing the phase-transformed cross-correlation, while using multiple microphone pairs can improve localization beyond two-microphone arrays.Known array geometry enables DoA estimation from inter-microphone delays; GCC-PHAT is especially common for two microphones.Source: §3 · pp. 6–7
  • SRP-PHAT localizes sources as maxima of a delay-and-sum acoustic power map and is more robust to reverberation, whereas sound-intensity methods are efficient but reflection-sensitive.SRP-PHAT averages GCC-PHAT across microphone pairs, while intensity-based DoA estimation avoids expensive grid searches in favorable acoustic conditions.Source: §3 · p. 7

Subspace methods

  • MUSIC uses eigenvalue-based signal and noise subspaces for accurate, noise-robust localization, while ESPRIT reduces search cost but often produces less accurate predictions; both are reverberation-sensitive.MUSIC and ESPRIT assume narrowband signals, although wideband extensions exist.Source: §3 · p. 7

Probabilistic methods

  • Mixture and Bayesian methods model sources or candidate positions probabilistically, with parameters typically estimated at test time or through offline-trained regression models.GMM-like methods exploit time-frequency sparsity but can be computationally intensive; GMR uses synthetic RIR training and generalized well to speech through latent activity modeling, while Bayesian methods estimate source number and DoAs with posterior sampling.Source: §3 · pp. 7–8

Sparse-recovery methods

  • Compressive-sensing and sparse-recovery methods exploit spatial sparsity and can achieve remarkable performance, but commonly require grid-aligned sources and strong dictionary-model assumptions.Grid-free methods and dictionary learning address these limitations to some extent, while sparse Bayesian learning combines Bayesian inference with sparse representations and often uses sparse arrays.Source: §3 · p. 8

Source-separation methods

  • Independent component analysis exploits statistical independence to retrieve sources and supports multi-source SSL, which is closely coupled with source separation because each task can assist the other.Conventional methods also provide baselines for deep-learning SSL and supply input features used by many deep-learning approaches.Source: §3 · pp. 6, 8

4 Neural network architectures for SSL

This section surveys neural network architectures proposed for sound source localization, emphasizing their progression toward more complex or efficient models and combinations inherited from broader deep-learning and signal-processing research. It organizes the literature by layer type using an inclusive complexity-based sequence from feedforward networks through CNNs, RNNs, CRNNs, residual and attention models, and encoder-decoder architectures.

Architecture evolution

  • SSL architectures have evolved with broader deep-learning and signal-processing advances, often adapting models from other domains and combining them in parallel or sequential configurations.Design commonly involves investigating architectures and tuning hyperparameters for the application.Source: §4 · p. 8

Taxonomy

  • The survey uses an inclusive organization by layer type, allowing each category to contain layers from previously presented categories.The sequence progresses from FFNNs to CNNs and RNNs, then CRNNs, residual and attention networks, and encoder-decoder systems.Source: §4 · pp. 8–9

Feedforward neural networks

  • Feedforward SSL systems include an MLP that extracts subband features from CPS-matrix eigenvectors, integrates them across subbands, and classifies candidate DoAs.The architecture uses multiple subband feedforward layers before a final output layer assigns the input to a candidate direction of arrival.Source: Figure 2 · p. 9

4.1 Feedforward neural networks

Feedforward neural networks for sound source localization span simple Perceptrons and MLPs with one or more hidden layers, using one-way fully connected processing. Studies range from early shallow networks and feature-specific MLPs to deeper, frequency-banded, weakly supervised, and enhancement-based designs.

Definition and scope

  • FFNNs process data one-way through layers, with Perceptrons having no hidden layer and MLPs defined here as networks with one or more hidden layers [Goodfellow et al., 2016, LeCun et al., 2015].The section focuses mainly on fully connected architectures and typically uses nonlinear activations after each layer except possibly the output layer.Source: §4.1 · p. 9

Early MLP methods

  • Early MLP-based SSL methods estimated source number before localization, processed complex two-microphone features, or used binaural ITD/ILD, GCC-PHAT classification, and horizontal-plane regression [Kim and Ling, 2011; Tsuzuki et al., 2013; Youssef et al., 2013; Xiao et al., 2015; Vesperini et al., 2016].These studies included reverberant evaluation despite an anechoic assumption, complex-valued processing that improved over real-valued MLPs, and improvement over conventional methods on simulated and real data.Source: §4.1 · p. 9

Deeper MLPs

  • Deeper MLPs used two, three, or four hidden layers for varied input types, azimuth-and-distance estimation, multi-source localization, and speech/non-speech classification [Roden et al., 2015; Yiwere and Rhee, 2017; He et al., 2018a].The four-hidden-layer system showed results similar to a 4-layer CNN for multi-source localization and speech/non-speech classification.Source: §4.1 · pp. 9–10

Frequency-specialized architectures

  • Frequency-specialized MLP architectures trained separate sub-band networks or parallel frequency branches, with integrated outputs outperforming a conventional 7-layer MLP and MUSIC on anechoic and reverberant single- and multi-source signals [Ma et al., 2015; Takeda and Komatani, 2016b,a, 2017; Takeda et al., 2018].Ma et al. [2015] assumed each band was mostly dominated by one source, while Takeda et al. used interchannel-correlation eigenvectors and progressively integrated frequency-dependent outputs.Source: §4.1 · p. 10

Alternative MLP uses

  • Other MLP applications estimated azimuth from relative transfer functions with weak supervision or enhanced interaural phase difference before downstream DoA estimation [Opochinsky et al., 2019; Pak and Shin, 2019].The weakly supervised approach computed part of its loss without ground-truth DoA labels, while the enhancement approach used a 3-layer MLP.Source: §4.1 · p. 10

4.2 Convolutional neural networks

CNNs support a broad range of sound source localization tasks, using diverse acoustic representations, prediction targets, and architectures. Reported studies cover single and multiple sources, reverberant and real environments, multitask learning, causal processing, and efficiency-oriented convolution designs.

Early CNN-based SSL

  • CNNs were first applied to SSL by Hirvonen [2015], classifying single-source audio into eight spatial regions from multichannel magnitude spectrograms without classical pooling.The four-convolutional-layer, four-fully-connected-layer network performed well on single sources and adapted without hand-engineering, but robustness to location shifts and hidden-feature interpretability remained concerns.Source: §4.2 · p. 10

Phase-based azimuth estimation

  • Chakrabarty and Habets predicted one or two speakers’ azimuths in reverberation from multichannel STFT phase, with filter dimensions and depth tailored to microphone correlations and WDO assumptions [Chakrabarty and Habets, 2017a, 2017b, 2019b].For an M-microphone array, they found M −1 convolutional layers optimal for exploiting neighboring-microphone phase correlations.Source: §4.2 · pp. 10–11

Multitask localization

  • CNNs have supported multitask localization, including azimuth with speech activity, joint VAD and DoA, and simultaneous source-number and DoA inference in simulated and real environments [He et al., 2018b, Vecchiotti et al., 2018, Nguyen et al., 2020a].Separate input branches or output branches were used to combine complementary features or jointly estimate localization and classification targets.Source: §4.2 · p. 11

Regression and input representations

  • CNNs also regress elevation, azimuth-elevation, or Cartesian coordinates from binaural, CPS, HRTF, raw-waveform, and spherical-harmonic inputs, while speaker masks can improve DoA estimation [Sivasankaran et al., 2018].These approaches include static and moving speakers, single and multiple sources, and two- or three-dimensional localization targets.Source: §4.2 · p. 11

Convolution design

  • 3D convolutions achieved better localization accuracy than 2D, complex, and depth-wise separable alternatives, while depth-wise separable convolutions provided a favorable accuracy–complexity trade-off at lower cost [Krause et al., 2020a].The 3D design operated across time, frequency, and channel axes but incurred high computational cost.Source: §4.2 · p. 11

Efficiency and robustness

  • Increasing dilation factors reduced the optimal depth of the original CNN while preserving similar SSL performance and lowering computational cost [Chakrabarty and Habets, 2019a, 2019b].Other efficiency-oriented systems used causal convolutions for real-time tracking or weighted SRP fusion to reduce noise and reverberation effects.Source: §4.2 · p. 12

4.3 Recurrent neural networks

Recurrent neural networks model temporal sequences, with LSTM and GRU variants addressing vanishing- and exploding-gradient difficulties. In sound source localization, few studies use only RNNs; reported applications align SED and DoA predictions or estimate TF masks for conventional DoA methods.

4.3 Recurrent neural networks

  • RNNs model temporal sequences, while LSTM and GRU cells mitigate the vanishing- and exploding-gradient problems affecting regular RNN training.These recurrent variants have become popular because they circumvent key training difficulties of standard RNNs.Source: §4.3 · p. 12
  • Few published SSL studies use only RNNs; Nguyen et al. [2021a] used one to match separately obtained sound-event-detection and direction-of-arrival predictions.The RNN determined which SED prediction corresponded to each DoA estimate.Source: §4.3 · p. 12
  • Wang et al. [2019] used a bidirectional LSTM to estimate a time-frequency mask, facilitating DoA estimation by conventional SRP or subspace methods.The mask enhanced the signal before conventional localization processing.Source: §4.3 · p. 12

4.4 Convolutional recurrent neural networks

CRNNs combine convolutional feature extraction with recurrent temporal integration and have been widely used for sound source localization since 2018. The Adavanne et al. CRNN became a DCASE SELD baseline that inspired numerous architectural, feature, training, and hybrid-method extensions.

Architecture

  • CRNNs exploit convolutional layers for SSL-relevant feature extraction and recurrent layers for integrating information over time.This combination explains their regular use for SSL since 2018.Source: §4.4 · p. 12

DCASE baseline

  • The Adavanne et al. [2019a] CRNN baseline directly estimated directions of arrival and localized overlapping sound events when they belonged to different sound types, inspiring many DCASE systems.Its architecture used convolutional and bidirectional GRU blocks with separate sound-event-detection and DoA estimation branches; earlier versions used an intermediate MUSIC-based spatial pseudo-spectrum output [Adavanne et al., 2018].Source: §4.4 · pp. 12–13, Figure 4 · p. 14

DCASE variants

  • Several DCASE-derived variants improved localization over the Adavanne et al. [2019a] baseline by modifying features, architectures, augmentation, or training, including Cao et al. [2019b]’s two-stage transfer-and-fine-tuning approach.Other changes included Gaussian-noise training, LSTM or GLU replacements, cross-channel power spectra, group delays, GCC-PHAT, data augmentation, and multi-branch inputs.Source: §4.4 · pp. 13–14

Alternative formulations

  • Beyond direct DoA estimation, CRNNs were used for source counting, task decomposition, TDoA-to-DoA inference, and sequential sound-event and localization processing.Nguyen et al. [2020c] combined SED, conventional histogram-based DoA estimation, and sequence matching to associate directions with sound classes, later extending the approach to moving sources with SMN ensembles [Nguyen et al., 2020b].Source: §4.4 · pp. 13–14

Reverberant environments

  • CRNN systems also achieved very good performance in simulated and real reverberant environments with static speakers, while later deeper architectures localized up to three simultaneous speakers.Perotin et al. used bidirectional LSTMs on FOA pseudo-intensity vectors for one or two speakers, and Grumiaux et al. [2021a] improved on the two-speaker CRNN by adding convolutional layers with less max-pooling.Source: §4.4 · p. 13

4.5 Residual neural networks

Residual connections support deeper sound source localization networks by enabling direct gradient flow and are combined with convolutional, recurrent, dilated, squeeze-excitation, and temporal convolutional components. Reported systems improve localization performance, efficiency, or challenge rankings across single-source localization and SELD tasks.

Residual connections

  • Residual connections bypass layer blocks, allowing gradients to flow directly through deeper networks and usually improving training [2016].They were introduced to address exploding or vanishing gradients and performance degradation in very deep networks.Source: §4.5 · p. 14

Residual and recurrent networks

  • Ranjan et al. [2019] combined ResNet with recurrent layers for SELD and reduced DoA error by more than 20° versus Adavanne et al. [2019a].Other residual systems used multichannel waveforms for single-source localization or combined ResNet with GRU and fully connected layers for SELD.Source: §4.5 · p. 15

Architectural extensions

  • Residual architectures were extended with squeeze-excitation modules, Conformer blocks, or indirect dereverberation and denoising pipelines for multi-source and FOA-based DoA estimation [2020; 2021].Squeeze-excitation modules model interdependencies between feature channels, while Yasuda et al. [2020] estimated DoA from a dereverberated and denoised pseudo-intensity vector.Source: §4.5 · p. 15

SELD architectures

  • Wang et al. [2020] achieved the best performance in the DCASE 2020 Challenge by ensembling residual neural network and recurrent-layer variants for DoA estimation.Shimada et al. [2020b] and Shimada et al. [2020a] also reported very good performance with residual convolutional-recurrent MMDenseLSTM systems for SELD.Source: §4.5 · p. 15

Temporal convolutional networks

  • Replacing recurrent layers with TCN blocks slightly improved SELD performance versus Adavanne et al. [2019a] while making hardware implementation more efficient [Guirguis et al., 2020].The TCN blocks comprised residual blocks with 1D dilated convolutions and increasing dilation factors.Source: §4.5 · p. 15

4.6 Attention-based neural networks

Attention mechanisms are increasingly used in sound source localization to emphasize relevant temporal or spectral information, with self-attention and Transformer-derived architectures often replacing recurrent units. Reported systems use attention for activity and direction estimation, source assignment, frequency masking, and cross-modal fusion.

Attention-based localization

  • Attention assigns learned weights to sequence vectors according to their relevance to outputs, motivating Transformer architectures that replace RNNs with attention models [Bahdanau et al., 2016; Vaswani et al., 2017].Self-attention models relationships among input vectors, while decoder attention captures their relevance to output vectors.Source: §4.6 · p. 15
  • Attention-based SSL systems combine convolutional layers and recurrent units with self-attention to estimate sound activity and azimuth/elevation, improve temporal information use, or focus on dominant frequency bins [Phan et al., 2020a,b; Schymura et al., 2020; Mack et al., 2020].Mack et al. use two attention stages with phase spectrogram inputs: one after the input and another after convolutional feature extraction.Source: §4.6 · pp. 15–16
  • Self-attention after a GRU estimated the association matrix between predictions and references, solving optimal assignment and producing large localization-error improvements [Adavanne et al., 2021].The association matrix matches predictions to references in multi-source localization.Source: §4.6 · p. 17

Cross-modal attention

  • Cross-modal attention separately embeds sound event detection and direction estimation outputs, aligns the embeddings through mirrored attention modules, and supports final estimation for up to three simultaneous sources [Lee et al., 2021a].The model first combines embeddings with a weighted linear operation, then applies a more complex alignment process.Source: §4.6 · p. 17

Overall trend

  • Recent SSL networks increasingly replace recurrent units with attention, which models longer-term dependencies at lower computational cost and benefits from parallel training [Vaswani et al., 2017].This trend is especially associated with multi-head self-attention modules.Source: §4.6 · p. 17

4.7 Encoder-decoder neural networks

Encoder-decoder networks learn compact representations and transform them into localization, separation, activity, or direction outputs. Surveyed variants include autoencoders, variational autoencoders, and U-Nets, with demonstrated gains in reverberant or weakly supervised settings.

4.7 Encoder-decoder neural networks

  • Encoder-decoder networks comprise an encoder that forms an input representation and a decoder that transforms it into the desired output, often yielding compact unsupervised representations [Goodfellow et al., 2016].Autoencoders are trained to reproduce their input, with a typically lower-dimensional bottleneck providing compressed encoding; convolutional and recurrent layers are also used.Source: §4.7 · p. 17, §4.7.1 · p. 17

4.7.1 Autoencoder

  • Autoencoders estimate DoA-related masks, grid-based multi-source activity, TDoA, and joint SED-DoA outputs, while adversarial training improves robustness to real data and unseen microphone arrays [Le Moing et al., 2021].An indirect AE-based TDoA method produced smoother estimates and outperformed classical GCC-PHAT, with an extension to two sources [Vera-Diaz et al., 2020; Vera-Diaz et al., 2021].Source: §4.7.1 · pp. 18–19

4.7.2 Variational autoencoder

  • Variational autoencoders enable semi-supervised sound source localization, outperforming SRP-PHAT and a supervised CNN in reverberant scenarios with limited labeled data [Bianco et al., 2020].The model generated inter-microphone RTF phases and jointly classified speaker DoA using abundant unlabeled data and limited labeled RTF data.Source: §4.7.2 · p. 19

4.7.3 U-Net architecture

  • U-Net systems support localization, separation, and SELD, including up to eight speakers on synthetic and real reverberant data and improved DoA error over a challenge baseline [Patel et al., 2020].Architectures use encoder-decoder skip connections; applications include angular-window separation, source-coordinate estimation, and event activity with DoA.Source: §4.7.3 · pp. 19–20

5 Input features · 5.1 Inter-channel features

The survey categorizes SSL inputs into signal representations, hand-crafted spatial features, and traditional signal-processing features, with inter-channel features emphasizing spatial and time-frequency information. This block focuses on relative transfer functions and binaural features, including their use for multi-source localization.

5 Input features

  • SSL systems commonly use signal-processing representations that encode spatial or time-frequency information, although different feature types are often concatenated at the input.The survey groups inputs into inter-channel, cross-correlation-based, spectrogram-based, Ambisonics, intensity-based, and multichannel-waveform features.Source: §5 · p. 20
  • Concatenating heterogeneous features usually improves performance by allowing the network to route pertinent information into an optimized higher-level representation.The cited explanation attributes this effect to the flexibility of the learning process, even when features contain redundant information.Source: §5 · p. 20

5.1.1 Relative transfer function (RTF)

  • An RTF is a frequency-dependent ratio of source-to-microphone acoustic transfer functions whose values depend strongly on source direction and microphone pairing.For more than two microphones, RTFs can be defined for each pair or relative to a reference microphone.Source: §5.1.1 · p. 20
  • RTF estimation becomes more sophisticated with background or sensor noise, while multiple sources can be handled across time-frequency bins by exploiting signal sparsity.Multiple estimated RTF sets across frequencies and, when appropriate, time frames can support multi-source localization.Source: §5.1.1 · p. 21
  • RTFs are commonly represented by real-valued pairs such as magnitude and phase, and Chazan et al. [2019], Hammer et al. [2021], and Bianco et al. [2020, 2021] used measured RTF arguments.Equivalent representations include real and imaginary parts or log-squared inter-channel power ratios with phase arguments.Source: §5.1.1 · p. 21

5.1.2 Binaural features

  • Binaural features use two-channel recordings designed to reproduce human hearing, with HRTFs describing anechoic head-related propagation and BRIRs incorporating room reverberation.Binaural features derive from binaural impulse responses and their frequency-domain representations, whose values depend on source direction.Source: §5.1.2 · p. 21
  • ILD, IPD, and ITD encode source position in frequency-dependent vectors, while their relationship to RTFs mainly reflects the recording setup.RTFs generalize the concept to multichannel systems, whereas binaural features specifically describe a two-channel binaural setup.Source: §5.1.2 · p. 21
  • Binaural SSL systems combine or separately process ILD, IPD, ITD, cross-correlation, and spectral features using MLP and CNN architectures.These representations have been used for denoising IPD, reducing feature dimension, estimating elevation, and providing information about multiple simultaneous sources.Source: §5.1.2 · pp. 21–22

5.2 Cross-correlation (CC)-based features · 5.3 Spectrogram-based features · 5.4 Ambisonic signal representation

The survey organizes localization inputs around cross-channel spatial features, spectrogram representations, and Ambisonics, spanning engineered transforms, raw multichannel tensors, and spherical-harmonic signals. Reported studies show gains from neural processing of these representations, while performance depends on feature choice, temporal context, and Ambisonic order.

5.2 Cross-correlation (CC)-based features

  • GCC-PHAT encodes microphone-pair delays for MLP or convolutional models and is less sensitive to speech variation, but noise and reverberation can degrade it.Neural networks have been used to improve robustness to these disturbances, with features formed across microphone pairs, delays, and frames.Source: §5.2 · p. 22
  • CPS, cross-spectrum, MUSIC pseudo-spectra, and power maps extend CC-based localization inputs across CRNN, CNN, MLP, and other neural architectures.CPS-based systems improved over the Adavanne et al. [2019a] baseline, while DNN acoustic imaging outperformed traditional deconvolution in performance or computation time.Source: §5.2 · p. 22
  • SRP-based power maps improved localization over traditional SRP-PHAT, while CNN and U-Net models produced deconvolved acoustic maps beyond the array’s intrinsic resolution.Salvati et al. [2018] fed normalized steered-response power over candidate TDoAs into a convolutional layer; later acoustic-imaging systems used CNNs or U-Nets.Source: §5.2 · p. 22

5.3 Spectrogram-based features

  • Spectrogram-based systems provide raw multichannel information as 3D time–frequency–channel tensors, allowing networks to learn inter-channel differences and exploit sparse time–frequency structure.This approach avoids explicit channel-dimension preprocessing while commonly using multichannel STFT spectrograms.Source: §5.3 · pp. 22–23
  • Models use frame-independent spectral matrices or temporally stacked spectrograms, with magnitude, phase, real–imaginary, and redundant complex features represented through varied network branches.Phase-only inputs can simplify training-data generation from white noise, while some systems combine magnitude and phase or dedicate separate branches to them.Source: §5.3 · p. 23
  • Mel- and Bark-scale spectrograms replace equally spaced STFT bins with perceptual non-linear subbands and have been explored in several SSL neural networks.The perceptual scales provide higher resolution in low-frequency subbands than in high-frequency subbands.Source: §5.3 · p. 23

5.4 Ambisonic signal representation

  • Ambisonics represents spatial sound with spherical-harmonic coefficients independently of microphone-array configuration; FOA has four channels, whereas HOA has more than four.Ambisonic channels are ideally in phase, but frequency-independent response is limited by high-frequency spatial aliasing and low-frequency noise amplification; reverberation and additive noise enter the multichannel model through reflected components and a noise vector.Source: §5.4 · pp. 23–24
  • Higher-order Ambisonics improved localization accuracy as order increased, although it remained below FOA pseudo-intensity features; quaternion inputs also improved over baseline.Poschadel et al. [2021a,b] compared orders 1–4, while Comminiello et al. [2019] adapted a network for quaternion-based FOA features.Source: §5.4 · p. 24

5.5 Intensity-based features · 5.6 Waveforms · 5.7 Other types of features

The survey covers intensity-based representations, raw multichannel waveforms, and other specialized features for deep-learning sound source localization. Intensity features exploit directional information, waveform methods pursue end-to-end representation learning, and unusual features add source-signal characteristics.

5.5 Intensity-based features

  • Intensity vectors encode directional information, with active intensity aligned to the source DoA under single-source free-field conditions.Sound intensity combines pressure and particle velocity; the real part is especially useful for localization, while particle velocity may require specialized sensors or pressure-based approximation.Source: §5.5 · p. 24
  • Particle velocity can be approximated from spatial pressure gradients or first-order Ambisonics, yielding complex pseudo-intensity vectors for localization.Approaches include finite differences and FOA channels, under conditions where particle velocity is proportional to the spatial gradient of sound pressure.Source: §5.5 · p. 24
  • Perotin et al. [2018b] reported that Ambisonics pseudo-intensity outperformed raw Ambisonics waveforms and traditional Ambisonics methods, with active and reactive components further improving performance.Energy normalization produced features similar to spherical-harmonic RTFs, while Yasuda et al. [2020] used two CRNNs to estimate cleaner intensity vectors by denoising, separating, and suppressing reverberation.Source: §5.5 · p. 24
  • Liu et al. [2021] estimated non-Ambisonics complex intensity from four-channel pressure and orthogonal velocity differences, retaining PHAT-weighted active intensity for reverberation robustness.The method used average pressure across channels and pressure differences from two microphone pairs to estimate particle velocity components.Source: §5.5 · pp. 24–25

5.6 Waveforms

  • Since 2018, researchers have fed raw multichannel waveforms directly to neural networks, aiming for end-to-end learned SSL representations without handcrafted features or preprocessing.Architectures include stacked 1D or 2D convolutions, residual and squeeze-excitation CNNs, autoencoders, and temporally shifted inputs based on TDoA or candidate source locations.Source: §5.6 · p. 25
  • Waveform-based models often require more complex feature-extraction architectures, and raw-input learning can become harder under noise or harm generalization.Despite these limitations, learned input weights can resemble conventional filterbanks used in SSL and speech/audio processing.Source: §5.6 · p. 25

5.7 Other types of features

  • Varzandeh et al. [2020] combined GCC-PHAT with a periodicity-degree feature that conveys the source’s harmonic content to a CNN.The periodicity degree is the ratio of harmonic power at a given period to total signal power for the frame.Source: §5.7 · p. 25

6 Output strategies

The survey organizes DoA output strategies into classification, regression, and non-direct approaches. Classification discretizes source space into zones, regression predicts continuous coordinates, and non-direct methods estimate intermediate quantities for conventional localization algorithms.

6.1 DoA estimation via classification

  • Classification networks divide the search space into zones and output source-activity probabilities, while peak picking yields single- or multi-source DoA estimates with known or unknown source count.Softmax supports single-source localization, whereas sigmoid independently scores zones for multi-source localization; thresholds or the J highest peaks determine active sources.Source: §6.1 · p. 26

6.1.1 Spherical coordinates

  • Classification grids commonly represent azimuth and elevation with separate branches, paired angular classes, or spherical grids, while distance estimation remains rarely investigated.Reported spherical grids include 429 classes for quasi-uniform full-sphere sampling and 432 classes for full azimuth with elevation limited to [−60°, 60°].Source: §6.1.1 · p. 26

6.1.2 Cartesian coordinates

  • Cartesian classification divides the horizontal (x, y) plane into equal-sized regions, but decreasing angular differences at larger distances make regression generally preferable.Le Moing et al. [2021, 2020] and Ma and Liu [2018] used this representation.Source: §6.1.2 · p. 26

6.2 DoA estimation via regression

  • Regression directly predicts continuous Cartesian or spherical coordinates without quantization, potentially improving accuracy but requiring known source count and addressing source permutation in multi-source localization.Permutation-invariant training is one proposed solution to the ambiguous target-output association during training and evaluation [Subramanian et al., 2021b].Source: §6.2 · p. 27

6.2.1 Spherical coordinates

  • Spherical regression predicts azimuth and elevation directly, through separate outputs, quaternion representations, or hybrid classification-regression designs that also estimate activity and distance.Sudo et al. [2019] used cosine and sine components in a quaternion to avoid discontinuities at angle boundaries.Source: §6.2.1 · p. 27

6.2.2 Cartesian coordinates

  • Cartesian regression estimates normalized (x, y) or (x, y, z) coordinates, sometimes jointly encoding source activity through the ACCDOA vector representation.ACCDOA multiplies normalized Cartesian coordinates by activity, allowing activity detection from vector norm and DoA recovery from normalized coordinates [Shimada et al., 2020a].Source: §6.2.2 · p. 27

6.3 Non-direct DoA estimation

  • Non-direct methods use neural networks to estimate masks, cleaned features, TDoA-related quantities, direct-path RTFs, or intermediate signals that conventional algorithms convert into DoA.Examples include SRP-PHAT, GCC-PHAT, MUSIC, intensity-based methods, dictionary matching, and correlation-based location selection [Pertilä and Cakir, 2017; Wang et al., 2019; Yang et al., 2021a].Source: §6.3 · p. 28
  • Other non-direct systems scan candidate directions or expose localization as an intermediate result within jointly trained separation, beamforming, and recognition pipelines.Jenrungrot et al. [2020] progressively narrowed an angular window, while Subramanian et al. [2021a] jointly trained differentiable modules with an ASR-specific cost.Source: §6.3 · p. 28

7 Data

SSL data strategies balance the scarcity of labeled real recordings against the need to model reverberant, noisy indoor environments. Surveys therefore emphasize synthetic RIR-based data, real recordings, and augmentation methods that expand training diversity or improve robustness.

7.1 Synthetic data

  • Training on speech or music, and especially combined speech, noise, and sound events, outperformed noise-only or matched-signal training for localization [Vargas et al., 2021; Krause et al., 2021].Synthetic data commonly convolve dry signals with simulated RIRs across varied source and array positions, room dimensions, and reverberation times.Source: §7.1 · pp. 28–29
  • Simulation with scattering effects and directional sources produced the best SSL performance, while a lower-complexity model matched usual ISM performance with greater computational efficiency [Gelderblom et al., 2021; Hübner et al., 2021].Geometrical-acoustics methods remain unable to precisely reproduce wave phenomena such as diffraction [Kuttruff, 2016].Source: §7.1 · p. 29

7.2 Real data

  • Only a few labeled real-recording datasets exist, including impulse-response corpora, DCASE reverberant and noisy sound-event data, and LOCATA recordings spanning static or mobile sources and microphones [Adavanne et al., 2019b; Politis et al., 2020a, 2021; Evers et al., 2020].Real data are crucial for assessing robustness in real-world environments, although collecting recordings with source-position metadata is cumbersome.Source: §7.2 · pp. 29–30

7.3 Data augmentation techniques

  • Data augmentation creates additional training examples without new recordings; FOA channel swapping or label-oriented rotation improved performance, whereas channel-oriented rotation performed worst [Mazzon et al., 2019].Other surveyed methods include SpecAugment, Mixup, pitch shifting, block mixing, spatial-spectral recombination, multi-source mixing, and random TF masking [Zhang et al., 2019a; Wang et al., 2021].Source: §7.3 · pp. 30–31

8 Learning strategies

Deep-learning sound source localization predominantly uses supervised learning, while semi-supervised and weakly supervised strategies address limited labeled real-world data. The surveyed methods adapt training objectives and data requirements to single-source, multi-source, regression, simulated, and real-data settings.

8.1 Supervised learning

  • Supervised learning dominates DNN-based SSL, but limited real-world labeled datasets motivate simulation, augmentation, and alternative training strategies.Supervised systems minimize average loss against labeled targets; common objectives include categorical cross-entropy for single-source classification, binary cross-entropy for multi-source classification, and mean square error for regression, with angular error [Perotin et al., 2019a] and the ℓ1-norm [Jenrungrot et al., 2020] also used.Source: §8 · p. 31, §8.1 · p. 31

8.2 Semi-supervised and weakly supervised learning

  • Semi-supervised learning pre-trains or combines supervised and unsupervised objectives to adapt SSL models to unseen conditions or real data without requiring labels for every example.Examples include entropy minimization with overfitting control [Takeda and Komatani, 2017; Takeda et al., 2018], a VAE-based system combining supervised and unsupervised terms [Bianco et al., 2020], and adversarial adaptation from simulated to real data [Le Moing et al., 2021; Goodfellow et al., 2014].Source: §8.2 · pp. 31–32
  • Weak supervision reduces label dependence by using weak source-count labels or triplet relationships to constrain localization predictions.He et al. [2019a, 2021a] adapted the cost function to use the presumed number of sources and reduce incoherent predictions, while Opochinsky et al. [2019] used query, positive, and negative samples in a triplet loss.Source: §8.2 · p. 32

9 Conclusions and perspectives

The survey organizes DL-based SSL literature from 2011–2021 by method characteristics and identifies future directions addressing real-world adaptation, model flexibility, multi-task learning, hybrid signal processing, and moving-source tracking. These perspectives target limited annotations, setup mismatch, and robust localization in complex acoustic environments.

9 Conclusions and perspectives

  • The survey categorizes DL-based SSL methods from 2011–2021 by source configuration, architecture, inputs, outputs, datasets, and learning strategy, with Tables II–V enabling targeted literature searches.The tables collect reviewed papers and report the main characteristics used in the taxonomy.Source: §9 · p. 32

9.1 Adaptation to (limited sets of) real-world data

  • Simulated-to-real performance drops because labeled real-world data are scarce and common simulations poorly model room geometry, scattering, diffraction, and practical microphone placement.Proposed remedies include more realistic but computationally heavier simulators, progressive fine-tuning from simulated to real SRIRs and recordings, domain adaptation, transfer learning, and deep manifold learning with limited labels [Kouw and Loog, 2019] [Bengio, 2012, Zhuang et al., 2020] [Le Moing et al., 2021] [Takeda and Komatani, 2017] [Laufer-Goldshtein et al., 2020] [Bianco et al., 2020, 2021].Source: §9.1 · pp. 32–33

9.2 Flexibility of the trained models

  • DL-based SSL models usually require fixed microphone-array geometry and signal parameters, motivating array-agnostic inputs, transfer learning, domain adaptation, and point-cloud processing [Adavanne et al., 2018, Perotin et al., 2019b, Grumiaux et al., 2021a] [Subramani and Smaragdis, 2021].Existing partial remedies include Ambisonics, CPS eigenvectors, spatial pseudo-spectra, and array-invariant techniques [Takeda and Komatani, 2016b] [Nguyen et al., 2020a, Wu et al., 2021b] [Luo et al., 2020].Source: §9.2 · p. 33

9.3 Multi-task learning

  • Multi-task SSL commonly shares feature extraction across SED and localization branches, while ACCDOA enables joint processing to the final layer; adding source counting improved SSL performance [Politis et al., 2020b] [Grumiaux et al., 2021a].Further directions include combining SSL with source separation or ASR, and task-oriented localization-beamforming-ASR systems can exploit abundant labeled speech through pre-trained ASR blocks [Subramanian et al., 2021a].Source: §9.3 · pp. 33–34

9.4 Combination of DL and conventional SP techniques

  • A major perspective is combining DL with conventional multichannel signal processing so models retain learned representations while exploiting explicit physical and propagation priors.Future hybrid systems could jointly optimize localization, diarization, separation, and enhancement using deep generative models and established joint-optimization frameworks [Goodfellow et al., 2014] [Kingma and Welling, 2014] [Girin et al., 2021] [Vincent et al., 2018, Gannot et al., 2017] [Engel et al., 2020, Shlezinger et al., 2020].Source: §9.4 · p. 34

9.5 Moving sources and deep tracking

  • Moving-source SSL and tracking remain underdeveloped; future systems should combine deep appearance and dynamical models, potentially replacing RNNs with Transformers and leveraging unsupervised multi-object tracking [Ciaparrone et al., 2020].Tracking must connect framewise SSL estimates, model source dynamics and identities, and handle track births and deaths; annotation scarcity makes unsupervised approaches especially relevant [Vo et al., 2015] [Luo et al., 2021].Source: §9.5 · pp. 34–35, Table 2 · p. 35

Related papers

Loading 2109.03465v3…