PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Sound Source Localization with Deep Learning Methods

Pierre-Amaury Grumiaux, Srđan Kitić, Laurent Girin, Alexandre Guérin

arXiv:2109.03465v3cs.SDcs.LGeess.AS

TL;DR

Sound source localization remains challenging in noise, reverberation, and multiple-source conditions, while conventional methods perform poorly in such scenarios. This paper surveys deep-learning-based localization methods from 2011 to 2021, organizing them by architectures, inputs, outputs, datasets, and learning strategies. It provides a structured overview and reference tables for finding methods by target characteristics.

  • Problem

    Sound source localization remains challenging because conventional signal-processing methods perform poorly when noise, reverberation, and several simultaneous sources are present.

  • Method

    The paper comprehensively surveys deep-learning-based sound source localization literature and categorizes methods by source configuration, architecture, input type, output strategy, datasets, and learning strategy.

  • Results

    The survey organizes the literature into a taxonomy and reference tables covering the main characteristics of reviewed deep-learning-based sound source localization methods.

  • Takeaways & Limitations

    The tables support quick searches for deep-learning localization methods with specified target characteristics.

  • Takeaways & Limitations

    Deep-learning localization methods generally assume the same microphone number, geometry, and directivity during training and inference, requiring retraining for different arrays.

Abstract

from arXiv · show

This article is a survey on deep learning methods for single and multiple sound source localization. We are particularly interested in sound source localization in indoor/domestic environment, where reverberation and diffuse noise are present. We provide an exhaustive topography of the neural-based localization literature in this context, organized according to several aspects: the neural network architecture, the type of input features, the output strategy (classification or regression), the types of data used for model training and evaluation, and the model training strategy. This way, an interested reader can easily comprehend the vast panorama of the deep learning-based sound source localization methods. Tables summarizing the literature survey are provided at the end of the paper for a quick search of methods with a given set of target characteristics.

1 Introduction

Sound source localization estimates source position or, commonly, direction of arrival from multichannel recordings, but remains difficult under noise, reverberation, and simultaneous sources. This survey maps deep-learning approaches by architecture, features, outputs, datasets, and training strategies.

  • Scope and motivation: SSL estimates one or several sound sources’ positions relative to a microphone array, usually focusing on azimuth and elevation rather than distance.The survey uses SSL and DoA estimation interchangeably unless otherwise specified.
  • Scope and motivation: Traditional signal-processing methods perform poorly in common difficult scenarios involving noise, reverberation, and several simultaneously emitting sources.These conditions motivate data-driven deep-learning approaches for indoor and domestic environments.
  • Deep-learning motivation: CNN, CRNN, and other DNN-based systems have reported higher DoA accuracy or lower angular error than SRP-PHAT, ICA, and MUSIC in low-SNR or reverberant conditions.Reported examples include a two-fold accuracy increase, a 25% accuracy increase, and a 50% reduction in average angular error.
  • Survey contribution: The paper reviews 156 papers published from 2011 to 2021 and organizes them into a taxonomy covering methods, configurations, datasets, and learning strategies.Four summary tables support rapid selection of methods with desired characteristics.
  • General pipeline: A typical DL-based SSL pipeline extracts features from multichannel microphone-array signals, feeds them to a DNN, and outputs a source location or DoA estimate.Recent systems may bypass feature extraction and feed multichannel raw data directly into the network.
  • General pipeline: DNNs are designed to learn the complex relationship between multichannel acoustic information and source location when supplied with sufficiently many representative training examples.Multichannel differences reflect distinct propagation paths, including direct sound and indoor reverberation.

2 Acoustic environment and sound source configurations

The survey focuses on indoor SSL under reverberation and noise, covering single- and multi-source configurations, source counting, and moving sources. It excludes standalone source tracking while noting how training data affect localization of moving sources.

  • Acoustic environment: Indoor SSL considers microphone arrays and sources in closed rooms, typically offices or domestic environments, where reverberation arises from multipath propagation.The survey treats reverberation as an inherent feature of its target environment.
  • Acoustic environment: Reverberation and environmental noise make SSL more difficult than anechoic localization.Noise may include interfering sound sources such as television, music, pets, or street noise.
  • Sound sources: Speech sources receive substantial attention because localization supports related tasks such as speech enhancement and recognition.These systems are often specialized for estimating speech-source DoAs.
  • Number of sources: Single-source SSL addresses mixtures with at most one active source, whereas multi-source SSL covers simultaneously emitting sources regardless of source type.Multi-source localization is described as substantially more difficult than single-source SSL.
  • Number of sources: Multi-source systems may assume the number of sources, estimate it separately, or jointly estimate source count and DoAs with multiple output branches.One example classifies up to four sources while another branch classifies azimuth regions.
  • Moving sources: The survey excludes source tracking as a standalone problem but reports that training with moving-source data can improve moving-source localization, while static-source training can still retain fair-to-good performance.The availability of real-world moving-source datasets is limited, and simulating moving sources is cumbersome.

3 Conventional SSL methods

Conventional SSL methods use signal models and signal-processing operations to infer source directions from multichannel recordings. The survey reviews time-delay, acoustic-map, subspace, mixture-model, sparse-recovery, and source-separation approaches as context and baselines for DL methods.

  • Time-delay methods: GCC-PHAT estimates a time difference of arrival by maximizing a phase-transformed cross-correlation between microphone signals.It is especially common for two-microphone arrays and can be extended across multiple microphone pairs.
  • Time-delay methods: Using multiple microphone pairs can improve GCC-based localization compared with relying on a single pair.The approach has been extended beyond two-microphone arrays.
  • Acoustic maps: SRP methods construct an acoustic power map over candidate spatial positions, whose local maxima indicate likely source directions.The PHAT version is popular because it is more robust to reverberation and can average GCC-PHAT values across microphone pairs.
  • Acoustic maps: Sound-intensity localization can avoid the computational cost of SRP grid searches and efficiently estimate DoA in favorable acoustic conditions.Its validity depends on sound intensity being parallel to the propagating wave direction.
  • Subspace methods: Subspace methods compute a time-averaged CPS matrix and use its eigenstructure to separate signal and noise subspaces.MUSIC probes directions using these subspaces, whereas ESPRIT directly infers DoA from source-subspace structure but can be less accurate.
  • Probabilistic methods: Probabilistic mixture models typically assign Gaussian components to sources or candidate positions and estimate parameters using likelihood optimization or EM.Gaussian mixture regression provides a locally linear but globally nonlinear regression formulation, while Bayesian variants incorporate priors.
  • Sparse and separation methods: Compressive sensing and sparse recovery formulate SSL around low-dimensional or sparse representations, often assuming spatial sparsity.ICA instead exploits statistical independence to retrieve sources and support multi-source localization.

4 Neural network architectures for SSL

The survey organizes neural SSL architectures by layer type using an inclusive progression from feedforward networks through CNNs, RNNs, CRNNs, residual and attention-based models. The reviewed architectures transform multichannel or derived acoustic features into DoA classifications or estimates.

  • Architecture taxonomy: The architecture review uses an inclusive complexity ordering, allowing a network in one category to contain layers from earlier categories.This ordering supports comparisons across increasingly composite neural designs.
  • Feedforward and subband designs: Takeda et al.'s MLP uses separate subband feedforward layers to process CPS-matrix eigenvectors as directional activation functions before progressively integrating subband representations.The final output layer classifies among candidate DoAs.
  • Architecture taxonomy: The survey presents FFNNs first, followed by CNNs, RNNs, CRNNs, residual networks, attention-based networks, and encoder-based systems.The progression is organized according to the types of layers used.

4.1 Feedforward neural networks

Feedforward SSL systems range from shallow perceptrons and MLPs to deeper and subband-specific designs. Reported variants estimate source count, DoA, distance, source activity, or enhanced features, while CNN comparisons provide architectural context.

  • Definitions and early systems: An FFNN passes data forward through fully connected layers, with an MLP defined here as an FFNN containing one or more hidden layers.A perceptron has no hidden layer.
  • Definitions and early systems: Early neural SSL studies often used shallow networks in simplified settings such as direct-path propagation only.These pioneering systems are characterized as applying to unrealistic setups.
  • MLP variants: Kim and Ling used separate MLPs for source-count estimation and localization conditioned on each considered source count, while Tsuzuki et al. used a complex-valued MLP for complex features.The complex-valued model produced better results than a real-valued MLP in the cited work.
  • MLP variants: Deeper MLPs were evaluated with different input features and hidden-layer widths, including a three-hidden-layer model estimating azimuth and distance.A four-hidden-layer MLP was also tested for multi-source localization and speech/non-speech classification.
  • CNN comparison: Hirvonen's CNN classified single-source speech or music into eight spatial regions using multichannel magnitude spectrograms and fully connected layers.The architecture used four convolutional layers and four fully connected layers without classical pooling.
  • Subband MLPs: Subband MLPs exploit the assumption that each frequency band is often dominated by one source, enabling training on single-source data and integration of azimuth probabilities across bands.Takeda et al. proposed related subband architectures.
  • Feature enhancement: An MLP can also enhance interaural phase difference features before a separate DoA estimator performs localization.This is an indirect use of an MLP rather than direct DoA prediction.

4.2 Convolutional neural networks

CNNs extract localization features through convolutional processing and have been adapted with varied dimensionality, receptive fields, inputs, and temporal aggregation. Reported studies explore accuracy–complexity trade-offs, robustness, real-time operation, and indirect feature weighting.

  • CNN foundations: CNNs are widely used for SSL because convolutional layers extract spatially structured patterns from multichannel acoustic features.They have been applied to spectrograms, SRP-PHAT maps, raw waveforms, binaural features, and other representations.
  • Convolution variants: 3D convolutions across time, frequency, and channel axes achieved better localization accuracy than several 2D alternatives, but incurred high computational cost.Depth-wise separable convolutions offered a favorable accuracy–model-complexity trade-off.
  • Architectural designs: CNN-based SSL systems use diverse architectures, including 2D-to-1D temporal aggregation, fully causal 3D and 1D convolutions, and multi-task branches.These designs target temporal integration, real-time adaptation, source tracking, or joint detection and localization.
  • Output and input strategies: CNNs have been used for direct DoA or coordinate prediction and for indirect localization by weighting narrow-band SRP components.The indirect approach reduced noise and reverberation effects in the reported test examples.
  • Dilated convolutions: Dilated convolutions can reduce the required number of convolutional layers while maintaining similar SSL performance and lowering computational cost.The dilation factor widens the effective kernel without increasing the number of parameters in the same way as a classical wider kernel.

4.3 Recurrent neural networks

Standalone RNN applications to SSL are relatively limited, while recurrent models are often combined with convolutional layers. Reported uses include aligning separately generated SED and DoA predictions and estimating masks for signal enhancement.

  • RNN foundations: RNNs model temporal sequences, with LSTM and GRU variants designed to mitigate vanishing- and exploding-gradient difficulties.These recurrent architectures are used to integrate information over time in acoustic processing.
  • Applications: Only a few published SSL systems use RNNs alone because recurrent layers are more often combined with convolutional layers.Examples include an RNN matching SED predictions to DoA estimates and a bidirectional LSTM estimating a time–frequency mask for enhancement.

4.4 Convolutional recurrent neural networks

CRNNs combine convolutional feature extraction with recurrent temporal integration and have become a major architecture for SSL and SELD. The surveyed systems vary in intermediate representations, task decomposition, inputs, and output design, with several reported baseline improvements.

  • CRNN rationale: CRNNs combine convolutional layers for feature extraction with recurrent layers for integrating information over time.This combination has been regularly exploited for SSL since 2018.
  • Adavanne architecture: Adavanne et al.’s CRNN used FOA features, convolutional and BGRU blocks, an intermediate MUSIC-derived SPS representation, and final azimuth–elevation classification.The SPS intermediary was intended to provide a representation useful for SSL learning.
  • Adavanne architecture: Later Adavanne variants removed the intermediary SPS and directly estimated DoA, becoming a DCASE Task 3 baseline that inspired many subsequent systems.The architecture could localize and detect overlapping events when the events had different types.
  • CRNN variations: CRNN extensions modified inputs, activations, pooling, recurrent units, output branches, and loss functions, with some studies reporting efficiency or performance improvements over baselines.Examples include Gaussian-noise augmentation, Bark-scale spectrograms, quaternion FOA, separate subtask networks, and alternative task splitting.
  • Hybrid designs: A two-step hybrid CRNN approach combined SED, conventional histogram-based DoA estimation, and sequence matching to associate the estimated event and localization sequences.The method used a second CRNN-based sequence matching network after the initial SED and DoA stages.

4.5 Residual neural networks

Residual architectures add skip connections to support deeper SSL networks and are combined with recurrent, dilated, squeeze-excitation, and temporal-convolution components. Reported systems target accuracy, computational efficiency, and robust SELD processing.

  • Residual foundations: Residual connections allow features and gradients to bypass layer blocks, supporting deeper networks without the degradation associated with difficult gradient flow.Residual architectures were introduced to address exploding or vanishing gradients and declining performance in very deep networks.
  • Residual SSL systems: Residual and dilated-convolution networks have been applied to raw multichannel waveforms, while residual–recurrent SELD systems have also been reported.One residual–recurrent system reduced DoA error by more than 20° relative to the Adavanne et al. baseline.
  • Residual SSL systems: Residual blocks combined with squeeze-excitation modules were placed before BGRU layers in a DCASE 2020 SELD architecture.The squeeze-excitation modules were intended to improve modeling of interdependencies between input feature channels.
  • Challenge systems: MMDenseLSTM adaptations and ensembles of residual and recurrent networks achieved strong DCASE 2020 Challenge results, with the ensemble reported as best-performing.MMDenseLSTM was originally developed for sound source separation before adaptation to SELD.
  • Temporal convolution: Replacing recurrent layers with TCN blocks made hardware implementation more efficient while slightly improving SELD performance over the Adavanne et al. baseline.The TCN blocks used residual structures and increasing dilation factors.
  • Indirect residual processing: A cascaded CRNN approach used residual processing to dereverberate FOA pseudo-intensity vectors and another CRNN to estimate a noise-attenuating time–frequency mask before DoA estimation.The final DoA estimate was obtained from the dereverberated and denoised pseudo-intensity vector.

4.6 Attention-based neural networks

Attention mechanisms are increasingly used in deep-learning SSL, including self-attention, multi-head self-attention, and cross-modal attention. They support activity and DoA estimation, association of predictions with references, and longer-term temporal modeling.

  • Attention mechanisms: Attention assigns different weights to temporal-sequence vectors according to their relevance for estimating output representations.The mechanism was originally introduced for sequence-to-sequence models and is now applied to SSL.
  • Self-attention for SSL: Attention-based SSL systems combine convolutional layers and recurrent or self-attention modules to estimate sound-event activity and DoA over time.Examples include BGRU followed by self-attention and CRNNs augmented with attention.
  • Self-attention for SSL: Self-attention after a GRU estimates an association matrix matching predictions with references, solving the optimal assignment problem and producing large localization-error improvements.The association mechanism directly addresses prediction-reference matching in multi-source localization.
  • Multi-head self-attention: Multi-head self-attention and Transformer-derived architectures model spectral characteristics and local and global context in recent SSL systems.Reported architectures include Transformer encoders, Conformers, and convolutional modules followed by MHSA.
  • Cross-modal attention: Cross-modal attention uses separate sound-event-detection and DoA-estimation embeddings as two data streams for SSL.This contrasts with systems whose initial blocks are shared between the two tasks.
  • Attention mechanisms: Attention modules, especially MHSA, tend to replace recurrent units because they model longer-term dependencies at lower computational cost and benefit from parallel training.The survey identifies this as a general recent trend in SSL DNNs.

4.7 Encoder-decoder neural networks

Encoder-decoder architectures transform input features into intermediate representations and then desired localization outputs. Autoencoders, variational autoencoders, and U-Nets support tasks including DoA estimation, source separation, source activity detection, and coordinate prediction.

  • Encoder-decoder networks: Encoder-decoder networks use an encoder to produce an input representation and a decoder to transform it into the desired output.This architecture is widely explored for compact data representations.
  • Autoencoders: Autoencoders are trained to reproduce their inputs, with a potentially lower-dimensional bottleneck providing a compressed representation.The survey notes convolutional and recurrent variants alongside feed-forward autoencoders.
  • Autoencoders: Autoencoder-based SSL methods estimate DoA-specific TF masks, source-position activity over an (x, y) grid, or smoothed TDoA representations.These designs support source separation and multiple-source localization.
  • U-Net architecture: U-Net systems use encoder-decoder stages with skip connections to preserve information and can output one TF mask per considered DoA.Applications include source separation and joint localization-separation, including moving-speaker scenarios.
  • Variational autoencoders: Variational autoencoders model output and latent probability distributions, enabling generation by sampling decoder distributions.Bianco et al. used a convolutional VAE to generate inter-microphone RTF phases jointly with a DoA classifier.
  • Variational autoencoders: A semi-supervised VAE using abundant unlabeled RTF data and limited labeled data outperformed SRP-PHAT and a supervised CNN in reverberant scenarios.The method was reported as an early VAE application to SSL.
  • U-Net architecture: U-Net variants combine convolutional blocks with BGRU layers for SELD or map GCC-PHAT features through ray space to source coordinates.One reported SELD system improved DoA error over the Adavanne et al. baseline.

5 Input features

Deep-learning SSL systems use waveform, spectrogram, hand-crafted, and conventional signal-processing features, often emphasizing spatial or time-frequency information. Common categories include inter-channel, cross-correlation, spectrogram, Ambisonics, intensity, and multichannel-waveform features.

  • Feature categories: SSL input features include low-level waveforms and spectrograms, hand-crafted binaural features, and features borrowed from MUSIC or GCC-PHAT.The survey organizes these into several feature categories for comparison.
  • Feature selection: Most SSL neural networks use signal-processing representations that emphasize spatial and time-frequency information rather than learning entirely from raw data.The survey relates this pattern partly to the relatively modest size of typical SSL architectures.
  • Feature selection: Concatenating features of different types commonly has a positive performance impact, as networks can route and merge pertinent information in later layers.The survey notes that such features may contain redundant information.
  • Inter-channel features: Relative transfer functions represent inter-channel spatial information as frequency-dependent ratios of source-to-microphone acoustic transfer functions.They can be estimated from channel STFT ratios under a single active directional source with negligible diffuse noise.
  • Inter-channel features: With multiple sources, TF-domain sparsity permits separate RTF estimation in bins assumed to contain at most one active source.Background or sensor noise requires more sophisticated RTF estimation procedures.
  • Inter-channel features: RTF representations are commonly converted to real-valued pairs using real and imaginary parts or modulus and argument.Inter-channel power ratios in decibels and phase differences are also used.
  • Binaural features: Binaural features derive from a specific two-channel dummy-head recording setup designed to reproduce human hearing effects relevant to localization.They include interaural level, phase, and time differences.
  • Binaural features: RTFs are a more general multichannel concept, whereas binaural features specialize the representation to a two-channel binaural setup.ILD and IPD are closely related to RTF log-power and argument representations.

5.2 Cross-correlation (CC)-based features

Deep-learning SSL systems use diverse spatial and spectro-temporal representations, including cross-correlation features, spectral maps, raw multichannel signals, spectrograms, and Ambisonics. These representations differ in how explicitly they encode inter-channel or spatial information before neural processing.

  • Cross-correlation features: GCC-PHAT features encode inter-channel delays and are commonly supplied to MLPs or convolutional networks for localization.GCC-PHAT is less sensitive to speech variations than standard cross-correlation but can be affected by noise and reverberation.
  • Cross-power spectrum: CPS features, which are linked to cross-correlation through a Fourier transform, have been used as CRNN inputs to improve localization over a baseline.
  • Spatial maps: MUSIC pseudo-spectra and steered-response power maps provide spatial representations that can be processed by CNNs or 3D convolutional layers.A reported power-map approach improved localization performance compared with traditional SRP-PHAT.
  • Spectrogram features: Multichannel spectrograms may use magnitude, phase, real and imaginary components, or combinations of these features as neural-network inputs.Some systems also use perceptually scaled mel- or Bark-frequency representations and separate branches for magnitude and phase.
  • Ambisonic representation: Ambisonics represents spatial sound-field properties with spherical-harmonic coefficients while remaining agnostic to microphone-array configuration.Its practical frequency-independent behavior is limited by spatial aliasing at high frequencies and noise amplification at low frequencies.

5.5 Intensity-based features

Intensity-based SSL features derive directional information from sound pressure and particle-velocity relationships, often using Ambisonic channels or multichannel pressure differences. Deep networks have been applied to these features alongside alternative waveform and periodicity representations.

  • Intensity principle: Active sound intensity is useful for localization because its real part is orthogonal to the sound-pressure wavefront.Particle velocity can under certain conditions be approximated from spatial pressure gradients or first-order Ambisonic channels.
  • Ambisonic intensity: The first Ambisonic pseudo-intensity-vector SSL system outperformed raw Ambisonic waveforms and traditional Ambisonic methods.Using both active and reactive intensity, with energy normalization by frequency band, improved the reported SSL performance.
  • Multichannel intensity: A non-Ambisonic approach estimated complex sound intensity from averaged pressure and pressure-difference estimates of two orthogonal particle-velocity components.Only active intensity was retained, with PHAT weighting applied for robustness against reverberation.
  • Waveforms: Waveform-based systems feed raw multichannel recordings directly to neural networks, often using convolutional layers to learn localization representations.This follows an end-to-end strategy that omits hand-crafted feature extraction.
  • Waveform limitations: Waveform inputs can require more complex architectures, and learning representations from noisy raw signals may be difficult or harm generalization.Some learned input-layer weights resemble conventional audio-processing filterbanks.
  • Other features: Periodicity degree features have been combined with GCC-PHAT in a CNN to convey harmonic-content information about the source signal.

6 Output strategies

Deep-learning SSL systems obtain DoA estimates mainly through classification or regression. Classification quantizes the search space and supports source-activity decisions, whereas regression produces continuous coordinates but introduces known-source-count and source-permutation challenges.

  • Classification: Classification divides the localization search space into zones and outputs a probability for each region.Multi-source estimates are commonly obtained by selecting the highest peaks, or peaks above a threshold when the source count is unknown.
  • Classification: Classification grids range from azimuth-only partitions to joint azimuth-elevation grids, with reported examples including 429 classes on a quasi-uniform sphere.Elevation-only estimation has been rarely studied, while recent systems more often estimate both azimuth and elevation.
  • Classification: Distance estimation is rarely investigated, with some systems quantizing distance into four or five classes alongside direction estimates.
  • Cartesian coordinates: Cartesian classification can suffer decreasing angular differences between regions farther from the microphone array, motivating regression for Cartesian coordinates.
  • Regression: Regression directly outputs continuous source coordinates and avoids localization-grid quantization.Its drawbacks include requiring a known or assumed source count and facing source permutation in multi-source localization.
  • Regression: Regression systems estimate azimuth, elevation, Cartesian coordinates, or activity-coupled Cartesian DoA representations, sometimes using separate branches or class-conditioned outputs.ACCDOA encodes normalized Cartesian direction together with source activity in one vector per sound class.
  • Hybrid strategies: Some regression networks estimate intermediate quantities that a non-neural algorithm subsequently converts into the final DoA.

7 Data

Because real labeled recordings are scarce and difficult to collect, DL-based SSL studies rely heavily on simulated data, recorded datasets, and augmentation. Surveyed approaches vary in acoustic realism, data sources, and evaluation settings, with simulation and augmentation used to address indoor noise and reverberation.

  • 7.1 Synthetic data: Real labeled SSL recordings are scarce and cumbersome to collect, so researchers often simulate large datasets for neural-network training.Recorded datasets with source-position metadata are difficult to produce across diverse spatial configurations and spectral content.
  • 7.1 Synthetic data: Simulated training data commonly convolve dry signals with room impulse responses generated across source, microphone-array, and room configurations.Room dimensions and reverberation time may also be varied to represent practical acoustic conditions.
  • 7.1 Synthetic data: Training-signal choice affects localization: speech or music can outperform noise-based training, while combining speech, noise, and sound events can outperform matched signals.White noise provides dense time-frequency coverage, but reported comparisons favored richer signal mixtures for localization performance.
  • 7.1 Synthetic data: Improved room-simulation models can reduce computational cost while maintaining comparable performance, and scattering effects with directional sources produced the best reported SSL performance among tested simulations.The comparable-performance result concerns a low-complexity model versus usual ISM training; the best-performance result comes from a separate simulation comparison.
  • 7.2 Recorded data: Real recordings and challenge datasets support robustness assessment, including reverberant and noisy DCASE data and LOCATA tasks spanning static or mobile sources and microphones.Publicly available datasets include recorded room impulse responses, binaural responses, and recordings from several microphone-array types.
  • 7.3 Data augmentation techniques: Data augmentation creates additional training examples without new recordings; DCASE systems used channel transformations, SpecAugment, and Mixup, with mixed outcomes across techniques.In one study, label-oriented and channel-inversion augmentation improved performance, whereas channel-oriented rotation performed worst.

8 Learning strategies

DNN-based SSL literature is dominated by supervised learning, but limited labeled real-world data motivates simulation, augmentation, and semi-supervised or weakly supervised alternatives. These strategies use unlabeled or weakly labeled data to adapt pretrained models or reduce incoherent predictions.

  • 8 Learning strategies: Most DNN-based SSL systems use supervised learning, while semi-supervised and weakly supervised approaches are also reported.Supervised training minimizes a loss comparing each network output with its corresponding target label.
  • 8 Learning strategies: Supervised SSL training is constrained by the small number and limited size of real-world labeled datasets, prompting simulation, augmentation, and alternative learning strategies.The available real datasets are described as insufficient for robust deep-learning training.
  • 8 Learning strategies: Unsupervised learning alone is not applicable to practical SSL when recorded acoustic signals lack source-position or direction labels.Semi-supervised learning instead combines labeled pretraining with unlabeled refinement or fine-tuning.
  • 8 Learning strategies: Semi-supervised learning has been used to improve performance on unseen conditions or real data by enriching limited labeled datasets.Reported methods include adapting pretrained networks and combining supervised and unsupervised learning.
  • 8 Learning strategies: Adversarial semi-supervised training adapted a DoA network from simulated data to infer from real data by distinguishing synthetic from real inputs.A discriminator classified incoming data as synthetic or real, while the generator learned to fool it.
  • 8 Learning strategies: Weak supervision fine-tuned pretrained networks with weak labels such as the number of sources, improving performance by reducing incoherent predictions.The approach modifies the cost function to account for weak labels.

9 Conclusions and perspectives

The survey synthesizes DL-based SSL research through a taxonomy of methods, training strategies, data, and configurations, then identifies persistent challenges and future directions. Key boundaries include poor simulated-to-real generalization, fixed microphone-array assumptions, limited DL–SP integration, and underdeveloped moving-source tracking.

  • 9 Conclusions and perspectives: The survey categorizes DL-based SSL literature by source configuration, neural architecture, input data, output strategy, datasets, and learning strategy.Tables II–V organize reviewed papers by these characteristics for quick method selection.
  • 9 Conclusions and perspectives: DNNs trained on simulated data generally lose performance on real-world signals because of train–test mismatch and scarce reliably annotated SSL data.Suggested responses include more sophisticated room-acoustics simulation, progressive training, domain adaptation, and transfer learning.
  • 9.2 Flexibility of the trained models: DL-based SSL models generally assume identical microphone-array setups during training and inference, requiring retraining when array geometry or directivity changes.Array-agnostic inputs, array-invariant methods, transfer learning, and domain adaptation are proposed as partial remedies.
  • 9.3 Multi-task learning: Multi-task learning is widely used in SELD systems, and explicit frame-level source counting was shown to improve SSL performance in one surveyed study.The survey also points to source separation and ASR as additional tasks that could be combined with SSL.
  • 9.3 Multi-task learning: The survey identifies end-to-end task-oriented learning as a way to use abundant labeled speech through pre-trained ASR blocks when DoA-annotated data are scarce.The described chain combines localization, DoA-parameterized beamforming, and ASR, with localization trained while other blocks can be frozen.
  • 9.4 Combination of DL and conventional SP techniques: DL–SP combinations remain poorly applied to SSL, while moving-source tracking and joint deep appearance–dynamics modeling remain important open directions.The survey also highlights combining physical priors with deep models and adapting unsupervised multi-object-tracking ideas to SSL.
Loading 2109.03465v3…