PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks

David Diaz-Guerra, Antonio Miguel, Jose R. Beltran

arXiv:2006.09006v2eess.AScs.LGcs.SDeess.SP

TL;DR

DOA tracking must account for temporal context, while some neural approaches use non-causal recurrence that limits real-time use. This paper applies causal 3D CNNs to SRP-PHAT maps and reports robust tracking, including satisfactory performance on LOCATA recordings.

  • Problem

    Two-step tracking can discard useful SRP-PHAT map information, and some DNN trackers use non-causal recurrent layers that impede real-time use.

    Source: §II · p. 3

  • Method

    The system uses causal 3D CNNs over SRP-PHAT power maps for joint DOA estimation and tracking, trained with simulated random trajectories generated as needed.

    Source: §I · p. 1, §VI · p. 11

  • Results

    Experiments found SRP-PHAT maps more robust to reverberation and noise than spectrogram inputs, and showed good tracking with causal convolutional layers without non-causal recurrent layers.

    Source: §VI · p. 11

  • Takeaways & Limitations

    Models trained with the random-trajectory procedure obtained satisfactory results on actual LOCATA recordings.

    Source: §VI · p. 11

    Generating simulations as training inputs makes training slower, despite providing an infinite-size dataset and flexibility to vary simulation conditions.

    Source: A. Training dataset · p. 4

Abstract

from arXiv · show

In this paper, we present a new single sound source DOA estimation and tracking system based on the well-known SRP-PHAT algorithm and a three-dimensional Convolutional Neural Network. It uses SRP-PHAT power maps as input features of a fully convolutional causal architecture that uses 3D convolutional layers to accurately perform the tracking of a sound source even in highly reverberant scenarios where most of the state of the art techniques fail. Unlike previous methods, since we do not use bidirectional recurrent layers and all our convolutional layers are causal in the time dimension, our system is feasible for real-time applications and it provides a new DOA estimation for each new SRP-PHAT map. To train the model, we introduce a new procedure to simulate random trajectories as they are needed during the training, equivalent to an infinite-size dataset with high flexibility to modify its acoustical conditions such as the reverberation time. We use both acoustical simulations on a large range of reverberation times and the actual recordings of the LOCATA dataset to prove the robustness of our system and its good performance even using low-resolution SRP-PHAT maps.

I. INTRODUCTION

The paper motivates joint DOA estimation and tracking because non-stationary sources need temporal context, while some neural trackers use non-causal recurrent layers that impede real-time use. It proposes causal 3D CNNs over SRP-PHAT power maps, whose patterns may help estimate direction even when adverse conditions produce ambiguous maxima.

I. INTRODUCTION

  • Non-stationary signals need tracking to exploit temporal correlation, but some DNN trackers use non-causal recurrent layers that make them unsuitable for real-time applications.Two-step methods may discard information by selecting only each map’s absolute maximum and treating DOA estimates with different peak widths alike.Source: §I · p. 1, §II · p. 3
  • The proposed 3D CNN jointly estimates and tracks a sound source from SRP-PHAT power maps, with causal processing that provides a new DOA estimate for each map.The system is evaluated in highly reverberant rooms and on actual LOCATA recordings; the paper also analyzes map resolution.Source: §I · p. 1

II. THE SRP-PHAT ALGORITHM

  • SRP-PHAT forms an acoustic power map by evaluating beamformer response across directions; its maximum should indicate the source position.The method can be computed using generalized cross-correlation functions and PHAT weighting [10],.Source: §II · p. 2
  • Computing SRP-PHAT sums for every direction may be costly, particularly across multidimensional search spaces; non-convex maps can also require many evaluations to find the maximum.Search strategies have been proposed to reduce the number of evaluations, including hierarchical searches over lower-resolution maps –.Source: §II · p. 2
  • With low SNR and greater reverberation, SRP-PHAT maps can have several local maxima, especially at low resolution, but other patterns may also encode DOA information.The examples contrast favorable conditions (SNR=30 dB, T60=0.3 s) with adverse conditions (SNR=5 dB, T60=0.9 s).Source: §II · pp. 2–3, Figure 1 · p. 2, Figure 2 · p. 2

III. DOA ESTIMATION WITH DEEP NEURAL NETWORKS

Neural-network approaches to DOA estimation have evolved from early perceptrons toward more varied architectures, including CNNs that can jointly estimate direction and track sources.

III. DOA ESTIMATION WITH DEEP NEURAL NETWORKS

  • An early neural-network approach used a fully connected perceptron with one hidden layer to classify DOA from GCCs.Source: §III · p. 3
  • Later techniques differ in their output format, input features, and network architecture.Source: §III · p. 3
  • An early neural-network approach estimated DOA from GCCs as a classification problem using a fully connected perceptron with a hidden layer; later techniques varied in output format, input features, and network architecture.Source: §III · p. 3

A. Output format

The section contrasts classification and regression output formats for DOA estimation, then motivates regression while noting its difficulty with multiple sources.

A. Output format

  • Classification requires a direction grid and one output per grid point, so estimating azimuth with elevation or XYZ coordinates can sharply increase output size, complexity, and training-data needs.The earlier network in had 359 outputs, corresponding to at most 1-degree azimuth resolution.Source: A. Output format · p. 3
  • Although reported worse results with regression, later work used three regression outputs for a unit vector, and this paper adopts regression based on those works’ results, –.Source: A. Output format · p. 3
  • Regression makes multiple-source DOA estimation harder; one proposed option is combining a single-source estimator with source cancellation to find sources iteratively,.Source: A. Output format · p. 3

B. Input features and network architecture

The method applies 3D CNNs to SRP-PHAT power maps, convolving over the maps and time to estimate and track source direction. Its layers are causal, producing an estimate for each new map without relying on future inputs.

B. Input features and network architecture

  • Proposed CNN inputs in prior work include GCCs, spatial-covariance eigenvectors, spectrograms, cepstrograms, Mel spectrograms, Ambisonics vectors, and raw audio.The authors previously used low-resolution SRP-PHAT maps with fully connected perceptrons.Source: B. Input features and network… · p. 3
  • Earlier 2D CNN approaches convolve over time and frequency, or over frequency and microphones without tracking time evolution.For spectrogram inputs, microphone signals are treated as channels in the time-frequency approach.Source: B. Input features and network… · p. 3
  • For ULA-based methods, adjacent microphone pairs can share phase-difference patterns, but this equivariance does not hold for other array geometries.The cited approach uses convolution across frequency and microphone dimensions.Source: B. Input features and network… · pp. 3–4
  • 3D CNNs process spherical SRP-PHAT maps over spatial and temporal dimensions, using compact-array geometry as the target setting.The authors note that 4D CNNs over 3D maps could extend the approach to 3D localization with distributed arrays.Source: B. Input features and network… · p. 4
  • Spherical maps favor rotational equivariance, but the authors note that Spherical CNNs are less computationally efficient than classical CNNs.Euclidean translations are not the preferred equivariance for spherical maps.Source: B. Input features and network… · p. 4
  • Causal convolutional layers make the system suitable for real-time use, unlike architectures that rely on bidirectional recurrent units.Each new power map receives a DOA estimate without using future input.Source: B. Input features and network… · p. 4

IV. PROPOSED TECHNIQUE

The proposed training procedure simulates source trajectories and acoustic conditions as needed during training, then converts simulated recordings into SRP-PHAT maps. It also applies voice activity detection to remove silent-frame information that could otherwise lead the model to track directional noise.

A. Training dataset

  • The authors use simulated signals because accurately hand-labeling moving-source recordings is difficult; measured RIRs would increase simulation complexity and training time.They report LOCATA results separately, and note that new microphone geometries require retraining.Source: A. Training dataset · pp. 4–5
  • Simulating inputs as needed creates an effectively infinite training set, reducing overfitting risk and allowing acoustic-condition distributions to vary during training.The authors cite SNR and reverberation time as parameters that can be varied for curriculum learning; this makes training slower.Source: A. Training dataset · p. 4
  • LibriSpeech utterances provide simulated source speech, and WebRTC VAD removes silent segments that could contain background noise.The authors remove those segments so the network does not learn to localize background noise as if it came from the source.Source: A. Training dataset · p. 4
  • Random starting and ending points are joined by straight paths with low-amplitude, per-axis sinusoidal variation to generate diverse trajectories.The trajectory-generation model is simple, and the authors assess it on LOCATA recordings.Source: A. Training dataset · p. 5
  • The simulations use a 12-sensor array designed for a NAO robot head, without modeling the robot head's acoustic effects.The microphone positions are taken from LOCATA documentation.Source: A. Training dataset · p. 5
  • Simulated signals are windowed into 4096-sample frames with a 3K/4 hop, converted to SRP-PHAT maps, and normalized to [-1,1].The simulation uses omnidirectional Gaussian noise for the desired SNR and approximates fractional delays to the nearest sample without interpolation.Source: A. Training dataset · p. 5
  • Because training simulations omit directional noise, the authors use WebRTC VAD to zero maps in frames classified as silent.This was intended to prevent tracking directional noise sources present in some LOCATA recordings.Source: A. Training dataset · p. 5

B. Model architecture

Cross3D processes SRP-PHAT maps and their maximum-location coordinates with a 3D convolutional architecture designed to preserve spatial information while tracking over time. Causal temporal convolutions produce a source-direction estimate for each map.

B. Model architecture

  • Explicitly supplying each map’s normalized maximum-location coordinates helps the network use DOA information that may be difficult to learn through the non-linear, non-differentiable argmax alone.These coordinates are added as input channels alongside the SRP-PHAT maps.Source: B. Model architecture · pp. 5–6
  • Two branches pool along perpendicular spatial axes, allowing each to retain positional information on the other axis.Each branch uses four 3D convolution layers; the number is reduced when map dimensions are too small for that many pooling layers.Source: B. Model architecture · p. 6
  • The architecture uses pooling to reduce spatial dimensions while retaining positional information needed for DOA estimation.The authors note that pooling must be used carefully because the desired output depends on position, not only on pattern presence.Source: B. Model architecture · p. 6
  • Causal temporal convolutions yield a DOA estimate for each new map using the last 37 maps, a tracking memory of 7.17 seconds.The architecture figure depicts a 16×32-map model with a 103-map input sequence, simplifying the display to one channel although the model uses three.Source: B. Model architecture · pp. 6–7, Figure 5 · p. 6
  • The network outputs three time sequences interpreted as the XYZ coordinates of unit vectors pointing toward the source.The model is trained to match these sequences to ground-truth direction vectors.Source: B. Model architecture · p. 7

C. Training

Training uses simulated trajectories with varied reverberation and SNR, plus a curriculum that begins at high SNR. The network is optimized to match source-direction unit vectors.

C. Training

  • The model minimizes Euclidean distance to ground-truth unit-vector sequences, which performed better than directly predicting spherical coordinates with great-circle distance.The passage reports this comparison in line with results in,,.Source: C. Training · p. 7
  • The authors define an epoch as 585 trajectories and train for 80 epochs using 20-second trajectories containing 103 SRP-PHAT maps.Optimization uses Adam.Source: C. Training · p. 7
  • Training spans reverberation times of 0.2–1.3 s and SNRs of 5–30 dB, beginning with 30 dB batches before switching to the full SNR range.The first 20 epochs use batches of five trajectories; later batches contain ten and the learning rate drops from 1e-4 to 1e-5.Source: C. Training · p. 7

A. Baseline methods

The evaluation uses alternative CNN baselines designed to compare SRP-PHAT maps with other inputs while retaining tracking information. These include maximum-coordinate, GCC, and spectrogram-based models.

A. Baseline methods

  • The alternative CNN baselines were designed to resemble the proposed model and share its temporal perceptive field for a fairer comparison.The evaluated model set is summarized in Table I.Source: A. Baseline methods · pp. 7–8
  • A 1D causal CNN uses the normalized coordinates of each SRP-PHAT map’s maximum as two input sequences.It has seven convolution layers and a 37-frame temporal perceptive field; the reported results use coordinates from 64×128 maps.Source: A. Baseline methods · p. 7
  • A second 1D causal CNN takes temporal sequences of GCC elements representing inter-microphone delays within the maximum inter-microphone distance.It reuses the architecture used for map-maximum coordinates.Source: A. Baseline methods · p. 7
  • The spectrogram baseline uses causal 2D convolutions followed by 1D causal convolutions, with magnitude and phase as inputs.The model uses the same analysis windows as the SRP-PHAT maps and normalizes magnitude and phase separately.Source: A. Baseline methods · pp. 7–8
  • The SELDnet replica replaces its sound-event-detection output with a single-source DOA output, but remains non-causal because it uses bidirectional GRUs.It also uses shorter analysis windows than the other evaluated methods; spectrogram-input models trained best with T60 sampled from 0 to 0.3 s.Source: A. Baseline methods · p. 8

B. Results

Across simulated conditions, Cross3D remains robust as reverberation worsens and can track trajectories even when SRP-PHAT map maxima are noisy. Its performance remains competitive at low map resolution, while full-map 3D CNNs outperform several baselines.

1) Simulated dataset:

  • Cross3D’s error degrades by only 5° at T60 = 1.5 s, beyond the training range, while SRP-PHAT cannot estimate the DOA properly.Models were trained across a range of reverberation times and SNRs, then tested at selected conditions.Source: 1) Simulated dataset · p. 8, Figure 8 · p. 9
  • In high reverberation and low SNR, Cross3D stays close to the true DOA despite noisy map maxima, and follows a sudden azimuth change.The example uses T60 = 0.9 s, SNR = 5 dB, and 32x64 maps; the figure distinguishes actual DOA, estimated DOA, map maxima, and silent frames.Source: 1) Simulated dataset · p. 9, Figure 7 · p. 9
  • With 4x8 maps, Cross3D estimates DOA more precisely than the map maxima, including beyond the grid’s 45° azimuth and 60° elevation spacing.The example uses T60 = 0.3 s and SNR=30 dB.Source: 1) Simulated dataset · p. 9, Figure 7 · p. 9
  • Cross3D with high-resolution maps performs best, while its low-resolution performance remains competitive across reverberation and SNR conditions.The comparison includes SNR=30 dB and SNR=5 dB; silent frames are included in the tracking RMSAE.Source: 1) Simulated dataset · p. 9, Figure 8 · p. 9
  • A 1D CNN on 64x128 map maxima performs worse than a 3D CNN on 4x8 maps, indicating the latter exploits map patterns beyond their maxima.GCC-based 1D CNN performance falls between Cross3D using 4x8 and 8x16 maps; spectrogram methods fare well in favorable conditions but are less robust to noise and reverberation.Source: 1) Simulated dataset · p. 9

2) LOCATA dataset:

The LOCATA evaluation tests models trained and validated on simulations against real recordings, with only a VAD modification made after testing began. Cross3D outperforms spectrogram baselines and, at sufficient map resolution, 1D CNN methods, though the small recording set limits conclusions.

2) LOCATA dataset:

  • The evaluation uses development recordings from LOCATA tasks 1, 3, and 5, while training and validation use simulated signals; a VAD was the only post-test modification.The tasks cover a static loudspeaker/static array, a moving talker/static array, and a moving talker/moving array.Source: 2) LOCATA dataset · p. 10
  • On LOCATA, Cross3D outperforms spectrogram-input baselines and 1D CNN methods when using maps of at least 16x32 resolution.The authors caution that the number of recordings is low; Table II reports RMSAE with silent frames included.Source: 2) LOCATA dataset · p. 10
  • At low map resolution, Cross3D degrades relative to simulation results, but that degradation disappears at higher resolution, where LOCATA errors are lower than simulated-test errors.Map-maximum 1D CNNs on high-resolution maps show little degradation, though Cross3D using the full maps gives far better results.Source: 2) LOCATA dataset · p. 10
  • In an example LOCATA recording, Cross3D gives the best analyzed estimates for both elevation and azimuth, while longer silences may explain lower results for some methods.The authors suggest including longer silences in training simulations and increasing temporal receptive field to improve CNN robustness to them.Source: Figure 9 · p. 10, 2) LOCATA dataset · p. 11

VI. CONCLUSIONS

The paper presents a causal 3D CNN tracker over SRP-PHAT maps, paired with on-demand simulated trajectory generation for training. Experiments report robust tracking against noise and reverberation, satisfactory LOCATA results, and a causal architecture suitable for real-time use.

VI. CONCLUSIONS

  • The fully causal architecture provides a DOA estimate every 192 ms and is feasible for real-time applications.It uses a 3D CNN over sequences of elevation and azimuth maps from a compact array; the authors say distributed arrays would make a 4D CNN extension straightforward.Source: §VI · p. 11
  • Experiments show SRP-PHAT maps are more robust to noise and reverberation than spectrogram inputs, and good tracking is possible using only causal convolutional layers.The authors conclude non-causal recurrent layers are not needed for this performance.Source: §VI · p. 11
  • On-demand random-trajectory simulation provides an effectively infinite training dataset whose acoustic parameters can be modified for training or targeted testing.The authors report satisfactory results when testing the simulation-trained models on LOCATA recordings.Source: §VI · p. 11
  • A causal 1D CNN over GCCs is presented as a robust baseline for applications where computational limits prevent SRP-PHAT map computation.The authors describe its performance as good and robust.Source: §VI · p. 11
Loading 2006.09006v2…