Source-linked AI summary
Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network
Sharath Adavanne, Archontis Politis, Tuomas Virtanen
TL;DR
The paper addresses multiple-source DOA estimation when the number of active sources is unavailable, including reverberant conditions. It proposes DOAnet, a CRNN that uses multichannel spectrogram magnitudes and phases to generate an SPS and DOA estimates in azimuth and elevation. Across anechoic and matched or unmatched reverberant datasets, DOAnet generally outperformed MUSIC and produced SPS outputs comparable to the baseline in evaluated contexts.
Problem
DOA estimation of multiple sources is challenging because the number of active sources may be unavailable, while MUSIC can suffer at low SNR and in reverberant scenarios.
Method
DOAnet uses multichannel spectrogram magnitudes and phases in a CRNN that sequentially generates an SPS and classifies multiple DOAs in azimuth-elevation space.
Results
DOAnet performed considerably better than MUSIC in most evaluated anechoic, matched-reverberant, and unmatched-reverberant scenarios while estimating source counts and 2-D DOAs.
Takeaways & Limitations
The results show potential for learning multiple overlapping sources’ 2-D directions directly from spectrograms without prior knowledge of the source count.
Abstract
from arXiv · showhide
This paper proposes a deep neural network for estimating the directions of arrival (DOA) of multiple sound sources. The proposed stacked convolutional and recurrent neural network (DOAnet) generates a spatial pseudo-spectrum (SPS) along with the DOA estimates in both azimuth and elevation. We avoid any explicit feature extraction step by using the magnitudes and phases of the spectrograms of all the channels as input to the network. The proposed DOAnet is evaluated by estimating the DOAs of multiple concurrently present sources in anechoic, matched and unmatched reverberant conditions. The results show that the proposed DOAnet is capable of estimating the number of sources and their respective DOAs with good precision and generate SPS with high signal-to-noise ratio.
I. INTRODUCTION
DOA estimation locates sound sources relative to a microphone and supports several microphone-array audio applications. DOAnet addresses source-count uncertainty and difficult acoustic conditions while estimating multiple-source directions and producing an SPS.
- DOA estimation identifies the relative positions of sound sources with respect to a microphone and supports enhancement, separation, and spatial audio coding.
- MUSIC and related subspace methods can handle multiple active sources but require the source count and may degrade at low SNR or under reverberation.
- It also generates an SPS alongside multiple-source DOAs without requiring the number of active sources, supporting directional visualization and higher-level sound analysis.
- DOAnet estimates azimuth and elevation by sampling the unit sphere uniformly, extending beyond approaches focused primarily on azimuth.
- The method uses Ambisonics as a common three-dimensional recording representation and evaluates a compact four-channel first-order format.
- DOAnet combines CNN and RNN layers to learn long-term temporal information and uses magnitude and phase spectrogram components as inputs.
II. METHOD
DOAnet maps multichannel spectrogram magnitudes and phases through a CRNN to generate an SPS and then classify source directions in azimuth-elevation space.
- DOAnet extracts each channel’s spectrogram and sequentially maps magnitude and phase features to SPS regression and DOA classification outputs.
- The SPS represents sound intensity along each direction, while DOA is specified by azimuth φ and elevation λ relative to the microphone.
- The method samples azimuth and elevation every 10 degrees, producing 614 SPS directions and 432 DOA directions restricted to elevations from -60 to 60 degrees.
A. Feature extraction
Feature extraction forms a multichannel tensor from spectrogram magnitudes and phases, preserving temporal frames for DOAnet input.
- Each channel’s spectrogram contributes 1024 positive-frequency magnitude and phase values to the network input tensor.
- Magnitude features from all channels are ordered before phase features, and the sequence length is L = 100, corresponding to 2 seconds.
B. Direction of arrival estimation network (DOAnet)
The DOAnet network uses CNN layers for local spectrotemporal features, bidirectional GRUs for temporal structure, and sequential SPS regression followed by multi-DOA classification.
- CNN layers extract local shift-invariant features, while bidirectional GRUs learn temporal structure without changing the time-axis length.
- The RNN output is mapped to SPS values through fully connected layers with linear activation, making SPS estimation a regression task.
- A second CRNN maps the SPS to DOA estimates using sigmoid outputs, allowing multiple directions to be predicted in one frame.
- DOAnet combines SPS and DOA estimation, training SPS with MUSIC-derived targets and DOA outputs with synthesized ground-truth directions.
- Training uses Adam, mean squared error for SPS, binary cross entropy for DOA, dropout, and early stopping based on the DOA metric.
A. Dataset
The evaluation uses synthetic datasets of static point sources assigned 2-D spatial coordinates in anechoic and reverberant contexts, with up to three overlapping events. Reverberant tests vary room size and reverberation time.
- Dataset rationale: Because suitable public datasets were unavailable, the study evaluates DOAnet on synthetic datasets comparable in size to prior DNN-based work.
- Dataset design: Synthetic recordings contain static point sources with spatial coordinates in anechoic and reverberant contexts, organized by up to one, two, or three overlapping sources.The datasets are labeled O1, O2, and O3, with A or R denoting anechoic or reverberant conditions.
- Dataset design: Each dataset uses three cross-validation splits with 240 recordings for training and 60 for testing; recordings are 30 seconds sampled at 44.1 kHz.
- Sound events: The source material comprises 11 DCASE 2016 sound-event classes, with disjoint training and testing examples selected during cross-validation.Examples include speech, coughing, door slams, page-turning, phone ringing, and keyboard sounds.
- Spatial conditions: Sound events are placed on a 10-degree azimuth-elevation grid, with overlapping events separated by at least 10 degrees and elevations constrained to −60 to 60 degrees.Anechoic sources are placed 1–10 m away, while reverberant sources are placed in a 10 × 8 × 4 m room with a central microphone.
- Scene synthesis: Anechoic scenes are generated in Ambisonics using real spherical harmonics and distance-attenuation gains, whereas reverberant scenes use image-source room simulation and spatial impulse responses.Matched and mismatched reverberant tests vary room size and reverberation time relative to training data.
B. Baseline
The baseline is MUSIC, a conventional high-resolution estimator that produces both spatial pseudo-spectra and DOAs for direct comparison with DOAnet. Its SPS derives from signal-noise subspace orthogonality.
- Baseline choice: MUSIC is selected as the conventional high-resolution baseline because it estimates both SPS and DOAs, enabling one-to-one comparison with DOAnet.
- MUSIC formulation: MUSIC forms a spatial covariance matrix from multichannel spectrograms and decomposes it into eigenvalue-ordered signal and noise subspaces.The signal subspace contains eigenvectors associated with the largest eigenvalues, while the remaining eigenvectors form the noise subspace.
- MUSIC formulation: The MUSIC SPS evaluates directional alignment with the noise subspace, and source DOAs are selected from the largest SPS peaks.
C. Metric
The evaluation measures SPS agreement with MUSIC and DOA angular error against synthesized ground truth, accommodating unequal numbers of estimated and true directions.
- SPS metric: SPS quality is measured by an SNR comparing DOAnet’s estimated SPS, SE(φ, λ), with MUSIC’s SPS, SGT(φ, λ).The metric is computed as 10 log10 of the relevant power ratio involving SPS differences.
- DOA metric: DOA accuracy is measured as the angular distance in degrees between each estimated direction and its synthesized ground-truth direction.The angular distance uses estimated and ground-truth azimuth and elevation coordinates.
- DOA metric: When estimated and ground-truth DOA counts differ, the evaluation reports minimum matching distances using the Hungarian algorithm and the percentage of frames meeting the matching criterion.
D. Evaluation procedure
The study fixes one DOAnet configuration across experiments, evaluates SPS and DOAs against MUSIC and synthesized ground truth, and examines closely located sources through SPS visualizations.
- Evaluation setup: Parameter tuning uses O1A test data, producing a 677 K-weight configuration that is reused throughout the subsequent studies.
- Evaluation setup: At test time, SPS is compared with MUSIC SPS, while DOA predictions from DOAnet and MUSIC are evaluated against the synthesized ground-truth DOAs.
- Controlled comparison: Because MUSIC uses the known number of active sources, DOAnet is also evaluated with that number supplied by selecting the corresponding top prediction probabilities.This supplements the threshold-based prediction procedure for a fair comparison.
- SPS visualization: Figure 2 plots SPS with azimuth on the horizontal axis and elevation on the vertical axis, marking ground-truth DOAs with black crosses for two closely located sources.
IV. RESULTS AND DISCUSSION
DOAnet generally outperformed MUSIC across evaluated reverberant and source-count conditions, while source-count estimation and MUSIC-SPS training introduced important limitations. The method showed consistent performance on unmatched reverberant data but degraded when training SPS estimates were unstable.
- Limitations: For three overlapping sources, unstable MUSIC SPS training produced poor DOAnet results.MUSIC approaches its theoretical limit in this setting, and the evaluation used four-channel audio features and MUSIC SPS.
- DOA estimation: DOAnet’s DOA error was considerably better than MUSIC even when the number of active sources was unknown.This comparison held across the reported datasets, although correct source-count predictions were relatively infrequent.
- Source-count estimation: 42.7% of estimated frames had the correct number of DOA predictions for anechoic recordings with up to two overlapping sound events.Correct-count prediction decreased further with three sources.
- Reverberant conditions: DOAnet’s performance on unmatched reverberant data was consistent with matched reverberant results and significantly better than MUSIC.The unmatched-room evaluation is summarized in Table II.
- Future improvements: The authors suggest smarter DOAnet peak picking, source-count input, more channels, or ground-truth SPS training as routes to improve performance.They also identify direct DOA training without SPS as an alternative and call for more extensive synthetic and real-data experiments.
V. CONCLUSION
The paper proposes DOAnet, a convolutional recurrent network for multiple-source localization that learns source counts and two-dimensional DOAs from input spectrograms. Across anechoic and reverberant datasets, it performed considerably better than MUSIC in most scenarios, demonstrating potential without prior source-count knowledge.
- Conclusion: DOAnet learns the number of active sources directly from input spectrograms and estimates precise DOAs in 2-D polar space.The method is a convolutional recurrent neural network for multiple-source localization.
- Conclusion: DOAnet performed considerably better than baseline MUSIC in most anechoic, matched reverberant, and unmatched reverberant scenarios.These evaluations support its potential for learning a highly computational algorithm without prior knowledge of the number of sources.