Source-linked AI summary
Towards End-to-End Acoustic Localization using Deep Learning: from Audio Signal to Source Position Coordinates
Juan Manuel Vera-Diaz, Daniel Pizarro, Javier Macias-Guarasa
TL;DR
Acoustic source localization matters for systems that analyze human activity, but this paper seeks to estimate source position directly from microphone-array signals. It uses a CNN with supervised fine-tuning on target-room data; its reported average improvement over SRP-PHAT exceeds that of GMBF.
Problem
The paper aims to estimate an acoustic source’s position directly from microphone-array signals, in a setting where precise human localization is presented as important for higher-level activity information.
Source: §1 · pp. 1–2Method
A CNN processes microphone-array signals, with supervised fine-tuning on a reduced set of recordings from the target environment.
Source: §1 · p. 2, §3.3.2 · p. 7Results
The CNN’s average absolute improvement over SRP-PHAT is more than 10 points higher than GMBF’s, reaching 31.3% for CNN and 20.7% for GMBF.
Source: §4.6 · p. 13Takeaways & Limitations
The authors conclude that fine-tuning with limited data is the largest contributor to improvement over bare CNN training, and additional fine-tuning material consistently improves results.
Source: §4.6 · p. 14Despite improvements with fine-tuning, the authors say the method remains far from suitable for competitive exploitation when GMBF is available.
Source: §4.6 · p. 13
Abstract
from arXiv · showhide
This paper presents a novel approach for indoor acoustic source localization using microphone arrays and based on a Convolutional Neural Network (CNN). The proposed solution is, to the best of our knowledge, the first published work in which the CNN is designed to directly estimate the three dimensional position of an acoustic source, using the raw audio signal as the input information avoiding the use of hand crafted audio features. Given the limited amount of available localization data, we propose in this paper a training strategy based on two steps. We first train our network using semi-synthetic data, generated from close talk speech recordings, and where we simulate the time delays and distortion suffered in the signal that propagates from the source to the array of microphones. We then fine tune this network using a small amount of real data. Our experimental results show that this strategy is able to produce networks that significantly improve existing localization methods based on \textit{SRP-PHAT} strategies. In addition, our experiments show that our CNN method exhibits better resistance against varying gender of the speaker and different window sizes compared with the other methods.
1. Introduction
The paper frames human localization as a prerequisite for higher-level activity information and interaction. It proposes using raw microphone-array signals with a CNN to estimate an acoustic source’s position, and reports better performance than traditional SRP-based solutions.
1. Introduction
- Human localization is presented as a fundamental task for perceptual systems to provide higher-level information about activities and support interaction with the physical environment.The passage states that without precise localization, further interaction cannot be carried out successfully.Source: §1 · p. 1
- A CNN is trained end-to-end on speech signals directly, avoiding hand-crafted feature vectors for acoustic source localization.The authors describe this as, to their knowledge, the first such approach in the literature.Source: §1 · p. 2
- The proposal outperforms traditional solutions based on Steered Response Power (SRP) in evaluations using semi-synthetic and real data.The paper identifies SRP-based methods as the basis of state-of-the-art systems.Source: §1 · p. 2
2. State of the Art
Prior acoustic source localization work includes signal-processing, optimization, and machine-learning approaches, with neural-network studies varying in input features, outputs, and use of real data. The paper identifies direct raw-signal input and direct three-dimensional coordinate estimation as gaps in the referenced work.
2. State of the Art
- Earlier neural-network approaches use localization-related features such as GCC or correlation values, binaural cues, spectral features, or cross-spectral maps.Other approaches use frequency-domain representations, leaving networks to learn relationships between spectral cues and location-related information.Source: §2 · p. 3
- Referenced neural-network works target DOA, angles, bearing and range, predefined classes, source maps, or time-frequency masks, but not direct source-coordinate values.The paper says none of the referenced works directly estimate acoustic-source coordinate values.Source: §2 · pp. 3–4
- Most reviewed neural-network studies use simulated data, while only some evaluate real recordings, which the authors consider necessary to assess performance in real conditions.The reviewed simulations commonly convolve clean speech with room, head-related, or direction-related impulse responses.Source: §2 · p. 4
- The paper describes a CNN that uses raw acoustic signals to estimate a source’s three-dimensional position and is tested on semi-synthetic and publicly available real data.The authors call the approach end-to-end because it covers the full acoustic source localization problem.Source: §2 · pp. 3–4
3. System Description
The system learns a mapping from windowed microphone signals to a source position with a CNN. To address limited real training data, it first trains on semi-synthetic signals and then fine-tunes using a small supervised subset recorded in the target room.
3.1. Problem Statement
- The problem is framed as regression from microphone signals to the speaker’s position, with finite audio windows and potential noise or distortion complicating analytic estimation.The system assumes microphone positions are fixed, and the CNN learns the mapping from the recorded signals.Source: §3.1 · p. 4
3.2. Network Topology
- The CNN takes windowed signals from the microphones and outputs the estimated source position, using five one-dimensional convolutional blocks and two fully connected blocks.The described convolutional filter sizes are 7, 5, and 3; the fully connected hidden layer has 500 nodes, with dropout probability 0.5 during training.Source: §3.2 · p. 5, Figure 1 · p. 5
3.3. Training Strategy
- The training strategy first uses semi-synthetic data and then fine-tunes the initialized network with a reduced real-data subset from the target environment.The authors propose this two-step strategy because real data available for training is generally limited.Source: §3.3 · p. 5
3.3.1. Semi-Synthetic Dataset Generation
- Semi-synthetic examples use close-talk recordings and randomly generated positions, with microphone signals simulated by applying source-to-microphone propagation delays.The delay is based on source-to-microphone distance, sampling frequency, and sound speed; the procedure accounts for sub-sample shifts.Source: §3.3.1 · p. 6
- The synthetic signals also add room-relevant disturbances, including a randomized 20–30 Hz tone and white Gaussian noise, to approximate the target room’s acoustic conditions.The tone range is based on a tone observed in the IDIAP room recordings; the noise scaling is chosen to produce a similar signal-to-noise ratio.Source: §3.3.1 · pp. 6–7
3.3.2. Fine Tuning Procedure
- Fine-tuning uses a short supervised set of real target-room recordings to introduce acoustic behavior such as multipath and reverberation that is more complex to simulate.The authors say only a few sequences are used for fine-tuning, leaving the rest for testing.Source: §3.3.2 · p. 7
4. Experimental Work
The experimental section describes the datasets, training details, general setup, and error metrics before presenting results against baseline performance.
4. Experimental Work
- The experiments cover the datasets used in both training stages, setup conditions, comparison metrics, and results beginning with baseline performance.Source: §4 · p. 7
4.1. Datasets
The evaluation and fine-tuning data come from the IDIAP Smart Meeting Room corpus, while Albayzin speech recordings supply material for semi-synthetic training. The paper uses annotated single-speaker sequences from IDIAP and the room layout to construct the test and training setup.
4.1.1. IDIAP AV16.3 Corpus: for testing and fine tuning
- IDIAP recordings provide audio and annotated 3D speaker-mouth positions, and the room layout is also used to generate semi-synthetic data.The sequences are synchronously sampled at 16 kHz, with a 40 ms frame shift.Source: §4.1.1 · pp. 7–8
- The IDIAP Smart Meeting Room is a 3.6 m × 8.2 m × 2.4 m room with two 8-microphone arrays on a central table.The arrays have 0.1 m radii and their centers are separated by 0.8 m.Source: §4.1.1 · p. 7, Figure 2 · p. 8
- The five selected annotated single-speaker sequences comprise three static-speaker sequences and two moving-speaker sequences.The sequences are denoted s01, s02, s03, s11, and s15.Source: §4.1.1 · p. 8, Table 2 · p. 9
4.1.2. Albayzin Phonetic Corpus: for Semi-Synthetic Dataset Generation
- The Albayzin phonetic corpus contributes 6,800 phonetically balanced utterances from which the semi-synthetic training data are generated.The source recordings come from close-talk microphones and are 16 kHz, 16-bit signals.Source: §4.1.2 · p. 9
4.2. Training and Fine Tuning Details
The network is trained separately for three audio-window lengths using randomly generated source positions and sampled speech windows. Fine tuning primarily uses moving-speaker sequences, with static-speaker sequences included in additional experiments.
4.2. Training and Fine Tuning Details
- Semi-synthetic source positions are sampled uniformly within the speaker-mouth coordinate ranges of the IDIAP room.The ranges are qx ∈ [0, 3.6] m, qy ∈ [0, 8.2] m, and qz ∈ [0.92, 1.53] m.Source: §4.2 · p. 9
- Each training epoch uses 7,200 randomly generated input frames and 800 validation frames, with ADAM optimization for 200 epochs and batches of 100.The optimization strategy is applied to the loss function described by equation (4).Source: §4.2 · p. 9
- Training uses 80 ms, 160 ms, and 320 ms windows, with a separate network model trained for each length.For each model, 200 recordings and 200 source positions are randomly selected, with 40 windows per recording and position.Source: §4.2 · p. 9
- Fine tuning mainly uses moving-speaker sequences s11 and s15, while s01, s02, and s03 are added in a later experiment.The experiments also examine gender and height in the fine-tuning and evaluation data.Source: §4.2 · p. 9
4.3. Experimental Setup
The experiments test whether the end-to-end CNN is competitive with SRP-PHAT and GMBF under a simple four-microphone setup. They evaluate baseline performance and progressively vary the fine-tuning data.
4.3. Experimental Setup
- Testing uses s01, s02, and s03, while the experiments vary fine-tuning from one sequence to adding sequences and ultimately static sequences.The sequence plan also compares semi-synthetic training plus fine tuning with training from scratch.Source: §4.3 · p. 10
- The proposal uses four of the 16 available microphones, grouped into two pairs, and tests 80 ms, 160 ms, and 320 ms acoustic frames.The microphone pairs are identified in Figure 2.c; the frame lengths assess performance across acoustic time resolutions.Source: §4.3 · p. 10
- The comparison evaluates the end-to-end CNN against SRP-PHAT and GMBF, a method based on fitting a generative model to GCC-PHAT signals with sparse constraints.GMBF had previously reported improvements over SRP-PHAT on the IDIAP dataset.Source: §4.3 · p. 10
4.4. Evaluation metrics
The section evaluates localization by comparing predicted speaker coordinates with annotated ground truth. It uses MOTP and relative MOTP improvement to compare methods.
4.4. Evaluation metrics
- MOTP is the evaluation metric used to compare estimated speaker positions with labeled ground-truth positions.The position estimates are compared using Euclidean distance.Source: §4.4 · p. 10
- The MOTP definition uses the total number of position estimates, NP, and the estimated and labeled position vectors at time k.The vectors are denoted s_k and s_kGT, respectively.Source: §4.4 · p. 10
- The comparison with SRP-PHAT reports each method’s relative improvement in MOTP.The passage introduces this relative-improvement measure for the CNN and GMBF comparisons.Source: §4.4 · p. 10
4.5. Baseline Results
The baseline compares SRP-PHAT, GMBF, and a CNN trained only on semi-synthetic data across three sequences and multiple window sizes. GMBF improves on SRP-PHAT, while the untuned CNN performs substantially worse than both methods.
4.5. Baseline Results
- GMBF’s average relative improvement over SRP-PHAT is around 20%, with peak values of almost 30%; best MOTP values are around 48cm versus around 69cm for SRP-PHAT.Average MOTP values are 59–78cm for GMBF and 76–96cm for SRP-PHAT.Source: Table 3 · p. 11, §4.5 · p. 11
- MOTP improves as frame size increases, which the authors attribute to better correlation estimates from longer signal windows.Source: §4.5 · p. 11
- The CNN trained only on semi-synthetic data performs far worse than SRP-PHAT and GMBF, suggesting real-data effects such as reverberation may affect the network.Source: §4.5 · p. 11
4.6. Results and Discussion
The experiments show that semi-synthetic training followed by real-data fine-tuning outperforms training from scratch and improves on SRP-PHAT. Adding fine-tuning sequences strengthens results, with the final configuration outperforming GMBF on average across window sizes.
4.6. Results and Discussion
- With s15 fine-tuning, CNN improved over SRP-PHAT by 1.8%–11.3% on average, but remained behind GMBF in all but one case.The exception was s02 at 80ms; CNN slightly degraded performance in two s03 cases.Source: §4.6 · p. 11, Table 4 · p. 12
- 5.3% average improvement across window lengths for semi-synthetic training plus s15 fine-tuning, versus −7.0% for training from scratch with s15.The fine-tuned approach ranged from 1.8% to 11.3% improvement, while training from scratch ranged from −20.6% to 4.3%.Source: §4.6 · p. 12, Table 5 · p. 12
- Adding s11 to s15 fine-tuning improved results over using s15 alone, with only one SRP-PHAT regression, a marginal −0.3%.s11 was a 33-second sequence of a randomly moving male speaker; CNN still trailed GMBF in all but two cases.Source: §4.6 · p. 13, Table 6 · p. 13
- 31.3% average absolute improvement over SRP-PHAT for CNN versus 20.7% for GMBF; CNN was better on s01 and s02 but slightly worse on s03.CNN’s maximum improvement was 41.6% (s01, 320ms), compared with GMBF’s 29.9% (s03, 320ms); static-sequence gains required tuning examples from similar, not identical, positions and varying heights.Source: §4.6 · pp. 13–14, Table 8 · p. 14
- Fine-tuning with limited data contributed most to gains over bare CNN training, and adding more fine-tuning material consistently improved results across speaker gender and static or dynamic activity.The authors conclude that the network learns acoustic cues relevant to localization and is a promising strategy for acoustic source localization.Source: §4.6 · p. 14
5. Conclusions
The paper presents an end-to-end CNN that localizes acoustic sources directly from audio, using semi-synthetic pretraining followed by real-data fine-tuning. The authors report improved localization over existing methods when sufficient fine-tuning data is available, alongside resistance to speaker gender and window-size variation.
5. Conclusions
- With sufficient fine-tuning data, the end-to-end CNN outperforms state-of-the-art methods and SRP-PHAT-based methods.The authors compare it with methods from and methods using SRP-PHAT.Source: §5 · p. 15
- The two-step strategy pretrains on semi-synthetic data and fine-tunes on a small amount of real data, improving performance over training from scratch with real data.Source: §5 · p. 15
- The CNN shows resistance to changes in speaker gender and signal window size compared with baseline methods.Source: §5 · p. 15
- Future work includes adding reverberation to semi-synthetic data, evaluating gender and language effects, expanding real recordings, and testing more complex microphone-pair setups.Source: §5 · p. 15