Source-linked AI summary
PG-SELD: Physics-Guided Sound Event Localization and Detection
Elad Cohen, Elad Dror Cohen, Arnon Netzer, Hai Victor Habi
TL;DR
SELD models face limited generalization across acoustic environments, where reverberation can introduce room-specific characteristics into learned representations. PG-SELD aligns reverberant features with free-field teacher features for matched scenes; on STARSS23, it consistently improves generalization across multiple baseline architectures.
Problem
SELD models have limited generalization across acoustic environments, where reverberation, noise, microphone characteristics, and room geometry can distort localization cues and encourage room-dependent representations.
Source: §1 · p. 1Method
PG-SELD uses a free-field teacher and physics-guided knowledge distillation to align intermediate features from matched reverberant and free-field recordings.
Source: §1 · p. 1, §3.1 · p. 2Results
Takeaways & Limitations
Reported improvements extend beyond synthetic data to real-world STARSS23 recordings across baseline architectures.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Sound event localization and detection (SELD) aims to jointly recognize sound events and estimate their directions of arrival from multichannel audio. Although recent deep learning approaches have achieved strong performance, their ability to generalize across acoustic environments remains limited, as room reverberation introduces environment-specific characteristics into the learned representations. In this work, we address this challenge by leveraging a physical free-field model as a room-independent reference. Specifically, we propose PG-SELD, a training framework that combines free-field with physics-guided knowledge distillation. Our approach aligns intermediate representations extracted from reverberant signals with those produced by a free-field teacher for matched acoustic scenes. This guidance encourages the model to preserve event- and localization-relevant information while reducing sensitivity to room-specific characteristics. Experimental results on the STARSS23 benchmark show that PG-SELD consistently improves the generalization performance of multiple baseline SELD architectures.
1. INTRODUCTION
SELD models still struggle to generalize across acoustic environments because room effects can distort localization cues and promote room-dependent representations. PG-SELD uses free-field references to encourage room-robust representations, with reported gains across multiple baseline architectures.
1. INTRODUCTION
- Reverberation, noise, microphone characteristics, and room geometry can distort localization cues, while synthetic training data may encourage room- or rendering-specific representations.Real multichannel recordings with accurate spatial annotations are costly to collect.Source: §1 · p. 1
- Data augmentation, multi-room simulation, synthetic-real combinations, and large-scale pretraining address robustness, but do not explicitly enforce room-robust representations.Prior work also explores echo-aware adaptation, physically informed spatial regularization, and intermediate-feature distillation.Source: §1 · p. 1
- PG-SELD uses free-field renderings as a room-independent reference to learn room-robust SELD representations.It aligns matched reverberant and free-field features through physics-guided knowledge distillation.Source: §1 · p. 1
- The proposed PGKD regularization aligns features from matched reverberant and free-field recordings to encourage room-robust latent representations.The framework is described as model-agnostic and adds no inference cost.Source: §1 · p. 1
- PG-SELD consistently improves generalization performance across multiple baseline SELD architectures on real acoustic environments.The reported evaluation includes CNN-Conformer, SELDNet, and HTS-AT.Source: §1 · p. 1
2. SIGNAL MODEL AND NOTATION
The signal model represents multichannel audio in the STFT domain as source signals transformed by the array response, with additive noise. It contrasts reverberant transfer functions with free-field responses that preserve predominantly direct-path information.
2. SIGNAL MODEL AND NOTATION
- The multichannel STFT signal is expressed as the sum of source signals transformed by the array response, plus additive noise.The model considers M microphones receiving signals from Q sources.Source: §2 · p. 2
- The STFT source coefficient, acoustic transfer function, and additive noise are denoted by S_q(t, f), H_q(f), and V, respectively.Here t and f denote the time-frame and frequency.Source: §2 · p. 2
- Reverberant transfer functions contain direct-path propagation and room-dependent reflections, whereas free-field acoustics preserve predominantly direct-path information.The passage characterizes free-field spatial representations as less affected by room-specific characteristics.Source: §2 · p. 2
3. METHOD
PG-SELD uses a free-field model as a room-independent reference to guide SELD representations under reverberation. It combines feature-level distillation with a training strategy that progressively shifts student inputs from free-field to reverberant recordings.
3. METHOD
- PG-SELD combines physics-guided knowledge distillation with a training strategy that uses the free-field model as a training objective.Source: §3 · p. 2
3.1. Physics-Guided Knowledge Distillation (PGKD)
- Matched reverberant and free-field inputs share sources, event activity, and spatial configuration, making the free-field representation a room-independent target.Source: §3.1 · p. 2
- A free-field teacher pretrained on a large-scale synthetic dataset guides the reverberant student by distilling intermediate representations.Source: §3.1 · p. 2
- A projection corrects teacher–student feature misalignment before backbone-output alignment, where representations encode higher-level semantic and spatial information.Source: §3.1 · p. 2
- The t-SNE visualization shows baseline features differing from free-field features, while PGKD brings features closer to the free-field conditions.Source: Figure 1 · p. 2, §3.1 · p. 3
3.2. Training Strategy
- The student’s inputs shift probabilistically from exclusively free-field to exclusively reverberant recordings, while the frozen teacher processes free-field inputs.The student does not directly observe the large-scale free-field dataset; the teacher transfers knowledge through feature-level regularization.Source: §3.2 · p. 3
4. EXPERIMENTAL RESULTS
PG-SELD is evaluated on synthetic and real recordings across three SELD architectures, with ablations examining its components and sensitivity to room diversity. It consistently improves reported results, including on STARSS23.
4.1. Implementation Details
- Experiments evaluate CNN-Conformer, SELDNet, and HTS-AT using simulated paired free-field/reverberant data and real STARSS23 recordings.The free-field teacher data derive from FSD50K; student pairs use TAU-SRIR-DB and FSD50K, with six training rooms and three validation rooms.Source: §4 · p. 3, §4.1 · p. 3
4.2. Results
- PG-SELD improves all evaluation metrics on DRIR and consistently reduces ESELD on STARSS23 across baseline models.The reported DRIR gains are primarily driven by improved localization.Source: Table 3 · p. 4, Table 2 · p. 4, §4.2 · p. 4
4.3. Ablation
- For CNN-Conformer on DRIR, PGKD improves performance over the baseline, curriculum learning adds a further gain, and λ = 0.5 performs best.The selected distillation weight is used in all experiments.Source: §4.3 · p. 4, Table 4 · p. 4
- With 15 hours of data fixed, PGKD provides larger gains when training-room diversity is low, and its added benefit decreases as room diversity grows.The authors suggest this pattern reflects reduced reliance on room-specific features under PGKD.Source: §4.3 · p. 4, Figure 3 · p. 4
- In t-SNE visualizations, PGKD-trained SELDNet features are more similar to free-field model features than baseline features.The authors interpret this as evidence that PGKD guides representations toward free-field characteristics and reduces sensitivity to room acoustics.Source: §4.3 · p. 4
5. CONCLUSIONS
PG-SELD aligns latent features from paired reverberant and free-field renderings to encourage room-robust representations. On STARSS23, it consistently improves generalization across multiple baseline models.
5. CONCLUSIONS
- On STARSS23, PG-SELD consistently improves generalization across multiple baseline models by aligning paired reverberant and free-field latent features.The method aims to preserve event- and localization-related information while suppressing room-dependent acoustic characteristics.Source: §5 · p. 4