Source-linked AI summary
SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection
Runbang Wang, Zining Liang, Yin Cao, Qiuqiang Kong
TL;DR
Existing sound-event localization predicts source classes and directions, but not the regions and energy represented by labeled acoustic maps. SAID pretrains a directional audio encoder on sound energy, then trains the complete model to predict separate source maps and classes. Its DCASE2026 Task 3 Track A submission ranks first on the official evaluation set.
Problem
Conventional SELD predicts classes and directions rather than source regions and energy, while acoustic imaging must separate nearby sources whose number and region sizes change.
Source: §1 · p. 1Method
SAID organizes audio features by direction, pretrains Audio2Sph on sound energy without class labels, and trains the model to predict a separate labeled map for each active source.
Source: §1 · p. 1Results
SAID ranks first on the official DCASE2026 Task 3 Track A evaluation set with 0.1080 Macro mAP and 0.3962 Macro Pearson r.
Source: §5 · p. 4Takeaways & Limitations
The model and data-generation pipeline provide a starting point for further research on audio-only semantic acoustic imaging.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In daily life, people hear speech, footsteps, and music around them. We can often recognize these sounds and judge where they come from. Each sound source can be shown on a separate acoustic map, a rectangular image covering $360^{\circ}$ horizontally and $180^{\circ}$ vertically. The map shows the directions occupied by the source as a region and the sound energy within that region. A class label identifies the sound. Predicting these labeled acoustic maps from audio is called semantic acoustic imaging. Such maps could help robots perceive their surroundings and allow augmented reality displays to show sound regions and classes over the real world. Existing models can recognize sound classes and estimate a direction for each source. However, a direction alone does not describe the source region or its energy. Acoustic imaging must also distinguish sound sources in nearby directions, while the number of active sources and the regions they occupy can change over time. We therefore propose the Semantic Acoustic Imaging Detector (SAID), which predicts a separate labeled acoustic map for each active source from audio. First, we pretrain Audio2Sph, SAID's audio encoder, through sound energy estimation across directions without class labels. Then, we train the complete SAID model to predict source regions, energy, and classes together. We also develop a pipeline that generates simulated recordings for pretraining and supports fine-tuning on real recordings. On the official DCASE2026 Task 3 Track A evaluation set, our submitted system ranks first with 0.1080 macro-averaged mean average precision (Macro mAP) and 0.3962 Macro Pearson $r$. Demos and code are provided at https://github.com/IN03X/SAID.
1. INTRODUCTION
Semantic acoustic imaging predicts a labeled directional map for each sound source, including its region and energy, to represent more than source direction alone. SAID organizes audio features by direction and decodes them into separate maps for active sources.
1. INTRODUCTION
- Semantic acoustic imaging estimates labeled acoustic maps from audio, showing source regions and energy across a 360°-by-180° directional image.The introduction identifies robotics and augmented reality as potential uses for those maps.Source: §1 · p. 1
- Earlier localization methods estimated directions, and SELD later combined class and direction; the DCASE2026 task instead represents sources as acoustic-map regions,,.The DCASE baseline adapts Mask R-CNN, while MaskFormer and Mask2Former use mask decoders,,.Source: §1 · p. 1
- Conventional SELD predicts classes and directions but not source regions or their energy, while acoustic imaging must separate nearby sources as their number and region sizes change,,.Audio recordings also lack features already arranged like an acoustic map.Source: §1 · p. 1
- SAID pairs Audio2Sph, which organizes audio features by direction, with Sph2Imaging, which predicts labeled maps; Audio2Sph is pretrained on directional sound energy without class labels [1].The paper also introduces a pipeline for simulated pretraining recordings and fine-tuning on DCASE recordings.Source: §1 · p. 1
2. METHOD
SAID encodes audio into features arranged by direction, then decodes them into a separate map, activity estimate, and class prediction for each candidate source. Its encoder also has a class-agnostic energy-map pretraining stage.
2. METHOD
- SAID converts four-channel audio into panoramic features, then predicts a map, activity score, and class for each candidate source.The overview separates Audio2Sph encoding from Sph2Imaging decoding.Source: §2 · p. 2
2.1. Audio2Sph: SAID’s Encoder
- Audio2Sph uses learned directional queries and spherical cross-attention to convert multi-scale time–frequency features into direction-organized panoramic features.Time Alignment combines adaptive average- and max-pooled features and produces outputs at 10 fps; during pretraining, a separate decoder predicts class-agnostic energy maps and is then removed.Source: §2.1 · p. 2
2.2. Sph2Imaging: SAID’s Decoder
- Sph2Imaging builds multi-scale spatial features and combines them with PaSST-derived class features to update learned slot queries.Each slot represents a candidate source rather than a fixed direction or class.Source: §2.2 · p. 2
- The Map Head predicts a map for each slot, whose preceding mask can restrict later spatial attention; the Active and Class Heads estimate activity and class.A Refine Block upsamples the final maps to 180×360, and each map receives its highest-probability class.Source: Figure 3 · p. 3, §2.2 · p. 3
3. DATA AND TRAINING
The pipeline generates simulated multichannel scenes for pretraining and class-labeled training, then fine-tunes SAID on DCASE recordings. Training proceeds from class-agnostic sound-energy estimation to matched source-map and class supervision.
3.1. Online Scene Generation
- Online scene generation uses the image source method to create two-second, four-channel recordings with 0–4 VCTK sources in sampled rooms.For class-labeled training, SourceBank contains 122,359 clips mapped to 13 DCASE classes; multiple emitting points driven by the same clip simulate an extended source.Source: §3.1 · p. 3
3.2. Staged Training
- Audio2Sph pretraining uses unweighted binary cross-entropy on 180×360, 100-fps energy maps, formed from the pointwise maximum of active-source spherical Gaussians.Frames without active sources receive all-zero targets; pretraining runs for 2.5 million steps with batch size one.Source: §3.2 · p. 3
- After pretraining, SAID trains on SourceBank scenes at 10 fps, with Hungarian matching between slots and sources using map and class agreement across ten prediction groups.The Map Head receives BCE and Dice supervision on matched maps, while the refined maps receive full-map BCE and Dice plus correlation and soft-IoU terms.Source: §3.2 · pp. 3–4
- Training supervises the Active Head with weighted cross-entropy for source classes and no-object slots, and the Class Head with class-frequency-weighted focal loss.The paper then fine-tunes on DCASE recordings with annotated maps and classes, using four microphone-configured views with azimuth rotations of 0°, 90°, 180°, and 270°.Source: §3.2 · p. 4
4. EXPERIMENTS
The experiments evaluate prediction compression, official DCASE2026 Task 3 Track A performance, and development-test ablations.
4.1. Evaluation and Results
- 97.81% less total JSON storage across 78 development-test recordings keeps every file below 20 MB; Macro mAP falls 0.0024 while Macro Pearson r rises 0.0219.Refined maps are stored as sparse points and reconstructed by the official evaluator using Gaussian smoothing.Source: §4.1 · p. 4, Table 1 · p. 4
- 0.1080 Macro mAP and 0.3962 Macro Pearson r rank first among Track A submissions, with the highest scores in both metrics.Table 2 compares each team's best-ranked submission on the official evaluation set.Source: Table 2 · p. 4, §4.1 · p. 4
4.2. Ablation Studies
- Development-test ablations compare encoder and pretraining variants using class-agnostic Mask AP and spatially matched Macro Class-F1 with 20° matching.Variants include replacing or removing the PaSST branch and training without Audio2Sph pretraining.Source: §4.2 · p. 4, Table 3 · p. 4
5. CONCLUSION
SAID predicts a separate labeled acoustic map for each active source from four-channel audio, using a staged training pipeline. Its challenge submission ranks first on the official DCASE2026 Task 3 Track A evaluation set.
5. CONCLUSION
- 0.1080 Macro mAP and 0.3962 Macro Pearson r earn SAID's challenge submission first place on the official DCASE2026 Task 3 Track A evaluation set.The model and data-generation pipeline are presented as a starting point for further audio-only semantic acoustic imaging research.Source: §5 · p. 4
- SAID predicts a separate labeled acoustic map for each active sound source from four-channel audio.Audio2Sph arranges panoramic features by direction, and Sph2Imaging decodes them into source regions, energy, and classes.Source: §5 · p. 4
- Training first pretrains Audio2Sph without class labels using online simulation, then trains SAID on individual source maps and classes before fine-tuning on real recordings.The paper presents the model and data-generation pipeline as a starting point for further audio-only semantic acoustic imaging research.Source: §5 · p. 4