PaperlayerView exact revision on arXiv

Source-linked AI summary

Hear Every Sound Object: Rescuing Overlooked Sources in Auditory Scene Analysis

Younghoo Kwon, Yongjoon Lee, Jung-Woo Choi

arXiv:2610.30874v1eess.AS

TL;DR

Object-based ASA models split source-specific features only at the final layer, while source-wise performance disparities remain substantial. PARASITE splits and progressively refines object features from earlier layers, and DeepASA++ achieves state-of-the-art results on ASA2 and STARSS23. On STARSS23, it attains a SELD score of 0.240.

  • Problem

    Object-based ASA models form source-wise features only at the final layer, and their performance varies substantially across sources.

    Source: §1 · p. 1

  • Method

    PARASITE splits object features from earlier scene features and progressively refines them with OGF and WCOA in DeepASA++.

    Source: §1 · pp. 1–2

  • Results

    DeepASA++ achieves state-of-the-art results on ASA2 and STARSS23, including a SELD score of 0.240 on STARSS23.

    Source: §6 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    DeepASA++ combines earlier object-feature splits with progressive refinement and achieves state-of-the-art performance on both evaluated datasets.

    Source: §6 · p. 4

Abstract

from arXiv · show

Auditory scene analysis (ASA) models extract information about individual sound sources from a complex auditory mixture. Recent object-based ASA models process the mixture through a sequence of modules and split it into source-wise features (i.e., object features) only at the last layer. However, substantial disparities in performance across sources limit overall performance. Our analysis using centered kernel alignment (CKA) reveals that CKA scores correlate strongly with per-source performance, with their disparities widening from intermediate to deeper layers. These findings motivate the early split and exchange of object features to address source-wise performance disparities before they widen in deeper layers. To this end, we propose PARASITE, a plug-in refinement architecture that splits object features at earlier layers and progressively refines them through sequential layers. Each layer integrates object features separated at different depths using Object-wise Gated Fusion (OGF) and then refines them using Windowed Cross-Object Attention (WCOA). We introduce the DeepASA++ model, which combines PARASITE with an ASA backbone and delivers state-of-the-art results on both ASA2 and STARSS23. DeepASA++ achieves a SELD score of 0.240 on STARSS23 and surpasses prior multi-stage inference architectures, including chain-of-inference.

1. INTRODUCTION

ASA seeks to characterize sources in complex mixtures, while object-based DeepASA forms source-specific features only after processing a shared scene representation. Source-wise performance disparities and widening CKA gaps motivate PARASITE, which splits and refines object features earlier; DeepASA++ combines it with DeepASA-Loco and reports state-of-the-art results on ASA2 and STARSS23.

1. INTRODUCTION

  • DeepASA splits a shared scene feature into object features only after the backbone’s final layer.Task-specific decoders map each object feature to its corresponding output.Source: §1 · p. 1
  • Per-source performance is associated with CKA values, and the CKA gap between sources widens from middle to final layers.The disparity motivates investigating intermediate features for improving less dominant sources.Source: §1 · p. 1
  • PARASITE splits object features from middle-layer scene features and progressively refines them alongside the scene backbone.OGF fuses same-object features from previous and current depths; WCOA enables exchange across objects over a temporal window.Source: §1 · pp. 1–2
  • DeepASA++ integrates PARASITE into DeepASA-Loco, whose scene backbone uses TF-Locoformer blocks.The paper reports results on ASA2 and STARSS23.Source: §1 · p. 2
  • 14.24 dB SI-SDRi on ASA2 and 0.240 SELD score on STARSS23 are reported for the proposed system as state-of-the-art results.DeepASA++ integrates PARASITE into DeepASA-Loco.Source: §1 · p. 2

2. SOURCE DOMINANCE ANALYSIS

The analysis relates source dominance in DeepASA-Loco’s scene features to per-source performance. CKA gaps are smallest at middle layers and widen toward the final layer, where the last-split strategy produces substantial rank-wise performance gaps.

2. SOURCE DOMINANCE ANALYSIS

  • The analysis ranks sources within mixtures by CKA averaged across layers, then measures layer-wise CKA for groups sharing a dominance rank.It uses Mel-spectrogram and SPEAR embeddings on about 19K five-source ASA2 training mixtures, with means and 95% confidence intervals over 50 subsets of 500 mixtures.Source: §2 · p. 2
  • The widening final-layer CKA gap accompanies the largest rank-wise performance gap under last-split decoding.Because only the final scene feature is split, the least dominant object feature is decoded from the representation in which that source is least present.Source: §2 · p. 2
  • From Layer 4 to the final layer, the CKA gap between highest and lowest dominance ranks grows from 0.32 to 0.45 with Mel-spectrogram references and from 0.17 to 0.21 with SPEAR embeddings.The rank-5 sources trail rank 1 by 10.60 dB in SI-SDR, 21.32° in localization error, and 27.4%p in classification accuracy with Mel-spectrogram references.Source: Figure 2 · p. 2, Table 1 · p. 2, §2 · p. 2

3. METHODOLOGY

DeepASA++ combines a scene backbone, split modules, PARASITE layers, and task-specific decoders. PARASITE fuses object features from successive depths within each object, then allows cross-object interaction over neighboring time frames.

3. METHODOLOGY

  • Split modules attached to the last L + 1 scene analysis blocks supply object features to PARASITE, which progressively fuses features from successive depths.The first PARASITE layer fuses features split from two adjacent scene features; later layers fuse the previous object features with features from the current scene feature.Source: §3 · p. 2
  • OGF uses three gated units to refine its two same-object inputs and fuse the resulting features.Its final output has a skip connection to the object feature carried from the previous layer.Source: §3 · p. 2
  • Each PARASITE layer applies Object-wise Gated Fusion (OGF) before Windowed Cross-Object Attention (WCOA).OGF combines features for the same object; WCOA lets objects interact by looking up adjacent time frames.Source: §3 · p. 2, Figure 3 · p. 3
  • WCOA attends across objects using keys from all objects within a temporal window around each query’s time-frequency bin.It sums attention probabilities for each object and applies each object’s resulting weight to its value at the query’s bin.Source: §3 · p. 3

4. EXPERIMENTAL SETUPS

The experiments use ASA2 for training and evaluation and STARSS23 for SELD evaluation. DeepASA-Loco is trained first, then PARASITE is trained or fine-tuned on a frozen backbone, with task-specific metrics used for evaluation.

4. EXPERIMENTAL SETUPS

  • ASA2 mixtures contain up to five sources from 13 classes, and ASA2 is used for both training and evaluation.STARSS23 is used to evaluate SELD performance; its model training uses augmented data.Source: §4 · p. 3
  • DeepASA-Loco uses six TF-Locoformer blocks with channel dimension 64, while PARASITE uses three layers and WCOA uses a five-frame temporal window.The number of objects is set to five.Source: §4 · p. 3
  • ASA2 training proceeds in two stages: DeepASA-Loco is trained for 150 epochs, then PARASITE is trained with the first-stage modules frozen.Each PARASITE layer’s object features are decoded for loss computation, but inference uses only last-block decoder outputs.Source: §4 · p. 3
  • For STARSS23, the ASA2-trained model is fine-tuned first on the dataset and then PARASITE is fine-tuned on the frozen backbone.SI-SDRi is reported for separation, and SELD score for detection and localization.Source: §4 · p. 3

5. RESULTS

DeepASA++ leads the reported ASA2 comparisons in separation and SELD, and its gains extend to STARSS23. Ablations and per-source results favor using middle-layer features with small CKA gaps and refining them across objects.

ASA2 results

  • 14.24 dB SI-SDRi and a SELD score of 0.147 make DeepASA++ best in both separation and SELD on ASA2.Attaching PARASITE to DeepASA also improves performance, so the gain does not depend on the backbone choice.Source: §5 · p. 3, Table 2 · p. 4

Model size

  • DeepASA-Loco (large) uses 1.69 times the parameters and 1.53 times the computation of DeepASA++ yet trails it by 0.45 dB in SI-SDRi and 0.017 in SELD score.The larger model doubles the Conv-SwiGLU channel expansion rate from 2 to 4.Source: §5 · p. 3, Table 3 · p. 4

Layer selection

  • Selecting Layers 4 and 5, which have small CKA gaps, outperforms using only Layer 6 or substituting Layers 2 and 3 among tested configurations.Using only Layer 6 improves the SELD score marginally over DeepASA-Loco, from 0.176 to 0.172.Source: §5 · pp. 3–4, Table 3 · p. 4

PARASITE modules

  • Replacing OGF with a single gated unit worsens SELD to 0.180 from 0.176 for DeepASA-Loco, and removing WCOA degrades both metrics versus DeepASA++.The authors attribute the WCOA ablation result to OGF processing each object feature independently.Source: §5 · p. 4, Table 3 · p. 4

Per-source performance

  • Across five-source ASA2 mixtures, performance degrades monotonically from dominance rank 1 to rank 5, while DeepASA++ outperforms both comparison models at every rank in both metrics.Its advantage over DeepASA-Loco grows from rank 1 to rank 5; enlarging DeepASA-Loco improves classification only for low-ranked sources and yields almost no localization gain.Source: §5 · p. 4, Figure 4 · p. 4

STARSS23 results

  • 0.240 SELD score makes DeepASA++ best on STARSS23, 0.013 below the previous best and 0.020 ahead of the top-ranked DCASE ensemble.Its localization error drops from 10.0° to 8.4°, the lowest among compared systems.Source: §5 · p. 4, Table 4 · p. 4

6. CONCLUSION

PARASITE and DeepASA++ address source-wise performance disparities by extracting object features early and progressively refining them. DeepASA++ achieves state-of-the-art performance on ASA2 and STARSS23.

  • DeepASA++ achieves state-of-the-art performance on the ASA2 and STARSS23 datasets.Source: §6 · p. 4
  • PARASITE extracts object features at earlier layers and integrates them with features from subsequent scene-backbone depths.It progressively refines object features through Object-wise Gated Fusion and Windowed Cross-Object Attention.Source: §6 · p. 4
  • Source-wise CKA disparities are small at intermediate layers but large at the final layer, motivating earlier object-feature extraction.Source: §6 · p. 4
  • Splitting object features where the CKA gap is small brings the largest gain, especially for less dominant sources.Source: §6 · p. 4
Loading 2610.30874v1…