Source-linked AI summary
Ontology-based Target Sound Extraction
Carlos Hernandez-Olivan, Marc Delcroix, Tsubasa Ochiai, Naohiro Tawara, Shoko Araki
TL;DR
Existing target sound extraction systems are tied to fixed sound categories and do not explicitly model hierarchical sound relationships. This paper introduces ontology-based TSE with ontology-aware class embeddings and CPCC regularization, enabling single-pass extraction at any ontology level. Experiments show that considering ontology structure improves extraction performance across levels, including a 2.4 dB leaf-level improvement over the baseline.
Problem
Existing TSE systems are typically conditioned on fixed sound-class representations and do not explicitly model relationships among semantically related ontology categories.
Method
The paper uses ontology-aware multi-hot or full-ontology one-hot class embeddings, regularized by CPCC loss to align embedding distances with ontology-tree shortest-path distances.
Results
Ontology-based multi-hot extraction with CPCC regularization provides stable performance across ontology levels and improves leaf-level extraction by 2.4 dB over the baseline.
Takeaways & Limitations
Ontology-based TSE enables single-pass extraction from fine-grained leaf classes to root categories without a separate sound-detection model.
Abstract
from arXiv · showhide
Target sound extraction (TSE) aims to isolate a sound source of interest from a mixture, given a semantic query. Existing TSE systems are conditioned on fixed class representations tied to individual sound categories, limiting their ability to handle the hierarchical relationships that naturally organize environmental sounds. In this paper, we introduce ontology-based TSE, a new task formulation in which a single model extracts sounds queried at any level of a sound ontology, from fine-grained leaf classes such as cat and dog to high-level categories such as animal. We propose a learnable class embedding table defined over all nodes of an AudioSet-derived ontology, regularized with a Cophenetic Correlation Coefficient (CPCC) loss that aligns embedding distances with shortest-path distances in the ontology tree. Our experiments across different approaches show the benefit of considering the ontology structure when training TSE systems.
1. INTRODUCTION
Ontology-based TSE addresses the fixed-granularity limitation of existing systems by using ontology-aware queries and conditioning embeddings to extract sounds from leaf to parent categories. The proposed model supports single-pass extraction without separate detection.
- Environmental sounds are organized hierarchically, but existing TSE systems typically extract at a fixed sound-class granularity.
- Existing class-conditioned systems lack explicit relationships between semantically related categories and may require separate detection plus one pass per active leaf class.
- The proposed TSE model extracts sounds at any ontology level with a single forward pass and no separate detection stage.
- Ontology-aware conditioning uses multi-hot queries or full-ontology one-hot queries to represent leaf and parent classes.The multi-hot query activates descendant leaf labels, whereas full-ontology one-hot assigns distinct vectors to ontology nodes.
- A CPCC loss regularizes the conditioning embedding space so pairwise latent distances reflect shortest-path distances in the ontology tree.This encourages semantically related classes to have similar representations.
2. ONTOLOGY-BASED TSE
Ontology-based TSE represents sound categories within an AudioSet-derived hierarchy and supports extraction at any ontology level with a single model and forward pass. The paper compares baseline, multi-hot, and full-ontology one-hot conditioning, and regularizes embeddings to reflect tree structure.
- Ontology-Based TSE: Ontology-based TSE queries any node in a rooted sound ontology, from broad roots through intermediate categories to fine-grained leaves, using one model and one forward pass.The ontology is derived from AudioSet; roots, intermediates, and leaves are evaluated as separate levels.
- Baseline One-Hot Conditioning: The baseline extracts every descendant leaf separately and sums their signals, without explicitly representing ontology relationships.At inference, it extracts all leaf descendants whether active or not, because active classes are unknown.
- Multi-Hot Hierarchical Conditioning: Multi-hot conditioning activates all leaf descendants of a queried node, enabling single-pass extraction of their summed active signals.The same descendant-based query is used during training and inference, and parent embeddings are implicitly formed by summing child embeddings.
- Full-Ontology One-Hot Conditioning: Full-ontology one-hot conditioning assigns an independent embedding to every leaf and internal ancestor, allowing ancestor categories to be modeled as independent semantic concepts.This extends the embedding matrix from leaf nodes to all ontology nodes.
- Training Strategy and Loss Function: Training selects one node per ontology level and reconstructs the sum of its active leaf descendants using negative SI-SNR plus CPCC regularization.CPCC is defined as the negative Pearson correlation between tree distances and Euclidean embedding distances; λ controls the hierarchical constraint weight.
3. EXPERIMENTS AND RESULTS
The experiments evaluate ontology-based extraction on a synthesized dataset and compare ontology-aware conditioning with leaf-based baselines. Multi-hot conditioning provides stronger extraction across ontology levels, including a 2.4 dB leaf-level improvement over the baseline.
- 3.2. Experimental Results: Table 1 reports mean SNR / SI-SNR improvements in dB on the test set.The unprocessed row gives the input SNR/SI-SNR between mixtures and targets.
- 3.1. Ontology-Annotated Dataset: The synthesized dataset contains 48K training, 4K validation, and 8K test mixtures.Mixtures contain 3–5 targets spanning at least two root branches, with 5-second duration and 5–15 dB target-to-background SNR.
- 3.2. Experimental Results: The baseline can perform leaf-level extraction but performs very poorly at intermediate and root ontology levels.Aggregating inactive leaf extractions for a parent degrades broader-level extraction; oracle detection mitigates this at the cost of a separate model.
- 3.2. Experimental Results: More than 1 dB separates multi-hot extraction from both the baseline and full-ontology one-hot conditioning for root- and intermediate-level extraction.This comparison covers systems 3–6 in Table 1.
- 3.3. Discussion: 2.4 dB improvement over the baseline is achieved for leaf-level extraction by ontology-based multi-hot extraction with CPCC regularization.The proposed approach also maintains stable extraction performance across ontology levels and operates in a single pass without a separate sound-detection model.
4. CONCLUSIONS
The paper introduces ontology-aware conditioning for TSE, enabling single-pass extraction at any level of a sound ontology. The framework uses class embeddings whose geometry reflects the AudioSet ontology, with future work targeting enrollment queries, sound event detection, and zero-shot separation.
- 4. CONCLUSIONS: Ontology-aware conditioning enables single-pass extraction from fine-grained leaf classes through root categories.The framework represents ontology structure through class embeddings reflecting the AudioSet ontology.
- 4. CONCLUSIONS: Future work includes audio enrollment queries, ontology-based sound event detection, and zero-shot separation.