QC-MI-OVSEL: Benchmarking Query-Conditioned Multi-Instance Open-Vocabulary Sound Event Localization

Motivation

• Prior formulations do not jointly model complete same-category source sets, time-varying cardinality, and suppression when the queried category is absent.SelectTSL supports multiple concurrent target sources per prompt but does not explicitly address scene-level absent queries, held-out categories, or complete 3-D source-set recovery.Source: §1 · p. 1

Contributions

• QC-MI-OVSEL comprises 40,000 FOA scenes and 80,000 balanced present/absent query views over 83 classes, including 17 novel classes excluded from training and development.The task predicts the unordered set of active physical instances matching a category-level language query, with their 3-D directions.Source: §1 · p. 1
• Q-RSet uses three learnable source slots to form query-conditioned hypotheses and aggregates source-specific acoustic evidence through non-exclusive responsibility weighting.The paper presents it as a source-set prediction model for recovering multiple query-matching sources.Source: §1 · p. 1

Task definition

• At each frame, the target set contains the active sources whose category matches the queried category, paired with their directions.The set is defined as G_t(q) = {(i, u_i,t) | a_i,t = 1, c_i = c_q}.Source: §2 · p. 1

Prediction target

• The prediction is an unordered framewise set, including the empty set when no queried source is active.A query can match multiple simultaneously active instances, with activities, directions, and target count varying over time.Source: §2 · p. 1

Target matching

• A source is a target only when its class matches the queried class; canonical queries and same-class paraphrases share identical reference targets.Source: §2 · p. 1

3. METHOD

• Q-RSet predicts up to three query-matching physical sources with framewise activity and 3-D direction through acoustic representation, source-hypothesis decoding, and responsibility evidence aggregation.These stages identify query-relevant acoustic content, form candidate source hypotheses, and associate each hypothesis with framewise acoustic evidence.Source: §3 · pp. 1–2

3.1. Query-Conditioned Acoustic Representation

• A learned query-relevance module scores acoustic tokens against the query, reweights them, and supplies shared memory for decoding and evidence aggregation.The acoustic representation uses a frozen GRAM-Ambisonics encoder and a pretrained CLAP text encoder; relevance is learned end-to-end rather than supervised as an explicit class probability.Source: §3.1 · p. 2

3.2. Query-Conditioned Source-Hypothesis Decoder

• Three learnable slots represent up to three sources; query-modulated slots cross-attend to shared acoustic memory, while permutation-invariant supervision encourages distinct source hypotheses.Slots have no predefined category, direction, or source identity; their distinct learned states and cross-attention support instance differentiation.Source: §3.2 · p. 2

3.3. Responsibility Evidence Aggregation

• Non-exclusive, slot-specific responsibilities measure how strongly each acoustic token supports a source hypothesis, enabling hypothesis-specific frame evidence.Responsibilities are computed independently across slots, so one token can support multiple hypotheses; pooled evidence feeds activity and normalized Cartesian DOA heads after temporal modeling.Source: §3.3 · p. 2

3.4. Set Matching and Training

• Training uses full-clip permutation-invariant matching on activity and active-frame DOA errors, then applies activity and active-frame cosine DOA losses.Absent queries contribute only activity loss; at inference, development-thresholded sigmoid activity scores determine active slots and framewise cardinality.Source: §3.4 · p. 2

Ontology and splits

• The benchmark has 83 ontology-antichain classes: 66 base and 17 novel, with novel classes excluded from training and development and reserved for testing.The 40,000 scenes are split into 30k/2k/8k train/dev/test, and source recordings are disjoint across splits.Source: §4 · p. 3
• The 8,000 test scenes are divided equally among Base-ID, Novel-ID, Acoustic-OOD, and Generalized conditions, varying query novelty and room exposure.Base-ID uses base queries and seen rooms; Novel-ID uses novel queries in seen rooms; Acoustic-OOD uses base queries in held-out rooms; Generalized combines base and novel queries with seen and held-out rooms.Source: §4 · p. 3

Query views and scene construction

• Each scene contains one to three instances of its present-query class plus non-target interferers, yielding framewise target counts from zero to three.Each scene supplies one present and one absent canonical query, for 80,000 balanced views; eight validated class paraphrases are reserved for linguistic-robustness evaluation.Source: §4 · p. 3

Spatial rendering and annotation

• Scenes are 10-second, 24-kHz FOA renderings using measured RIRs, with 10-Hz annotations of instance identity, activity, azimuth, and elevation for query-matching sources.DOAs are defined only on active frames.Source: §4 · p. 3

5.1. Evaluation Metrics

• Evaluation uses F1, localization recall, localization error, source-count error, exact-set recovery, and absent-query false-positive rate, with a 20° localization tolerance unless stated otherwise.Exact requires the complete source set and all matched directions within 20°; absent-query false-positive rate counts scenes with any predicted target activity.Source: §5.1 · p. 3

5.2. Training and Model Selection

• Models are trained and selected using canonical queries, with paraphrases reserved for linguistic-robustness evaluation and results averaged over three seeds.Frozen GRAM and CLAP encoders are used; checkpoints and activity thresholds are selected by development-set F1.Source: §5.2 · p. 3

5.3. Baselines

• The baselines compare semantic filtering after query-independent SELD with query-conditioned track prediction, including Q-MACCDOA and TQ-SEL-Adapt.Q-MACCDOA predicts three activity–DOA tracks without source slots or responsibility aggregation; TQ-SEL-Adapt uses three tracks with full-clip permutation-invariant training.Source: §5.3 · p. 3

6.1. Main Comparison

• With the same query-conditioned acoustic representation, Q-RSet reaches 50.36% F1 versus 43.09% for direct multi-track regression and improves LR, LE, K-MAE, Exact, and Abs-FPR.TQ-SEL-Adapt has an Abs-FPR of 96.13%.Source: §6.1 · pp. 3–4
• Against OVSELD-Adapt, Q-RSet improves F1 by 3.24 percentage points and LR by 8.64 points, while reducing K-MAE from 0.47 to 0.41 and Abs-FPR from 30.01% to 20.55%.Exact rises from 21.02% to 25.89%; OVSELD-Adapt has lower LE, 7.93° versus 8.86°.Source: §6.1 · p. 3, Table 1 · p. 4

6.2. Generalization and Language Robustness

• Novel-ID reduces F1 from 57.07% to 34.27% and LR from 56.56% to 26.90% versus Base-ID, while Acoustic-OOD remains close to Base-ID.Across four conditions, LE varies only from 8.74° to 9.10°, indicating a larger bottleneck in source discovery and count recovery than directional estimation.Source: §6.2 · p. 4, Table 2 · p. 4
• Paraphrasing reduces F1 from 50.36% to 37.58% and LR from 46.11% to 30.36%, while K-MAE rises from 0.41 to 0.55.LE changes by only 0.24°, so linguistic variation mainly affects identifying and counting sources rather than localization accuracy for recovered sources.Source: §6.2 · p. 4, Table 2 · p. 4

6.3. Multi-Instance Analysis

• As maximum target count rises from Kmax = 1 to Kmax = 3, K-MAE increases from 0.43 to 0.89 and Exact falls from 39.27% to 14.89%.F1 remains between 52.72% and 56.04%, and LE increases moderately from 8.16° to 9.22°; F1 does not decline monotonically, unlike K-MAE and Exact.Source: Table 3 · p. 4, §6.3 · p. 4

6.4. Ablation Study

• Increasing the decoder from 3 to 6 to 12 layers raises F1 from 47.23% to 49.28% to 50.36%, and the 12-layer model performs best on LR, LE, K-MAE, Exact, and Abs-FPR.The 12-layer decoder is used in the final Q-RSet.Source: Table 4 · p. 4, §6.4 · p. 4
• Removing each component reduces F1 across all three matched seeds; the largest mean drop is 6.27 percentage points without the Temporal Transformer, followed by 4.01 points without Audio Cross-Attention.Removing Query FiLM, Responsibility, and Query Relevance reduces F1 by 2.32, 2.14, and 1.17 points, respectively; some removals slightly improve isolated secondary metrics, but none improves overall source recovery.Source: Table 4 · p. 4, §6.4 · p. 4

• Q-RSet improves source recovery, cardinality estimation, complete-set recovery, and absent-query suppression over complementary baselines.Source: §7 · p. 4
• Novel categories, paraphrased queries, and increasing source multiplicity mainly impair source discovery, counting, and set completion.Source: §7 · p. 4
• Directional accuracy remains comparatively stable for successfully recovered sources, making query-conditioned source-set recovery a central challenge beyond DOA estimation alone.Source: §7 · p. 4