PaperlayerView exact revision on arXiv

Source-linked AI summary

Overview and Meta-Analysis of DCASE 2026 Challenge Task 6: Audio Moment Retrieval from Long Audio

Hokuto Munakata, Tatsuya Komatsu, Keisuke Imoto, Taichi Nishimura, Huang Xie, Tuomas Virtanen

arXiv:2609.12484v2eess.AScs.SD

TL;DR

AMR seeks query-matching temporal moments in long audio, a setting where manual search is time-consuming. This paper describes the DCASE 2026 challenge, its datasets and evaluation, and an MS-CLAP/DETR baseline; the top systems reached 48.59% Recall1@0.7, about 3.5 times the baseline score.

  • Problem

    Manually finding query-relevant portions of long audio is time-consuming and does not scale, motivating text-based retrieval of matching moments.

    Source: §1 · p. 1

  • Method

    The paper presents the challenge datasets and evaluation protocol, including a baseline combining MS-CLAP features with a DETR-based moment-detection network.

    Source: §6 · p. 4

  • Results

    48.59% Recall1@0.7 was achieved by the three best systems, roughly 3.5 times the baseline score.

    Source: §6 · p. 4

  • Takeaways & Limitations

    The top-performing teams enhanced feature extractors and used confidence-score calibration or ensembling across temporal resolutions.

    Source: §1 · p. 1

    Systems tuned only on synthetic development data sometimes transferred poorly to real data, and synthetic validation improvements occasionally accompanied worse real-data performance.

    Source: §5.2 · p. 4

Abstract

from arXiv · show

This paper presents an overview of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2026 Challenge Task 6, Audio Moment Retrieval (AMR) from Long Audio. Given a several-minute-long audio recording and a free-form text query, AMR aims to retrieve temporal moments in the recording that match the query, where each moment is represented by a pair of start and end timestamps. This task requires effective cross-modal alignment and long-range temporal modeling. We describe the task definition, the evaluation metrics, the development and evaluation datasets, and a baseline system that combines a pre-trained MS-CLAP feature extractor with a Detection Transformer (DETR)-based moment-detection network. On the development data, the baseline trained on a manually annotated dataset and a synthetic dataset achieved Recall1@0.7 of 13.56%, indicating that AMR in long audio remains a challenging problem. The challenge attracted 21 teams, which submitted 59 systems in total. The three best systems achieved Recall1@0.7 of 48.59%, roughly 3.5 times the baseline score. The results show that strengthening the audio-text feature extractor and the moment-detection network led to substantial performance improvements. Furthermore, the top three teams boosted performance by applying confidence score calibration or ensembling across different temporal resolutions of features.

1. INTRODUCTION

Audio Moment Retrieval (AMR) finds query-relevant temporal intervals in long, untrimmed recordings, addressing the time-consuming task of manually searching large audio archives. DCASE 2026 Task 6 brought this task to the challenge and its top systems substantially outperformed the baseline.

1. INTRODUCTION

  • AMR retrieves contiguous start- and end-timestamp intervals matching a complex free-form query in a long recording, which may contain several matching moments.For example, a query can describe spectators cheering and shouting during a sports game.Source: §1 · p. 1
  • By letting users search long recordings with free-form text, AMR can help index and navigate archives that are time-consuming to search manually.The motivating archives include meetings, lifelogs, broadcasts, podcasts, and acoustic monitoring.Source: §1 · p. 1
  • Unlike clip-level audio retrieval, AMR detects when queried content occurs in long, untrimmed audio; unlike sound event detection, it uses open-vocabulary queries that may describe overlapping events or broader scenes.The paper relates AMR's temporal-detection paradigm to video moment retrieval, while noting that AMR must infer temporal cues from sound alone.Source: §1 · p. 1
  • AMR faces a scarcity of annotated long-audio data while requiring long-range temporal context and strong audio-text alignment; prior work found synthetic pretraining followed by annotated-data fine-tuning substantially improves accuracy.The initial study introduced the AM-DETR baseline and Clotho-Moment, later complemented by manually annotated CASTELLA.Source: §1 · p. 1
  • 48.59% Recall1@0.7 was achieved by the three best systems, roughly 3.5 times the baseline score.The challenge drew 21 teams and 59 systems.Source: §1 · p. 1

2. TASK DEFINITION

AMR predicts timestamped moments in long audio that match a free-form query, ranking predictions by confidence. The task evaluates temporal overlap and prioritizes whether the highest-confidence moment matches a reference.

2.1. Problem formulation and notation

  • AMR takes a long recording and free-form query and predicts one or more matching timestamped moments, ranked by confidence.A query may correspond to multiple reference moments, and participants must submit at least one predicted moment per query.Source: §2.1 · pp. 1–2

2.2. Evaluation metrics

  • Recall1@θ evaluates whether the highest-confidence predicted moment reaches IoU ≥θ, while mAP@θ accounts for all confidence-ranked predictions matched one-to-one to references.Averaged mAP spans θ values from 0.50 to 0.95 in 0.05 steps; Recall1 is reported at θ = 0.5 and θ = 0.7.Source: §2.2 · p. 2
  • Recall1@0.7 is the primary ranking metric, requiring the top-ranked moment to attain IoU ≥0.7; additional predictions affect mAP but not this ranking.A predicted moment is correct when its temporal IoU with a reference moment meets the chosen threshold.Source: §2.2 · p. 2

3. DATASETS

The challenge provides two development datasets and a separately collected evaluation dataset, all pairing long recordings with English captions and timestamp annotations. The development data combines synthetic and manually annotated resources, while evaluation ground-truth timestamps are withheld.

3.1. Development dataset

  • Clotho-Moment contains 51,240 one-minute recordings and 44,261 captions, created by overlaying Clotho foreground clips onto long Walking Tours recordings at random intervals.The overlay procedure supplies timestamps automatically, avoiding manual annotation.Source: §3.1 · p. 2
  • CASTELLA provides 1,862 manually annotated recordings spanning 120 hours, with one-to-five-minute durations and up to five moments per recording.Crowdsourced captions and one-second-resolution timestamps were reviewed by listening to audio to remove errors caused by visual bias; each recording averages 2.1 moments.Source: §3.1 · p. 2
  • Development-training combines the training splits of CASTELLA and Clotho-Moment, while development-validation and development-testing use CASTELLA’s validation and test splits.These split names are redefined using DCASE terminology.Source: §3.1 · p. 2

3.2. Evaluation dataset

  • The evaluation set contains 100 real YouTube recordings of one to five minutes and 177 text queries, with ground-truth timestamps withheld during the challenge.Captions and timestamps were collected in the same manner as CASTELLA, and queries may have multiple ground-truth timestamp pairs.Source: §3.2 · p. 2

4. BASELINE SYSTEM

The AM-DETR baseline combines shared audio-text embeddings with a transformer-based moment detector. It contextualizes audio features using the text query, predicts ranked candidate moments, and trains them against reference moments with multiple losses.

4. BASELINE SYSTEM

  • AM-DETR combines an audio-text feature extractor with a moment-detection network adapted from QD-DETR for acoustic recordings.The network builds on Detection Transformer and Moment-DETR.Source: §4 · p. 2
  • Pretrained MS-CLAP maps text queries and one-second audio frames into a shared 768-dimensional embedding space that captures audio-text alignment.Using precomputed embeddings rather than raw audio keeps the detection network lightweight.Source: §4 · pp. 2–3
  • A transformer encoder contextualizes audio embeddings conditioned on the text query, and a decoder predicts K candidate moments with confidence scores.The baseline uses K = 10 candidates and ranks them by confidence, using the top candidate for Recall1@0.7.Source: §4 · p. 3
  • Training matches predicted and reference moments bipartitely and combines center-width L1, generalized IoU, confidence cross-entropy, highlight-detection, and negative-pair losses.Models train on development-training, are selected on development-validation, and are evaluated on development-testing; two configurations use CASTELLA alone or with Clotho-Moment.Source: §4 · p. 3

5. RESULTS AND DISCUSSION

The top three systems tied at Recall1@0.7 of 48.59%, roughly 3.5 times the baseline. Discussion of the systems highlights feature extraction, moment detection, confidence and timestamp refinement, and differences between synthetic and real data.

5.1. Results

  • 48.59% Recall1@0.7 was achieved by each of the top three systems, roughly 3.5 times the baseline; the fourth-ranked system scored 46.89%.The three leaders tied on the primary metric but differed on Recall1@0.5 and average mAP.Source: Table 2 · p. 3, §5.1 · p. 3

5.2. Discussion

  • Teams replacing the feature extractor dominated the leaderboard, with median Recall1@0.7 of about 41%, versus 18% for systems retaining MS-CLAP.With architecture unchanged, replacing MS-CLAP with M2D-CLAP raised development-testing Recall1@0.7 from 26.80% to 40.70%.Source: §5.2 · p. 3
  • Top systems used DETR-based network variants, while a Qwen2-Audio system compressed sequence features to predict moments but remained below the top system.Kim and Choi reported transformer self-attention replacement with a Mamba temporal encoder as one of their largest single-component gains; most training-free systems did not outperform the baseline.Source: §5.2 · pp. 3–4
  • Total parameters correlated weakly with performance (r = 0.22), versus trainable parameters (r = 0.57); gains also involved losses and post-processing.The discussion names confidence-score refinement and model ensembling as examples of post-processing.Source: §5.2 · pp. 3–4
  • Varifocal loss improved Recall1@0.7 from 20.30% to 27.62%, while an MLP calibrator improved it from 34.45% to 40.01%.Teams addressed misalignment between confidence scores and localization quality using loss functions or score calibrators.Source: §5.2 · p. 4
  • Sugawara targeted moment-width estimation with varied-resolution ensembling, while Kim and Kibata were better at roughly detecting more difficult moments.The comparison suggests Sugawara's timestamps were more precise for easier-to-detect moments; the three teams tied on Recall1@0.7.Source: §5.2 · pp. 3–4
  • Synthetic-only validation sometimes improved scores while worsening real-data performance, so real development-testing validation proved essential.Some teams used augmentation during pre-training to better exploit synthetic data.Source: §5.2 · p. 4
  • Development-testing and evaluation scores showed very strong correlation, but widening gaps between Recall1@0.7 and average mAP indicate that predicting multiple intervals remains challenging.The discussion also notes widening gaps between Recall1@0.7 and Recall1@0.5 as scores increase, suggesting room to improve moment-width prediction.Source: §5.2 · p. 4

6. CONCLUSION

The paper presents DCASE 2026 Task 6, the first DCASE challenge for retrieving text-queried moments from minutes-long recordings, and reports results from its datasets, baseline, and evaluation protocol. The three best systems substantially outperformed the baseline, while the gap between stricter and looser IoU metrics suggests room for improvement.

6. CONCLUSION

  • The three best systems achieved Recall1@0.7 of 48.59%, roughly 3.5 times the baseline score.Source: §6 · p. 4
  • The paper presents the first DCASE task for retrieving text-queried temporal moments from minutes-long recordings and provides datasets, a baseline model, and an evaluation protocol.Source: §6 · p. 4
  • All top-performing systems enhanced feature extractors and moment-detection networks, while the three top-ranked systems also used score calibration or ensembling across feature temporal resolutions.Source: §6 · p. 4
  • The gap between Recall1@0.5 and Recall1@0.7 suggests further improvement is possible, and future work will integrate and refine these approaches into a simple, higher-performing model.Source: §6 · p. 4
Loading 2609.12484v2…