PaperlayerView exact revision on arXiv

Source-linked AI summary

Segmental Posterior Decoding for Audio Moment Retrieval

Seungdeok Choi, Seongmin Choi, Inhan Choi, Junho Kim, Jeong-gyu Ban, Yong-Hwa Park

arXiv:2609.16495v1eess.AS

TL;DR

Audio moment retrieval systems often score fixed-slot proposals without normalizing over competing explanations of the full timeline. This paper uses globally normalized segment marginals to score candidate moments; on CASTELLA, the approach consistently outperforms proposal-wise confidence scoring on the same network.

  • Problem

    Existing AMR scores are proposal-level rather than globally normalized, so they do not explicitly quantify a candidate moment’s plausibility relative to competing timeline explanations.

    Source: §1 · p. 1

  • Method

    Segmental posterior decoding scores candidate moments by exact segment marginal posteriors under a globally normalized model, with training also treating spans and adjacent subdivisions as competing segmentation hypotheses.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    On CASTELLA, segment marginal decoding consistently outperforms proposal-wise confidence scoring on the same network.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The main gain arises from structured marginal decoding, with a larger advantage for queries with multiple relevant moments and additional improvement from expanded-space normalization.

    Source: §5 · p. 4

Abstract

from arXiv · show

Audio moment retrieval (AMR) identifies temporal segments in long recordings that best match a free-form text query. Existing systems largely rely on fixed-slot DETR decoders that assign proposal-level confidence scores without explicitly normalizing over competing explanations of the full timeline. We propose segmental posterior decoding, which defines a globally normalized distribution over temporal segmentations and scores each candidate moment by its exact segment marginal posterior computed through forward-backward inference. We further expand the training segmentation space by treating a foreground span and its adjacent subdivisions as distinct hypotheses, thereby increasing competition among alternative segmentations. On CASTELLA, our method achieves 41.15% R1@0.7 and 34.68% mAP, outperforming the same network decoded with DETR slot confidence by 10.91 and 9.20 percentage points, respectively.

1. INTRODUCTION

AMR systems typically score fixed DETR proposals locally, without explicitly comparing a candidate moment with competing timeline explanations. Segmental posterior decoding scores moments by their marginal across globally normalized segmentations and consistently outperforms proposal-wise confidence scoring on the same network on CASTELLA.

  • Existing AMR systems largely use fixed-slot DETR proposals with proposal-level confidence that does not explicitly measure relative plausibility among competing timeline explanations.Source: §1 · p. 1
  • Segmental posterior decoding ranks each candidate by its exact marginal posterior across possible segmentations, computed through forward-backward inference.The global normalization makes the score reflect preference relative to alternative timeline explanations.Source: §1 · p. 1
  • On CASTELLA, segment marginal decoding consistently outperforms proposal-wise confidence scoring on the same network.Source: §1 · p. 1

2. PROPOSED METHOD

The method replaces inference-time DETR proposal confidence with a segmental CRF decoder that ranks candidate moments by exact segment marginal posteriors. Training expands the segmentation space to include adjacent foreground subdivisions and combines NLL with an IoU-risk objective.

2. PROPOSED METHOD

  • At inference, the DETR decoder is discarded and candidate moments are ranked by segment marginal posteriors, followed by temporal NMS.The segmental decoder shares the audio memory with the DETR branch during training.Source: §2 · p. 1

2.1. Segmental CRF Formulation

  • Foreground segments represent query-relevant moments, while uncovered frames are background; each foreground potential measures unnormalized log-compatibility from its start and end boundaries and a learned bias.The audio memory is already query-conditioned, so the boundary-based potential remains query-conditioned despite having no explicit span-interior term.Source: §2.1 · pp. 1–2

2.3. Training Objective

  • During training, a foreground span and its two adjacent subdivisions are distinct hypotheses, increasing competition with the ground-truth segmentation under global normalization.NLL favors the ground-truth segmentation, while IoU-risk encourages higher segment marginals for temporally accurate candidates; both are combined with the original DETR loss.Source: §2.1 · p. 2, §2.3 · pp. 2–3

2.2. Segment Marginal Posterior Decoding

  • Semi-Markov forward-backward inference computes exact segment marginal posteriors, which serve directly as retrieval scores; greedy temporal NMS removes redundant top candidates.The partition function and segment marginals are computed over valid segmentations.Source: §2.2 · p. 2

3. EXPERIMENTAL SETTINGS

The experiments use two-stage training followed by evaluation on CASTELLA and zero-shot transfer evaluation on UnAV-100.

3. EXPERIMENTAL SETTINGS

  • Evaluation uses the CASTELLA dev-test split with 1,347 queries and UnAV-100 for zero-shot transfer.Training pre-trains on Clotho-Moment and fine-tunes on CASTELLA, following the two-stage DCASE 2026 Task 6 protocol.Source: §3 · p. 3
  • Main results are averaged over three seeds and reported as mean ± standard deviation.Marginal decoding retains the top 10 after drawing 50 candidates and applying temporal NMS at IoU 0.7.Source: §3 · p. 3

4. RESULTS & DISCUSSIONS

Segmental posterior decoding outperforms proposal-wise scoring on CASTELLA, retains an advantage on UnAV-100, and benefits from structured marginalization and an expanded training segmentation space. Its R1@0.7 advantage grows as queries contain more ground-truth moments.

4.1. Overall Performance and Cross-Dataset Transfer

  • 8.02 points in R1@0.7 and 8.16 points in mAP separate segmental posterior decoding from VFL on CASTELLA.The comparison also reports that the method outperforms QFL and VFL, which incorporate localization quality into proposal confidence.Source: §4.1 · p. 3
  • 41.15 R1@0.7 places the method at the level of the strongest single-model DCASE 2026 Task 6 submissions, versus 45.14 for the overall best result.The overall best result additionally uses model ensembling and LLM-based refinement.Source: §4.1 · p. 3
  • On UnAV-100, segmental posterior decoding retains a clear advantage across all metrics without additional training.QFL and VFL no longer improve over the DETR baseline on this dataset.Source: §4.1 · p. 3

4.2. Ablation on Segmental Decoding

  • Using the same trained checkpoints, segment marginal decoding outperforms DETR slot confidence, local-potential ranking, and Viterbi decoding in both R1 and mAP.Marginals aggregate each candidate's probability over compatible segmentations, accounting for competing timeline explanations.Source: §4.2 · pp. 3–4, Table 2 · p. 4
  • Adding span-mean audio-text content gives no meaningful improvement despite nearly tripling active segmental-head parameters from 67k to 199k.Adding either duration prior further degrades localization, particularly for long moments.Source: §4.2 · p. 4

4.3. Training Objective and Expanded Segmentation Space

  • Allowing adjacent foreground segments lowers posterior entropy from 2.02 to 1.79 and increases marginal mass assigned to ground-truth and peak segments across all three seeds.Disallowing adjacent foreground segments degrades both R1 and mAP; removing the IoU-risk also substantially degrades localization, particularly for long moments.Source: §4.3 · p. 4

4.4. Effect of the Number of Moments

  • As ground-truth moments per query increase, the R1@0.7 advantage of segment marginal decoding grows while mAP decreases for both methods.Both methods retain only ten predictions; segment marginal decoding remains robust in the high-count regime and maintains an advantage across moment-count groups.Source: Figure 2 · p. 4, §4.4 · p. 4

5. CONCLUSION

The conclusion presents segmental posterior decoding as a structured alternative to proposal-level confidence scoring, with its main gain coming from marginal decoding.

5. CONCLUSION

  • Segmental posterior decoding replaces proposal-level confidence with exact segment marginal posteriors under a globally normalized segmental model.The conclusion attributes the main gain to structured marginal decoding, with an increasing advantage for queries containing multiple relevant moments; competitive normalization over expanded segmentations provides an additional improvement.Source: §5 · p. 4

7. COMPLIANCE WITH ETHICAL STANDARDS

The study used publicly available datasets and did not directly experiment on human or animal subjects.

7. COMPLIANCE WITH ETHICAL STANDARDS

  • No additional ethical approval was required because the research used publicly available datasets and involved no direct human or animal experimentation.Source: §7 · p. 5
Loading 2609.16495v1…