PaperlayerView exact revision on arXiv

Source-linked AI summary

TAG-Bench: Benchmarking Temporal Audio Grounding in Large Audio Language Models

Yuhang Dai, Xin Shu, Zengxi Li, Lei Xie, Xiangang Li, Jianwei Yu

arXiv:2609.01542v2eess.AS

TL;DR

Temporal audio grounding—locating every queried event in time—remains under-evaluated in large audio language models. TAG-Bench evaluates 21 systems on a human-verified benchmark and finds that even the strongest model achieves only 31.2 mIoU and 21.5% recall at IoU≥0.7.

  • Problem

    Existing large audio language model evaluations emphasize understanding and captioning rather than explicit timestamp localization, despite its relevance to practical audio applications.

  • Method

    TAG-Bench evaluates temporal audio grounding across 1,750 human-verified query-recording pairs, eight diagnostic subsets, and 21 systems using localization and occurrence-count metrics.

  • Results

    31.2 mIoU is the highest overall score, but no model exceeds 13.2% one-to-many count accuracy and the strongest system reaches only 21.5% recall at IoU≥0.7.

  • Takeaways & Limitations

    The results show that precise localization and complete occurrence recovery remain difficult for current large audio language models.

  • Takeaways & Limitations

    Subset comparisons are confounded by source domain, query type, duration, and recurrence rate, while the longest recording is 20 minutes.

Abstract

from arXiv · show

Large audio language models (LALMs) can describe what is heard, but their ability to localize when queried content occurs remains less systematically evaluated. We present TAG-Bench, a benchmark for temporal audio grounding in which a model returns every time interval that matches a natural-language query. TAG-Bench contains 1,750 human-verified query-recording pairs covering 149.5 hours, with eight source-dependent subsets spanning query categories and audio durations from 7 s to 20 min; 22.1% of the queries have multiple ground-truth intervals. Across 21 evaluated systems, the best-performing model achieves 31.2 mIoU and is the only system above 20 mIoU on the two long subsets, yet even this top performer reaches only 21.5% recall at IoU >= 0.7. Moreover, 9 of 21 systems fall below 5 mIoU, and every model under-reports the number of occurrences on one-to-many queries, with none exceeding 13.2% count accuracy. Because responses are free-form, we report parsing-failure rate and MAE coverage: parsing failures remain in mIoU, Recall, gIoU, and count metrics as empty predictions but do not enter MAE. The results separate precise localization, occurrence enumeration, and output-format reliability within a benchmark whose cross-subset comparisons are descriptive rather than controlled estimates of query abstraction or duration. We will release the TAG-Bench data and evaluation code to support future research.

1. INTRODUCTION

TAG-Bench defines Temporal Audio Grounding as retrieving every query-matching audio interval and addresses a gap in evaluating LALMs’ localization, recurrence handling, and timestamp precision. It introduces a source-diverse benchmark and diagnostics while treating cross-subset comparisons as descriptive rather than causal.

  • Problem definition: Temporal Audio Grounding requires models to return the start and end times of every segment matching a natural-language query.Relevant segments typically occupy a median of 7% of benchmark audio, creating a needle-in-a-haystack localization problem.
  • Benchmark design: 1,750 human-verified queries span 1,750 distinct recordings totaling 149.5 hours across eight subsets covering varied query types and durations from 7 s to 20 min.The benchmark includes words, acoustic events and descriptions, semantics, and emotion.
  • Core challenges: 22.1% of queries contain multiple ground-truth intervals, requiring models to recall disjoint occurrences rather than only one event.Recurring footsteps and repeatedly discussed topics exemplify the one-to-many setting, while precise timestamp prediction remains difficult for autoregressive multimodal decoders.
  • Evaluation contributions: TAG-Bench introduces union-based IoU and count-accuracy diagnostics to distinguish recovering all occurrences from placing temporal boundaries precisely.Its unified evaluation covers 21 systems, including general-purpose LALMs, omni-modal systems, and temporal-grounding experts, while analyzing parsing coverage and model series.
  • Scope and limitations: Cross-subset contrasts are descriptive because query category, source domain, audio duration, and recurrence rate are not varied independently.The benchmark therefore does not support causal claims about abstraction or context length without matched-data studies.

2. RELATED WORK

Prior audio benchmarks emphasize question answering, reasoning, representation learning, or temporal control, while newer audio LLMs increasingly incorporate explicit temporal awareness and grounding mechanisms.

  • Audio understanding benchmarks: MMAU, MMAU-Pro, and MMAR assess multi-task audio reasoning through question answering rather than timestamp-output evaluation.Their published task descriptions emphasize question answering and reasoning.
  • Audio understanding benchmarks: AudioTime and TACOS provide temporally aligned data on clips no longer than 30 seconds, principally for temporal-control or language–audio representation research.Text-to-audio grounding was formalized on AudioCaps-style clips.
  • Time-aware audio LLMs: TimeAudio, LAT-Audio, GigaChat Audio, time prompts, and retrieval-based pipelines introduce temporal awareness through bridging, iterative local reasoning, explicit timestamps, post-training, or event-grounded retrieval.These approaches span frame-to-time bridging, global-to-local reasoning, long-recording timestamp answers, audio-side prompts, and question answering over hours of audio.

3. TAG-BENCH

TAG-Bench evaluates whether large audio language models can return every time interval matching a natural-language query, including recurring one-to-many targets. It spans eight query–recording subsets across varied audio durations and uses staged annotation with human verification and quality control.

  • Task formulation: Models output a set of time intervals covering all recording segments whose content matches a natural-language query.Ground truth likewise contains one or more intervals, with one-to-one and one-to-many queries sharing the same format.
  • Dataset composition: 1,750 queries paired with distinct recordings comprise TAG-Bench, totaling 149.5 hours of audio.The evaluation unit is a query–recording pair rather than an hour of audio.
  • Dataset composition: Eight subsets organize query–recording pairs by query category and audio length, including two long subsets with 440 query-level outcomes across 127.6 hours.Each recording contributes exactly one query, so the benchmark does not measure within-recording consistency across different queries.
  • Audio-length dimension: 25% of queries account for 85% of total audio duration in the two long subsets, which average about 17 minutes and reach 20 minutes.The other six subsets use recordings averaging 7–93 seconds.
  • One-to-many targets: 387 queries (22.1%) contain at least two ground-truth intervals, with up to 18 per query and an average of 4.02 intervals.A correct answer must recall all occurrences; recurring-target shares range from 1% for word queries to 42% for acoustic-event queries.
  • Benchmark construction: Three construction stages combine dataset sourcing, automatic timestamp and transcript pre-annotation, and human annotation with cross-validation and boundary quality control.Boundary deviations exceeding 0.5 s are re-adjudicated, and inconsistent samples are discarded.

4. EXPERIMENTAL SETUP

TAG-Bench evaluates 21 systems using standardized benchmark prompts and free-form timestamp parsing, while reporting complementary metrics that separate interval overlap, temporal displacement, and occurrence counting. The setup includes general-purpose audio LLMs, omni-modal baselines, and temporal-awareness experts, with deterministic handling of recoverable timestamps and one-to-many interval sets.

  • Systems: 21 systems are benchmarked across general-purpose audio LLM, omni-modal, and temporal-awareness expert groups.Several model families include thinking and non-thinking variants, enabling within-family comparisons.
  • Systems: Every system receives the same item-specific benchmark question through its model-native chat and audio wrapper.This standardizes the benchmark input while preserving each system’s native interaction interface.
  • Answer parsing: Timestamps are recovered from free-form responses with a deterministic cascade that normalizes multiple time formats without semantic rescue.The cascade matches interval expressions, start/end patterns, and timestamp arrays, but does not infer missing intervals from descriptions.
  • Metrics: mIoU and Recall@τ evaluate merged interval-set overlap at τ ∈{0.3, 0.5, 0.7}, while CntAcc measures exact occurrence-count agreement on one-to-many queries.Under-reporting and over-reporting rates replace the count equality condition and satisfy CntAcc + Under + Over = 100.
  • Metrics: gIoU penalizes displacement within the smallest enclosing interval and ranges from −1 to 1, assigning −1 to responses without recoverable predictions.Unlike union IoU, generalized IoU distinguishes near misses from widely displaced answers.
  • Metrics: Midpoint absolute error measures threshold-free localization bias by matching each ground-truth midpoint to its nearest predicted midpoint on one-to-many queries.The sample-level MAE averages the resulting midpoint errors across ground-truth intervals.

5. RESULTS AND ANALYSIS

TAG-Bench results show that temporal grounding remains difficult: FireRedAudio leads overall and on long-audio subsets, but systems struggle with strict localization, multiple occurrences, and recoverable free-form outputs. Rankings depend on the metric and subset, while cross-subset differences do not isolate duration or query abstraction.

  • Overall performance: 31.2 mIoU is FireRedAudio’s highest overall point estimate, yet its R@0.7 is only 21.5 and nine of 21 models score below 5 mIoU.The median target covers 7% of its recording, making strict-overlap localization particularly difficult.
  • Subset performance: 20.6/21.7 mIoU makes FireRedAudio the long-subset leader, ahead of Gemini-3.1-Pro at 12.9/7.2 and LAT-Audio at 8.6/5.5.MOSS-Audio-8B-Thinking falls to 4.8/4.1 on long audio despite leading or nearly leading short acoustic subsets.
  • Subset performance: 34.5 mIoU leads word queries for FireRedAudio, while its R@0.3-to-R@0.7 drop from 47.7 to 25.8 shows that precise word boundaries remain difficult.FireRedAudio also leads semantic, rough-semantic, and emotion grounding at 38.4, 40.6, and 25.2 mIoU.
  • Metric trade-offs: 31.2 mIoU, 3.5 gIoU, and 34.3 s parsed-only MAE identify different leaders and trade-offs rather than a single model ranking.FireRedAudio has 15.1% parsing failures and 84.9% MAE coverage; MOSS-Audio-8B-Thinking has 99.9% coverage but 923.1 s MAE.
  • Model comparisons: 8B thinking–instruct mIoU improves from 12.8 to 29.6 for MOSS-Audio, but Step-Audio-2-mini-Think scores 2.7 versus 6.3 for the non-thinking mini.The results do not support a family-independent benefit from explicit reasoning.

6. CONCLUSION

TAG-Bench introduces a human-verified benchmark for temporal audio grounding with broad audio coverage, diagnostic subsets, and count-aware evaluation. Across 21 systems, FireRedAudio leads overall mIoU but remains limited in high-IoU recall and one-to-many occurrence reporting.

  • TAG-Bench comprises 1,750 query–recording pairs spanning 149.5 hours of audio across eight source-dependent diagnostic subsets.
  • 31.2 mIoU is FireRedAudio’s highest overall result across 21 evaluated systems.It also reaches 20.6/21.7 mIoU on the two approximately 17-minute subsets.
  • 21.5 R@0.7 shows FireRedAudio’s limited high-IoU recall despite its leading overall mIoU.FireRedAudio under-reports 91.0% of one-to-many queries.

APPENDICES · A. TAG-BENCH DETAIL VISUALIZATIONS

TAG-Bench targets are short and sparse, typically occupying only a small portion of recordings. Their onsets span nearly the full audio timeline, while recording durations form a bimodal distribution separating shorter subsets from 17–20-minute recordings.

  • A. TAG-BENCH DETAIL VISUALIZATIONS: 2.7 s is the median target-segment duration, indicating that queried events are typically brief.
  • A. TAG-BENCH DETAIL VISUALIZATIONS: 1,190 s is the latest reported target onset, so targets can occur far beyond the beginning of the audio.
  • A. TAG-BENCH DETAIL VISUALIZATIONS: The target-onset range extends from the first seconds through 1,190 s, covering recordings with substantially different temporal positions.
  • A. TAG-BENCH DETAIL VISUALIZATIONS: Audio durations are bimodally distributed, separating short-to-medium subsets from 17–20-minute subsets.
  • A. TAG-BENCH DETAIL VISUALIZATIONS: 7% is the median target-to-audio coverage ratio, showing that targets are sparse within recordings.
  • A. TAG-BENCH DETAIL VISUALIZATIONS: 1,471 recordings contribute to the duration panel, which spans the full range up to 1,200 s and excludes word items.

B. SUPPLEMENTARY VISUALIZATIONS OF TEST RESULTS

Figs. 5–7 visualize model–subset localization, one-to-one versus one-to-many performance, and interval-count diagnostics. They show distinct specialization profiles, degraded mIoU on multi-interval queries, and pervasive occurrence under-reporting.

  • Fig. 5: Model–subset heatmap: FireRedAudio leads four short-audio subsets and both long-audio subsets, while MOSS thinking variants lead the two short acoustic subsets.The heatmap reports mIoU across 21 models and eight subsets; MOSS thinking models reach approximately 50–52 mIoU on short acoustic event and description queries but fall below 5 on long audio.
  • Fig. 6: One-to-one versus one-to-many: 34.1 to 21.0 is FireRedAudio’s mIoU drop from one-to-one to one-to-many queries, compared with 31.4 to 23.6 for MOSS-Audio-8B-Thinking and 27.9 to 20.9 for Gemini-3.1-Pro.Every competitive model degrades on queries with multiple ground-truth intervals; weak-system inversions can reflect intersection with larger multi-interval unions rather than correct enumeration.
  • Fig. 7: Interval-count diagnostics: 79.3–100% under-reporting dominates all 21 rows, with Step-Audio-R1 showing the largest count-accuracy segment at 13.2%.Gemini-3.1-Pro follows with 7.0% count accuracy, while FireRedAudio reaches 4.7%.
  • Fig. 7: Interval-count diagnostics: 3.67 predicted intervals per one-to-many query is close to Gemini’s 4.02 ground-truth average, but 82.1% under-reporting and 10.9% over-reporting show output volume alone is insufficient.FireRedAudio remains predominantly a single-interval responder despite leading overall localization scores.

C. COMPLETE PER-SUBSET RESULTS

Complete per-subset results report overlap, Recall, gIoU, parsed-only MAE, count accuracy, and parsing failures across eight subsets, with failed parses counted as empty predictions except for MAE. Performance leaders vary by query type, while long-audio grounding remains substantially weaker than short-subset results.

  • Evaluation protocol: Failed parses count as empty predictions for mIoU, Recall, gIoU, and count accuracy, but are excluded from MAE; MAE coverage equals 100 −Fail.Fail is diagnostic rather than ranked, and all metrics except MAE are percentages.
  • Short subsets: 34.5 mIoU and 25.8 R@0.7 make FireRedAudio the word-subset leader, while MOSS thinking models lead acoustic events at 51.9 mIoU and 40.9 R@0.7.On words, MOSS instruct variants exceed thinking variants at R@0.5 despite lower mIoU; FireRedAudio’s acoustic-event 40.9% Fail limits MAE coverage to 59.1%.
  • Long-audio subsets: 20.6 mIoU makes FireRedAudio the only positive-gIoU model on long semantic audio, with Gemini-3.1-Pro at 12.9 and LAT-Audio at 8.6.FireRedAudio’s 33.7 s parsed-only MAE has 99.4% coverage, and other general-purpose systems remain at or below 4.8 mIoU.
  • Long-audio subsets: 21.7 mIoU and 17.3 R@0.7 lead long acoustic-description audio for FireRedAudio, despite 185.4 s MAE covering only 79.2% of samples.Gemini-3.1-Pro is second in mIoU at 7.2, while LAT-Audio is second at R@0.7 with 3.8; FireRedAudio’s Fail is 20.8%.
Loading 2609.01542v2…