PaperlayerView exact revision on arXiv

Source-linked AI summary

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

Luoyi Sun, Xiao Zhou, Zeqian Li, Ya Zhang, Yanfeng Wang, Weidi Xie

arXiv:2604.13023v2cs.SDcs.MM

TL;DR

Large audio-language models remain unreliable at precise temporal grounding because training annotations are coarse and existing benchmarks underrepresent short events in complex audio. SpotSound addresses this with timestamp-interleaved modeling, hallucination-aware training, and the SpotSound-Bench benchmark. Across multiple benchmarks, it achieves state-of-the-art or highly competitive grounding while maintaining strong sound event detection performance.

  • Problem

    Large audio-language models lack reliable fine-grained temporal grounding because common training data use clip-level supervision and benchmarks inadequately test short events in complex audio.

  • Method

    SpotSound interleaves timestamp tokens with audio tokens and trains with positive and negative queries to localize events while suppressing timestamps for absent events.

  • Results

    SpotSound delivers state-of-the-art or highly competitive temporal grounding across multiple benchmarks while retaining strong sound event detection performance.

  • Takeaways & Limitations

    SpotSound narrows the gap between coarse audio understanding and fine-grained temporal reasoning under realistic short-event grounding conditions.

  • Takeaways & Limitations

    SpotSound remains less precise and less generalizable on short-window transient sounds, and its localization accuracy is bounded by temporal annotation granularity and quality.

Abstract

from arXiv · show

Large Audio-Language Models (ALMs) have recently demonstrated remarkable capabilities in holistic audio understanding, yet they remain unreliable for temporal grounding, i.e., the task of pinpointing exactly when an event occurs within long-form audio. This limitation stems from two factors: training data dominated by clip-level supervision lacking precise timestamps, and benchmarks that fail to simulate real-world scenarios where short events are obscured by dense background sounds. In this paper, we introduce SpotSound, an audio language model designed for grounding audio events. SpotSound incorporates a novel training objective, specifically designed to suppress hallucinated timestamps for events absent from the input. Additionally, we present SpotSound-Bench, a challenging temporal grounding benchmark where target events occupy less than ~10% of each clip, creating a rigorous 'needle-in-a-haystack' evaluation. Experiments demonstrate that SpotSound achieves state-of-the-art results on temporal grounding benchmarks while maintaining robust performance across general downstream audio-language tasks. Code, models and benchmark are released on https://loiesun.github.io/spotsound/

1 Introduction

SpotSound addresses unreliable temporal grounding in large audio-language models, where precise event timing is needed but training data and benchmarks often lack fine-grained, realistic supervision. It combines timestamp-aware modeling, hallucination suppression, and challenging evaluation to improve grounding while retaining broad audio-language capability.

  • Existing ALMs handle holistic audio tasks well but struggle to precisely localize events in continuous audio streams.This limits applications such as security surveillance and media forensics, where timing matters alongside event classification.
  • Clip-level annotations teach models to associate events with entire recordings, while existing benchmarks often contain long, isolated sounds that are easy to locate.Real-world audio instead embeds short events within continuous background noise.
  • SpotSound interleaves timestamp tokens with audio embeddings to support fine-grained boundary localization for open-vocabulary queries.Its training data use positive and negative queries, requiring the model to distinguish present events from absent ones and suppress hallucinated timestamps.
  • SpotSound trains on 77.6k temporally aware instruction-tuning samples, combining 10k curated samples with 67.6k samples from existing audio-language datasets.
  • SpotSound-Bench provides a challenging benchmark for long-duration, realistic temporal reasoning, while experiments report state-of-the-art performance across multiple grounding benchmarks and strong SED accuracy.

2 Methods

SpotSound formulates temporal grounding as staged existence detection followed by interval localization, using timestamp-interleaved audio-language sequences. Its architecture and training pipeline combine timestamp tokens, language queries, autoregressive target prediction, and synthetic foreground-background mixtures.

  • Problem formulation: The model first predicts whether a queried event exists, then localizes all matching intervals when the answer is yes.The existence output is binary, while grounding outputs start and end timestamps for each matching interval.
  • Problem formulation: SpotSound localizes time spans in recordings that correspond to natural-language queries using large audio-language models.The study uses Qwen2-Audio and Audio Flamingo 3 as representative backbone models.
  • Timestamp-interleaved sequence: SpotSound constructs an interleaved sequence of timestamp tokens and audio tokens, followed by an instruction and language query, for LLM prediction.The architecture feeds this sequence to the LLM to generate either an existence judgment or temporal boundaries.
  • Timestamp-interleaved sequence: The default timestamp granularity is 1 second, with each textual timestamp token placed immediately before its corresponding audio frame features.The LLM reads inserted timestamp tokens to generate boundaries instead of decoding dense positional encodings.
  • Dataset generation: The dataset pipeline generates foreground captions with LLMs, trims foreground clips, and mixes foreground and background sounds while preserving insertion timestamps as ground truth.
  • Training strategy: Training minimizes autoregressive negative log-likelihood over target tokens, using binary targets for existence instructions and interval targets for grounding instructions.

3 Training Dataset and Benchmark

SpotSound’s training data combines diverse temporally aligned sources with synthetic long-form mixtures and explicit negative queries. SpotSound-Bench evaluates grounding when short target events occupy a small fraction of realistic audio, while existing benchmarks cover different temporal regimes.

  • Training Dataset: 77.6k samples span varied audio durations and query formats, combining publicly available datasets with newly generated densely annotated data.
  • Training Dataset: The unified training corpus combines temporal grounding datasets with densely time-stamped classification corpora under a standardized textual-query and (start, end) timestamp format.Sources include AudioGrounding, Clotho-Moment, UnAV-100, and AudioSet Strong Label.
  • Training Dataset: The synthetic long-form dataset places trimmed foreground events randomly within 40–60-second Walking Tours background ambience and varies foreground and background gains.Foreground gain is jittered by ±5 dB, while background gain is fixed at −10 ± 5 dB relative to the foreground.
  • Training Dataset: Negative-query training pairs absent events with each clip and uses presence and localization questions to improve hallucination robustness.Negative queries are absent from annotations and share no lexical overlap with the positive query.
  • Benchmark: Existing benchmarks span short-form transient-event localization and long-form sustained-span localization across AudioGrounding, Clotho-Moment, and the UnAV-100 subset.
  • Benchmark: Existing benchmarks have average target-window coverage of 26% on AudioGrounding, 33% on Clotho-Moment, and 28% on the UnAV-100 subset.The paper describes these higher coverage ratios as narrowing the search space and simplifying grounding.
  • Benchmark: SpotSound-Bench contains 400 audio-query-timestamp triplets with 54.2-second average clips, 3.9-second target events, and 7.2% temporal density.Its long, unstructured recordings create a large search space dominated by background content; audio streams and timestamp annotations are released for reproducible evaluation.

4 Experiments

Experiments evaluate SpotSound across temporal grounding, hallucination resistance, joint existence-and-localization assessment, and ablations. SpotSound achieves strong cross-benchmark performance, with timestamp interleaving providing the main temporal-resolution improvement while timestamp granularity trades accuracy against latency.

  • 4.1 Audio Temporal Grounding: Task-specific models generalize poorly across distributions, while existing ALMs retain semantic understanding but struggle with precise temporal localization.WTATG reaches 51.4 mIoU on AudioGrounding but 9.1 on Clotho-Moment; TimeAudio reaches 67.4 on AudioGrounding but only 7.6 on SpotSound-Bench.
  • 4.1 Audio Temporal Grounding: SpotSound-A achieves 91.7% predicted IoU in a qualitative SpotSound-Bench example, whereas baselines show semantic misalignment, trivial intervals, or autoregressive hallucinations.The comparison includes Qwen2-Audio, Kimi-Audio, TimeAudio, and Audio Flamingo 3.
  • 4.2 Hallucination for Negative Samples: SpotSound models improve average existence accuracy over Audio Flamingo 3 by +18.8% on Clotho-Moment and +8.1% on AudioGrounding.The evaluation combines positive and negative queries to test whether the queried event is present.
  • 4.3 Two-stage Joint Assessment: In the two-stage joint assessment, large ALMs underperform because of hallucinated non-existent events and weak temporal grounding, while SpotSound remains highly competitive.TimeAudio cannot complete the evaluation because of hallucinations.
  • 4.5 Ablation Study: Interleaved absolute timestamps produce larger gains than removing the 30-second encoder limitation, while finer timestamp granularity improves some benchmarks at higher inference latency.The study sets granularity to 1 second to balance benchmark performance and computational efficiency; LoRA performs best at r=8 and α=16.

5 Related Work

Audio-language research has progressed toward unified audio understanding, but temporal localization remains weaker, especially for environmental sounds and long-form recordings. Audio temporal understanding addresses this gap through grounding language queries in specific audio segments, while video methods face distinct long-video challenges.

  • ALMs extend audio understanding beyond clip-level tasks, yet precise temporal event localization remains an unresolved capability.
  • Unified audio-language models support versatile audio processing and reasoning but remain weaker on environmental-sound perception and event localization.
  • Audio temporal understanding aligns language queries with specific audio segments, progressing from fully supervised grounding to weak supervision, long-form retrieval, and high-resolution temporal control.
  • Video temporal grounding methods span short and long videos, but sparse relevant moments and differences between temporal regimes hinder unified long-video models.

6 Conclusion

SpotSound combines timestamp-interleaved alignment, hallucination-mitigating training, and a realistic benchmark for precise grounding of short acoustic events. Across multiple benchmarks, it reports state-of-the-art or highly competitive temporal grounding while retaining strong sound event detection results.

  • SpotSound combines timestamp-interleaved alignment with training that explicitly mitigates hallucinations to localize short acoustic events in continuous audio.
  • SpotSound-Bench evaluates temporal acuity using short-window events embedded in complex scenes under a realistic needle-in-a-haystack setting.
  • Across multiple benchmarks, SpotSound achieves state-of-the-art or highly competitive temporal grounding performance while maintaining strong sound event detection results.
  • The paper provides supplementary dataset statistics, experiments, implementation details, qualitative results, and limitation discussions in the appendix.

A.1 Dataset Statistics

The synthetic dataset samples 10,000 audio-visual examples from VGGSound and ASSL while preserving source class priors. SpotSound-Bench instead emphasizes roughly 60-second untrimmed recordings containing highly localized 0–10-second target events, creating a sparse temporal search problem.

  • Synthetic Dataset: 10,000 synthetic audio-visual samples are drawn evenly from VGGSound and ASSL, with sampling intended to preserve the source distributions’ class priors.The dataset contains 5,000 instances from each source.
  • Synthetic Dataset: ASSL contributes a skew toward high-frequency anthropogenic classes, whereas VGGSound contributes a comparatively uniform semantic distribution.
  • SpotSound-Bench: SpotSound-Bench predominantly contains untrimmed videos centered around 60 seconds, while most target event windows last 0–10 seconds.
  • SpotSound-Bench: The disparity between global recording duration and local event duration creates a needle-in-a-haystack benchmark requiring long-range context modeling and fine-grained grounding.
  • The appendix includes additional dataset and benchmark statistics alongside experiments, implementation details, qualitative results, and limitation discussions.

B.1 Ablation Study of SpotSound-Q

SpotSound-Q ablations examine timestamp granularity, training-data composition, trainable-parameter capacity, latency, and robustness. The reported configuration uses 1-second timestamps, hybrid ASSL-plus-synthetic training, and LoRA, while finer temporal resolution increases inference cost and paraphrase tests show reliable localization.

  • Ablation Factors: SpotSound-Q ablations evaluate timestamp granularity, ASSL and synthetic-data integration, and trainable-parameter capacity using mIoU.
  • Best Configuration: SpotSound-Q reaches peak performance across four benchmarks with 1-second timestamp granularity, a 5k ASSL plus 10k synthetic corpus, and LoRA with r=8 and α=16.
  • Efficiency: Finer timestamp granularity increases inference latency because it requires autoregressive generation of more textual timestamp tokens.
  • Robustness: Robustness testing perturbs event positions and paraphrases queries to assess sensitivity to temporal distribution bias and prompt formulation.
  • Robustness: Across synonymous and interrogative query reformulations, the method maintains accurate temporal localization, indicating alignment beyond fixed prompt templates.

B.4 Hallucination for Negative Samples

The supplementary evaluation examines hallucinations on absent sound events using SpotSound-Bench and an UnAV-100 subset. SpotSound achieves state-of-the-art performance across all four reported metrics on both benchmarks.

  • The evaluation measures positive and negative query accuracy, with “/” indicating hallucination on non-existent events.The table distinguishes prediction accuracy for positive and negative queries and marks inability to determine event presence as hallucination.
  • SpotSound establishes a new state-of-the-art across all four evaluation metrics on both benchmarks.TimeAudio frequently confuses event presence and absence, while other contemporary ALMs show only marginal robustness against false positives.

C Additional Implementation Details

The implementation uses a two-stage verification-and-localization pipeline, synthetic captioned training data, and qualitative analyses across grounding benchmarks. SpotSound first checks whether a queried event exists, then predicts its temporal window.

  • Inference pipeline: SpotSound first verifies whether the queried sound event occurs, then localizes its start and end timestamps only after positive detection.This decoupled pipeline is designed to avoid directly predicting timestamps in cases that may induce hallucination.
  • Synthetic data: The synthetic training set contains 10,000 samples built from ASSL and VGGSound foreground events.Qwen2-Audio generates detailed acoustic captions for VGGSound, while DeepSeek-v3 converts ASSL labels and timestamps into chronological narratives.
  • Synthetic data: DeepSeek-v3 transforms ASSL event labels and temporal boundaries into cohesive, chronologically accurate 10–15-word audio captions.The transformation supplies the labels and their start and end times to the captioning model.
  • Qualitative analysis: SpotSound-Bench qualitative analysis reports robust grounding for sparse events but occasional missed occurrences in multi-instance clips.The analysis presents success and failure cases from SpotSound-A across different benchmarks.
  • Qualitative analysis: On AudioGrounding, SpotSound shows robust overall localization but sometimes produces slight fine-grained boundary misalignments.A reported example predicts 9.65s instead of the 9.00s ground-truth endpoint, which disproportionately affects mIoU.

D.3 Qualitative Results for UnAV-100 subset

The UnAV-100 subset contains extended-duration events whose annotations may merge repeated occurrences across silent intervals. SpotSound can detect these gaps precisely, but that precision can reduce automated scores against coarse ground truth.

  • UnAV-100 annotations frequently merge consecutive occurrences of the same sound class into one continuous temporal window.The merged window can absorb silent intervals between repeated occurrences.
  • SpotSound’s fine-grained predictions detect silent gaps that the benchmark’s coarse annotations absorb.This creates a structural mismatch between acoustically precise predictions and the reference temporal windows.
  • The annotation mismatch artificially penalizes SpotSound in automated evaluation metrics despite acoustically more precise predictions.Figure S5 presents success and failure cases for the UnAV-100 subset.

E Limitation and Future Work

The paper identifies limitations in transient-event grounding, annotation quality, and complex-scene understanding. Future work targets denser data, overlapping events, and complete multi-instance localization for real-world audio.

  • Short-window, highly transient sounds remain a bottleneck for SpotSound’s temporal precision and generalization.The current framework performs especially well on distinct, sustained acoustic events, leaving transient benchmarks as a limitation.
  • Localization accuracy is bounded by the granularity and quality of temporal annotations in the training corpus.Greater robustness therefore requires larger datasets with dense, fine-grained, challenging acoustic samples.
  • Future work will address polyphonic environments with overlapping events and improve localization of every repeated-event time window.The paper frames these constraints as the next step toward robust real-world deployment.
Loading 2604.13023v2…