Source-linked AI summary
What Are You Listening to? Temporal Music Grounding for Audio-to-Text Large Language Models
Kun Fang, Ziyu Wang, Ichiro Fujinaga
TL;DR
Audio-language models can produce plausible musical responses without clear evidence that they used the audio, motivating temporal music grounding as an explicit evaluation task. The paper introduces MusicGroundingBench with controlled MIDI-to-audio alignment across note-level and structured excerpts, finding that grounding is difficult for current models but improves substantially with task-specific training.
Problem
Existing music evaluations do not directly establish whether plausible model responses are grounded in temporally localized audio evidence.
Method
The paper introduces MusicGroundingBench, using algorithmically generated MIDI rendered to audio with exact alignment across MGBench-3N and MGBench-2B.
Results
Temporal music grounding remains challenging for current audio-language models but improves substantially through task-specific training.
Takeaways & Limitations
The benchmark provides a controlled testbed for assessing whether audio-language model outputs are grounded in temporally localized musical evidence.
Abstract
from arXiv · showhide
Large audio-language models can produce fluent and musically plausible responses, yet it often remains unclear whether those responses are grounded in the audio input. We introduce temporal music grounding, a task in which a model returns one or more time spans corresponding to a queried musical note, event, or pattern. To evaluate this capability, we present MusicGroundingBench, a controlled benchmark suite built by rendering algorithmically generated piano MIDI to audio, yielding exact symbolic-to-audio alignment. The suite comprises two subsets: MGBench-3N, which evaluates note-level grounding in clips containing up to three notes, and MGBench-2B, which evaluates structured grounding and short-form music understanding in two-bar excerpts. Experiments show that temporal music grounding remains challenging for current audio-language models, whereas task-specific training yields substantial gains. We further report exploratory evidence on the relationship between grounding supervision and music understanding. These results establish MusicGroundingBench as a controlled testbed for assessing whether audio-language models ground their responses in temporally localized musical evidence.
1. INTRODUCTION
The paper frames temporal music grounding as a way to test whether audio-language models connect answers to localized musical evidence. It introduces MusicGroundingBench, a controlled benchmark spanning note-level and structured music grounding, and reports that task-specific training substantially improves performance.
- Temporal music grounding requires models to return time spans corresponding to queried musical events or patterns.
- Existing music evaluations largely focus on final text outputs, leaving temporal grounding in localized musical evidence directly untested.
- MusicGroundingBench uses algorithmically generated MIDI rendered to audio, providing exact symbolic-to-audio alignment for controlled evaluation.
- MGBench-3N evaluates note-level grounding, while MGBench-2B evaluates structured grounding and short-form music understanding in richer two-bar excerpts.
- Experiments show that temporal music grounding is difficult for current audio-language models but improves substantially with task-specific training.
2. RELATED WORK
Prior work applies grounding and multimodal language models across vision, video, audio, and music, but music grounding remains insufficiently evaluated. Existing music benchmarks usually assess final text outputs rather than fine-grained temporal localization.
- Grounding research has linked natural-language queries to spatial or temporal regions in vision, video, and emerging audio settings.
- Audio grounding settings focused on environmental sounds, speech events, or generic alignment do not directly capture music’s relational and structurally defined targets.
- Multimodal language models have been adapted to music captioning, question answering, reasoning, and instruction following.
- Existing music evaluations generally emphasize caption quality or answer correctness instead of fine-grained temporal grounding.
3. BENCHMARK CONSTRUCTION
MusicGroundingBench is a controlled benchmark suite built from algorithmically generated, MIDI-rendered piano audio with exact symbolic-to-audio alignment. Its two subsets test sparse note-level localization and richer two-bar grounding and understanding, with automatically derived questions and aligned temporal references.
- MusicGroundingBench uses synthetic piano audio rendered from symbolic MIDI so temporal references can be defined exactly from aligned note events.
- MGBench-3N: MGBench-3N contains short monophonic clips with up to three sequential notes, spanning all 88 piano keys and four basic three-note contours.
- MGBench-3N: MGBench-3N organizes grounding queries into absolute, relative, ordinal, and pattern-based families targeting note attributes, ordering, and simple relations.
- MGBench-2B: MGBench-2B extends the benchmark to monophonic two-bar excerpts with rhythmic, tonal, intervallic, harmonic, and repetition structure.
- Validation: Samples are validated for musical consistency, and excerpts longer than 10 seconds are removed before question-answer rendering.
- QA construction: Grounding questions use note-linked time spans, while understanding questions use free-form text derived from the same excerpt and its annotations.
4. EVALUATION PROTOCOL
The evaluation separates formatting compliance from localization or semantic quality. Grounding uses interval overlap and boundary metrics, while understanding uses an LLM judge that permits semantically equivalent answers.
- Grounding models return temporal spans, whereas understanding models answer concept-level questions with text.
- Grounding: Grounding instruction-following rate measures whether an output can be parsed into a valid grounding-style answer containing time spans.
- Grounding: Predicted and reference spans are greedily matched one-to-one at IoU thresholds of 0.5 and 0.7 before computing precision, recall, and F1.
- Grounding: MergedIoU compares merged predicted and reference time regions, while onset and offset MAE quantify boundary precision only over matched spans.
- Understanding: Understanding predictions are judged for instruction following and binary semantic correctness, with mean correctness reported as Global Accuracy.
5. EXPERIMENTS
Experiments compare non-LLM baselines, zero-shot audio-language models, and LoRA-adapted instruction-tuned models across note-level grounding, structured grounding, and understanding. Grounding is difficult without task-specific training, while training substantially improves performance and grounding supervision helps understanding inconsistently across backbones.
- Experimental setup: The experiments evaluate proposal-based FiLM and XAttn baselines, zero-shot Music Flamingo and Qwen2-Audio, and LoRA-adapted LLaMA-3-8B and Qwen3.5-9B-Base models.
- Attention analysis: A qualitative attention example shows broad note-boundary scanning in an earlier layer and concentration near the target span in a later layer.
- Grounding results: Temporal music grounding remains difficult for existing audio-language models, whereas task-specific training substantially improves localization performance.
- MGBench-3N: On MGBench-3N, both instruction-tuned models achieve strong localization, with LLaMA-3-8B slightly ahead in MergedIoU and F1.
- MGBench-2B: On MGBench-2B, Qwen3.5-9B-Base consistently outperforms LLaMA-3-8B, including under the stricter IoU ≥0.7 criterion.
- Benchmark comparison: MGBench-2B separates models more clearly than MGBench-3N and provides the more demanding test of grounded temporal reasoning on structured excerpts.
- Understanding results: Instruction-tuned models outperform external audio-language baselines on MGBench-2B understanding, especially where zero-shot systems struggle with tonality, chord, and pattern questions.
- Grounding supervision: Grounding supervision improves understanding for Qwen3.5-9B-Base but not global accuracy for LLaMA-3-8B, providing no uniform cross-backbone conclusion.
6. CONCLUSION
The paper establishes temporal music grounding and MusicGroundingBench as a controlled way to test whether audio-language model outputs are grounded in temporally localized musical evidence. Results indicate that grounding is difficult for current models but can be learned effectively through task-specific training, with preliminary links to music understanding.
- Temporal music grounding evaluates whether audio-to-text models can localize answer-relevant musical events in time.
- MusicGroundingBench provides a controlled benchmark for evaluating grounding in audio-language models.
- Current audio-language models find temporal music grounding challenging, but task-specific training can improve this capability substantially.
- Grounding supervision may relate to music understanding under some model and data settings.