PaperlayerView exact revision on arXiv

Source-linked AI summary

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich, Ramani Duraiswami, Dinesh Manocha

arXiv:2608.29999v1cs.SDcs.AIcs.LG

TL;DR

Most LALMs describe audio without assigning timestamps, limiting a capability needed for temporally structured understanding. TEMPO unifies five timestamping tasks through SFT innovations and GRPO refinement, and substantially outperforms strong timestamp-trained baselines while SFT supplies most gains.

  • Problem

    Most LALMs treat audio understanding as a clip-level task and lack timestamping for identified events, speakers, and sounds.

  • Method

    TEMPO uses a unified decoder with atomic timestamp tokens, a time-aware projector, distance-aware Gaussian loss, synthetic-to-real SFT, and GRPO with verifiable temporal rewards.

  • Results

    TEMPO substantially outperforms Audio Flamingo Next and Qwen3-Omni across five tasks, with the largest gains on multi-speaker ASR and diarization.

  • Takeaways & Limitations

    Ablations indicate that careful SFT design is the primary lever for adding timestamping capabilities, while GRPO provides consistent but moderate refinements.

  • Takeaways & Limitations

    TEMPO does not handle overlapping multi-domain events requiring joint segmentation and labeling of concurrent streams.

Abstract

from arXiv · show

Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.

1 Introduction

TEMPO addresses the lack of timestamping in clip-level LALMs with a unified five-task model, an SFT recipe, and GRPO refinement. It substantially outperforms strong timestamp-trained baselines, with the largest gains on speech.

  • Motivation: Current LALMs generally describe audio at the clip level, discarding temporal structure needed to associate events with precise intervals.Timestamping supports applications including meeting transcription and music analysis.
  • Results: TEMPO substantially outperforms Audio Flamingo Next and Qwen3-Omni, including reducing multi-speaker ASR WER from 0.70 to 0.44.Diarization mIoU also improves from 0.44 to 0.71 over Qwen3-Omni, with consistent gains across audio and music tasks.
  • Unified model: TEMPO unifies multi-speaker ASR, speaker diarization, audio temporal grounding, dense audio captioning, and timestamped music captioning in one decoder.All five tasks segment audio into intervals and assign textual labels.
  • Post-training: Its SFT recipe combines 0.1 s atomic timestamp tokens, sinusoidal wall-clock encodings in a time-aware projector, and a distance-aware Gaussian loss.The recipe is trained without prior temporal supervision and follows a synthetic-to-real curriculum.
  • Post-training: GRPO provides a unified reinforcement-learning refinement using task-specific verifiable temporal rewards, yielding modest additional gains on speech and sound tasks.It starts from the SFT checkpoint and directly optimizes evaluation objectives.
  • Data and evaluation: 119K training examples and 10K evaluation examples cover all five timestamping tasks under a unified output format.The training corpus combines 51K synthetic and 68K real-world examples drawn from established corpora.

2 Related Work

Prior LALMs show strong clip-level capabilities but persistent weaknesses in temporal grounding and fine-grained temporal reasoning. TEMPO extends unified timestamping across speech, sound, and music, including structured timestamped chord progressions and temporal GRPO rewards.

  • Large Audio-Language Models and Timestamping: Recent LALMs perform strongly on captioning, question answering, and open-ended reasoning, but temporal grounding remains a persistent weakness.Studies also report difficulty with event ordering, duration, and onset/offset estimation.
  • Large Audio-Language Models and Timestamping: TAC and TimeAudio use unified models for timestamping but remain restricted to limited tasks.TEMPO is presented as a unified alternative spanning five timestamping tasks.
  • Speech Timestamping: Unlike speech-only diarization and ASR systems, TEMPO jointly predicts who, what, and when across speech and non-speech tasks.The framework is trained across non-speech tasks as well as speech tasks.
  • Sound: Event Detection, Temporal Grounding and Dense Captioning: Clip-level benchmarks such as AudioCaps and Clotho can compress distinct events into global summaries, whereas dense captioning and audio grounding provide temporal localization.These tasks respectively segment and describe events or localize text queries to intervals.
  • Music Timestamping: No existing audio-language model is reported to produce structured, timestamped chord progressions from audio before TEMPO.Earlier music systems either specialize in chord recognition or generate theory-aware captions without structured timestamped chord labels.
  • Reinforcement Learning for Audio-Language Models: TEMPO applies GRPO with temporally verifiable rewards across a unified multi-task LALM, extending emerging reinforcement-learning work in audio.The cited prior audio efforts apply GRPO to audio reasoning or audio question answering rather than unified temporal timestamping.

3 Tasks and Datasets

TEMPO is trained on five speech, sound, and music timestamping tasks using real and synthetic data under a unified output format. The corpus combines established task-specific datasets with a synthetic calibration stage.

  • Tasks and data: The training setup spans five timestamping tasks across speech, sound, and music using real and synthetic data.Examples use a unified output format, with task coverage illustrated in Table 1.
  • Speech: 16,045 ASR and 16,053 diarization samples come from AMI, ICSI, and Switchboard conversational corpora with speaker-labeled transcripts and precise timestamps.These data support joint transcript-timing prediction and speaker-turn segmentation.
  • Sound: 13,500 audio temporal-grounding samples are derived from AudioSet Strong by converting over 500 sound classes into paraphrased natural-language queries.The source provides onset and offset times for the sound classes.
  • Sound: 9,858 dense audio-captioning examples come from TACOS, which pairs recordings with captions and onset/offset timestamps.The dataset directly supplies temporally localized descriptions.
  • Music: 13,000 timestamped music-captioning samples are derived from Slakh2100 using aligned MIDI annotations for chords, tempo, and instruments.Slakh2100 contains 2,100 multi-track songs synthesized from MIDI.
  • Synthetic curriculum: The synthetic Stage-1 corpus contains 51,512 examples across ASR, diarization, audio grounding, dense captioning, and music tasks.Timestamp tokens are balanced uniformly across possible values, with 10,000 Slakh2100 samples added to ensure task exposure.

4 Method

TEMPO combines a unified timestamping model with a synthetic-to-real SFT curriculum and GRPO refinement. Its method addresses timestamp tokenization, wall-clock alignment, and the mismatch between timestamp errors and evaluation metrics.

  • Unified Architecture: TEMPO is built on Audio Flamingo 3 and uses task-specific prompt tags to handle five timestamping tasks with one decoder.The audio encoder remains frozen during training.
  • Timestamp Representation: Atomic timestamp tokens represent each 0.1 s timestamp as one categorical decision instead of fragmented BPE pieces.The vocabulary covers approximately 600 timestamp candidates.
  • Time-Aware Projector: The time-aware projector adds fixed sinusoidal wall-clock encodings to frame-indexed audio features before the MLP.Encodings span periods from 0.08 s to 60 s, covering short acoustic events and longer musical structure.
  • Distance-Aware Loss: A distance-aware Gaussian loss assigns greater supervision to timestamps near the ground truth, preserving ordinal structure relevant to tIoU.The training objective combines cross-entropy with the time-aware loss using λtime = 0.5 and σt = 0.3 s.
  • Synthetic-to-Real Curriculum: The two-stage curriculum trains first on 51,512 synthetic examples and then on 32,726 real-world examples before reinforcement learning.Synthetic training establishes temporal calibration before real-data fine-tuning.
  • Reinforcement Learning: GRPO starts from the SFT checkpoint and uses verifiable, metric-aligned rewards with format gating across the timestamping tasks.Rewards are bounded in [0, 1], while task-specific terms combine localization, boundary, transcription, captioning, or music metrics.

5 Experiments and Results

TEMPO is evaluated across five timestamping tasks using held-out datasets and task-specific metrics, against zero-shot LALMs, frontier models, prior timestamping systems, specialists, and pipeline ablations. Full SFT provides the main gains over LALM baselines, while multi-task RL adds generally modest refinements.

  • Evaluation setup: TEMPO’s evaluation spans multi-speaker ASR, speaker diarization, audio temporal grounding, dense audio captioning, and timestamped music captioning.The benchmark uses held-out test sets and metrics covering textual accuracy, temporal overlap, and boundary precision.
  • Compared methods: The comparison includes zero-shot LALMs, Gemini frontier models, TimeAudio, specialized systems, and SFT/RL ablations.Naive SFT, curriculum-stage checkpoints, and single-task versus multi-task RL isolate training and optimization effects.
  • Speech results: WER drops from 69.7% to 43.5% for multi-speaker ASR, while mIoU rises from 44.4% to 71.1% for diarization over Qwen3-Omni after multi-task RL.Full SFT reaches 44.7% ASR WER and 70.5% diarization mIoU before RL refinement.
  • Sound results: TEMPO reaches 59.3% dense-captioning eF1 and 68.5% mIoU, exceeding TimeAudio by 14.9 and 3.7 absolute points before RL.On audio grounding, multi-task RL reaches 46.5% F1 and 49.4% mIoU, exceeding TimeAudio by 6.1 and 2.2 absolute points under SFT.
  • Music results: On music captioning, Stage 1+2 reaches 19.6% chord-root accuracy and 32.7% chord-quality accuracy, whereas zero-shot LALMs remain below 1% chord F1.The provided results identify substantial gaps on chord-related metrics between trained TEMPO and zero-shot baselines.
  • Ablations: Stage 2 real-data fine-tuning sharply improves Stage 1 synthetic-only performance, while Stage 1+2 usually exceeds Stage 2 alone.Grounding F1 rises from 34.3% to 46.2%, and ASR WER falls from 94.0% to 44.7%; multi-task RL improves four of five tasks but not music.

6 Baseline format-failure analysis

The analysis tests whether zero-shot LALM gaps could arise from output-format failures and reports the remaining qualitative error patterns. Performance gaps persist after tolerant parsing, while TEMPO’s residual errors center on segment merging and boundary misalignment.

  • Format-failure analysis: Tolerant, model-specific parsers preserve substantial performance gaps after removing baseline format failures.The analysis attributes the dominant remaining errors to incorrect content and temporal localization rather than parsing.
  • Qualitative errors: TEMPO’s residual errors are dominated by segment merging and boundary misalignment even when semantic content is correct.

7 Transfer to real world music

The music component is trained on synthesized MIDI-derived recordings and additionally evaluated on real piano recordings from MAESTRO to test transfer beyond synthesized data.

  • Data and transfer: Music training uses Slakh2100, whose recordings are synthesized from human-annotated MIDI files, while MAESTRO supplies real-world piano recordings for evaluation.The evaluation randomly selects 1,000 MAESTRO samples using the dataset’s test-set processing pipeline.

8 Conclusion

TEMPO unifies five timestamped generation tasks in one decoder through specialized SFT design, synthetic-to-real training, and GRPO refinement. It substantially outperforms key LALM baselines, with the largest gains in ASR and diarization, while SFT remains the primary contributor.

  • Unified model: TEMPO performs multi-speaker ASR, speaker diarization, audio temporal grounding, dense audio captioning, and timestamped music captioning in a single decoder.
  • Training approach: Its SFT recipe combines atomic timestamp tokens, a time-aware projector, a distance-aware Gaussian loss, and a synthetic-to-real curriculum, followed by GRPO refinement.
  • Main results: WER falls from 69.7% to 43.5% in multi-speaker ASR and diarization mIoU rises from 44.4% to 71.1%.The comparison is against Qwen3-Omni in the reported benchmark results.
  • Conclusion: Ablations identify careful SFT design as the primary lever for installing timestamping, with RL providing additional refinement.

Limitations

TEMPO’s scope is bounded by its task coverage, temporal resolution, encoder choice, training variability, and music-data domain. These constraints limit applicability to overlapping events, sub-100 ms timing, other audio frontends, and broader real-world music.

  • TEMPO does not handle overlapping multi-domain events requiring simultaneous segmentation and labeling of concurrent streams.
  • A fixed 0.1 s temporal resolution excludes applications requiring sub-100 ms precision, including phoneme-level alignment and fine-grained percussive onset detection.The authors report that quantization is not dominant for the five evaluated tasks, where boundary MAE is approximately 0.77–1.19 s.
  • Because TEMPO uses a frozen Whisper-large encoder, its results may not generalize to other audio frontends.
  • The study omits multi-seed training because repeating the full multi-stage 7B pipeline is computationally costly.The authors nevertheless report large baseline margins and consistent ablation ordering as support for robustness.
  • Music captioning is trained exclusively on synthesized Slakh2100 data, so performance may differ substantially on broader real-world music with natural acoustics and production effects.The checkpoint is additionally evaluated on real-world MAESTRO to test transfer beyond synthesized MIDI recordings.

C Per-Component SFT Ablation

The controlled ablation shows that the distance-aware loss supplies the main isolated gains, while the time-aware projector helps when paired with that learning signal on temporally dense tasks.

  • The distance-aware loss alone improves all three representative tasks, including a 14.6-point gain in dense-captioning event F1.The same variant also improves diarization speaker-labeled F1 and grounding F1.
  • The time-aware projector alone is not useful, suggesting that wall-clock positional information requires a corresponding learning signal.
  • With the distance-aware loss, the projector improves diarization DER from 86.2% to 79.3%.
  • With the distance-aware loss, the projector improves grounding F1 from 38.0% to 38.7%, MAE from 1.73 s to 1.69 s, and mIoU from 41.9% to 42.5%.

D Baseline Prompts and Format failure analysis

The format-validation analysis separates syntax failures from substantive timestamping errors by parsing model-specific conventions and reevaluating only parseable outputs. Baseline performance changes modestly on valid outputs, leaving substantial gaps attributable to content and temporal localization.

  • Zero-shot baselines use model-specific timestamp conventions and tolerant parsers that accept several common output variants.Outputs without recoverable timestamped intervals are scored as maximally incorrect rather than discarded.
  • Restricting ASR evaluation to parseable outputs changes baseline scores only modestly; AF3 WER decreases from 209.7% to 206.7%.TEMPO is evaluated on the identical valid-example subset for each baseline.
  • After format failures are removed, the performance gap remains substantial, indicating that incorrect content and temporal localization dominate baseline errors.
  • TEMPO’s remaining errors are dominated by temporal granularity and segment-boundary mistakes rather than degenerate generations.
  • In severe dense-captioning temporal failures, event descriptions remain semantically correct despite poor overlap: 10.2% of examples with mIoU < 0.1 score METEOR 0.28 versus 0.23 dataset-wide.

E Qualitative Diagnostic Analysis

Qualitative diagnostics show that TEMPO can preserve transcript and event temporal structure, but failures often collapse adjacent or repeated segments into fewer intervals while retaining semantic content.

  • A successful ASR example achieves 0 WER and 0.99 mIoU, with onset and offset errors below 0.05 s.
  • An ASR failure collapses five reference segments into two predictions, raising WER to 125% and lowering mIoU to 0.21.
  • TEMPO preserves repeated sound-event granularity in one dense-captioning example, achieving 0.75 mIoU.
  • A contrasting dense-captioning failure merges seven reference events into one interval, reducing mIoU to 0.04 while retaining a semantically correct description with METEOR 0.99.
  • The work uses publicly released scientific artifacts under their stated licenses for non-commercial academic research.
Loading 2608.29999v1…