Source-linked AI summary
GigaChat Audio: Time-aware Large Audio Language Model
Aleksandr Kutsakov, Mariia Sadovina, Georgii Gospodinov, Alexandr Maximenko, Oleg Kutuzov, Pavel Bogomolov, Fyodor Minkin
TL;DR
Temporal grounding in long recordings remains unreliable, even though timestamps let users navigate to supporting audio. GigaChat Audio interleaves periodic time anchors with audio tokens and uses synthetic supervision; the authors report stable long-form grounding with anchors and better length generalization from mixed-duration training.
Problem
Temporal grounding in long-form audio remains unreliable, despite timestamps being important for letting users verify answers against supporting audio segments.
Source: §1 · p. 1Method
GigaChat Audio interleaves periodic temporal anchors with continuous audio tokens and uses a cascaded synthetic-data pipeline to train timestamped answers, descriptions, and summaries.
Source: §1 · p. 1Results
Mixed-duration training outperforms single-duration regimes across nearly all audio lengths, while periodic anchors are essential for stable long-form grounding.
Source: §4.2 · p. 4, §5 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Temporal grounding in long recordings remains challenging for audio-conditioned LLMs. We present a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input. Our approach interleaves periodic time markers with continuous audio tokens using large-scale synthetic supervision from a cascaded pipeline. Our model achieves strong temporal-grounding accuracy on short and long benchmarks and supports time-anchored fragment descriptions and summaries. Extensive ablations examine how time representation, marker frequency, tokenization, and duration-mixture design affect accuracy and computational cost. We release model weights and datasets to support further research on time-aware audio understanding, available at https://huggingface.co/ai-sage/GigaChat3.1-Audio-10B-A1.8B.
1 SaluteDevices, Russia
The section identifies SaluteDevices, Russia, and lists associated author handles.
1 SaluteDevices, Russia
- The listed handles are askutsakov, sadovinama, georgygospodinov, ae.maximenko, olegkutuzov01, and bobrosoft98.Source: §1 · p. 1
1. Introduction
GigaChat Audio addresses unreliable temporal grounding in long recordings by interleaving periodic time anchors with audio tokens and training with synthetic supervision. It supports timestamped answers, fragment descriptions, and summaries for inputs up to 120 minutes.
1. Introduction
- In long recordings, temporal grounding makes answers verifiable by letting users navigate to supporting audio, but existing audio-conditioned LLMs often emit non-parseable timestamps, coarse references, or unsupported temporal claims.Time is not naturally represented in standard audio token streams, and long recordings exacerbate the mismatch with timestamped outputs.Source: §1 · p. 1
- Short-only training does not extrapolate to long recordings, long-only training degrades short-audio performance, and periodic anchors are essential, though once-per-minute anchors can suffice.Removing anchors collapses long-recording accuracy; anchor frequency and format affect the accuracy–compute trade-off.Source: §1 · p. 1
- GigaChat Audio supports up to 120-minute inputs and produces time-anchored answers, fragment descriptions, and summaries by interleaving continuous audio tokens with periodic temporal anchors.A cascaded synthetic-data pipeline uses transcript slicing to reduce front-loading bias and a global verifier to enforce consistency.Source: §1 · p. 1
- The work releases an open-weight time-aware audio LLM and a 10k+ hour temporal dataset spanning seconds-to-hours recordings, with temporal-QA and time-anchored summarization supervision.The reported contributions also include robust evaluation by multi-sampling aggregation.Source: §1 · p. 1
2. Related Work
Prior audio LLMs support speech and audio instruction following, transcription, and general audio QA, but their temporal grounding—especially on long recordings—has rarely been systematically isolated. This work distinguishes itself through time-anchored open-ended QA and summarization for recordings up to 120 minutes, examining temporal representations and duration-mixture training.
Related Work
- Temporal grounding in long-form recordings is rarely isolated and systematically analyzed in systems that otherwise support transcription and general audio QA.Modern multimodal systems include GPT-4o [1], Gemini [2], Qwen3-Omni [4], and Voxtral [3], while earlier open AudioLLMs [5] [6] demonstrate broad audio understanding and dialogue capabilities.Source: §2 · pp. 1–2
- Unlike transcription-focused systems such as VibeVoice-ASR [7], MOSS Transcribe Diarize [8], and WhisperX, this work targets time-anchored open-ended QA and summarization.WhisperX provides word-level timestamp alignment for long recordings.Source: §2 · p. 2
- Unlike short-event grounding, audio moment retrieval, and localization systems such as TimeAudio, this work targets recordings up to 120 minutes and studies time representation and duration-mixture training.The design choices include anchor frequency, time format, and tokenization, with duration-mixture training addressing length generalization.Source: §2 · p. 2
- Inspired by video temporal reasoning and separator-token localization [16], the evaluation uses LLM-as-a-judge for timed summaries and fragment descriptions, with interval-overlap metrics verifying temporal grounding.The related work also notes text-to-audio grounding benchmarks [11] and retrieval datasets Clotho-Moment and CASTELLA [14].Source: §2 · p. 2
3. Method
The method combines a time-aware audio LLM with periodic temporal anchors, synthetic supervision, and task-specific evaluation for grounding, fragment description, and timed summarization. Its data pipeline filters and timestamps long recordings, while evaluation uses interval metrics or transcript-informed LLM judging.
3.1. Model
- The model attaches an audio front end to a 10B-A1.8B MoE text checkpoint with 256k-token context, then interleaves timestamps with audio tokens every 60 seconds.Its encoder, subsampler, and projector produce embeddings aligned to text at a 160 ms frame rate; audio encoder pretraining uses KMeans distillation targets from 2M hours of multilingual audio.Source: §3.1 · p. 2
3.2. Base Audio Post-Training Data
- Base audio post-training uses a fixed heterogeneous SFT mixture spanning captioning, dialogue, emotion recognition, and general voice tasks.The mixture remains fixed across time-aware ablations to isolate temporal design choices.Source: §3.2 · p. 2
3.3. Time-aware Tasks
- Temporal grounding predicts event intervals, fragment description describes a supplied interval, and timed summarization creates a multi-part summary while choosing its own segment boundaries.Grounding relates to audio moment retrieval; fragment description to conditioned captioning and long-audio speech summaries.Source: §3.3 · p. 2
3.4. Audio and timestamped transcripts
- The transcript pipeline filters 24k hours of English YODAS2 to 14k hours, adds word-level timestamps, then buckets recordings by duration and balances bucket sizes.English filtering retains 16k hours at p(English) > 0.7 before silence-ratio filtering; buckets are ≤20 minutes, 20–40 minutes, and ≥40 minutes.Source: §3.4 · p. 3
3.5. Data generation
- Synthetic grounding QA is generated from ∼10-minute transcript slices by text-only GPT-OSS-120B [29], verified against full transcripts, and split at the audio level.Slicing reduces front-loading bias; timed summarization instead provides the full timestamped transcript in one pass, without sampled fragments.Source: §3.5 · p. 3
3.6. Evaluation
- Temporal grounding is scored by interval mIoU and midpoint MAE, while fragment descriptions and timed summaries use LLM judging against references with the full timestamped transcript.Timed summaries are judged on timing structure, factual accuracy, errors or contradictions, and style; an automatic metric also reports the proportion of whole-minute segments.Source: §3.6 · p. 3
4. Experiments
Experiments show that grounding remains strong on long recordings with periodic temporal anchors, while duration-mixture design, anchor frequency, and marker encoding affect performance and token cost.
4.2. Length extrapolation
- Training on all duration buckets outperforms other regimes across nearly all evaluation lengths up to 120 minutes, while short-only training fails on long audio and long-only training hurts short-audio performance.For clean separation, early-recording questions are discarded from evaluation; Figure 3 reports temporal-grounding mIoU across training-duration windows and evaluation lengths.Source: Figure 3 · p. 4, §4.2 · p. 4
4.1. Main comparison across tasks and datasets
- 53.8 mIoU on 20–40 min improves to 65.2 mIoU with 7-second anchors, while removing anchors drops it to 14.2 and degrades timed descriptions and summaries.Qwen3-Omni reaches 3.6 mIoU on 20–40m TGr and 290.5 s interval MAE on AMI. Evaluation spans short-clip grounding, meeting phrase timing, DCASE start-time questions, and the paper’s grounding, fragment-description, and timed-summarization tasks.Source: §4.1 · pp. 3–4
4.3. Special timing tokens and scaling the timing ratio
- Special timing tokens require a much larger timing-token share in SFT than plain-text timestamps to reach comparable mIoU.The comparison keeps the visible hh:mm:ss format, replacing its characters with special tokens initialized from the corresponding digit and colon embeddings.Source: §4.3 · p. 4, Table 2 · p. 4
4.4. Inter-timing frequency
- More frequent inter-timing anchors improve grounding at added token cost; 60-second anchors yield 3 s median error versus 1.5 s with 7-second anchors.The reported median errors show that the model interpolates second-level precision between minute-spaced anchors.Source: §4.4 · p. 4, Table 3 · p. 4
4.5. Inter-timing format
- Pure-seconds markers substantially degrade performance, whereas minute-index formats stay close to hh:mm:ss with much lower token overhead.The comparison uses one anchor per minute and evaluates shorter encodings without changing marker semantics.Source: §4.5 · p. 4, Table 4 · p. 4
5. Conclusion
The paper presents a time-aware Audio LLM for answers and summaries anchored in recordings up to 120 minutes long. It finds periodic temporal anchors and mixed-duration training important for reliable long-form grounding and length generalization, and releases a large temporal-audio dataset.
5. Conclusion
- Periodic temporal anchors are essential for stable long-form grounding; anchors as sparse as once per minute support reliable grounding, and mixed-duration training is necessary for length generalization.Existing multimodal models often degrade sharply beyond a few minutes of audio.Source: §5 · p. 4
- The time-aware Audio LLM supports up to 120 minutes of input and produces answers and summaries explicitly anchored in time.Source: §5 · p. 4
- The authors release a 10k+ hours dataset spanning seconds-to-hours recordings for temporal QA and timed summarization.The release aims to facilitate further research on time-aware audio understanding.Source: §5 · p. 4
6. Generative AI Use Disclosure
Generative AI was used for limited manuscript editing and specified experimental tasks, but not to formulate scientific claims, interpret results, or determine conclusions. The authors reviewed and validated outputs and take responsibility for the final content.
6. Generative AI Use Disclosure
- AI tools generated synthetic supervision from timestamped transcripts and served as LLM-as-a-judge evaluators for fragment description and timed summarization.These uses were part of the experimental framework described in Sections 3.5 and 3.6.Source: §6 · p. 5
- For manuscript preparation, AI tools provided limited grammar, clarity, and stylistic editing; the authors reviewed, edited, and validated all outputs.The authors take full responsibility for the final content.Source: §6 · p. 5
- Generative AI tools did not formulate scientific claims, interpret results, or determine research conclusions.Source: §6 · p. 5