Source-linked AI summary
GigaChat Audio: Time-aware Large Audio Language Model
Aleksandr Kutsakov, Mariia Sadovina, Georgii Gospodinov, Alexandr Maximenko, Oleg Kutuzov, Pavel Bogomolov, Fyodor Minkin
TL;DR
Temporal grounding in long recordings remains unreliable because standard audio token streams do not naturally represent time. GigaChat Audio interleaves periodic temporal anchors with audio tokens and achieves stable long-form grounding while supporting time-anchored answers, descriptions, and summaries.
Problem
Temporal grounding in long-form audio remains unreliable because standard audio token streams do not naturally represent time and long recordings worsen timestamped output alignment.
Method
GigaChat Audio interleaves continuous audio tokens with periodic temporal anchors and uses cascaded synthetic supervision to produce timestamped answers, descriptions, and summaries.
Results
The model maintains 53.8 mIoU on 20–40-minute recordings, rising to 65.2 mIoU with 7-second temporal anchors, while mixed-duration training generalizes across lengths.
Takeaways & Limitations
Periodic temporal anchors and mixed-duration training are necessary for stable long-form grounding and length generalization, while sparse anchors can remain reliable.
Abstract
from arXiv · showhide
Temporal grounding in long recordings remains challenging for audio-conditioned LLMs. We present a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input. Our approach interleaves periodic time markers with continuous audio tokens using large-scale synthetic supervision from a cascaded pipeline. Our model achieves strong temporal-grounding accuracy on short and long benchmarks and supports time-anchored fragment descriptions and summaries. Extensive ablations examine how time representation, marker frequency, tokenization, and duration-mixture design affect accuracy and computational cost. We release model weights and datasets to support further research on time-aware audio understanding, available at https://huggingface.co/ai-sage/GigaChat3.1-Audio-10B-A1.8B.
1 SaluteDevices, Russia {askutsakov, sadovinama, georgygospodinov, ae.maximenko, olegkutuzov01, bobrosoft98, · 1. Introduction
GigaChat Audio addresses unreliable temporal grounding in long-form audio by interleaving audio tokens with periodic time markers and training on large-scale synthetic supervision. It supports timestamped answers, fragment descriptions, and summaries for recordings up to 120 minutes, with experiments examining representation, supervision, and duration generalization.
- 1. Introduction: Temporal grounding makes long-recording interfaces verifiable by linking answers to the audio segment supporting what happened and when.The motivation covers meetings, podcasts, lectures, and call-center logs, where users ask questions, request summaries, and navigate to evidence.
- 1. Introduction: Long-form audio remains difficult because models can produce plausible content with non-parseable timestamps, coarse references, or unsupported temporal claims.Standard audio token streams do not naturally represent time, and longer recordings exacerbate the problem.
- 1. Introduction: Experiments ask how to represent time, generate temporal supervision economically, and generalize across recording lengths.The paper states that manual annotation for long recordings is prohibitively expensive and examines duration-regime transfer.
- 1. Introduction: Training only on short audio does not extrapolate to long recordings, motivating training on a mixture of audio lengths.The supplied result passage reports strong asymmetry between short-only and long-only duration regimes, while the contribution passage states that temporal models must use mixed audio lengths.
- 1. Introduction: The architecture interleaves text tokens, continuous audio tokens, and timing tokens represented as hh:mm:ss text or special tokens.These token streams are shown as interleaved in the architecture description.
- 1. Introduction: GigaChat Audio supports up to 120 minutes of input and produces time-anchored answers, fragment descriptions, and summaries with explicit timestamps.The model interleaves continuous audio tokens with periodic inter-timings that act as temporal anchors.
- 1. Introduction: A cascaded synthetic-data pipeline generates temporal supervision at scale from timestamped transcripts for long recordings.The pipeline uses transcript slicing to reduce front-loading bias; the contribution passage also describes verification and robust evaluation by multi-sampling aggregation.
- 1. Introduction: The released resources include open model weights and a 10k+ hours temporal dataset covering seconds-to-hours recordings with temporal QA and time-anchored summarization supervision.The dataset and model are presented as support for research on time-aware audio understanding.
2. Related Work
Prior audio-language systems support broad audio understanding, but temporal grounding—especially in long recordings—remains insufficiently isolated and systematically analyzed. This work targets time-anchored open-ended question answering and summarization over recordings up to 120 minutes, with verifiable evaluation.
- Audio-language models: Existing multimodal and open AudioLLMs support speech-driven instruction following, broad audio understanding, and dialogue, but rarely isolate long-form temporal grounding.Examples include GPT-4o, Gemini, Qwen3-Omni, Voxtral, and earlier open AudioLLMs.
- Transcription-centric systems: Long-form ASR and transcription tools provide speaker, timing, or word-level alignment, whereas this work addresses time-anchored open-ended questions and summaries.Related systems include VibeVoice-ASR, MOSS Transcribe Diarize, and WhisperX.
- Temporal grounding: Short-audio grounding, temporal localization, and audio moment retrieval study related problems, while this work focuses on recordings up to 120 minutes and practical time-representation choices.Related benchmarks and approaches include text-to-audio grounding, TimeAudio, Clotho-Moment, and CASTELLA.
- Evaluation: Inspired by video temporal reasoning and separator tokens, the work uses judge-based evaluation for timed summaries and fragment descriptions with interval-overlap verification.This combines open-ended generation assessment with explicit temporal-grounding checks.
3. Method
The method attaches an audio front end to a 10B-A1.8B MoE language model and fine-tunes it with explicit temporal signals for long-audio understanding. It defines three time-centric generation tasks and evaluates grounding with interval metrics alongside judge-based assessments for descriptions and summaries.
- Model and training: The model combines an audio front end with a 10B-A1.8B MoE text checkpoint using a 256k-token context and audio SFT with explicit time signals.Training uses data parallelism only, without tensor or sequence parallelism.
- Model and training: The audio stack uses an encoder, subsampler, and projector with FlashAttention and chunk-wise encoder attention over 8 s chunks with 40 ms stride.The resulting embeddings are aligned to the text embedding space at a 160 ms frame rate.
- Time-centric tasks: The method supports temporal grounding, interval-conditioned fragment description, and timed summarization, with summarization requiring the model to choose segment boundaries.Grounding predicts an interval, fragment description receives an interval, and summarization produces a multi-part time-anchored summary.
- Data construction: Synthetic training data begins from 24k hours of English YODAS2 shards, filters to 16k hours by language probability, and retains 14k hours after silence-ratio filtering.The pipeline obtains word-level timestamps and aligned transcripts with WhisperX before filtering and duration bucketing.
4. Experiments
Experiments evaluate temporal grounding, fragment description, and timed summarization across short and long audio benchmarks, showing that the model remains effective as duration increases. Ablations characterize duration-mixture generalization, timestamp representation, anchor frequency, and marker encoding trade-offs.
- Benchmarks: The evaluation covers AUDIOGROUNDING, AMI Meeting Corpus, and time-aware DCASE Audio QA, using mIoU or interval MAE for temporal grounding.AUDIOGROUNDING uses 7–10 s clips and acoustic-event intervals; AMI uses 15–50 min meetings and phrase-level timestamps.
- Metrics: Timed summarization is measured with Tm, AES, and Rd, while fragment description uses an LLM-as-a-judge overall score.AES is computed from Acc, 1 − Err, and Style; Rd is the share of segments whose duration is a multiple of 60 seconds.
- Main results: 53.8 mIoU on 20–40 min shows stable long-form grounding, whereas Qwen3-Omni reaches 3.6 mIoU and 290.5 s interval MAE on long-form benchmarks.TimeAudio is reported only at its original benchmark duration because it often produces UNK or non-parseable timestamps beyond two minutes.
- Duration generalization: Training only on short audio fails on long recordings, while training only on long audio hurts short-audio performance across evaluations up to 120 minutes.Figure 3 evaluates checkpoints trained on different duration buckets over a shared duration grid and finds asymmetric generalization.
- Timing ablations: More frequent inter-timing anchors improve grounding but add tokens: 60 s anchors yield 3 s median error versus 1.5 s for 7 s anchors.Minute-index marker formats remain close to hh:mm:ss with lower token overhead, whereas pure-seconds encodings substantially degrade performance.
5. Conclusion
The paper presents a time-aware Audio LLM that handles up to 120 minutes of input and produces answers and summaries anchored explicitly in time. It identifies long-recording temporal grounding as a bottleneck and highlights periodic temporal anchors as essential for stable grounding.
- Conclusion: The model supports up to 120 minutes of audio input and generates answers and summaries explicitly anchored in time.This extends time-aware audio understanding to long recordings.
- Conclusion: Temporal grounding remains a major bottleneck for existing multimodal models, which often degrade sharply beyond a few minutes of audio.The conclusion specifically identifies long recordings as the challenging setting.
- Conclusion: Periodic temporal anchors, or inter-timings, are essential for stable grounding in long-form audio.The passage notes that even sparse anchors, such as once per minute, can be sufficient for reliable grounding.
6. Generative AI Use Disclosure
Generative AI supported limited manuscript language editing and two experimental functions: synthetic-supervision generation and LLM-as-a-judge evaluation. The authors reviewed and validated all outputs and retain responsibility for the final content.
- Generative AI tools were used for limited language editing, including grammar, clarity, and stylistic refinement.
- All generated outputs were carefully reviewed, edited, and validated by the authors, who take full responsibility for the final content.
- Within the experimental framework, large language models generated synthetic supervision from timestamped transcripts and served as LLM-as-a-judge evaluators for fragment descriptions.