TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding

1 Introduction

• AV-MSG localizes only the occurrences matching a query, unlike dense event captioning, which describes every event.Audio can delimit events such as barking or gunshots more clearly than visual appearance.Source: §1 · p. 2, Figure 1 · p. 2
• Existing methods can collapse recurring occurrences into one interval or over-segment them, while overlap-focused objectives do not explicitly penalize missing or extra segments.The paper identifies this count miscalibration as a key bottleneck for robust AV-MSG.Source: §1 · p. 2
• Time Token Interleaving (TTI) inserts special absolute-time tokens into the audio-visual stream, marking the times of the audio and visual tokens that follow.The tokens provide an explicit, generable time axis distinct from ordinary numerals.Source: §1 · pp. 2–3
• TiTok achieves the most accurate AV-MSG results among the compared models, outperforming AVicuna and ChronusOmni fine-tuned on the same training data.The evaluation uses the UnAV-100-based protocol.Source: §1 · p. 3
• TiTok combines global, local, count, precision, and format rewards with GDPO, and the paper establishes an UnAV-100-based evaluation protocol.The introduction reports that TiTok outperforms AVicuna and ChronusOmni, including a ChronusOmni model fine-tuned on the same training data.Source: §1 · pp. 2–3

2.1 Audio-Visual Temporal Grounding with LLMs

• Prior work addresses audio-visual grounding or multi-segment prediction, but the paper says none addresses both jointly; AVicuna can emit multiple segments yet its predicted counts deviate substantially from ground truth.Most LLM-based temporal grounding methods rely heavily on visual input and predict a single event timestamp.Source: §2.1 · p. 3

2.2 Temporal Representation in MLLMs

• Temporal representations vary between input-side timestamps and output-side relative or absolute time; TiTok's TTI uses dedicated time tokens at both input and output.Dedicated absolute-time tokens have been used for visual-only grounding, but not interleaved into an audio-visual stream.Source: §2.2 · p. 4

2.3 Reinforcement Learning for Temporal Grounding

• Temporal-grounding RL often combines format and IoU rewards with GRPO, while GDPO separately normalizes distinct rewards to avoid collapsing different reward combinations into identical advantages.The paper motivates GDPO as a way to preserve distinctions among reward signals.Source: §2.3 · p. 4

3.1 Problem Statement

• The model must infer how many temporal segments to predict while locating every occurrence of a queried event in audio-visual video.Ground truth consists of start–end intervals; samples with one interval are single-segment, and those with at least two are multi-segment.Source: §3.1 · p. 4

3.2 Time Token Interleaving

• Time Token Interleaving inserts each timestamp into the audio-visual block at that time and uses the same special tokens for input perception and output timestamps.The shared representation is intended to support boundary placement and distinguish recurring events as separate segments.Source: §3.2 · p. 5

3.3 Two-Stage Training

• Training first uses cold-start SFT to teach valid special time-token generation, then GDPO-based RL to build AV-MSG ability.GDPO independently normalizes each reward before combining advantages and batch-normalizing them, preserving fine-grained localization signals that easier-to-learn rewards could suppress.Source: Stage 1: Cold-Start SFT · p. 6, Stage 2: GDPO-based Reinforcement… · p. 6

3.4 Reward Design

• The five rewards cover global precision–recall agreement, local segment placement, event count, temporal precision, and valid output format.Local reward pairs predictions with the best unpaired ground-truth segment and divides by the larger segment count, so count mismatches are reflected; global reward uses precision and recall F1.Source: Local Reward · p. 7, Global Reward · p. 7, §3.4 · p. 7
• The count reward penalizes both over- and under-prediction, with a special zero reward when multiple ground-truth events are collapsed into one segment.The separate precision reward discourages inflated boundaries, while format reward accepts any number of well-formed segments.Source: Count Reward · p. 8, Precision Reward · p. 8, Format Reward · p. 8

4.1 Experimental Settings

• The UnAV-100 protocol groups each video's intervals by event category into one query and a chronologically ordered target, while Charades-STA adds single-segment training examples.Evaluation uses held-out UnAV-100 and Charades-STA test splits; CountF1 complements overlap metrics by measuring under- and over-segmentation.Source: Datasets · pp. 8–9, Metrics · pp. 9–10

4.2 Quantitative Results

• TiTok achieves the best results on every UnAV-100 metric, with a wider advantage over fine-tuned ChronusOmni in F1 and CountF1 than in mIoU.Because mIoU compares segment unions, merging nearby occurrences can still score well; F1 and CountF1 require occurrences to be predicted separately.Source: Table 2 · p. 11, Performance on UnAV-100 · p. 11

4.3 Qualitative Results

• On the illustrated clip, TiTok predicts segment counts and boundaries closest to ground truth for both visually cued volleyball and clearly audible clapping.The comparison shows that audio alone does not prevent count errors, while visual-only input provides little evidence for the clapping query.Source: §4.3 · p. 12, Figure 4 · p. 12

4.4 Ablation Studies

• TTI's interleaving contributes more to boundary placement, while special time tokens contribute more to count calibration; removing audio also lowers every metric.Among reward ablations, removing the global reward causes the largest mIoU drop and lowers all F1 scores; removing precision lowers F1 at every threshold. Replacing GDPO with GRPO trades higher CountF1 for lower mIoU and F1@0.7, while both optimizers improve every metric over SFT.Source: Effect of TTI · pp. 12–13, Effect of Audio · p. 13, RL Design Choice · pp. 13–14

4.5 Training Dynamics

• During GDPO training, global and precision rewards rise early and plateau, while multi-segment count reward starts lower and trends upward afterward.The local reward is higher for single-segment samples, whose targets are easier to cover and do not incur the multi-segment count-mismatch penalty.Source: §4.5 · p. 14

• TiTok achieves the highest overlap and CountF1 scores among the compared baselines on the UnAV-100-based protocol.Source: §5 · p. 14
• TiTok combines Time Token Interleaving with multi-segment-oriented rewards optimized using GDPO.Source: §5 · p. 14
• The paper introduces a UnAV-100-based evaluation protocol and CountF1, which depends only on annotations and can be recomputed for other benchmarks.Source: §5 · p. 14