TRACE: Temporal Grounding Video LLM via Causal Event Modeling

1 INTRODUCTION

• VTG underlies moment retrieval, dense video captioning, highlight detection, and video summarization, while video LLMs are increasingly used for versatile, zero-shot prediction.Earlier non-generative models excel at moment retrieval and highlight detection but are task-specific, inflexible, and require substantial fine-tuning.Source: §1 · p. 1
• Natural-language-only video LLM outputs blend timestamps and captions, whereas structured video understanding also requires timestamps and salient scores.The paper argues that this mismatch may make it difficult for video LLMs to model events effectively and achieve satisfactory VTG results.Source: §1 · pp. 1–2
• TRACE improves zero-shot results across Youcook2, Charades-STA, and QVHighlights, and is competitive with task-specific models after fine-tuning.Reported gains include 3.1 and 4.9% on Youcook2 (CIDEr and F1 Score), 6.5% and 3.7% in Recall (IOU = {0.5, 0.7}) on Charades-STA, and 10.3% and 9.2% for mAP and HIT@1 on QVHighlights.Source: Figure 1 · p. 2, §1 · p. 2
• Causal event modeling represents outputs as timestamped events containing salient scores and captions, predicted from video, instructions, and preceding events.TRACE implements the framework by handling frames, timestamps, scores, and text as separate tasks with interleaved tokens and an adaptive head-switching method.Source: §1 · p. 2

2 RELATED WORKS

• VTG tasks include identifying event timestamps for moment retrieval, dense video captioning, summarization, and highlight detection.The paper defines VTG as precisely identifying event timestamps within a video.Source: §2 · p. 2
• Figure 2 depicts sampled-frame timestamps integrated into visual tokens and answer tokens sequenced as time, score, then text.The figure caption also points to Figure 4 for TRACE’s generation process.Source: Figure 2 · p. 3
• Traditional VTG systems use task-specific prediction heads and can be resource-intensive, lack zero-shot capability, and require downstream fine-tuning.Dense-captioning architectures also tend to be task-specific, difficult to adapt to LLM structures, and lack zero-shot capability.Source: §2 · p. 3
• Video LLMs have difficulty pinpointing event timestamps, prompting methods such as TimeChat, VTimeLLM, and Hawkeye to fine-tune on VTG datasets.Later approaches add fast-slow visual tokens, time encoders, or special time tokens, but the paper says they do not account for videos’ inherent structure.Source: §2 · pp. 3–4

3 TRACE

• TRACE is proposed as a video LLM implementation of a framework for outputs aligned with video structure.The section frames the method around two questions: how to model structured outputs and how to implement that model.Source: §3 · p. 4

3.1 MODELING THE INHERENT STRUCTURES OF VIDEOS

• Each event contains a timestamp, salient score, and caption, and the next event is conditioned on prior events, instructions, and visual inputs.Within an event, time, score, and caption are decoded sequentially; events are ordered by timestamp.Source: §3.1 · p. 4

3.2 TRACE: TASK-INTERLEAVED TEMPORAL GROUNDING VIDEO LLM

• TRACE treats frames, instructions, timestamps, scores, and captions as distinct tasks, using separate encoders and decoding heads with interleaved task tokens.The design also uses adaptive head switching to choose the decoder for the next generated token.Source: §3.2 · p. 4

3.2.1 SEPARATED MULTI-TASK PROCESSING

• TRACE uses the LLM backbone’s text tokenizer and head, while timestamp and score tasks use separate encoders and decoding heads.The timestamp and score tokenizers use number, separator, and synchronization tokens; the frame inputs combine compressed visual tokens with encoded frame times.Source: §3.2.1 · p. 5

3.2.2 TASK-INTERLEAVED SEQUENCE MODELING

• Event tokens are ordered chronologically, with each event sequenced as time, score, then text tokens.This ordering implements the factorization of the event prediction into successive conditional predictions.Source: §3.2.2 · p. 5

3.2.3 ADAPTIVE HEAD-SWITCHING MECHANISM FOR GENERATION

• TRACE switches between time, score, and text decoding heads when it generates a ⟨sync⟩ token.The heads cycle in the order time, score, then text.Source: §3.2.3 · p. 6, Figure 4 · p. 7

3.3 TRAINING STRATEGY AND DATA PREPARATION

• Training has two stages: initialize task modules while freezing the vision encoder and LLM, then fine-tune the LLM and task modules while keeping the vision encoder fixed.Stage 1 uses uniformly sampled 128 frames per video; Stage 2 samples one frame from each of 128 clips.Source: §3.3 · p. 6
• Stage 2 uses 635K VTG instruction-tuning samples and 284K compressed video-caption samples, plus video question-answering datasets.VTG data include VTG-IT, ActivityNet Captions, and a subset of InternVid; compressed caption data retain one sample per video when instructions differ.Source: §3.3 · p. 6

4.1 EVALUATION DATASETS, METRICS, AND BASELINE MODELS.

• Evaluation covers dense video captioning, moment retrieval, and video highlight detection, using their respective dataset-specific caption, localization, and highlight metrics.Baselines include general video LLMs and models designed specifically for VTG.Source: §4.1 · p. 7

4.2 PERFORMANCE OF TRACE

• TRACE improves over video LLMs across three zero-shot datasets, including Youcook2 CIDEr by 3.1% and F1 by 4.9%, Charades-STA Recall by 6.5% and 3.7%, and QVHighlights mAP by 10.3% and HIT@1 by 9.2%.The reported Charades-STA Recall improvements use IoU thresholds 0.5 and 0.7, respectively; Table 2 marks some 13B results as unfair comparisons.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 ABLATION STUDIES OF TRACE.

• Removing causal event modeling reduces performance; the ablation compares TRACE with natural-language-style inputs on VTG-IT.The authors also report that TRACE can perform better while sampling fewer video frames.Source: §4.3 · p. 8, Table 3 · p. 9
• Using shared encoders and decoding heads instead of separate task-specific ones disrupts prelearned LLM knowledge and can produce irrelevant responses.In the ablation, time and score tokens are added directly to the text tokenizer.Source: §4.3 · p. 8, Table 3 · p. 9
• TRACE’s performance increases with more sampled frames and is comparable to or better than VTG-LLM and TimeChat with only eight frames.The comparison is reported in the frame-count ablation.Source: §4.3 · p. 8, Table 3 · p. 9
• Adding InternVid and ActivityNet Captions training data improves results on long-video datasets but slightly lowers performance on short-video Charades-STA.The data ablation compares training with VTG-IT alone against the original TRACE training data.Source: §4.3 · p. 8, Figure 5 · p. 9

4.2 PERFORMANCE OF TRACE

• On ActivityNet Captions, TRACE leads in moment retrieval and matches VTimeLLM comparably on dense video captioning, though most compared algorithms trained on this dataset.Momentor is the exception to the training-data qualification.Source: §4.2 · p. 8, Table 4 · p. 10

4.4 FINE-TUNED PERFORMANCE OF TRACE.

• After three fine-tuning epochs, TRACE reaches new SOTA results on Youcook2 without audio and is competitive with non-generative models on Charades-STA.The authors also report that TRACE benefits more from fine-tuning than TimeChat and VTG-LLM, which struggle to attain satisfactory performance.Source: §4.4 · p. 9, Table 5 · p. 10

5 CONCLUSION AND FUTURE WORKS

• TRACE is presented as a potential foundation for future VTG research, following superior zero-shot results against video LLM baselines and competitive fine-tuned performance against traditional models.The paper frames this as demonstrating video LLMs’ potential for VTG tasks.Source: §5 · p. 10
• Because TRACE uses pretrained decoder-only LLMs and predicts each event from previous events, it may miss complex event relationships.The authors suggest adding outputs from causality-discovery models as supplementary inputs.Source: §5 · p. 10
• The authors also propose expanding annotations to include QA-pair timestamps and question-answer matching scores to improve performance.This is presented as a future direction for adding more video-understanding tasks.Source: §5 · p. 10

A.1 DETAILS OF DATA FORMAT

• VTG training data combines ActivityNet Captions and InternVid with annotations collected by VTG-IT.The appendix organizes the annotations into four task types.Source: §A.1 · p. 16
• General-task responses omit timestamps and scores; dense captioning and moment retrieval use timestamps and captions, highlight detection returns the query, and summarization returns each event caption.A ⟨sync⟩ placeholder marks absent timestamps and scores for general tasks and absent scores for dense captioning.Source: §A.1 · p. 16

A.2 PROCESSING INTERNVID

• InternVid-Full videos are excluded from dense-caption data if captions are too short or repetitive, event counts fall outside 5–50, or special characters occur.Repetitive captions are screened with a fuzzywuzzy threshold of 70.Source: §A.2 · pp. 16, 18
• For moment retrieval, TRACE uses InternVid-10M-FLT-INFO rather than InternVid-Full because direct use of the full annotations yielded suboptimal performance, likely from imprecise timestamps.The selected annotations are a filtered subset of InternVid-Full provided by its authors.Source: §A.2 · p. 18

A.3 PROCESSING VTG-IT

• VTG-IT highlight and summarization data are refined because their supplied salient scores are often uniform within each video.Dense-caption and moment-retrieval annotations are used directly.Source: §A.3 · p. 18
• For highlight detection, each dense-caption event is split into up to 20 clips, whose frame-caption similarities are computed with EVA-CLIP and mapped to Gaussian-distributed scores.The similarity scores are normalized and assigned highlight scores according to their relative ranks among clips.Source: §A.3 · p. 18
• For video summarization, the highest-scoring clip is selected as the summary clip for each dense-caption event.The summarization data is built from the video-highlight and dense-caption tasks.Source: §A.3 · p. 18

• Training takes about five days for Stage 1 and five days for Stage 2.Source: §B.1 · p. 20

• TRACE-uni outperforms TRACE on both VTG tasks and general video understanding tasks, while the original TRACE setting improves performance on long videos but slightly reduces it on short Charades-STA videos.TRACE-uni incorporates general video understanding data from parts of LLaVA-Video-178k.Source: §B.2 · p. 20
• TRACE significantly outperforms other video LLMs on fine-tuned QVHighlights, according to the reported comparison.Source: §B.2 · p. 20, Table 9 · p. 21
• TRACE handles general video understanding tasks while excelling in VTG tasks, as reported across the cited evaluation tables.Source: §B.2 · p. 21, Table 10 · p. 21, Table 11 · p. 21
• TRACE-uni surpasses TRACE across all three VTG evaluation datasets and improves general video understanding without additional VTG training data.It uses about 2M training examples and performs on par with or better than general video LLMs using the same LLM backbone and vision encoder.Source: §B.2 · p. 21
• Increasing the number of slots per frame significantly improves TRACE’s performance under the reported Youcook2 setting, which uses VTG-IT in Stage 2 and samples 64 frames.The authors recommend more slots per frame when computational or efficiency constraints are not a concern.Source: §B.2 · p. 21

B.3 CASE STUDIES

• TRACE’s case studies demonstrate accurate event identification and traditional video captioning.Figure 11 presents the case study.Source: §B.3 · p. 21, Figure 11 · p. 23
• The cited zero-shot results table covers Youcook2, Charades-STA, and QVHighlights, while the slot-count tables report Youcook2 results using VTG-IT in Stage 2 and 64 sampled frames.Source: Table 12 · p. 22, Table 13 · p. 22, Table 14 · p. 22

C DISCUSSION ON CAUSAL EVENT MODELING

• Causal event modeling structures video LLM responses as event triplets, unlike causal language modeling’s whole-text output.The discussion contrasts this event-based formulation with complete causal relationship modeling and discovery.Source: C DISCUSSION ON CAUSAL EVENT MODELING · p. 22

C.1 CAUSAL LANGUAGE MODELING VS. CAUSAL EVENT MODELING.

• The discussion identifies output format as causal event modeling’s key improvement over causal language modeling.Source: §C.1 · p. 23
• Event triplets model inter-event relations through next-event prediction and encode intra-event dependencies from timestamps to scores, then from both to captions.Source: §C.1 · p. 23
• Separating timestamps, scores, and text allows independent encoders and decoders, avoids adding tokens to the text tokenizer, and retains general video understanding capacity.Source: §C.1 · p. 24

• Complete causal relationship research includes methods that model event relations, benchmark complex causal reasoning, or discover causality graphs from videos.Examples include VAR and Jin et al. (2022) for relationship modeling, and studies introducing reasoning benchmarks or causality graphs.Source: §C.2 · p. 24
• On Event-Bench, TRACE outperforms open-source video LLMs and performs comparably to GPT-4o on event description, contextual reasoning, and episodic reasoning despite using less causal-reasoning training data.Source: §C.2 · p. 24, Table 15 · p. 25
• TRACE still conditions later event triplets on previously generated events because of its decoder-only backbone; proposed integrations include causality graphs as inputs, reasoning examples, or visual attention masks.Source: §C.2 · pp. 24–25