PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

Linli Yao, Yuancheng Wei, Yaojie Zhang, Lei Li, Xinlong Chen, Feifan Song, Ziyue Wang, Kun Ouyang, Yuanxin Liu, Lingpeng Kong, Qi Liu, Pengfei Wan, Kun Gai, Yuanxing Zhang, Xu Sun

arXiv:2602.08711v3cs.CV

TL;DR

Existing captioning methods lack dense temporal coverage across audio-visual content, while scene boundaries make joint timestamp and caption evaluation difficult. The paper introduces Omni Dense Captioning, structural captions, benchmark and metric infrastructure, and a model trained with SFT and GRPO; the model achieves strong benchmark performance and transfers to audio-visual reasoning and temporal grounding. The authors also identify limited generalization to hour-long videos as a scope boundary.

  • Problem

    Existing audio-visual captioning lacks explicit timestamps and dense supervision, visual dense captioning neglects audio, and semantic scene boundaries complicate joint evaluation.

  • Method

    The paper defines Omni Dense Captioning with six-dimensional structured audio-visual captions, builds OmniDCBench and SodaM, and trains TimeChat-Captioner with SFT followed by GRPO.

  • Results

    TimeChat-Captioner-GRPO achieves state-of-the-art SodaM performance with a score of 35.0, surpasses Gemini-2.5-Pro on that metric, and transfers strongly to Daily-Omni, World-Sense, and temporal grounding.

  • Takeaways & Limitations

    Dense, multi-dimensional captions provide richer supervision for downstream omni-video understanding and generation, including audio-visual reasoning and temporal grounding.

  • Takeaways & Limitations

    The model has limited generalization to varying video durations, particularly hour-long content, so long videos are processed by segmenting them into shorter clips.

Abstract

from arXiv · show

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code are available at https://github.com/yaolinli/TimeChat-Captioner.

1. Introduction

Omni Dense Captioning addresses the lack of temporally granular, audio-visual captioning by generating continuous scene segments with fine-grained structured descriptions. The paper introduces structural captions, a benchmark, an evaluation metric, and a trained baseline for this task.

  • Existing audio-visual captioning mainly produces global descriptions without timestamps, while dense video captioning largely omits audio semantics.
  • Omni Dense Captioning segments videos into continuous scenes and generates fine-grained audio-visual descriptions for each segment.
  • Dense captions combine continuous temporal coverage with details such as spatial attributes, actions, dialogue, and acoustic cues.
  • A six-dimension schema organizes captions around events, environment, camera state, shot editing, dialogue, and acoustic cues.
  • The paper provides OmniDCBench, SodaM, and TimeChat-Captioner-7B to support benchmarking, evaluation, and modeling for omni-video captioning.

2. Related Work

Related work spans audiovisual temporal coherence, video captioning, dense temporal localization, long-range modeling, structured video organization, and reinforcement learning. OmniDenseCaptioning is positioned as scene-aligned supervision for broader multi-shot understanding and generation.

  • Recent omni-modal captioning work shifts video understanding from vision-centric descriptions toward joint audio-visual understanding.
  • Dense video captioning localizes temporal segments and generates event descriptions, with research progressing from pipeline-based to end-to-end frameworks.
  • LongVALE models extended video durations, while ARC-Chapter organizes videos into chapter-level units for structured descriptions.
  • Multi-shot understanding and generation rely on dense, scene-aligned supervision that OmniDenseCaptioning is designed to provide.
  • Reinforcement-learning approaches align multimodal video models with task-specific objectives through verifiable rewards, structured reasoning, or content coverage.

3. OmniDenseCaptioning Task and A New Benchmark

OmniDenseCaptioning generates continuous, timestamped, fine-grained audio-visual narratives organized as semantically coherent scenes. The paper introduces OmniDCBench and SodaM to support annotation and evaluation of these structured descriptions despite ambiguous scene boundaries.

  • Task Definition: OmniDenseCaptioning segments videos into successive scenes and generates detailed paragraph-level descriptions with explicit timestamps.Each scene is semantically coherent in time, location, or narrative context and may contain multiple shots.
  • Task Definition: Its six-dimensional schema covers audiovisual events, background, camera state, shot editing, dialogue, and acoustic cues.The schema is designed to capture spatial, temporal, cinematic, spoken, and acoustic information comprehensively.
  • Task Definition: Compared with sparse event-centric captioning, the task targets comprehensive successive multi-scene narratives covering significant visual and auditory content.The output is intended to preserve subtle, fine-grained details rather than only salient moments or brief descriptions.
  • Benchmark Dataset Curation: OmniDCBench contains 1,122 human-annotated videos with descriptions averaging 995 words per video.The benchmark uses diverse movie clips and YouTube videos, with annotations created and double-checked by human annotators.
  • Evaluation Metric Design: SodaM jointly evaluates timestamp accuracy and caption quality while aligning variable-length scene sequences to mitigate boundary ambiguity.Its evaluation design addresses temporal segmentation, multi-dimensional descriptions, and many-to-one prediction alignments.

4. TimeChat-Captioner Framework

TimeChat-Captioner combines a Qwen2.5-Omni backbone with staged data construction, SFT, and GRPO to generate structured, time-aware captions. Task-specific rewards target valid formatting, appropriate length, timestamp accuracy, and SodaM caption quality.

  • Model Architecture: TimeChat-Captioner uses Qwen2.5-Omni with temporally interleaved audio-visual tokens and M-RoPE for synchronous comprehension and scene-boundary localization.The backbone combines joint audio-visual perception with temporal position encoding.
  • Data Construction: The training pipeline samples 3-minute clips, generates coarse segmentations followed by six-dimensional detailed captions, and filters invalid or unsuitable data.Filtering removes clips with too few segments, missing audio, malformed JSON or fields, and overly short segments.
  • Data Construction: 42K high-quality time-aware video-caption pairs remain after filtering, with training videos and annotation schema independent from the manually annotated benchmark.This separation supports evaluation of generalization between synthetic training annotations and manual evaluation annotations.
  • Training Strategy: SFT teaches the structured output format, while GRPO jointly improves timestamp accuracy and caption quality for the challenging OmniDenseCaptioning task.The model first learns next-token prediction over timestamped, multidimensional captions before reinforcement optimization.
  • Training Strategy: The framework addresses weak temporal-learning signals because timestamp tokens comprise only 0.7% of outputs and SFT can overfit scene-count distributions.GRPO is introduced to address both token imbalance and limited generalization.
  • Reward Design: GRPO combines format, length, timestamp, and time-aware caption rewards into a weighted objective.The timestamp reward averages F1 across IoU thresholds {0.3, 0.5, 0.7, 0.9}, while the caption reward uses SodaM.

5. Experiments

Experiments show that TimeChat-Captioner performs strongly on dense captioning and transfers to audio-visual reasoning, temporal grounding, and open-source data generation. Ablations indicate that increased data and SodaM-guided GRPO improve performance, while DP merging addresses evaluation bias from differing segment granularities.

  • OmniDCBench results: 35.0 SodaM score makes TimeChat-Captioner-GRPO state of the art for time-aware captioning, surpassing Gemini-2.5-Pro.The model ranks second for scene-boundary localization, behind Gemini-2.5-Pro, while outperforming other open-source baselines.
  • Audio-visual reasoning: 52.8 and 22.6 on Daily-Omni and World-Sense show strong cross-task transfer to caption-based audio-visual VideoQA.The model outperforms all open-source baselines despite being optimized primarily for temporal-aware dense captioning.
  • Temporal grounding: TimeChat-Captioner-GRPO outperforms established temporal-understanding models and Qwen2.5-Omni-7B across all Charades-STA evaluation metrics.The results support transfer from OmniDenseCaptioning training to downstream temporal grounding.
  • Caption quality as training data: 44.9 versus 44.4 Daily-Omni accuracy shows that captions from TimeChat-Captioner can supervise Qwen2.5-Omni-3B comparably to Gemini-2.5-Pro captions.The off-the-shelf baseline reaches only 7.9 accuracy under the same setup.
  • Ablations: Increasing SFT data from 20K to 40K raises the OmniDCBench score from 31.3 to 32.6 across benchmarks.The ablation reports consistent performance gains with larger supervised training data.
  • Ablations: Adding the SodaM reward improves temporal understanding and caption completeness, with 2K-sample GRPO outperforming scaling SFT data from 20K to 40K.The reward targets time-aware caption quality, while base rewards cover format, length, and temporal alignment.
  • Evaluation analysis: DP merging is needed for fair evaluation because hard one-to-one matching can reward incidental differences in predicted segment granularity.Both evaluated models over-segment relative to the 14.2-second ground-truth average, but their predicted segment durations differ.

6. Conclusions

The paper introduces OmniDenseCaptioning, its benchmark and SodaM metric, and a TimeChat-Captioner model trained with synthetic data and task-specific rewards. The model surpasses Gemini-2.5-Pro and generalizes to related omni-video understanding tasks while serving as an open-source captioning data engine.

  • OmniDenseCaptioning generates temporally aligned, multi-dimensional, and structurally rich video captions.
  • OmniDCBench and SodaM provide a human-annotated benchmark and tailored evaluation for this task.
  • TimeChat-Captioner, trained with synthetic audio-visual data and task-specific rewards, outperforms Gemini-2.5-Pro on the reported evaluations.
  • The model functions as a cost-effective, open-source captioning data engine that matches closed-source APIs for supervising downstream Omni-VideoLLMs.

Impact Statement

The paper identifies accessibility and education benefits from dense audiovisual captioning, alongside risks from pretrained-model biases and misuse for misinformation. It reports public academic data sources, documented limitations, and responsible-use licensing as safeguards.

  • Dense, temporally grounded audiovisual captioning may improve accessibility for impaired users and support video-based education.
  • Potential risks include inherited biases from pretrained models and misuse for misinformation.
  • The authors use publicly available academic datasets, document data sources and model limitations, and release resources under responsible-use licenses.

A. Limitations and Future Work.

The paper reports limitations from the 32K context window and limited generalization to hour-long videos, while proposing longer-form data and token compression as future directions. Supplementary SodaM studies examine judge robustness, human alignment, and evaluation design.

  • Limitations: The 32K context-window constraint limits training for lengthy video inputs and captions averaging 1K words.The authors state that extending the context window is essential for accommodating more frames and comprehensive captions.
  • Limitations: Generalization remains limited for varying video durations, particularly hour-long content.The current workaround divides long videos into approximately one-minute clips and captions them sequentially.
  • Future work: Future work targets more diverse long-form videos and efficient token compression to improve duration generalization, timestamp accuracy, and training cost.
  • SodaM validation: SodaM is re-evaluated with four LLM judges using fixed prompts and DP merging, with Kendall’s W measuring ranking agreement.
  • SodaM validation: W=0.925 indicates highly consistent system rankings across judges, with the GRPO model ranking first under every judge.

B.2. Human Alignment: SodaM vs. Traditional Captioning Metrics

The study evaluates whether SodaM aligns with human preferences better than traditional captioning metrics for lengthy, multi-dimensional captions. SodaM shows stronger case-level and system-level agreement, while DP merging is presented as necessary for fair evaluation.

  • Human alignment: SodaM reaches 77.9% case-level agreement with human pairwise preferences, exceeding CIDEr, METEOR, and SODA c.The comparison uses 122 non-tie judgments from 129 blind pairwise evaluations.
  • Human alignment: At the system level, SodaM variants achieve Pearson r>0.93, whereas traditional metrics yield r=0.17–0.55 without statistical significance.The Pearson analysis covers five systems and is treated as supportive because of the small sample size.
  • DP merging: Without DP merging, fine-grained models predicting shorter segments are systematically and unfairly under-rated.The appendix motivates DP merging as a safeguard against evaluation errors caused by mismatched segment granularity.
  • DP merging: With DP merging, the SodaM gap widens to +9.7, matching the 75% human preference for Qwen3-Omni; without merging, it shrinks to +1.2.Hard-matching F1 and mIoU additionally penalize pathological over-segmentation.
  • Overall findings: The combined studies report judge robustness, strong human alignment, and DP merging as necessary for fair evaluation rather than a source of bias.The reported safeguards include Kendall’s W=0.925, 77.9% case-level agreement, and Pearson r>0.93.

C. Additional Experimental Results

Additional experiments examine reward-weight sensitivity, SFT duration, and the composition and statistics of the TimeChatCap-42K training data. The results indicate limited sensitivity to coherence-weight changes and improved performance with longer SFT training.

  • Reward weights: Varying the coherence reward weight from 1.0 to 1.5 changes F1, mIoU, and SodaM by less than 0.5%.The ablation reports marginal differences across all three metrics.
  • Training-data statistics: 73.9% of TimeChatCap-42K videos fall within 50–60 seconds, with annotations averaging 877 words per video across six dimensions.The dataset’s average segment length is 10.04 seconds.
  • SFT training: Extending SFT training from 1 to 2 epochs yields consistent improvements across all evaluation metrics.The results are interpreted as evidence that the task requires sufficient SFT training to learn structured, multi-dimensional outputs.

D.1. Training Data Construction

TimeChatCap-42K is constructed through a synthetic, staged pipeline that produces temporally segmented and richly structured audio-visual annotations. The paper also describes human review procedures for the OmniDCBench benchmark and situates the method among several baseline categories.

  • Synthetic data pipeline: A three-stage pipeline synthesizes high-quality training samples for OmniDenseCaptioning.The paper provides detailed prompts for the pipeline stages.
  • Dataset statistics: TimeChatCap-42K annotations average 877 words per video across six dimensions, with average segments lasting 10.04 seconds.Most videos, 73.9%, have durations between 50 and 60 seconds.
  • Human annotation: OmniDCBench samples are annotated by human experts, reviewed by at least one additional annotator, and supported by an intuitive interface with real-time feedback.The annotation process is described as spanning approximately one month.
  • Training procedure: The training procedure uses supervised fine-tuning followed by reinforcement learning via Group Relative Policy Optimization.The implementation description specifies 2 SFT epochs on 40K samples and a subsequent GRPO phase using 2K samples.
  • Baselines: Evaluation compares the proposed method with closed-source Gemini systems and open-source general-purpose omni-modal models.The baseline taxonomy includes Gemini-2.5-Pro, Gemini-2.5-Flash, Qwen2.5-Omni, Qwen3-Omni, MiniCPM-o-2.6, and video-SALMONN-2.

F. Additional Qualitative Analysis

Qualitative analysis shows that TimeChat-Captioner aligns with ground truth across the six annotation dimensions and preserves fine-grained scene details. Comparisons report hallucination or missed events in competing systems, while the accompanying prompts operationalize annotation and evaluation.

  • Qualitative comparison: TimeChat-Captioner achieves fine-grained ground-truth alignment across all six annotation dimensions.The qualitative analysis covers events, background, camera state, shot editing, dialogue, and acoustic information.
  • Events: The model captures named characters, appearance, temporally continuous actions, and fine-grained behaviors such as searching through a car window.These examples illustrate detailed event narration and scene continuity.
  • Audio-visual structure: Its descriptions also represent dialogue, camera positioning, shot transitions, and the narrative purpose of interior shots.The examples connect conversational content with visual narrative and editing choices.
  • Comparative errors: Gemini-2.5-Pro misidentifies the male driver as a woman, while Qwen-3-Omni focuses on irrelevant background elements instead of the main conversation.The reported errors distort scene semantics or omit the central event.
  • Prompt design: The training-data and evaluation prompts respectively guide multi-dimensional annotation generation and checklist-based verification of atomic keypoints.The prompts cover detailed events, camera state, background, storyline, shooting style, speech, acoustics, and six-dimensional recall.
Loading 2602.08711v3…