Source-linked AI summary
AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, Liang Wang, Tieniu Tan
TL;DR
Audiovisual captioning needs semantically rich descriptions that align visual and auditory events over time, but existing approaches often underuse audio or decouple the modalities. AVoCaDO combines temporally aligned SFT data with GRPO rewards for audiovisual captioning, and it outperforms existing open-source audiovisual models while remaining competitive in visual-only settings.
Problem
Existing captioning approaches are predominantly vision-centric or decouple audio and visual captions, limiting joint temporal reasoning over audiovisual content.
Method
AVoCaDO uses a two-stage post-training pipeline: SFT on 107K high-quality audiovisual captions followed by GRPO rewards for temporal coherence, dialogue accuracy, caption length, and repetition collapse.
Results
AVoCaDO outperforms existing open-source audiovisual models and surpasses Gemini-2.5-Pro on UGC-VideoCap, while achieving competitive performance under visual-only settings.
Takeaways & Limitations
Joint audiovisual temporal alignment and holistic reward design support semantically rich captions across audiovisual benchmarks and visual-only evaluation settings.
Abstract
from arXiv · showhide
Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present AVoCaDO, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities. We propose a two-stage post-training pipeline: (1) AVoCaDO SFT, which fine-tunes the model on a newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) AVoCaDO GRPO, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC and DREAM-1K benchmark under visual-only settings.
1 INTRODUCTION
Audiovisual captioning must jointly reason over visual and auditory events because decoupled captions can miss their temporal relationships. AVoCaDO addresses this need with a two-stage post-training pipeline and achieves strong audiovisual and visual-only results.
- Existing captioning approaches are predominantly vision-centric, despite dialogues, voiceovers, and music supplying essential semantic cues for holistic video understanding.The paper identifies separate audio-caption generation followed by concatenation as a common but decoupled workaround.
- Jointly processing audio and visual inputs improves pilot question-answering accuracy by 15.8% on average and by 27.8% for AV Event Alignment versus concatenated captions.The pilot compares separately generated concatenated captions with jointly generated time-aligned captions.
- A two-stage pipeline combines 107K high-quality audiovisual caption pairs for SFT with GRPO rewards targeting temporal coherence, dialogue accuracy, caption length, and repetition collapse.The SFT dataset emphasizes temporal alignment between visual and audio events.
- AVoCaDO integrates visual and auditory events in temporal synchrony to produce semantically rich, temporally aligned captions.The model is built on Qwen2.5-Omni and is designed specifically for audiovisual captioning.
- AVoCaDO outperforms all existing open-source audiovisual models and surpasses Gemini-2.5-Pro on UGC-VideoCap, while remaining competitive under visual-only settings.The model is evaluated across audiovisual captioning benchmarks and visual-only settings.
2 RELATED WORKS
Prior video captioning work has largely emphasized vision, while audiovisual models and reinforcement-learning methods have not been tailored comprehensively to audiovisual captioning. AVoCaDO instead uses holistic rewards targeting temporal coherence, dialogue accuracy, caption length, and collapse.
- Most existing VideoLLM captioners remain vision-centric, while audiovisual VideoLLMs generally incorporate both modalities without specific optimization for captioning.The related work contrasts vision-focused systems with audiovisual systems that are not captioning-specialized.
- Prior reinforcement-learning methods apply rule-based rewards to general video understanding or isolated visual-captioning goals, leaving detailed audiovisual caption verification challenging.Long descriptions remain prone to visual omissions and hallucinations, complicating verification.
- 2.2 REINFORCEMENT LEARNING FOR VIDEOLLMS: AVoCaDO distinguishes itself from prior work through holistic rewards tailored to audiovisual captioning across temporal coherence, dialogue accuracy, caption length, and repetition collapse.The design targets multiple fine-grained caption-quality dimensions rather than a single visual-only aspect.
3 AVOCADO
AVoCaDO uses a two-stage post-training pipeline combining supervised fine-tuning on curated audiovisual captions with GRPO guided by complementary rewards. The pipeline targets temporally aligned multimodal descriptions, dialogue fidelity, coverage, and controlled caption length.
- 3.1 AVoCaDO SFT: AVoCaDO sequentially applies SFT and GRPO, using Qwen2.5-Omni-7B as a base model with interleaved visual-audio token alignment.SFT trains on 107K curated audiovisual video-caption pairs, while GRPO further optimizes caption quality.
- 3.1 AVoCaDO SFT: The SFT dataset contains 107K high-quality audiovisual video-caption pairs curated from diverse video sources.The collected videos emphasize rich auditory elements and complex audiovisual interactions.
- 3.1 AVoCaDO SFT: A two-stage captioning strategy first generates modality-specific captions, then fuses them with the original video to produce semantically rich, temporally aligned captions.This strategy addresses omissions that can occur when jointly generating audiovisual captions directly.
- 3.2 AVoCaDO GRPO: GRPO optimizes captions with checklist, dialogue, and length-regularized rewards targeting audiovisual coverage, dialogue accuracy, and repetition control.It samples grouped responses, computes their rewards and advantages, and optimizes the policy objective without a separate critic model.
- 3.2.2 Checklist-Based Reward: The checklist reward evaluates coverage of fine-grained keypoints spanning cross-modal logic, actions, auditory elements, spatio-temporal cinematography, and static entities.Ground-truth captions are decomposed into structured keypoint inventories, and a judge model scores whether generated captions mention each keypoint.
- 3.2.3 Dialogue-Based Reward: The dialogue reward matches generated and reference dialogue by content, verifies speaker consistency, and combines these signals through an F1-based score.The content-matching problem is formulated as a dynamic-programming search analogous to Longest Common Subsequence, with similarity threshold γ = 0.6.
- 3.2.4 Length-Regularized Reward: The length-regularized reward encourages complete captions while penalizing excessive length to mitigate repetition collapse and improve inference efficiency.Its thresholds are τ1 = 2048 and τ2 = 4096.
4 EXPERIMENTS
AVoCaDO is evaluated on direct captioning, QA-based caption assessment, visual-only captioning, and post-training ablations. It achieves strong benchmark performance, while ablations attribute gains to both SFT data construction and tailored GRPO rewards.
- 4.2.1 DIRECT CAPTION EVALUATION: AVoCaDO achieves state-of-the-art performance among open-source models on both video-SALMONN-2 and UGC-VideoCap.The comparison uses direct caption-quality metrics, with Table 1 reporting the benchmark results.
- 4.2.1 DIRECT CAPTION EVALUATION: AVoCaDO balances comprehensiveness and accuracy, outperforming all open-source models in both Video-SALMONN-2 Total and UGC-VideoCap average score.Some baselines have lower hallucination rates but produce overly brief captions, increasing miss rates and weakening UGC-VideoCap performance.
- 4.2.2 QA-BASED CAPTION EVALUATION: AVoCaDO outperforms open-source and Qwen3-Omni baselines by 20.2% on Daily-Omni and 7.5% on Worldsense.These benchmarks evaluate caption quality through QA answered from textual captions, with unanswered questions counted as incorrect.
- 4.2.2 QA-BASED CAPTION EVALUATION: AVoCaDO also demonstrates competitive performance on the VDC Detailed subset and DREAM-1K in visual-only settings.These benchmarks specifically measure captioning performance for videos evaluated without audio.
- 4.2.3 ABLATION STUDIES: SFT improves benchmark scores, dialogue quality, and repetition-collapse reduction, while GRPO rewards further target dialogue, audiovisual events, and caption length.The SFT gains are consistent across direct and QA-based evaluation; the ablation is summarized in Table 4.
- 4.2.3 ABLATION STUDIES: The dialogue reward raises dialogue F1 by over 2% on both benchmarks, and length regularization reduces repetition collapse while improving other metrics.Fine-tuning on the same 2K GRPO samples without the curated rewards provides no significant gains and degrades Video-SALMONN-2 performance.
5 CONCLUSION
AVoCaDO addresses audiovisual video captioning through temporal coordination of visual and auditory events. Its two-stage post-training improves audiovisual captioning and remains competitive in visual-only settings.
- AVoCaDO substantially outperforms existing open-source models on four audiovisual video captioning benchmarks.
- The two-stage pipeline combines temporally aligned supervised fine-tuning with reward-guided optimization for coherence, dialogue accuracy, caption length, and repetition collapse.
- AVoCaDO achieves competitive results on the VDC Detailed subset and DREAM-1K visual-only captioning benchmarks.
A DETAILS OF THE TRAINING DATA
The training data combines multiple video sources to provide diverse audiovisual content. These sources cover short-form posts, multi-shot videos, and videos with varied durations and styles.
- The training dataset draws from multiple sources to ensure diverse audiovisual content.
- TikTok-10M contributes 24K selected videos representing short-form content with varied visual styles, background music, voiceovers, and themes.
- Shot2Story contributes 20K multi-shot videos, enabling the model to capture events within shots and associate them across shots.
- ShortVideo contributes 18K randomly selected videos spanning durations from under 30 seconds to over 5 minutes.
B DETAILS OF BENCHMARKS
The benchmarks evaluate audiovisual caption quality across diverse videos and tasks. Evaluation uses judge models to identify errors, score caption dimensions, or measure downstream question-answering effectiveness.
- The benchmark suite includes video-SALMONN-2, UGC-VideoCap, Daily-Omni, and additional temporally synchronized audiovisual clips and tasks.
- video-SALMONN-2 evaluates missing, incorrect, and hallucinated events across 483 videos from 14 domains.
- UGC-VideoCap scores captions on visual, audio, and detail dimensions for 1,000 short TikTok videos containing meaningful audio segments.
- Daily-Omni and the temporally synchronized clips assess caption utility through judge-model question answering over diverse downstream tasks.
C IMPLEMENTATION DETAILS
Implementation uses separate training settings for supervised fine-tuning and GRPO while keeping the video and audio encoders frozen. Video sampling, token limits, and hardware are fixed for training and evaluation.
- AVoCaDO SFT runs for 2 epochs with batch size 128 and learning rate 2 × 10^-5, while GRPO runs for 1 epoch with batch size 64 and learning rate 1×10^-5.
- GRPO samples 8 responses per query at temperature 1.0 and uses KL-divergence regularization coefficient β = 0.04.
- The video and audio encoders remain frozen throughout training.
- Inputs are sampled at 2 fps, frames are limited to 512 × 28 × 28 pixels, and total video tokens are constrained by the 32K-token context window.
D.1 ANALYSIS OF THE AUDIOVISUAL VIDEO CAPTION GENERATION BY GEMINI
Direct audiovisual caption generation can omit information from either modality, whereas the two-stage approach provides more comprehensive coverage and was adopted for SFT data construction.
- The two-stage strategy provides more comprehensive audiovisual coverage than direct caption generation.
- Direct caption generation tends to omit information from either the audio or visual modality.
- The authors adopted two-stage captioning to ensure higher-quality SFT data.
D.2 ANALYSIS OF THE THRESHOLDS IN LENGTH-REGULARIZED REWARD
The length-regularized reward uses thresholds of 2,048 and 4,096 tokens, balancing typical caption lengths against the model’s context and video-duration limits.
- Figure 5 analyzes caption-token-length distributions across video durations to support the threshold choices.
- 4,096 tokens is the upper threshold, chosen because captions for videos up to 100 seconds rarely exceed 3,982 tokens.A 100-second input approaches the 32K context limit when audio, video, prompts, and generated captions are combined.
- Figure 8 compares direct captioning with the proposed two-stage approach, distinguishing added audio-related and visual-related information by color.
- The dataset is restricted to videos of 100 seconds or less to prevent context overflow and truncated captions.At 100 seconds, audio consumes 2,500 tokens and maximum-resolution video consumes 25,600 tokens.
- 2,048 tokens is the lower threshold, set above the mean caption length of 1,437 tokens for videos under 100 seconds.Outputs between 2,048 and 4,096 tokens receive a linearly decreasing length reward.
E ADDITIONAL QUALITATIVE RESULTS
Additional qualitative results show AVoCaDO producing more accurate, temporally synchronized, dialogue-faithful, and semantically comprehensive audiovisual captions than the compared models.
- Caption Comparison across Models: video-SALMONN-2 misrecognizes dialogue and temporal order, while UGC-VideoCaptioner omits dialogue and adds redundant descriptions.
- Caption Comparison across Models: In another comparison, video-SALMONN-2 misaligns audio and visual chronology, misidentifies speaker gender, and misses narration, whereas UGC-VideoCaptioner neglects spoken content.
- Caption Comparison across Models: AVoCaDO accurately synchronizes audiovisual events, faithfully transcribes dialogue, and maintains strong semantic coverage in both qualitative examples.
- Prompts for SFT Caption Data: The SFT-data prompts separately generate video-frame and audio captions before fusing them into a coherent narrative.
- Prompts for Reward Evaluation: The reward prompts decompose captions into audiovisual keypoints, assess keypoint accuracy, extract dialogues, check speaker consistency, and answer questions from captions.
- Prompts for SFT Caption Data: The fusion prompt preserves every sentence from both modality captions while allowing temporal reordering for a logically aligned narrative.