Source-linked AI summary
Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, Yuan Lin
TL;DR
Detailed video description remains a central challenge for video understanding, especially as existing systems and datasets have limited scale, quality, or temporal detail. Tarsier2 addresses this with expanded video-text pre-training, fine-grained temporal supervision, and DPO-based preference optimization. The resulting 7B model outperforms leading proprietary and open-source systems on detailed description and achieves state-of-the-art results across diverse video-understanding benchmarks.
Problem
Detailed video description remains challenging because existing video-text data and models have limited scale, quality, domain coverage, or temporal detail.
Method
Tarsier2 uses 40 million video-text pairs, fine-grained frame-event alignment during supervised fine-tuning, and automatically generated preference data for DPO training.
Results
Tarsier2 outperforms proprietary and open-source models on detailed video description and sets new state-of-the-art results across video question-answering, hallucination testing, grounding, and embodied QA.
Takeaways & Limitations
Tarsier2 demonstrates that scaling training-data volume and diversity, fine-grained temporal alignment, and DPO training support a versatile generalist video-language model.
Takeaways & Limitations
Evaluation uses a frame-sampling strategy that may differ from official benchmark settings, including 128 rather than 768 frames for Video-MME.
Abstract
from arXiv · showhide
We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM) designed for generating detailed and accurate video descriptions, while also exhibiting superior general video understanding capabilities. Tarsier2 achieves significant advancements through three key upgrades: (1) Scaling pre-training data from 11M to 40M video-text pairs, enriching both volume and diversity; (2) Performing fine-grained temporal alignment during supervised fine-tuning; (3) Using model-based sampling to automatically construct preference data and applying DPO training for optimization. Extensive experiments show that Tarsier2-7B consistently outperforms leading proprietary models, including GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K benchmark, Tarsier2-7B improves F1 by 2.8% over GPT-4o and 5.8% over Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6% performance advantage over GPT-4o and +24.9% over Gemini-1.5-Pro. Tarsier2-7B also sets new state-of-the-art results across 15 public benchmarks, spanning tasks such as video question-answering, video grounding, hallucination test, and embodied question-answering, demonstrating its versatility as a robust generalist vision-language model.
1 Introduction
Tarsier2 is a 7B LVLM designed to improve detailed video description and broader video understanding through expanded data, fine-grained temporal alignment, and preference optimization. It outperforms leading proprietary and open-source models on description and multiple video understanding benchmarks.
- Tarsier2 is a 7B LVLM that outperforms GPT-4o and Gemini-1.5-Pro on detailed video description and achieves state-of-the-art results across video question-answering benchmarks.
- 40 million video-text pairs expand pre-training from 11 million, with commentary videos supplying contextual information for video understanding.Experiments confirm that increasing pre-training data consistently improves performance.
- 150K fine-grained video descriptions pair events with their corresponding frames for supervised temporal alignment.Compared with traditional video-caption alignment training, this improves video-description accuracy and reduces hallucinations.
- Model-generated preference data, including corrupted-video negatives and AutoDQ filtering, supports DPO training after supervised fine-tuning.The authors report continued performance improvements over the supervised fine-tuning stage.
- Tarsier2-7B achieves leading video-description results against proprietary and open-source models and sets new state-of-the-art results in question-answering, hallucination testing, grounding, and embodied QA.Human side-by-side evaluations report advantages of +7.8% over GPT-4o, +12.3% over Gemini-1.5-Pro, and +51.4% over Tarsier-34B.
2 Related Work
Video-LLM research has advanced through architectural improvements and larger video-text datasets, but data scale, quality, domain coverage, and description detail remain constrained. Tarsier2 addresses these limitations with 40 million diverse pairs, fine-grained descriptions, and a three-stage training process.
- Video LLMs: Video-LLM research emphasizes visual representations, token resampling, and Vision Transformer integration, while Tarsier2 uses a visual encoder, visual adaptor, and LLM.
- Video LLMs: Existing video-text datasets remain limited in quantity, quality, domain coverage, or description detail, with several open-source models trained on fewer than 5 million pairs.
- Video LLMs: Tarsier2 uses 40 million video-text pairs spanning diverse genres and adds 150K fine-grained video descriptions for supervised fine-tuning.
- Video Description: Earlier video-description datasets focused on single-sentence summaries, whereas recent work seeks more detailed outputs requiring richer annotations and temporal modeling.
- Video Description: Tarsier2 builds commentary-based video-text pairs containing low-level actions and high-level plot information, then applies DPO after supervised fine-tuning to refine description quality.
3 Approach
Tarsier2 uses large-scale, diverse pre-training data and a three-stage training process combining fine-grained temporal grounding with automated preference optimization. The approach targets more accurate, comprehensive, and human-like video descriptions.
- Pre-training: Tarsier2 uses 40 million video-text pairs for pre-training, including public datasets and newly collected data spanning multiple video genres.The pre-training stage covers captioning, question answering, action recognition, grounding, image understanding, and text generation.
- Pre-training: Commentary videos provide both low-level atomic actions and high-level plot information, strengthening video-text alignment across granularities.The dataset pipeline filters and processes commentary and other online videos before using their text or generated captions.
- Supervised fine-tuning: The first SFT stage annotates 150K video clips with descriptions aligned to the specific frames corresponding to each event.This fine-grained frame-event supervision is designed to improve event identification and reduce omissions in generated descriptions.
- Supervised fine-tuning: A second SFT stage adds natural, human-like description data to reduce fragmentation caused by event-level grounding requirements.The SFT data combines 150K fine-grained video-description pairs with diverse instructions and refined descriptions.
- Direct Preference Optimization: Tarsier2 constructs DPO preference data by comparing responses from original and corrupted videos, then filters pairs using AutoDQ score differences.Corruptions include clip switching, reversing, cropping, and down-sampling; the policy is optimized with DPO after SFT.
- Direct Preference Optimization: The filtering threshold δ controls preference-pair validity using differences in AutoDQ recall and precision scores.This filtering step addresses cases where a corrupted-video response is not strictly worse than the original response.
4 Experiments
Tarsier2-7B achieves strong results across detailed video captioning, short- and long-video question answering, temporal reasoning, hallucination evaluation, grounding, and embodied question answering. Ablations indicate that expanded pre-training data, fine-grained temporal alignment, supervised fine-tuning, and DPO contribute to these gains.
- Video Captioning: 42.0% overall F1 on DREAM-1K surpasses GPT-4o’s 39.2%, with Tarsier2-7B leading open-source models in precision and recall across categories.The benchmark evaluates detailed descriptions of fine-grained actions and events.
- Video Captioning: Human evaluators give Tarsier2 a slight advantage over GPT-4o and a larger advantage over Gemini in side-by-side description comparisons.Tarsier2 ties GPT-4o in 40% of cases and has a 45.6% versus 20.7% advantage against Gemini.
- Short-Video Question Answering: 71.5% on MVBench and 71.6% on PerceptionTest establish state-of-the-art short-video question-answering performance against proprietary and open-source models.These benchmarks assess visual comprehension and reasoning.
- Temporal Reasoning: 54.7% on TVBench, 42.0% on TOMATO, and 65.8%/38.0%/28.8% on Vinoground Text/Video/Group demonstrate strong temporal reasoning.Tarsier2 also reaches 75.3%/75.1%/80.6%/66.6% on TempCompass mc/yn/cm/cg tasks.
- Hallucination Evaluation: Tarsier2-7B achieves 67.0% on VideoHallucer and 84.6% on EventHallusion question answering, while surpassing GPT-4o by 7.1% on EventHallusion description matching.These evaluations target hallucinations in video understanding and generated descriptions.
5 Conclusion
Tarsier2 advances detailed video description and broad video understanding through larger, more diverse training data, fine-grained temporal alignment, and DPO training. The paper also identifies longer videos, real-time processing, and richer video-audio-text interaction as future directions.
- Tarsier2 outperforms existing proprietary and open-source models in generating detailed and accurate video descriptions.
- Tarsier2 sets new benchmarks across a wide range of video understanding tasks.
- The model’s improvements are driven by scaling training-data volume and diversity, fine-grained temporal alignment, and DPO training.
- Future work targets longer videos, real-time processing, and richer interactions among video, audio, and text.
A Training hyper-parameters
This appendix reports Tarsier2’s training hyper-parameters, including a layer-wise learning-rate decay used for visual-encoder training.
- Training hyper-parameters are reported for pre-training, SFT-1&2, and DPO stages.
- A layer-wise learning-rate decay of 0.9 is applied during visual-encoder training.
- Table 12 presents the training hyper-parameters of Tarsier2.
B Public datasets of pre-training stage
Tarsier2’s pre-training data combines public and in-house datasets, adding newly released, OCR-related, dynamic-event, and long-video resources.
- The pre-training datasets contain approximately 20 million public and 20 million in-house data.
- The public-data collection adds newly released open-source and OCR-related datasets beyond most datasets inherited from Tarsier1.
- WebVid-10M contributes 2.9 million video-text pairs selected for likely dynamic events, alongside newer long-video understanding datasets.
C Annotation process for SFT data
Tarsier2’s SFT data were built through manual, fine-grained temporal annotation followed by GPT-4o-based instruction-sample generation. The process included quality control, re-annotation, and removal of data with potential copyright risks.
- Manual annotators described dynamic video content, including actions, events, scene transitions, and camera movements, while avoiding unnecessary static elements.
- Annotators mapped described dynamic information to corresponding frame numbers to provide fine-grained temporal grounding.
- Quality inspections sent substandard annotations back for re-annotation, while data with potential copyright risks were discarded.
- GPT-4o generated varied instruction-tuning samples from manual annotations and 16 uniformly sampled video frames.
D Detail setting of DPO training
DPO training uses sampled preference pairs constructed with negative sampling and preference filtering, with fixed sampling settings and a 20K-pair training set.
- D Detail setting of DPO training: The default DPO data construction uses negative sampling and preference-pair filtering.Both positive and negative samples use top p=0.7 and temperature=0.7.
- D Detail setting of DPO training: The re-annotation prompt for diverse instruction data is used in the second supervised fine-tuning stage.The prompt asks for sequential descriptions of changes in actions, environments, object states, attributes, and camera movements.
- D Detail setting of DPO training: The preference-pair filtering threshold is δ=0.3, and 20K pairs are randomly sampled for DPO training.The ablations remove negative sampling or filtering while keeping other parameters unchanged.
E Detailed results of individual datasets at different stages
The paper reports detailed results by training stage and dataset, using F1, precision, and recall for DREAM-1K and accuracy for other benchmarks.
- E Detailed results of individual datasets at different stages: Detailed ablation results are provided separately for pre-training, supervised fine-tuning, and DPO.The corresponding results appear in Tables 14, 15, and 16.
- E Detailed results of individual datasets at different stages: The recaptioning experiment is reported separately from the stage-specific ablations.Its results are listed in Table 17.
- E Detailed results of individual datasets at different stages: DREAM-1K uses F1, precision, and recall, while the other benchmarks use accuracy.This reporting convention applies to the detailed individual-dataset results.
F Tarsier2-Recap-585K Data Composition
Tarsier2-Recap-585K combines video-caption, action-recognition, and intent-recognition data, while staged training improves temporal fidelity and reduces hallucination.
- F Tarsier2-Recap-585K Data Composition: Tarsier2-Recap-585K primarily comprises video-caption datasets, supplemented by Kinetics-700, SSV2, and Oops.Kinetics-700 and SSV2 provide human-action clips, while Oops contributes rare actions and unexpected events.
- F Tarsier2-Recap-585K Data Composition: WebVid-10M contributes about 30% of Tarsier2-Recap-585K through a pre-filtered subset likely to contain dynamic events.The dataset composition table records the source partitions and sampled clips.
- F Tarsier2-Recap-585K Data Composition: Ego4D clips are randomly merged into multi-action videos, with 50K clips sampled for recaptioning.The merging process produces around 1M clips in total.
- F Tarsier2-Recap-585K Data Composition: Kinetics-700 and SSV2 contribute 50K and 10K sampled training clips, respectively.These datasets supply additional human-action videos for the recaptioning mixture.
- F Tarsier2-Recap-585K Data Composition: Temporal localization in the first SFT stage significantly reduces hallucination, but can subdivide events into finer actions.The second training stage improves description accuracy and output style.
H DREAM-1K cases
DREAM-1K qualitative cases compare Tarsier2-7B with other video-MLLMs across five video subsets, marking correct and incorrect descriptions for visual inspection.
- H DREAM-1K cases: Figures 10–14 compare video descriptions from Tarsier2-7B, GPT-4o, Gemini-1.5-Pro, and LLaVA-Video-7B-Qwen2.The comparisons cover live-action, animation, stock, YouTube, and Shorts subsets.
- H DREAM-1K cases: Green marks identify correct descriptions of key objects, actions, and events, while red marks identify incorrect descriptions.The figure cases are intended for qualitative comparison against the raw videos.