PaperlayerView exact revision on arXiv

Source-linked AI summary

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, Yuan Lin

arXiv:2501.07888v3cs.CVcs.AI

TL;DR

Detailed video description remains a central challenge for video understanding, especially as existing systems and datasets have limited scale, quality, or temporal detail. Tarsier2 addresses this with expanded video-text pre-training, fine-grained temporal supervision, and DPO-based preference optimization. The resulting 7B model outperforms leading proprietary and open-source systems on detailed description and achieves state-of-the-art results across diverse video-understanding benchmarks.

  • Problem

    Detailed video description remains challenging because existing video-text data and models have limited scale, quality, domain coverage, or temporal detail.

    Source: §1 · p. 3, §3.1 · p. 8

  • Method

    Tarsier2 uses 40 million video-text pairs, fine-grained frame-event alignment during supervised fine-tuning, and automatically generated preference data for DPO training.

    Source: §1 · p. 3, §3 · p. 6, §3.3 · p. 9

  • Results

    Tarsier2 outperforms proprietary and open-source models on detailed video description and sets new state-of-the-art results across video question-answering, hallucination testing, grounding, and embodied QA.

    Source: §1 · pp. 3, 5, §5 · p. 21

  • Takeaways & Limitations

    Tarsier2 demonstrates that scaling training-data volume and diversity, fine-grained temporal alignment, and DPO training support a versatile generalist video-language model.

    Source: §5 · p. 21

    Evaluation uses a frame-sampling strategy that may differ from official benchmark settings, including 128 rather than 768 frames for Video-MME.

    Source: §4.2.1 · p. 18

Abstract

from arXiv · show

We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM) designed for generating detailed and accurate video descriptions, while also exhibiting superior general video understanding capabilities. Tarsier2 achieves significant advancements through three key upgrades: (1) Scaling pre-training data from 11M to 40M video-text pairs, enriching both volume and diversity; (2) Performing fine-grained temporal alignment during supervised fine-tuning; (3) Using model-based sampling to automatically construct preference data and applying DPO training for optimization. Extensive experiments show that Tarsier2-7B consistently outperforms leading proprietary models, including GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K benchmark, Tarsier2-7B improves F1 by 2.8% over GPT-4o and 5.8% over Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6% performance advantage over GPT-4o and +24.9% over Gemini-1.5-Pro. Tarsier2-7B also sets new state-of-the-art results across 15 public benchmarks, spanning tasks such as video question-answering, video grounding, hallucination test, and embodied question-answering, demonstrating its versatility as a robust generalist vision-language model.

1 Introduction

Tarsier2 is a 7B LVLM designed to improve detailed video description and broader video understanding through expanded data, fine-grained temporal alignment, and preference optimization. It outperforms leading proprietary and open-source models on description and multiple video understanding benchmarks.

  • Tarsier2 is a 7B LVLM that outperforms GPT-4o and Gemini-1.5-Pro on detailed video description and achieves state-of-the-art results across video question-answering benchmarks.Source: §1 · p. 3
  • 40 million video-text pairs expand pre-training from 11 million, with commentary videos supplying contextual information for video understanding.Experiments confirm that increasing pre-training data consistently improves performance.Source: §1 · p. 3
  • 150K fine-grained video descriptions pair events with their corresponding frames for supervised temporal alignment.Compared with traditional video-caption alignment training, this improves video-description accuracy and reduces hallucinations.Source: §1 · p. 3
  • Model-generated preference data, including corrupted-video negatives and AutoDQ filtering, supports DPO training after supervised fine-tuning.The authors report continued performance improvements over the supervised fine-tuning stage.Source: §1 · p. 5
  • Tarsier2-7B achieves leading video-description results against proprietary and open-source models and sets new state-of-the-art results in question-answering, hallucination testing, grounding, and embodied QA.Human side-by-side evaluations report advantages of +7.8% over GPT-4o, +12.3% over Gemini-1.5-Pro, and +51.4% over Tarsier-34B.Source: §1 · p. 5

2 Related Work

Video-LLM research has advanced through architectural improvements and larger video-text datasets, but data scale, quality, domain coverage, and description detail remain constrained. Tarsier2 addresses these limitations with 40 million diverse pairs, fine-grained descriptions, and a three-stage training process.

Video LLMs

  • Video-LLM research emphasizes visual representations, token resampling, and Vision Transformer integration, while Tarsier2 uses a visual encoder, visual adaptor, and LLM.Source: §2 · p. 5
  • Existing video-text datasets remain limited in quantity, quality, domain coverage, or description detail, with several open-source models trained on fewer than 5 million pairs.Source: §2 · pp. 5–6
  • Tarsier2 uses 40 million video-text pairs spanning diverse genres and adds 150K fine-grained video descriptions for supervised fine-tuning.Source: §2 · p. 6

Video Description

  • Earlier video-description datasets focused on single-sentence summaries, whereas recent work seeks more detailed outputs requiring richer annotations and temporal modeling.Source: §2 · p. 6
  • Tarsier2 builds commentary-based video-text pairs containing low-level actions and high-level plot information, then applies DPO after supervised fine-tuning to refine description quality.Source: §2 · p. 6

3 Approach

Tarsier2 uses large-scale, diverse pre-training data and a three-stage training process combining fine-grained temporal grounding with automated preference optimization. The approach targets more accurate, comprehensive, and human-like video descriptions.

Pre-training

  • Tarsier2 uses 40 million video-text pairs for pre-training, including public datasets and newly collected data spanning multiple video genres.The pre-training stage covers captioning, question answering, action recognition, grounding, image understanding, and text generation.Source: §3 · p. 6, §3.1 · pp. 7–8
  • Commentary videos provide both low-level atomic actions and high-level plot information, strengthening video-text alignment across granularities.The dataset pipeline filters and processes commentary and other online videos before using their text or generated captions.Source: §3.1 · pp. 7–8

Supervised fine-tuning

  • The first SFT stage annotates 150K video clips with descriptions aligned to the specific frames corresponding to each event.This fine-grained frame-event supervision is designed to improve event identification and reduce omissions in generated descriptions.Source: §3.2 · p. 8, Figure 4 · p. 9
  • A second SFT stage adds natural, human-like description data to reduce fragmentation caused by event-level grounding requirements.The SFT data combines 150K fine-grained video-description pairs with diverse instructions and refined descriptions.Source: §3.2 · p. 9

Direct Preference Optimization

  • Tarsier2 constructs DPO preference data by comparing responses from original and corrupted videos, then filters pairs using AutoDQ score differences.Corruptions include clip switching, reversing, cropping, and down-sampling; the policy is optimized with DPO after SFT.Source: §3.3 · pp. 9–10
  • The filtering threshold δ controls preference-pair validity using differences in AutoDQ recall and precision scores.This filtering step addresses cases where a corrupted-video response is not strictly worse than the original response.Source: §3.3 · p. 11

4 Experiments

Tarsier2-7B achieves strong results across detailed video captioning, short- and long-video question answering, temporal reasoning, hallucination evaluation, grounding, and embodied question answering. Ablations indicate that expanded pre-training data, fine-grained temporal alignment, supervised fine-tuning, and DPO contribute to these gains.

Video Captioning

  • 42.0% overall F1 on DREAM-1K surpasses GPT-4o’s 39.2%, with Tarsier2-7B leading open-source models in precision and recall across categories.The benchmark evaluates detailed descriptions of fine-grained actions and events.Source: §4.1.1 · p. 12
  • Human evaluators give Tarsier2 a slight advantage over GPT-4o and a larger advantage over Gemini in side-by-side description comparisons.Tarsier2 ties GPT-4o in 40% of cases and has a 45.6% versus 20.7% advantage against Gemini.Source: §4.1.1 · p. 13

Short-Video Question Answering

  • 71.5% on MVBench and 71.6% on PerceptionTest establish state-of-the-art short-video question-answering performance against proprietary and open-source models.These benchmarks assess visual comprehension and reasoning.Source: §4.1.2 · p. 13

Temporal Reasoning

  • 54.7% on TVBench, 42.0% on TOMATO, and 65.8%/38.0%/28.8% on Vinoground Text/Video/Group demonstrate strong temporal reasoning.Tarsier2 also reaches 75.3%/75.1%/80.6%/66.6% on TempCompass mc/yn/cm/cg tasks.Source: §4.1.2 · pp. 13, 15

Hallucination Evaluation

  • Tarsier2-7B achieves 67.0% on VideoHallucer and 84.6% on EventHallusion question answering, while surpassing GPT-4o by 7.1% on EventHallusion description matching.These evaluations target hallucinations in video understanding and generated descriptions.Source: §4.1.4 · pp. 15–16

5 Conclusion

Tarsier2 advances detailed video description and broad video understanding through larger, more diverse training data, fine-grained temporal alignment, and DPO training. The paper also identifies longer videos, real-time processing, and richer video-audio-text interaction as future directions.

  • Tarsier2 outperforms existing proprietary and open-source models in generating detailed and accurate video descriptions.Source: §5 · p. 21
  • Tarsier2 sets new benchmarks across a wide range of video understanding tasks.Source: §5 · p. 21
  • The model’s improvements are driven by scaling training-data volume and diversity, fine-grained temporal alignment, and DPO training.Source: §5 · p. 21
  • Future work targets longer videos, real-time processing, and richer interactions among video, audio, and text.Source: §5 · p. 21

A Training hyper-parameters

This appendix reports Tarsier2’s training hyper-parameters, including a layer-wise learning-rate decay used for visual-encoder training.

B Public datasets of pre-training stage

Tarsier2’s pre-training data combines public and in-house datasets, adding newly released, OCR-related, dynamic-event, and long-video resources.

C Annotation process for SFT data

Tarsier2’s SFT data were built through manual, fine-grained temporal annotation followed by GPT-4o-based instruction-sample generation. The process included quality control, re-annotation, and removal of data with potential copyright risks.

D Detail setting of DPO training

DPO training uses sampled preference pairs constructed with negative sampling and preference filtering, with fixed sampling settings and a 20K-pair training set.

D Detail setting of DPO training

  • The default DPO data construction uses negative sampling and preference-pair filtering.Both positive and negative samples use top p=0.7 and temperature=0.7.Source: D Detail setting of DPO training · p. 37
  • The re-annotation prompt for diverse instruction data is used in the second supervised fine-tuning stage.The prompt asks for sequential descriptions of changes in actions, environments, object states, attributes, and camera movements.Source: D Detail setting of DPO training · p. 38
  • The preference-pair filtering threshold is δ=0.3, and 20K pairs are randomly sampled for DPO training.The ablations remove negative sampling or filtering while keeping other parameters unchanged.Source: D Detail setting of DPO training · p. 38

E Detailed results of individual datasets at different stages

The paper reports detailed results by training stage and dataset, using F1, precision, and recall for DREAM-1K and accuracy for other benchmarks.

E Detailed results of individual datasets at different stages

F Tarsier2-Recap-585K Data Composition

Tarsier2-Recap-585K combines video-caption, action-recognition, and intent-recognition data, while staged training improves temporal fidelity and reduces hallucination.

F Tarsier2-Recap-585K Data Composition

H DREAM-1K cases

DREAM-1K qualitative cases compare Tarsier2-7B with other video-MLLMs across five video subsets, marking correct and incorrect descriptions for visual inspection.

H DREAM-1K cases

  • Figures 10–14 compare video descriptions from Tarsier2-7B, GPT-4o, Gemini-1.5-Pro, and LLaVA-Video-7B-Qwen2.The comparisons cover live-action, animation, stock, YouTube, and Shorts subsets.Source: H DREAM-1K cases · p. 41, Figure 10 · p. 42, Figure 11 · p. 43
  • Green marks identify correct descriptions of key objects, actions, and events, while red marks identify incorrect descriptions.The figure cases are intended for qualitative comparison against the raw videos.Source: H DREAM-1K cases · pp. 41–42

Related papers

Loading 2501.07888v3…