PaperlayerView exact revision on arXiv

Source-linked AI summary

Tarsier: Recipes for Training and Evaluating Large Video Description Models

Jiawei Wang, Liping Yuan, Yuchen Zhang, Haomiao Sun

arXiv:2407.00634v2cs.CVcs.LG

TL;DR

Fine-grained video description requires comprehensive, faithful coverage of complex events and poses a non-trivial evaluation problem. Tarsier addresses this with a simple frame-encoding architecture, two-stage training, and DREAM-1K with AutoDQ, achieving stronger description performance than existing open-source models and competitive results against proprietary systems.

  • Problem

    Fine-grained video description must cover notable events faithfully despite subtle motions, complex scenes, and the absence of a unique reference answer.

  • Method

    Tarsier combines separate CLIP-ViT frame encoding with an LLM and trains it through multi-task pre-training followed by instruction tuning; DREAM-1K and AutoDQ evaluate detailed descriptions.

  • Results

    Tarsier significantly outperforms existing open-source models and achieves state-of-the-art results across multi-choice VQA, open-ended VQA, and zero-shot video captioning benchmarks.

  • Takeaways & Limitations

    Tarsier demonstrates that a simple architecture paired with extensive multi-task pre-training and human-annotated, multi-grained descriptions can support strong generalist video understanding.

Abstract

from arXiv · show

Generating fine-grained video descriptions is a fundamental challenge in video understanding. In this work, we introduce Tarsier, a family of large-scale video-language models designed to generate high-quality video descriptions. Tarsier employs CLIP-ViT to encode frames separately and then uses an LLM to model temporal relationships. Despite its simple architecture, we demonstrate that with a meticulously designed two-stage training procedure, the Tarsier models exhibit substantially stronger video description capabilities than any existing open-source model, showing a $+51.4\%$ advantage in human side-by-side evaluation over the strongest model. Additionally, they are comparable to state-of-the-art proprietary models, with a $+12.3\%$ advantage against GPT-4V and a $-6.7\%$ disadvantage against Gemini 1.5 Pro. When upgraded to Tarsier2 by building upon SigLIP and Qwen2-7B, it further improves significantly with a $+4.8\%$ advantage against GPT-4o. Besides video description, Tarsier proves to be a versatile generalist model, achieving new state-of-the-art results across nine public benchmarks, including multi-choice VQA, open-ended VQA, and zero-shot video captioning. Our second contribution is the introduction of a new benchmark -- DREAM-1K (https://tarsier-vlm.github.io/) for evaluating video description models, consisting of a new challenging dataset featuring videos from diverse sources and varying complexity, along with an automatic method specifically designed to assess the quality of fine-grained video descriptions. We make our models and evaluation benchmark publicly available at https://github.com/bytedance/tarsier.

1 Introduction

Tarsier targets fine-grained video description, requiring comprehensive and faithful coverage of complex events while addressing difficult evaluation. It combines a simple LVLM architecture with two-stage training and introduces DREAM-1K and AutoDQ for evaluation.

  • Motivation: Fine-grained video description requires long, free-form answers that comprehensively and faithfully cover notable video events.The challenge increases with subtle movements, quick motions, multiple events, subjects, and shots.
  • Motivation: Existing open-source models lag far behind proprietary models on fine-grained video description.
  • Tarsier: Tarsier uses a CLIP-ViT encoder, projection layer, and LLM trained through multi-task video-to-text pre-training followed by instruction tuning.Both stages use high-dynamic videos with accurately matched text.
  • Evaluation: DREAM-1K provides diverse, complex videos, while AutoDQ automatically evaluates fine-grained descriptions through event extraction and entailment.The benchmark includes videos from diverse sources and complexity levels.
  • Results: +52% advantage in human side-by-side evaluation over the strongest open-source model; Tarsier-34B is favored over GPT-4V by +12.3% but trails Gemini 1.5 Pro by −6.7%.Tarsier-34B also achieves strong automatic evaluation results and state-of-the-art performance across multiple public video-understanding benchmarks.

2 Related Work

Prior video-language models connect visual encoders and language decoders through varied architectures, but common video description benchmarks remain too simple to stress-test open-world, fine-grained understanding. Tarsier instead uses a simple projection-based connection with larger, more diverse pre-training and high-quality fine-grained instruction data.

  • Video-Language Modeling: Video-language models combine visual encoders with text decoders, using either separate-frame encoding or architectures that jointly model spatial-temporal relationships.
  • Video-LLMs: Video-LLMs commonly connect pre-trained video encoders to LLMs through adapters or projection layers and fine-tune them on instruction data.Tarsier adopts a simple projection-layer architecture.
  • Video-LLMs: Tarsier differs from prior systems by using larger, more diverse pre-training data and high-quality fine-grained description data for fine-tuning.
  • Video Description: Standard video captioning benchmarks often use simple single-action videos and therefore do not fully stress-test video understanding.Earlier systems can score highly on specific datasets but often generalize poorly to open-world videos.

3 The Tarsier model

Tarsier separately encodes video frames with CLIP-ViT, projects them into the LLM token space, and relies on the LLM for autoregressive generation and inter-frame reasoning. Its two-stage recipe combines broad multi-task pre-training with human-annotated instruction tuning focused on complex, multi-grained descriptions.

  • Model Architecture: Each frame is encoded separately by CLIP-ViT, mapped into the LLM token space through an MLP, concatenated with separators, and sent to the LLM for autoregressive generation.
  • Model Architecture: The LLM captures inter-frame relationships, while the CLIP-ViT and MLP remain frozen during training.
  • Training Strategy: Tarsier uses multi-task pre-training followed by instruction tuning on moderate-scale, human-annotated data.
  • Pre-training: Pre-training unifies video captioning, question answering, action recognition, multi-image understanding, and text generation as text-generation tasks.
  • Instruction Tuning: Instruction tuning includes 100K complex movie clips annotated with descriptions spanning different levels of granularity.Additional tasks cover camera motions and creative writing.

4 Evaluation of Video Description Models

This section introduces DREAM-1K, a challenging benchmark for fine-grained video description, and AutoDQ, an interpretable automatic evaluation method. It also illustrates that Tarsier descriptions capture more events and hallucinate less than existing models.

  • DREAM-1K: DREAM-1K contains 1,000 diverse video clips with detailed annotations designed to evaluate complex video descriptions.The dataset includes videos from five sources, multiple events, subjects, and shots, with fine-grained human descriptions.
  • DREAM-1K: Each DREAM-1K clip includes dynamic content that cannot be identified from a single frame alone.Videos average 8.9 seconds, 6.3 events, 2.2 subjects, 1.9 shots, and 59.3 annotated words.
  • Qualitative comparison: Tarsier descriptions capture more events and exhibit fewer hallucinations than existing models in the illustrated example.The figure compares descriptions for a video containing six actions.
  • Evaluation motivation: Traditional n-gram metrics are inadequate for long descriptions because valid descriptions can have little lexical overlap with a reference.Manual quality scoring is also costly and subjective, motivating an interpretable automatic alternative.
  • AutoDQ: AutoDQ extracts events from reference and generated descriptions, then uses entailment to compute recall, precision, and F1.Recall measures reference events entailed by the model description, while precision measures model events entailed by the reference.

5 Experiments

The experiments evaluate Tarsier on DREAM-1K, human side-by-side comparisons, and zero-shot public video-understanding benchmarks. Tarsier consistently outperforms open-source systems and achieves competitive results against proprietary models.

  • DREAM-1K automatic evaluation: Tarsier models achieve significantly higher AutoDQ F1, precision, and recall than all open-source models across DREAM-1K categories.Tarsier-34B also exceeds GPT-4V and Gemini 1.5 Pro in overall automatic evaluation, while Tarsier2-7B reaches 40.1 overall F1 and slightly exceeds GPT-4o.
  • DREAM-1K automatic evaluation: F1 scores generally decline as video complexity increases, but Tarsier-34B remains ahead of open-source models and GPT-4V across complexity levels.Its scores remain comparable to Gemini 1.5 Pro throughout the complexity range.
  • Human evaluation: Human comparisons on 300 DREAM-1K videos show Tarsier-34B preferred over PLLaVA and GPT-4V, while slightly trailing Gemini 1.5 Pro.Tarsier2-7B has a +4.8% advantage over GPT-4o and ties it in more than half of cases.
  • Other public benchmarks: Tarsier is evaluated zero-shot on multi-choice VQA, open-choice VQA, and video-captioning benchmarks without using their training sets.The evaluation uses 8 frames per video by default, except for 16 frames on EgoSchema.
  • Other public benchmarks: Tarsier establishes state-of-the-art results across the reported multi-choice VQA, open-choice VQA, and zero-shot video-captioning benchmarks.Tarsier-34B leads existing models on MSVD-QA, ActivityNet-QA, and TGIF-QA, while the Tarsier family leads the reported zero-shot captioning comparisons.

5.3 Ablation studies

Ablation studies show that Tarsier’s performance depends strongly on its two-stage training design and training data, with removing or replacing components reducing results on DREAM-1K and MVBench.

  • Ablation studies: Each ablation modification causes a significant performance decline on both DREAM-1K and MVBench.The tested changes remove training stages or replace the paper’s pre-training and instruction-tuning data.
  • Ablation studies: Removing stage 1 pre-training lowers MVBench accuracy by 16.0 absolute points.The study also reports a notable decrease in DREAM-1K scores.
  • Ablation studies: Replacing the pre-training dataset with WebVid-10M lowers MVBench accuracy by 14.2 absolute points.This substitution also reduces DREAM-1K performance.

6 Conclusion and Future Work

The paper concludes that Tarsier outperforms existing open-source video-description models and attributes this performance to multi-task pre-training and human-annotated, multi-grained description data. Future work targets larger data and models and improved instruction following.

  • Conclusion: Tarsier outperforms existing open-source video-description models in both automatic and human evaluations.The conclusion summarizes the family’s performance across the paper’s evaluation settings.
  • Conclusion: The ablations attribute Tarsier’s strong performance to extensive multi-task pre-training and fine-tuning on human-annotated, multi-grained video-description data.These training factors are identified as contributors to the reported performance.
  • Future work: Future directions include scaling pre-training data, scaling the visual encoder and LLM together, and improving complex instruction following.The paper presents these as areas for further exploration.

A Training hyperparameters

The appendix presents the training hyperparameters used for Tarsier’s two training stages.

  • Training hyperparameters: The paper provides a table of training hyperparameters for Stage 1 and Stage 2.The cited appendix passages identify the hyperparameter table but do not specify individual settings.

B AutoDQ prompts

AutoDQ uses ChatGPT prompts to extract atomic dynamic events from video descriptions and then classify each event's relationship to the description.

  • Event extraction: AutoDQ extracts at most 10 key events from a video description.Events must describe actions, motion, or movement rather than static information.
  • Event extraction: Each event is a brief, atomic sentence with a subject and predicate, while scene cuts and camera motions are excluded.The prompt also discourages repeated events and unnecessary appearance descriptions.
  • Event extraction: The extraction prompt returns events as a Python dictionary whose events value is a list of strings.The prompt requires no additional explanatory output.
  • Entailment prediction: The entailment prompt takes a video description and an event list, then outputs each event, relationship, and reason in JSON format.The input placeholders represent the description and previously extracted events.
  • Entailment prediction: AutoDQ classifies each extracted event as entailment, neutral, or contradiction relative to the video description.Entailment means the description supports the event, contradiction means it conflicts with some detail, and neutral means neither applies.

C Video description prompts

The DREAM-1K experiments use prompts that ask models to describe video subjects, actions, background scenes, and especially dynamic events, with model-specific instructions.

  • Default prompt: The default DREAM-1K prompt asks models to describe the background, characters, and actions in the provided video.This prompt is presented as the default for the experiments.
  • Model-specific prompts: GPT-4V receives 8 uniformly sampled frames and is asked to describe the video rather than individual images.Its prompt emphasizes main subjects, actions, and the background scene while excluding feelings and atmosphere.
  • Model-specific prompts: Gemini 1.5 Pro is asked to describe the video in one paragraph, mainly focusing on dynamic events and excluding feelings or atmosphere.The prompt is explicitly centered on dynamic events.
  • Model-specific prompts: PLLaVA-34B is instructed to produce a precise, detailed paragraph based on key information from the overall video.Its instructions advise avoiding over-detailed colors and object counts and omitting uncertain information.
  • Model-specific prompts: LLaVA-NeXT-Video is prompted to provide a detailed video description focused on main subjects, their actions, and background scenes.The prompt is identified as recommended by a cited reference.

D Human evaluation details

The human evaluation compares anonymized model-description pairs using judgments of helpfulness for a blind reader, emphasizing accuracy, comprehensiveness, and dynamic events.

  • Evaluation setup: The evaluation randomly samples 300 DREAM-1K videos and uses three experienced annotators.For each model pair, each annotator evaluates 100 description pairs with randomly swapped ordering.
  • Evaluation setup: Annotators label each pair as Description A is better, Description B is better, or Same quality.The descriptions are presented without identifying which model produced them.
  • Annotation criteria: Annotators judge which description better helps a blind person understand the video's content.The guide frames the decision around helpfulness, accuracy, and comprehensiveness.
  • Annotation criteria: When one dimension is equal, the description that is more comprehensive or more accurate is preferred.The guide separately specifies comprehensiveness when accuracy is equal and accuracy when comprehensiveness is equal.
  • Annotation criteria: When both descriptions have accuracy and comprehensiveness flaws, annotators judge which flaw has less impact on understanding the main content.The guide permits subjective judgment and common sense in this case.
  • Annotation criteria: Dynamic events receive priority, while static details earn no extra credit unless they help explain key events.Annotators can ignore sounds because the model does not accept audio input.
  • Annotation procedure: Annotators are asked to make a meaningful comparison and use Same quality as little as possible.
Loading 2407.00634v2…