PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

Mukhtiar Ali, Harsh Dubey, Sugam Mishra, Chulwoo Pack

arXiv:2608.04302v1cs.CVcs.IRcs.MM

TL;DR

Existing video-language evaluations provide limited evidence about accurate long-form, paragraph-level description, especially for holistic stories and fine-grained details. CLIP-CC-Bench addresses this gap with a 200-clip movie dataset and a five-judge, coarse-to-fine semantic evaluation protocol, finding VideoLLaMA3 leading with perfect Borda consensus. The benchmark is controlled and reproducible, but remains bounded to English narrative-film content and does not explicitly model event ordering.

  • Problem

    The central gap is limited evaluation of long-form, paragraph-level video descriptions that capture both overarching stories and fine-grained details.

  • Method

    CLIP-CC-Bench uses 5 hours of movie content in 200 approximately 90-second clips with expert-written descriptions and evaluates outputs through five MTEB judges using coarse- and fine-grained semantic matching with Borda aggregation.

  • Results

    VideoLLaMA3 achieves perfect 80/80 Borda consensus and a 0.67 mean score, leading the evaluation of 17 VLMs.

  • Takeaways & Limitations

    The benchmark provides a practical and publicly available framework for reproducible long-form video-description evaluation.

  • Takeaways & Limitations

    The benchmark is limited to English narrative-film content, uses one reference per clip, and does not explicitly model temporal or causal event ordering.

Abstract

from arXiv · show

Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The evaluation suite employs an ensemble of five state-of-the-art LLM-based embedding models to increase reliability and mitigate single-model bias, and applies two complementary methodologies: (i) coarse-grained semantic matching and (ii) fine-grained semantic matching to compare model-generated descriptions against CLIP-CC-Bench references. Using this framework, we evaluate 17 state-of-the-art video-language models and report both their Borda-aggregated rankings and their average scores on CLIP-CC-Bench. We further quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability. We release standardized evaluation scripts, model outputs, and aggregation tools at https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench to support reproducibility. CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks.

1 Introduction

CLIP-CC-Bench addresses the challenge of evaluating long-form, paragraph-level video descriptions, where existing methods struggle to measure both overarching stories and fine-grained details. It introduces a bias-mitigated dataset and a multi-granular ensemble evaluation protocol for 17 video-language models.

  • Long-form video descriptions remain difficult to evaluate because models must capture overarching stories and fine-grained details.
  • Traditional lexical metrics often miss semantic understanding and discourse structure in paragraph-level descriptions.
  • Existing benchmarks generally assess isolated short clips, independent segments, or discrete questions rather than holistic paragraph-level description.
  • CLIP-CC-Bench contains 5 hours of movie content segmented into 200 approximately 90-second clips with expert-written descriptions excluding proper nouns and cultural references.
  • The framework combines coarse- and fine-grained semantic matching across five embedding judges, aggregates rankings with Borda scores, and evaluates 17 state-of-the-art VLMs.
  • All data, evaluation code, and baseline results are publicly released to support reproducible research.

2 Related Work

Related benchmarks and metrics leave paragraph-level video description insufficiently assessed: existing resources emphasize short clips, localized events, discrete recognition, lexical overlap, short-context similarity, or opaque judging. CLIP-CC-Bench targets this gap with transparent multi-granular semantic matching using long-context embedding models.

  • Video Description Benchmarks: Short-clip and dense-captioning datasets provide limited or independently evaluated temporal coverage, while video QA tests discrete recognition rather than coherent description generation.
  • Video Description Benchmarks: No existing benchmark offers holistic paragraph-level description assessment for minute-scale videos.
  • Evaluation Metrics: N-gram metrics provide reproducible lexical-overlap scores but miss semantically equivalent paraphrases, discourse structure, and narrative coverage.
  • Evaluation Metrics: BERTScore and Sentence-BERT capture semantic similarity but typically use short context windows unsuitable for paragraph-level comparison.
  • Evaluation Metrics: LLM-based judges can correlate with human judgments on short captions but retain transparency and consistency concerns for long-form descriptions.
  • CLIP-CC-Bench: CLIP-CC-Bench uses five long-context MTEB embedding models with complementary paragraph- and sentence-level matching for transparent, reproducible evaluation.

3 Methodology

The methodology builds a diverse, controlled movie-clip dataset and pairs it with detailed paragraph references designed to test temporal, visual, and action understanding. Evaluation uses standardized clips, bias-reducing annotation procedures, and multi-granular semantic comparisons.

  • Dataset Construction: CLIP-CC-Bench curates movie content for temporal diversity, visual richness, and challenging narrative structures in long-form description.
  • Bias Control: Proper nouns, locations, brands, cultural references, and fictional-universe terminology are excluded so evaluation emphasizes visual understanding over memorized associations.
  • Selection Criteria: All clips are approximately 90 seconds, self-contained, technically consistent, and selected for multiple visual elements and temporal dynamics.
  • Annotation: Four trained annotators narrate clips in real time; transcripts are manually verified and grammar-cleaned without changing content or adding names.
  • Dataset Statistics: The dataset contains 5 hours of movie content in 90-second clips with corresponding expert-generated descriptions.
  • Dataset Statistics: Description lengths vary substantially, with bimodal word-count patterns around 100–300 and 400–800 words for simpler and more complex scenes.
  • Dataset Diversity: The dataset draws from more than 140 films and series spanning 1959–2024, with 200 clips distributed across diverse narrative types.
  • Reference Characteristics: References track chronological event progression and include detailed visual elements, spatial relationships, foreground actions, and background context.

4 Experiments

The experiments evaluate 17 Video Language Models with standardized paragraph generation and a multi-granular, five-judge semantic matching protocol. The protocol combines holistic paragraph alignment, sentence-level content matching, and Borda aggregation across judges.

  • 4.1 Model Selection and Inference: 17 Video Language Models are evaluated using their maximum supported frame capacity rather than an imposed frame limit.All models generate paragraph-length descriptions under standardized inference settings.
  • 4.1 Model Selection and Inference: The evaluation standardizes prompts and generation settings, including temperature=0.0 and max_tokens=5000.The prompt requests detailed coverage of events, actions, camera movements, attributes, and scenes without identifying characters or entities.
  • 4.3 Evaluation Methodologies: Five MTEB embedding models act as independent judges, each encoding sentences or complete paragraphs into dense vectors.The ensemble is used to support multi-judge semantic evaluation rather than reliance on one embedding model.
  • 4.3 Evaluation Methodologies: Coarse matching compares complete reference and predicted paragraphs, while fine-grained matching compares sentence embeddings through an n×m cosine-similarity matrix.Fine-grained precision and recall use best sentence matches in opposite directions, and sentence-level F1 balances semantic precision and coverage.
  • 4.3 Evaluation Methodologies: Each judge combines coarse and fine scores with a harmonic mean, penalizing models that perform well at only one granularity.The resulting per-judge HM-CF score is averaged over the dataset before model ranking.
  • 4.3.3 Multi-Judge Consensus Ranking: The final consensus ranks 17 VLMs by Borda aggregation across judges, assigning 16 points to first place and 0 to last place.The maximum attainable Borda score is 80 across five judges, reducing dependence on any single judge’s ranking.

5 Results

The ensemble evaluation ranks 17 VLMs using coarse- and fine-grained semantic matching across five embedding judges, with strong agreement and stable rankings. VideoLLaMA3 leads consistently, while fine-grained detail matching remains weaker than holistic alignment.

  • Per-Judge Evaluation Statistics: VideoLLaMA3 achieves the highest scores across all five embedding models, while KaLM-Gemma3-12B reaches HM-CF 0.79.Embedding models differ in absolute scales and variance, motivating ensemble evaluation.
  • Overall Ranking: VideoLLaMA3 ranks first with Borda 80/80 and mean 0.67, followed by mPLUG-Owl3 with Borda 75 and mean 0.66.LLaVA-OneVision ranks third after a full-precision mean HM-CF tiebreaker against ViLAMP.
  • Inter-Judge Agreement: System-level judge agreement is high, with Spearman ρ ranging from 0.96 to 0.99 and mean Kendall τ of 0.92.At the instance level, mean pairwise Pearson correlation is 0.90 across 3,383 model-clip pairs.
  • Ranking Stability: Bootstrap rankings agree with the original at mean Kendall τ 0.98, preserve the top-ranked model in 100% of replicates, and keep nearly all rank intervals within one position.The exception is InternVL2, whose 95% bootstrap rank interval lies within two positions.
  • Analysis and Key Findings: Coarse-grained scores exceed fine-grained scores across models; VideoLLaMA3 scores 0.82 coarse versus 0.76 fine on KaLM-Gemma3-12B.The largest reported gap is on NV-Embed-v2, where scores are 0.68 coarse versus 0.47 fine.
  • Performance Stratification: Performance tiers separate top-tier models at Borda ≥60, mid-tier models at 30 ≤ Borda < 60, and lower-tier models below Borda 30.Top-tier models have mean scores above 0.63, whereas lower-tier models fall below 0.58.

6 Limitations

CLIP-CC-Bench is a controlled but bounded benchmark: it focuses on English narrative films, omits proper nouns, and uses one reference description per clip. Its semantic protocol is reliable and stable under resampling, but event ordering, human validation, scale, domain coverage, multiple references, and multilingual evaluation remain open boundaries.

  • Its narrative-film, English-only scope excludes instructional, surveillance, and user-generated videos, limiting tested settings.
  • Each clip has one reference description, which cannot represent every valid description of a complex scene.
  • Semantic matching does not explicitly model temporal or causal event ordering, motivating metrics that reward correct sequencing.
  • The protocol’s rankings are internally reliable and stable under dataset resampling, while targeted human meta-evaluation remains future work.
  • Proposed extensions include larger scale, broader domains, multiple references per clip, and cross-linguistic evaluation.

7 Conclusion

CLIP-CC-Bench evaluates long-form video description with an ensemble-based semantic protocol and Borda aggregation, illustrated through qualitative and per-judge analyses of two 90-second clips. Across 17 VLMs, transformer architectures lead the ranking, while overall semantics remain stronger than fine-grained detail capture.

  • CLIP-CC-Bench combines five MTEB embedding models with Borda count aggregation to evaluate long-form video descriptions.
  • Evaluation of 17 VLMs finds transformer architectures dominant, with VideoLLaMA3 achieving perfect 80/80 Borda consensus.
  • The consistent coarse–fine granularity gap indicates stronger overall-semantic capture than precise-detail capture, with the top model reaching 0.67 mean.
  • Qualitative analysis presents sampled frames, expert references, and all 17 candidate descriptions for clips 112 and 053.
  • Tables report per-VLM descriptions and five-judge harmonic-mean fine-grained/coarse-grained similarities, sorted by Borda rank.
  • The longva description for clip 053 is truncated because its approximately 10 837-character output rapidly degenerates into repetitive nonsense.

B.1 Cross-Exhibit Performance Patterns

Cross-exhibit results show stratified VLM quality and judge-specific sensitivity across a winter encounter and a complex security checkpoint scene. Stronger models retain advantages, while judge behavior differs notably in sensitivity to description length.

  • In clip 112, llava_one_vision ranks first with Borda 84 and scores 0.830, 0.770, 0.678, 0.710, and 0.781 across KaLM, GTE, NV, NeMo, and Qwen3.
  • Top-tier VLMs remain above 0.70 across most judges, mid-tier models fall in the 0.60–0.75 range, and lower-tier models drop below 0.65.
  • In clip 053, videollama3 ranks second with Borda 83 and scores 0.809, 0.726, and 0.694 on KaLM, GTE, and Qwen3.
  • KaLM, GTE, NeMo, and Qwen3 are largely length-invariant, whereas NV-Embed penalizes candidate–reference length mismatches more strongly.

B.3 The Pathological Case of LongVA on Clip

LongVA’s clip 053 output becomes extremely verbose, repetitive, and hallucinatory, producing very low scores and failing to track the scene’s temporal structure. The case illustrates how pathological verbosity can accompany fabricated content and weak narrative coherence.

  • LongVA generates 10 837 characters for clip 053, more than four times the reference length, with repetitive content grounded nominally in “The Adjustment Bureau”.
  • Its scores are 0.600, 0.526, 0.294, 0.413, and 0.563 for KaLM, GTE, NV, NeMo, and Qwen3, respectively.
  • The description fabricates movie references, invents scenes, and repeats numbered card-holding scenarios without advancing narrative understanding.
  • The Borda rankings in both qualitative exhibits align with the aggregate VLM ranking, placing longva among the consistently lower-ranked models.
  • Relative model orderings persist across the winter encounter and security choreography, linking benchmark performance to temporal reasoning, visual detail extraction, and narrative coherence.

C.1 Embedding-Ensemble Methodology

The framework combines five embedding judges with coarse- and fine-grained semantic matching to evaluate paragraph-level descriptions reliably. It reveals stable model rankings while exposing a gap between narrative understanding and precise visual detail.

  • Ensemble-based evaluation framework: Five embedding judges and Borda aggregation reduce single-model bias by producing consensus-based VLM rankings.The judges may differ in absolute scores, but their relative VLM rankings remain remarkably consistent.
  • Observed evaluation pattern: 0.65–0.85 coarse-grained cosine scores exceed the 0.45–0.70 fine-grained precision/recall scores typical of current VLMs.This disparity indicates stronger capture of scene context, interactions, and temporal flow than object attributes, spatial relations, and subtle actions.
  • Coarse–fine semantic matching: Coarse-grained matching measures holistic paragraph alignment, whereas fine-grained matching verifies detailed content.The two granularities are combined through harmonic-mean HM-CF to discourage success through only generalities or disconnected details.
  • Motivation for automated evaluation: Human evaluation of 200 clips paired with 17 descriptions would require thousands of annotation hours and remains subjective for long-form semantic equivalence.Subjectivity reflects differing preferences over verbosity, detail granularity, and narrative structure.

D Dataset Topical Distribution

The supplied passages identify a table reporting the primary narrative-type distribution of 200 clips and describe GPT-4o cleanup procedures used for transcribed narrations.

  • Primary narrative type: Table 10 reports the distribution of all 200 clips by primary narrative type.It complements source- and topical-diversity statistics summarized in table 1.
  • Output format: The cleaned narration is returned as plain prose inside a single JSON object whose only field is summary.The procedure is specified for reproducible transcript processing.
  • GPT-4o cleanup: The cleanup prompt fixes grammar, punctuation, spelling, and spoken disfluencies without changing described content or event order.It also prohibits proper nouns and conversational framing.
Loading 2608.04302v1…