PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Video-Text Pretraining with Multi-View Captions

Fida M. Thoker, Renaud Vandeghen, Karen Sanchez, Marc Van Droogenbroeck, Bernard Ghanem

arXiv:2609.35090v1cs.CV

TL;DR

Single sparse captions often miss rich spatiotemporal video semantics, while generated descriptions can be noisy. The paper creates multi-view captions with refinement and separate representations for summary and detailed views, and reports improved retrieval across benchmark types with a data-efficient alternative to scaling video-text pairs.

  • Problem

    Web-scale video-text datasets typically use a single caption that captures only a limited aspect of video content, while generated descriptions may add unsupported details.

    Source: §1 · pp. 1–2

  • Method

    The paper generates complementary summary and detailed captions, refines them through visual reasoning, adds semantic positive captions, and models summary and detailed views with separate CLS tokens.

    Source: §1 · p. 2

  • Results

    Multi-view captions substantially improve pretraining across standard, fine-grained, and detailed video-text retrieval benchmarks and offer a data-efficient alternative to scaling video-text pairs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper's findings support richer and complementary textual supervision as an effective direction for video-text pretraining.

    Source: §1 · p. 2

Abstract

from arXiv · show

Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/

1 INTRODUCTION

Video-text pretraining has scaled models and datasets, but the quality of language supervision remains underexplored. Single sparse captions can omit rich video semantics, motivating more diverse and reliable supervision.

1 INTRODUCTION

  • Web-scale video-text datasets often pair each video with one short, sparse, weakly aligned, or noisy caption that compresses rich content.A high-level caption can omit temporal progression, object interactions, and contextual details.Source: §1 · p. 1
  • Single-view supervision can discard complementary semantic interpretations and fine-grained dynamics in videos.The introduction asks whether rich video content can be learned from a single textual view.Source: §1 · pp. 1–2
  • Longer MLLM-generated captions alone may introduce irrelevant details, incorrect information, or drift from dominant visual content.The paper frames effective supervision as needing diversity, visual fidelity, and semantic comprehensiveness.Source: §1 · p. 2
  • The paper proposes an automated MLLM framework that generates summary and detailed captions, refines them through visual reasoning, and adds semantic positives.It also uses separate CLS tokens for summary and detailed views and recaptions approximately 10 million videos.Source: §1 · p. 2

2 RELATED WORK

Prior work advances video-text learning through scaling, stronger architectures, and synthetic captions. This paper focuses instead on supervision design, using automatically generated complementary video descriptions.

2 RELATED WORK

  • Video-text pretraining has advanced through larger datasets, stronger visual encoders and architectures, scaling strategies, and masking methods.Earlier approaches aligned video and text with contrastive learning and cross-modal matching objectives.Source: §2 · p. 2
  • Despite large-scale video-text models, relatively few prior works investigate supervision design as a source of stronger learning signals.This paper explores richer, multi-view textual supervision rather than focusing on model and data scaling.Source: §2 · p. 2
  • Video recaptioning approaches include frame-caption summarization and multiple generated candidates, while Panda-70M uses human verification before automatic selection.The proposed framework is fully automated and generates multiple detailed, complementary textual views.Source: §2 · pp. 2–3

3 METHODOLOGY

The method constructs complementary summary and detailed captions, refines details against video evidence, and generates semantic positives. It trains with separate text representations for summary and detailed views.

3.1 MLLM-BASED MULTI-VIEW SUPERVISION GENERATION

  • The framework builds a multi-view corpus through caption generation, reasoning-based refinement, and semantic-positive generation to improve supervision diversity, fidelity, and coverage.It starts from InternVid-10M-FLT and generates multiple semantically complementary signals for each video.Source: §3.1 · p. 3

3.1.1 MULTI-VIEW CAPTION GENERATION.

  • Different MLLMs generate concise summaries and detailed descriptions as complementary semantic views of each video.The summary prompt targets the primary activity; the detailed prompt requests visible objects, actions, interactions, and scene context.Source: §3.1.1 · p. 3

3.1.2 REASONING-BASED CAPTION REFINEMENT.

  • A reasoning-capable model checks detailed captions against video evidence, preserving grounded content while removing unsupported details and potential hallucinations.Residual inaccuracies may remain because of limitations of the underlying MLLM.Source: §3.1.2 · p. 4

3.1.3 SEMANTIC POSITIVE CAPTION GENERATION.

  • Semantic positive captions describe different visually grounded aspects of a video rather than merely paraphrasing the same description.They are conditioned on the video and a selected refined caption, and describe the actor, action, interacting object, and outcome.Source: §3.1.3 · p. 4

3.2 VIDEO-TEXT PRETRAINING

  • Pretraining uses separate CLS tokens for summary and detailed or semantic-positive captions, with both views sharing the video representation.The framework applies video-text contrastive learning, video-text matching, and masked language modeling to both views; zero-shot retrieval averages their similarity scores.Source: §3.2 · p. 5
  • The training corpus exposes the model to multiple semantically consistent, complementary textual views instead of one caption per video.The paper presents this as enabling stronger alignment between visual content and language semantics.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate video-text retrieval on standard and fine-grained benchmarks, with main pretraining runs using 5M or 10M InternVid videos and ablations using a 1M subset. Results cover zero-shot and fine-tuned retrieval, supervision recipes, and the effect of separate summary and detailed text representations.

4.1 STANDARD TEXT-TO-VIDEO RETRIEVAL

  • 34.2→38.2 MSR-VTT R@1 for MVC-B over same-data Baseline-B; MVC-L also reaches 62.0 vs 31.5 on DiDeMo against InternVideo trained on 646M samples.The controlled comparison uses the same backbone, training framework, and 5M videos; MVC-L uses 10M videos for the DiDeMo comparison.Source: Table 1 · p. 6, §4.1 · p. 6
  • At 5M, fine-tuned MVC-L exceeds UMT-L by +21.6 R@1 on DiDeMo and +14.0 on ActivityNet; same-data MVC-B gains persist after fine-tuning.With the same architecture and pretraining videos, MVC-B improves from 61.2 to 72.9 on DiDeMo and from 55.7 to 64.4 on ActivityNet.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 FINE-GRAINED AND DETAILED TEXT-TO-VIDEO RETRIEVAL

  • MVC improves R@1 by 38.2 points on CARE-S over Baseline-B, with gains across all six fine-grained settings and stronger results than UMT-L despite fewer videos and a smaller backbone.The six settings cover spatial, temporal, detailed-description, event, whole-clip, and single-shot retrieval; MVC-L also improves over ViCLIP-L across all six.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 ABLATION STUDIES

  • On six ablation benchmarks, adding original captions to generated supervision yields the best or equal-best result, including an LSMDC increase from 16.7 to 18.2.The ablations use a 1M-video subset; summary captions alone also improve over original captions across all six benchmarks.Source: Table 4 · p. 8, §4.3 · p. 8
  • Detailed captions alone underperform summary captions, but combining granularities, refining detailed captions, and adding semantic positives each improve retrieval results.Refinement improves performance consistently across all six benchmarks, and semantic positive captions further improve every benchmark.Source: Table 5 · p. 9, §4.3 · p. 9
  • With identical supervision and training, dual-CLS outperforms single-CLS on all six benchmarks, including a 2.3-point gain on CARE-T.The design separately represents summary and detailed views; on ActivityNet, R@1 rises from 50.1 to 51.3.Source: Table 6 · p. 9, §4.3 · p. 9

5 CONCLUSION

The paper presents multi-view MLLM-generated supervision and granularity-aware text representations, reporting retrieval gains across zero-shot and fine-tuned settings, often with smaller pretraining datasets.

5 CONCLUSION

  • Experiments across standard, fine-grained, and detailed retrieval benchmarks show consistent gains in zero-shot and fine-tuned settings, often surpassing methods trained on larger datasets.The conclusion identifies supervision quality and granularity as important dimensions of scalable video-text learning.Source: §5 · p. 9
  • The framework combines multi-view caption generation, reasoning-based refinement, and semantic positive generation to improve supervision diversity, visual fidelity, and semantic coverage.It generates richer video-text annotations than conventional single-caption supervision.Source: §5 · p. 9
  • A granularity-aware text representation separately models short and detailed descriptions to exploit their complementary supervision..Source: §5 · p. 9

A.1 VQA RESULTS

Video question answering provides a complementary test of learned video-text representations, and MVC improves results on ActivityNet-QA, with more modest or comparable outcomes on the other reported datasets.

A.1 VQA RESULTS

  • 50.0 vs. 47.3 on ActivityNet-QA: MVC-L using 10M clips exceeds UMT-L trained on 17M reported pretraining videos.MVC-L reaches 50.0 on ActivityNet-QA with 10M clips; UMT-L reaches 47.3 with 17M reported pretraining videos.Source: §A.1 · p. 15
  • For 5M entries, MVC-B improves ActivityNet-QA from 43.5 to 47.3 (+3.8), and MVC-L improves it from 45.1 to 49.1 (+4.0).The passage reports more modest MVC-L improvement on MSR-VTT-QA (45.5 to 46.0) and comparable MSVD-QA performance (51.3 vs. 51.5).Source: §A.1 · p. 15
  • Increasing MVC-L from 5M to 10M video clips further improves all three benchmarks, reaching 50.0 on ActivityNet-QA, 46.3 on MSR-VTT-QA, and 51.9 on MSVD-QA.The benchmarks are ActivityNet-QA, MSR-VTT-QA, and MSVD-QA.Source: §A.1 · p. 15

A.2 MLLM-BASED MULTI-VIEW SUPERVISION GENERATION

The supervision pipeline creates summary and detailed captions, refines detailed captions against video content, and adds event-focused semantic positives. Caption statistics and examples describe the distinct roles of these views and the effects of refinement.

A.2 MLLM-BASED MULTI-VIEW SUPERVISION GENERATION

  • Summary captions are prompted to describe only the dominant activity in one concise sentence, while detailed captions expose richer visual evidence with bounded output.The supplied summary prompt requests a maximum of 16 words; the detailed-caption prompt requests a maximum of 128 words.Source: §A.2 · p. 16
  • Refined detailed captions are generated from individual draft captions by checking them against video content, with factual correction emphasized over free-form regeneration.Qwen3-VL-8B-Thinking receives both the video and one draft detailed caption in each call.Source: §A.2 · p. 16
  • Semantic positive captions focus on specific visible events and differ from generic paraphrases by shifting semantic focus while remaining consistent with the video.The prompt calls for two captions grounded in concrete actors, actions, objects, and outcomes.Source: §A.2 · pp. 16–17
  • Generated detailed captions are longest and most variable; refinement reduces their average length and variability, while semantic positives remain concise and add event-specific supervision.Summary captions provide concise anchors, detailed captions broader context, and semantic positives focus on specific visible events.Source: §A.2 · p. 17, Table 8 · p. 17
  • The content-word Jaccard measure captures lexical overlap between detailed and refined captions, but ignores word order, frequency, and sentence structure.It therefore measures lexical overlap rather than semantic correctness or claim preservation.Source: §A.2 · p. 17
  • Refinement shortened audited Qwen captions from 96.0 to 60.6 words and Tarsier captions from 80.1 to 60.3 words, without imposing a target length.The prompt instructs the model to remove overlays, unsupported content, and non-visual inferences; the shortening is consistent with removing irrelevant or unsupported information.Source: §A.2 · pp. 16–17
  • Gemma classified 98.54% of Qwen and 98.72% of Tarsier refinements as semantically consistent or compatible, but this text-only assessment does not measure visual correctness.Contradictions were classified in 0.79% and 0.90% of the respective refinements.Source: §A.2 · p. 17
  • In the refinement example, Qwen’s refined caption corrects the animal from a dog to a cat and removes unsupported actions while retaining the visible drying activity.Figure 3’s example contrasts the original misidentification and unsupported actions with the visually grounded refined captions.Source: §A.2 · p. 18, Figure 3 · p. 19

A.3 DATASETS

The experiments use the InternVid-10M-FLT subset for pretraining and evaluate retrieval across general, temporal, fine-grained, detailed-description, and multi-shot benchmarks.

A.3 DATASETS

  • InternVid-10M-FLT supplies 10M filtered video clips with original captions for large-scale video-text pretraining.The work states that datasets are publicly available and used under their original licenses and terms; generated captions are planned for release after acceptance.Source: §A.3 · p. 19
  • MSR-VTT evaluates general video-text alignment, DiDeMo emphasizes temporally localized event retrieval, and ActivityNet Captions contains long videos with dense activity and event annotations.These benchmarks differ in content and temporal structure.Source: §A.3 · pp. 19–20
  • LSMDC covers complex movie scenes and narrative content, while MSVD is a smaller benchmark of short videos paired with human-written captions.LSMDC clips are paired with text derived from scripts and audio descriptions.Source: §A.3 · p. 20
  • CaReBench separates appearance-focused spatial retrieval from temporal retrieval using separate spatial and temporal annotations.The benchmark contains 1,000 test samples.Source: §A.3 · p. 20
  • DREAM-1K evaluates detailed-description and event-based retrieval, while Shot2Story20K tests whole-clip and single-shot retrieval using its 2,000-sample test split.Shot2Story20K focuses on multi-shot video understanding through shot-level captions and video-level summaries.Source: §A.3 · p. 20

A.4 ADDITIONAL ABLATIONS

The ablations examine how video masking ratio and pretraining duration affect text-to-video retrieval under multi-view caption supervision.

Effect of Masking Ratio

  • Reducing masking from 50% to 10% improves MSVD (38.8→39.7), ActivityNet (50.1→52.5), DiDeMo (53.0→54.3), and CARE-T (54.1→55.9).Table 9 keeps caption supervision fixed and reports text-to-video R@1.Source: §A.4 · p. 20, Table 9 · p. 20
  • At a 10% masking ratio, LSMDC decreases from 18.2 to 15.7, while Shot2Story-S remains similar.Thus, the lower ratio benefits most benchmarks, but not LSMDC.Source: §A.4 · p. 20, Table 9 · p. 20

Effect of Training Schedule

  • Performance improves through 20 pretraining epochs, with smaller gains from 15 to 20 epochs than at earlier intervals.The duration comparison holds model, supervision, and training configuration fixed and reports text-to-video R@1 across six benchmarks.Source: §A.4 · p. 20, Table 10 · p. 21

A.5 FINE-TUNING HYPERPARAMETERS

The section specifies fine-tuning settings for text-to-video retrieval and video question answering, with results reported from single runs.

A.5 FINE-TUNING HYPERPARAMETERS

  • Fine-tuning settings cover text-to-video retrieval and video question answering.The reported experiments use standard evaluation protocols from prior work.Source: §A.5 · p. 21
  • All reported results come from a single training run.Source: §A.5 · p. 21
Loading 2609.35090v1…