PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum

Zhuoning Guo, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Xiaowen Chu

arXiv:2510.27571v1cs.CVcs.AIcs.CLcs.IRcs.LG

TL;DR

Video retrieval remains centered on coarse text-to-video tasks, leaving complex queries and diverse domains insufficiently evaluated. The paper co-designs a benchmark, synthesized training data, and a curriculum for GVE; GVE achieves state-of-the-art zero-shot generalization on UVRB.

  • Problem

    Existing approaches struggle with fine-grained, long-context, and non-text retrieval scenarios, while task and domain evaluation remains fragmented.

    Source: §1 · p. 2

  • Method

    The paper co-designs UVRB diagnostics, V-SynFlow's 1.55 million retrieval pairs, and a Modality Pyramid curriculum to train GVE.

    Source: §1 · pp. 2–3

  • Results

    Across UVRB, GVE-7B achieves state-of-the-art zero-shot results, with mean scores of 0.573 across datasets and 0.600 across task categories.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    The diagnostics find conventional benchmarks unrepresentative of overall retrieval ability and identify partially relevant retrieval as typical and generalizable.

    Source: §1 · p. 3

    Evaluations use vision-only inputs, excluding audio, transcripts, and metadata that can matter for real-world disambiguation.

    Source: §A.14 · p. 29

Abstract

from arXiv · show

The prevailing video retrieval paradigm is structurally misaligned, as narrow benchmarks incentivize correspondingly limited data and single-task training. Therefore, universal capability is suppressed due to the absence of a diagnostic evaluation that defines and demands multi-dimensional generalization. To break this cycle, we introduce a framework built on the co-design of evaluation, data, and modeling. First, we establish the Universal Video Retrieval Benchmark (UVRB), a suite of 16 datasets designed not only to measure performance but also to diagnose critical capability gaps across tasks and domains. Second, guided by UVRB's diagnostics, we introduce a scalable synthesis workflow that generates 1.55 million high-quality pairs to populate the semantic space required for universality. Finally, we devise the Modality Pyramid, a curriculum that trains our General Video Embedder (GVE) by explicitly leveraging the latent interconnections within our diverse data. Extensive experiments show GVE achieves state-of-the-art zero-shot generalization on UVRB. In particular, our analysis reveals that popular benchmarks are poor predictors of general ability and that partially relevant retrieval is a dominant but overlooked scenario. Overall, our co-designed framework provides a practical path to escape the limited scope and advance toward truly universal video retrieval.

1 Introduction

The introduction argues that coarse retrieval practice leaves diverse video-query scenarios and task/domain generalization insufficiently addressed. It presents evaluation, data synthesis, and curriculum design as coupled components of a universal-retrieval framework.

1 Introduction

  • Current MLLM retrievers achieve strong coarse-grained text-to-video results but struggle with fine-grained, long-context, and non-text query scenarios.The stated query formats include composed text-and-image queries and purely visual queries.Source: §1 · p. 2
  • The framework treats diagnostic evaluation, quality-controlled data synthesis, and interconnected multi-task learning as coupled challenges for universal video retrieval.It highlights task hierarchy: foundational spatial perception can support higher-order temporal reasoning.Source: §1 · p. 2
  • UVRB comprises 16 test datasets across diverse domains and tasks, with analysis of 14 state-of-the-art models to expose capability limitations.Source: §1 · pp. 2–3
  • V-SynFlow produces over 1.55 million high-quality cross-domain, multi-task pairs, and Modality Pyramid trains GVE from foundational tasks toward more complex dependent tasks.The curriculum prioritizes data-abundant foundational tasks before complex ones.Source: §1 · p. 3
  • Experiments validate GVE on UVRB; diagnostics find conventional benchmarks unrepresentative of overall ability and partially relevant retrieval typical and generalizable.Figure 2 summarizes performance across 16 datasets and 9 abilities.Source: Figure 2 · p. 2, §1 · p. 3

2 Related Works

Related work describes a fragmented retrieval landscape: benchmarks and models remain specialized by task or domain, while query formats and video-embedding approaches have expanded.

2 Related Works

  • Video-retrieval benchmarks add fine-grained annotations for grounding, scene understanding, and camera motion, but remain specialized.Examples include CaReBench, UltraVideo, and CameraBench.Source: §2 · p. 3
  • Retrieval now includes composed text-image queries and purely visual egocentric queries, yet evaluation and models remain siloed across tasks and domains.This fragmentation prevents holistic assessment of model generalization.Source: §2 · p. 3
  • Video embedders have shifted from image-centric models with temporal modules to MLLMs, but training data remains narrow for generalizable embeddings.The related work also says these embedders have not been systematically evaluated across more complex tasks and domains.Source: §2 · p. 3

3 Methodology

The methodology co-designs a diagnostic benchmark, synthesized multi-task training data, and a curriculum embedder for diverse video retrieval formats and domains.

3 Methodology

  • The framework integrates diagnostic evaluation, targeted data synthesis, and model optimization as a solution for universal video retrieval.It links UVRB diagnostics, V-SynFlow synthesis, and Modality Pyramid training.Source: §3 · pp. 3–4

3.1 Unifying and Benchmarking Video Retrieval

  • UVR compares query and video embeddings across textual, composed, and visual queries and coarse-grained, fine-grained, partially relevant, and long-context domains.Relevance is measured by cosine similarity between query and video embeddings.Source: §3.1 · p. 4
  • UVRB spans 16 datasets and tests 14 baselines to diagnose strengths and weaknesses across video retrieval scenarios.The diagnostics guide the paper’s data synthesis and model training design.Source: §3.1 · p. 4

3.2 Scalable Synthesis of Cross-Domain Multi-Task Video Retrieval Data

  • Over 1.55 million descriptive pairs form UVRD, synthesized from weakly annotated web videos through filtering, MLLM enrichment, and multi-task assembly.The workflow targets diverse query formats and data domains.Source: §3.2 · pp. 4–5
  • The workflow filters noisy or static assets, generates enriched captions and frame/clip pairs, then assembles composed retrieval and basic alignment tasks.The final corpus also maps selected synthesized captions into text-to-text pairs.Source: §3.2 · p. 5

3.3 Modality Pyramid: Customized Curriculum Contrastive Learning for Generalizable Embeddings

  • GVE fuses tokenized text or prompts with projected visual features, then applies last-token pooling and L2 normalization to produce retrieval embeddings.The model is derived from Qwen2.5-VL and uses its pretrained vision-language-aligned knowledge.Source: §3.3 · p. 5
  • Modality Pyramid schedules tasks from atomic to composite, using positive-pair cosine similarity to estimate alignment and progressively exploring harder tasks.Its temperature anneals from 0.1 to 1.0 by default, shifting focus from well-aligned tasks toward challenging ones.Source: §3.3 · p. 6

4 Experiments

GVE is evaluated against 14 baselines on UVRB, with results covering dataset- and ability-level performance, ablations, scaling, and correlations among retrieval capabilities. The findings highlight GVE’s zero-shot performance and show that benchmark representativeness, capability relationships, and scaling behavior vary across retrieval dimensions.

4.1 Experimental Setups

  • The evaluation compares 14 baselines spanning CLIP- and MLLM-based architectures and parameter sizes from 87M to 8.3B, while noting that some may have trained on UVRB test data.Metrics include R@1, R@10 for fuzzy-query datasets, and P@1 for composed tasks with multiple positive candidates.Source: §4.1 · p. 6

4.2 Main Results

  • GVE-7B scores 0.573 across datasets and 0.600 across task categories, outperforming Unite-7B by +6.5% and +7.3% respectively in zero-shot evaluation.GVE-7B leads in TXT (0.657 versus 0.609), CMP (0.312 versus 0.254), and LC (0.814 versus 0.746); GVE-3B scores 0.571 versus Unite-7B’s 0.559.Source: Table 1 · p. 7, Table 2 · p. 8, §4.2 · p. 8
  • UVRD raises 3B composed-task performance by 27% relative, while the Modality Pyramid lifts the 7B overall score from 0.594 to 0.600 and TXT from 0.648 to 0.657.Together, the dataset and curriculum yield a reported total gain of 1.8% to 3.1% over the baseline.Source: Table 3 · p. 8, §4.2 · p. 9
  • GVE-3B gains +7.4% (+7.1%) per 10× data increase on average across datasets (abilities), compared with GVE-7B’s +5.4% (+5.4%).Performance improves logarithmically with more training data but with diminishing returns; GVE-7B starts from a higher baseline.Source: Figure 6 · p. 9, §4.2 · p. 9

4.3 Analysis of Dimensional Capabilities

  • MSRVTT’s average correlation with UVR performance is ρavg = 0.58, while partially relevant retrieval has the highest average correlation at ρavg = 0.97.Fine-grained, partially relevant, and long-context retrieval exhibit strong mutual correlations (ρ ≥0.90).Source: §4.3 · p. 9
  • Spatial and temporal representations are weakly correlated (ρ = 0.12), while temporal skills correlate more strongly with fine-grained understanding (ρT-FG = 0.98) than spatial skills do (ρS-FG = 0.39).The analysis identifies a gap in jointly modeling when and where.Source: §4.3 · p. 9
  • CLIP-based models show spatial bias (ρS-CG = 0.99) and weak temporal ability (ρT-CG = 0.46), whereas MLLM-based models show stronger PR-CG correlation (0.98 vs. 0.70) and temporal–long-context coupling (ρLC-T = 0.64).For CLIP-based models, compositional representation inversely correlates with visual accuracy (ρCMP-VIS = −0.71), and temporal and long-context skills are nearly unlinked (ρT-LC = −0.14).Source: §4.3 · pp. 9–10
  • CLIP4Clip’s 87M parameters outperform Unite-7B’s 8B parameters on VIS, scoring 0.714 versus 0.702.Visual fidelity has a weak correlation with overall retrieval success (ρAVG-VIS = 0.26).Source: §4.3 · p. 10

5 Conclusions

The paper presents evaluation, data synthesis, and curriculum design as a co-designed framework for general video retrieval. It reports that GVE achieves state-of-the-art generalization on UVRB.

5 Conclusions

  • The authors present UVRB, 1.55 million high-fidelity multi-task training pairs, and a curriculum-based GVE as a co-designed framework for general video retrieval.They report state-of-the-art generalization on UVRB and characterize the framework as a path toward a more robust, versatile, and generalizable retriever.Source: §5 · p. 10

A.1 Overview of Appendix

The appendix collects supplementary material on the benchmark, data synthesis, model and training, experiments, and limitations. It points readers to detailed methods and additional results.

A.1 Overview of Appendix

  • Benchmark supplements cover UVRB dataset statistics, construction strategies, and the evaluation protocol.They also describe query prompts and retrieval metrics.Source: §A.1 · p. 14
  • Data supplements describe the training mixture, dataset-construction pipeline, and synthetic prompts for captioning, composed retrieval, and frame-level tasks..Source: §A.1 · p. 14
  • Model supplements detail GVE’s Qwen2.5-VL-derived architecture, input fusion, embedding extraction, LoRA fine-tuning, and contrastive learning setup.Key hyperparameters are also covered.Source: §A.1 · p. 14
  • Experimental supplements include baseline specifications, GVE training dynamics, training- and test-time scaling, and video classification results.The appendix also indicates that it discusses limitations concerning scope and settings.Source: §A.1 · p. 14

A.2 UVRB Details

UVRB’s dataset details describe statistics, construction strategies, and query prompts across retrieval tasks. The suite includes coarse-grained, fine-grained, long-context, composed, and visual retrieval settings.

A.2 UVRB Details

  • UVRB reports query and corpus counts, video durations, and text lengths, and presents dataset construction strategies and evaluation prompts and metrics.All videos use eight uniformly sampled frames.Source: §A.2 · pp. 14, 16, Table 4 · p. 15, Table 5 · p. 16
  • Coarse-grained datasets include MSRVTT, DiDeMo, and CRB-G; their queries use clip-text pairs, concatenated video descriptions, and general captions, respectively.CRB-S and VDC-O provide spatial or object-centric queries, while CRB-T uses temporal captions.Source: §A.2 · pp. 14–15
  • Partially relevant retrieval uses event descriptions from DREAM-1K, theme annotations from LoVR, and keyword annotations from PE-Video.These sources support event-to-video, theme-to-clip, and keyword-based partially relevant matching.Source: §A.2 · p. 15
  • Long-context retrieval includes full-length LoVR video captions as queries and detailed VDC descriptions as fine-grained queries.LoVR-V pairs long text with long video.Source: §A.2 · p. 15
  • Composed retrieval includes text-image and text-video queries adapted from MomentSeeker, while visual queries use a sampled MSRVTT frame or a LoVR clip.The visual tasks retrieve the source video from an image or clip query.Source: §A.2 · pp. 15–16

A.3 Evaluation Pipeline.

The evaluation pipeline standardizes model evaluation while supporting diverse architectures and task-specific prompting. Its three phases run tasks, execute models, and produce metrics and diagnostics.

A.3 Evaluation Pipeline.

  • The pipeline has three phases: dynamic task orchestration, configurable model execution, and metric and diagnostic generation.It supports diverse model architectures through a standardized interface built atop MTEB.Source: §A.3 · p. 16
  • Instruction-sensitive tasks receive dynamically injected domain-specific prompts to approximate real-world conditions and provide actionable diagnostics.Custom benchmarks integrate through dedicated loaders.Source: §A.3 · p. 16
  • Efficiency and reliability features include deferred expensive tasks, abstracted model initialization, online or offline data loading, and multi-GPU safeguards.The pipeline also includes clean shutdowns and an offline-first retry logic.Source: §A.3 · p. 16

A.4 Training Data

Training uses a diverse mixture of public and synthesized text, image, and video data, with all tasks formulated as instruction-guided retrieval. The authors also vary sampling to balance training signal and computational load.

A.4 Training Data

  • All tasks are formulated as instruction-guided retrieval across public datasets and the synthesized Universal Video Retrieval Dataset (UVRD).The training mixture spans text, image, and video data.Source: §A.4 · p. 16
  • Video data receives tailored preparation: VAST uses one vision caption, InternVid-FLT drops about 300K low-quality videos, PE-Video uses refined human captions, and WebVid excludes about 5M videos.The WebVid exclusion targets queries captioning over one video.Source: §A.4 · pp. 16–17
  • Contrastive training combines pre-mined explicit hard negatives with in-batch negatives, while sampling controls dataset contributions and computational load.Extremely large datasets receive fixed-size samples; selected high-value or complex datasets may be upsampled by repetition.Source: §A.4 · p. 17
  • For the 3B model, training uses 12.55 million instances; for the 7B model, it uses 13.73 million.The 7B model’s total rises through increased sampling rates for several large datasets.Source: §A.4 · p. 17

A.5 Dataset Construction Pipeline Details

This appendix describes a reusable dataset-construction framework and synthetic-data pipeline for video retrieval, along with the GVE architecture and training and additional analyses of scaling, classification, and limitations.

A.5 Dataset Construction Pipeline Details

  • The pipeline standardizes retrieval datasets as a Corpus, Queries, and Relevance Mapping, enabling consistent structure and compatibility across retrieval tasks.Its abstract base class supports tasks ranging from text-to-video to composed queries.Source: §A.5 · p. 18
  • Inheritance and method overrides support modality changes and composed-task variants, while chunked processing, validation, and repair address scale and video integrity.For example, the framework can convert MSRVTT to image-to-video and uses ffmpeg to repair common video errors.Source: §A.5 · p. 18

A.6 Prompt for Synthetic Video Retrieval Data Generation

  • The synthetic-data prompts cover video recaptioning, text-image and text-video composed retrieval, frame captioning, and text-to-text retrieval from captions.Text-to-text examples are formed by randomly matching two sentences from the caption list.Source: §A.6 · pp. 18–19
  • Captioning prompts require factual, visible-content descriptions with varied styles, while composed-retrieval prompts require specific, coherent queries that use the reference and add retrieval value.Image-based queries should depend on both image and video; clip-based queries should reflect practical relationships such as continuation, viewpoint, or condition changes.Source: §A.6 · pp. 19–21
  • The synthetic-data workflow uses structured prompts with dynamic controls, multimodal input routing, JSON validation, and sharded, resumable generation.It produces grounded, stylistically varied instances at million scale, with caption templates spanning concise summaries, rich descriptions, abstract interpretations, keywords, and partially relevant information.Source: §A.6 · p. 23

A.7 Model Architecture Details

  • GVE derives from Qwen2.5-VL Bai et al. (2025), removes its autoregressive head, and maps text, images, and video into a shared embedding space.Visual tokens are inserted into text sequences with temporal encoding for sampled video frames, and the final attended token supplies the embedding, motivated by its alignment with retrieval intent in instruction-tuned models.Source: §A.7 · p. 23

A.8 Training Implementation and Hyperparameters

  • Training applies LoRA Hu et al. (2021) to language modules while freezing the visual backbone and token embeddings, then uses normalized last-token embeddings for cosine similarity.Training uses AdamW with a 3 × 10^-5 learning rate, weight decay 0.1, bf16, and an InfoNCE-style loss at temperature 0.03; cross-device negatives expand the negative pool without increasing per-device memory.Source: §A.8 · p. 24

A.9 Baseline Details; A.10 Training Dynamics

  • Table 10 presents baseline model names, abbreviations, architectures, sizes, and training-data types; Figures 9 and 10 show GVE-3B and GVE-7B training dynamics across four metrics.Source: §A.9 · p. 24, Table 10 · p. 25, Figure 9 · p. 26

A.11 Experiments of Training-time Scaling: More Results of Data Scaling

  • GVE-3B gains +0.039 (14.7%) in compositional retrieval per decade of data, versus GVE-7B’s +0.025 (8.7%).GVE-3B also gains +0.057 (11.1%) in coarse-grained, +0.042 (7.1%) in textual, and +0.040 (7.9%) in fine-grained retrieval, versus GVE-7B’s +0.037 (6.6%), +0.037 (5.8%), and +0.031 (5.6%); visual gains are +0.024 (3.8%) and +0.013 (2.1%), respectively. Long-context retrieval is the exception: GVE-7B gains +0.042 (5.4%) versus GVE-3B’s +0.029 (3.8%), indicating task-dependent scaling efficiency.Source: §A.11 · pp. 24–25, Figure 11 · p. 27

A.12 Experiments of Test-time Scaling

  • At inference with maximum tokens fixed at 200, increasing sampled frames generally improves performance, with long-context gains of +19.6% for GVE-3B and +12.8% for GVE-7B, while compositional performance slightly degrades.As frames increase, the GVE-7B/GVE-3B AVG-D gap narrows from 0.029 to 0.024.Source: §A.12 · p. 27, Figure 12 · p. 28

A.13 Experiments of Video Classification

  • LanguageBind achieves the highest mean video-classification accuracy (0.553), followed by GVE-7B and InternVideo2-6B (0.526 each).The five benchmarks are evaluated zero-shot for GVE, though other baselines may have trained on them; performance remains limited on temporally complex Breakfast (best: 0.453) and fine-grained SSV2 (best: 0.569).Source: Table 11 · p. 29, §A.13 · p. 29

A.14 Limitations

  • The study is limited to vision-only evaluation, fixed 8-frame and 8192-token inference, general-domain coverage, and resource-intensive training.These choices exclude audio, transcripts, and metadata; may disadvantage sparse-event or hour-long-video tasks; omit medical, industrial, and surveillance domains; and constrain access, particularly for GVE-7B.Source: §A.14 · p. 29
Loading 2510.27571v1…