Survey: Transformer based Video-Language Pre-training

• Transformers support pre-training on large data followed by fine-tuning on smaller downstream datasets to learn transferable representations.Source: §1 · p. 1
• Transformer pre-training achieved major gains in NLP and was subsequently adapted to computer vision, motivating its application to video processing.Source: §1 · p. 1
• Video-Language understanding is challenging because video combines multimodal information and temporal structure with language.Source: §1 · p. 1
• The survey reviews transformer foundations, Video-Language proxy and downstream tasks, datasets, model structures, performance comparisons, challenges, and future directions.Source: §1 · p. 1

2.1.1. Self-Attention

• Self-attention maps input tokens to keys, queries, and values, using relevance scores and weighted values to produce contextual representations.The decoder’s encoder-decoder attention uses keys and queries from the encoder and values from the preceding decoder block.Source: §2.1.1 · p. 2

2.1.2. Multi-Head Attention

• Multi-head attention projects inputs into h K-Q-V groups, processes each group separately, and concatenates their outputs through a learned projection.Source: §2.1.2 · p. 2, Figure 1 · p. 2

2.1.3. Position Encoding

• Position encoding is added because self-attention does not inherently capture sequence order; fixed and learned encodings achieve similar reported performance.Source: §2.1.3 · p. 2

2.1.4. Transformer Structure

• The standard transformer stacks encoder and decoder blocks, with decoder blocks adding encoder-decoder attention and all sub-layers using residual connections and normalization.The original configuration uses six encoder blocks, six decoder blocks, d_model=512, and hidden dimension d_h=2048.Source: §2.1 · pp. 1–2, §2.1.4 · pp. 2–3

2.1. Transformer

• Compared with CNNs and RNNs, transformers capture global information in parallel and use a stackable architecture suited to large-scale pre-training.Source: §2.1.4 · p. 3

2.2. Pre-training & Fine-tuning

• Pre-training on large datasets is followed by fine-tuning on smaller task-specific datasets to transfer learned representations rather than train each model from scratch.Source: §2.2 · p. 3

2.2.1. Proxy Tasks

• Video-Language proxy tasks comprise completion, matching, and ordering objectives that learn intra-modal, inter-modal, or temporal relationships.Self-supervised learning dominates because it automatically generates training labels for massive pre-training datasets.Source: §2.2.1 · p. 3
• Completion tasks reconstruct masked language, video, token, or modality content, including MLM, MFM, MTM, MMM, and left-to-right language reconstruction.MFM variants use cross-entropy, regression, or contrastive learning, while MMM reconstructs one entire modality from the other.Source: §2.2.1 · pp. 3–4
• Matching tasks align video and language globally or locally using classification, contrastive objectives, or frame-level start-and-end prediction.Source: §2.2.1 · pp. 4–5
• Ordering tasks shuffle sentences or frames and train models to recover their original sequence order, exploiting temporal and textual structure.Source: §2.2.1 · p. 5

2.2.2. Video-Language Downstream Tasks

• Downstream transfer includes retrieval, action recognition, segmentation, step localization, VideoQA, and captioning, using classifiers, matching scores, or transformer decoders.Video captioning is the only generative downstream task described, and some methods add a decoder or reuse an encoder for word generation.Source: §2.2.2 · pp. 5–6

2.3. Video-Language Datasets

• Transformer-based video models depend heavily on dataset scale and quality, motivating the survey’s organization of commonly used datasets by annotation type.Table 2 summarizes the three dataset groups used for pre-training and fine-tuning.Source: §2.3 · p. 6, Table 2 · p. 7

2.3.1. Label Based Datasets

• Label-based datasets provide video-level annotations and are widely used for classification tasks such as action recognition.HMDB51 contains 6,841 videos across 51 action categories; UCF101, MPII Cooking, Kinetics, and AVA are other representative datasets.Source: §2.3.1 · p. 6

2.3.2. Caption Based Datasets

• Caption-based datasets provide descriptions for videos or segments and support both Video-Language pre-training and downstream retrieval, captioning, localization, and related tasks.Howto100M contains 136M video clips paired with captions, while ActivityNet, Charades, TGIFs, and YouCookII support downstream applications.Source: §2.3.2 · pp. 6–7

2.3.3. Other Datasets

• Other annotation types support question answering, action segmentation, and action-step localization through temporal boundaries, hierarchical steps, or ordered descriptions.TVQA supplies multiple-choice questions with temporal annotations, COIN organizes instructional videos from domains to steps, and CrossTask provides ordered task steps.Source: §2.3.3 · p. 7

Single-Stream Transformers

• Single-Stream models feed different modality features or embeddings into one transformer to capture intra-modal and inter-modal information.The overview includes VideoBERT, HERO, ClipBERT, DeCEMBERT, VLM, VATT, and VICTOR.Source: §3 · p. 7, Figure 2 · p. 8

Multi-Stream Transformers

• Multi-Stream models encode each modality with independent transformers and then construct cross-modal relationships with an additional component such as another transformer.CBT, ActBERT, and Univl are categorized as Multi-Stream Transformers.Source: §3 · pp. 7–8, Figure 2 · p. 8

Model Comparisons

• Across structures, model distinctions include modality inputs, pre-training proxy tasks, downstream tasks, and evaluation benchmarks.Most models use caption and video tokens, while DeCEMBERT adds ASR captions, ActBERT adds object regions, and VATT adds audio.Source: §3 · p. 8, Figure 2 · p. 8

3.1.1. VideoBERT

• VideoBERT pioneered single-stream Video-Language pre-training by combining video and text tokens and predicting masked tokens to learn semantic and temporal representations.It uses MLM, MFM, and VLM objectives and demonstrates downstream performance on zero-shot action classification and video captioning in the cooking domain.Source: §3.1.1 · p. 8

3.1.2. HERO

• HERO combines a cross-modal transformer with a temporal transformer and introduces LVLM and FOM objectives that benefit retrieval and temporal-reasoning tasks.Its experiments also show that text-based video-moment retrieval is sensitive to domain gaps, with domain-consistent data achieving comparable or better performance using less pre-training data.Source: §3.1.2 · pp. 9–10

3.1.3. ClipBERT

• ClipBERT reduces computation by sparsely sampling frames and clips with a 2D CNN, finding that 1 or 2 frames per clip and 2 clips per video suffice for effective pre-training.Image-text pre-training on COCO and Visual Genome Captions also benefits video-text tasks.Source: §3.1.3 · p. 10

3.1.4. DeCEMBERT

• DeCEMBERT addresses noisy or misaligned ASR captions by adding dense captions and using constrained attention to select the best-matched ASR caption.The model is pre-trained with MLM and VLM and fine-tuned for captioning, retrieval, and videoQA.Source: §3.1.4 · p. 10

3.1.5. VLM

• VLM supports text, video, or both as input through a task-agnostic cross-modal transformer and introduces masked modality and masked token modeling.MMM is especially effective for text-based video retrieval, while MTM performs better than the combined MLM and MVM objectives; VLM maintains competitive performance with fewer parameters.Source: §3.1.5 · pp. 10–11

3.1.6. VATT

• VATT learns video, audio, and text representations end to end from raw inputs using pure attention-based processing and contrastive video-audio and video-text matching.It is pre-trained on Howto100M and AudioSet and can transfer from video to image classification.Source: §3.1.6 · p. 11

3.1.7. VICTOR

• VICTOR adapts single-stream pre-training to Chinese Video-Language data with encoder-decoder transformers, memory queues, and proxy tasks for masked frame order, masked sentence order, and dual alignment.Its pre-trained weights transfer to classification, recommendation, captioning, and cross-modal retrieval tasks.Source: §3.1.7 · p. 11

3.2.1. CBT

• CBT combines text, visual, and cross-modal transformers with noise contrastive estimation to preserve fine-grained video information for Video-Language learning.Its pretrained features are evaluated on action recognition, action anticipation, video captioning, and video segmentation.Source: §3.2.1 · p. 11

3.2.2. ActBERT

• ActBERT uses global action and local object inputs with Tangled Transformer blocks, whose co-attention enables communication across modality-specific streams.The model is pretrained with MLM, MAM, MOM, and VLM, then transferred to five downstream tasks.Source: §3.2.2 · pp. 11–12
• ActBERT’s experiments show that global action information and local object clues are complementary.Source: §3.2.2 · p. 12

3.2.3. Univl

• Univl addresses the mismatch between understanding-oriented pretraining and generative downstream tasks by combining encoders, cross-modal interaction, and a generation decoder.It uses VLM, MFM, MLM, and language reconstruction objectives across retrieval, captioning, segmentation, localization, and sentiment tasks.Source: §3.2.3 · p. 12
• Univl’s three-stage training first aligns modality encoders, then trains the full model with a smaller learning rate, while masking whole text inputs to strengthen video representations.The step-by-step strategy consistently improves the pre-training process.Source: §3.2.3 · p. 12

Model Structure

• Transformer adaptations include shared attention weights in VATT, task-specific attention masks in VLM, and Tangled Transformer blocks in ActBERT.Source: §3.3 · p. 12
• Most methods use WordPiece text embeddings and fixed S3D visual features, while VICTOR, HERO, ClipBERT, and VATT use alternative backbones or end-to-end visual processing.Source: §3.3 · p. 12

Proxy Tasks

• Masking and matching proxy tasks dominate, while ordering tasks target temporal reasoning and whole-modality masking benefits retrieval.Source: §3.3 · pp. 12–13

Training Strategy

• Stage-by-stage pretraining in Univl and curriculum learning in CBT make pretraining progress smoother than training the whole model in one step.Source: §3.3 · p. 13

Downstream Tasks

• VLM generally performs best among single-stream models, while Univl generally outperforms other multi-stream models on text-based video retrieval and video captioning.The comparison uses MSR-VTT retrieval and YouCookII captioning benchmarks; VICTOR is excluded because it uses a Chinese dataset.Source: §3.3 · p. 13, Table 4 · p. 13, Table 5 · p. 13

4.1. Pre-training Dataset

• Pre-training datasets remain limited in quantity, caption quality, and domain diversity, while domain gaps can persist despite increased data volume.Howto100M provides scale, but automatically generated captions introduce mistakes and misalignments, and datasets often focus on single domains.Source: §4.1 · p. 13

4.2. Video-Language Transformer Designs

• Visual backbone design remains unresolved because independent backbones create feature-extraction gaps and end-to-end 2D CNNs may lose temporal information.The survey also calls for unified evaluation because models currently use different tasks, datasets, and settings; VALUE covers 11 datasets across retrieval, captioning, and videoQA but is not yet widely adopted.Source: §4.2 · pp. 13–14
• Future models should improve generalization by exploiting scene, character, object, audio, and other multimodal information across broader domains.The survey specifically identifies medical and surveillance videos as underexplored application areas beyond activity, film, and television data.Source: §4.2 · p. 14

4.3. Transformer Efficiency

• Transformer efficiency is a major constraint because quadratic time and memory complexity hinder scalability in Video-Language settings.Video-Language processing requires deeper models and larger parameter counts than pure-text processing, increasing hardware and computation demands.Source: §4.3 · p. 14

5. Conclusion

• The survey covers transformer background, pre-training and fine-tuning tasks, datasets, model structures, downstream performance, challenges, and future research directions.It compares state-of-the-art transformer models while highlighting their key strengths.Source: §5 · p. 14