PaperlayerView exact revision on arXiv

Source-linked AI summary

HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, Jingjing Liu

arXiv:2005.00200v2cs.CVcs.CLcs.LG

TL;DR

Video-and-language pre-training is constrained by weak temporal alignment, image-text-derived objectives, and narrow video sources. HERO uses hierarchical local-global multimodal encoding, sequence-aware pre-training tasks, and diverse video data, achieving state-of-the-art results across multiple tasks while introducing How2R and How2QA.

  • Problem

    Existing models lose temporal alignment, underuse video sequence structure, and rely on restricted cooking or instructional video sources.

  • Method

    HERO combines Cross-modal and Temporal Transformers with MLM, MFM, VSM, and FOM, and trains jointly on HowTo100M and large-scale TV data.

  • Results

    HERO achieves new state of the art across evaluated retrieval, question answering, inference, and captioning tasks, including TVR, TVQA, VIOLIN, and TVC.

  • Takeaways & Limitations

    The study provides a hierarchical video-and-language pre-training framework and two new downstream benchmarks, How2R and How2QA.

  • Takeaways & Limitations

    The authors identify extending HERO to other video-and-language tasks and designing additional pre-training tasks as future work.

Abstract

from arXiv · show

We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.

1 Introduction

HERO addresses limitations of prior video-and-language pre-training by modeling local and global temporal context, adding sequence-aware objectives, and broadening training data and evaluation. It achieves state-of-the-art results across multiple video-and-language tasks and introduces How2R and How2QA.

  • Motivation: Existing models often lose temporal alignment, reuse image-text objectives, and rely on cooking or instructional videos lacking complex social interactions.These constraints limit modeling of sequential video structure and diverse visual content.
  • Model: HERO hierarchically fuses subtitle sentences with local video frames using a Cross-modal Transformer, then builds global sequential context with a Temporal Transformer.The architecture transfers frame-level multimodal context into video-level temporal representations.
  • Pre-training tasks: VSM learns global subtitle-video matching and local temporal localization, while FOM restores the original order of shuffled video frames alongside MLM and MFM.Ablation studies find that VSM and FOM play critical roles in video-and-language pre-training.
  • Training data: HERO is jointly trained on HowTo100M and large-scale TV data to learn from instructional content and plots involving emotions, social dynamics, and causal relations.The TV dataset supplements HowTo100M with more complex plots and varied genres.
  • Results: HERO achieves new state of the art across retrieval, question answering, inference, and captioning tasks, including TVR, TVQA, VIOLIN, TVC, DiDeMo, and MSR-VTT.The paper also introduces How2R and How2QA as new video-moment retrieval and question-answering benchmarks.

2 Related Work

Video-and-language pre-training remains less mature than image-and-text pre-training, with existing work spanning several downstream tasks. HERO targets this gap through improvements to architecture, objectives, training-corpus diversity, and evaluation benchmarks.

  • Research landscape: Language and image-text pre-training advanced rapidly after BERT, but video-and-language pre-training remains in its infancy.The related work contrasts broad progress in language and image-text settings with the relatively early stage of video-and-language research.
  • Existing work: Existing video-and-language models cover text-based retrieval, video question answering, and video captioning.The cited methods include VideoBERT, CBT, MIL-NCE, ActBERT, and UniViLM.
  • HERO’s scope: HERO aims to advance video-and-language omni-representation learning through architecture, pre-training-task, training-corpus, and benchmark design.These four dimensions define the paper’s stated research agenda.

3 Hierarchical Video+Language Encoder

HERO hierarchically fuses subtitle sentences with aligned local video frames, then models global temporal context, and trains this representation with four pre-training tasks.

  • Input Embedding: The input embedder projects ResNet and SlowFast visual features and WordPiece token representations into a shared lower-dimensional space with positional embeddings.Visual features combine 2D and 3D representations before projection; token and frame embeddings are layer-normalized.
  • Cross-modal Transformer: HERO first applies a Cross-modal Transformer to each subtitle sentence and its associated visual frames for fine-grained multimodal fusion.The transformer outputs contextualized embeddings for subtitle tokens and video frames.
  • Temporal Transformer: A Temporal Transformer then processes encoded frame embeddings from the whole clip to learn global video context.Residual connections add the original visual embeddings back to preserve positional information.
  • Pre-training Tasks: Pre-training uses MLM and MFM alongside VSM, which learns local frame-sentence alignment and global query-video matching.Tasks are sampled one per mini-batch, while MLM and MFM mask only one modality at a time.
  • Pre-training Tasks: MFM masks visual features with 15% probability and reconstructs them from remaining frames and subtitles using regression or noise-contrastive objectives.MFFR regresses masked-frame representations toward visual features, whereas MNCE distinguishes the correct frame from negative distractors.
  • Architecture Comparison: HERO’s early multimodal fusion uses subtitle-frame alignment more finely than XML, whose modality-specific matching scores are combined later.Experiments report that HERO outperforms a flat BERT-like baseline and that its design far outperforms the flat architecture.

4 Experiments

HERO is evaluated through extensive ablations and across diverse benchmarks, datasets, and video modalities. Results support its hierarchical design, temporal-alignment pre-training tasks, mixed pre-training data, and broad downstream performance.

  • Pre-training ablations: FOM improves tasks requiring temporal reasoning, while VSM particularly benefits video-moment retrieval through local and global subtitle-frame alignment.FOM yields large gains on TVQA and significant gains on How2R and How2QA; VSM is especially effective on TVR and How2R.
  • Pre-training ablations: HERO’s optimal pre-training setting combines MLM, MNCE, FOM, and VSM on HowTo100M and TV data.The ablation study identifies this combination as optimal.
  • Model ablations: The hierarchical design explicitly aligns subtitles and frames, whereas flat architectures learn this alignment only through implicit attention.Pre-training substantially lifts HERO but provides much smaller gains for flat and alternative hierarchical baselines.
  • Downstream results: HERO achieves new state of the art across all six downstream benchmarks, spanning retrieval, question answering, inference, and captioning.The comparisons use task-specific baselines including XML, HowTo100M, STAGE, Multi-stream, and MMT.
  • Downstream results: +0.55/+4.72/+10.65 on DiDeMo R@1/10/100 and +1.9/+3.2/+4.9 on MSR-VTT R@1/5/10 over the cited baselines.On multi-channel tasks, HERO nearly doubles XML on TVR R@1, improves TVQA by +3.28%, VIOLIN by +0.75%, and TVC Cider by +4.60%.

5 Conclusion

HERO combines hierarchical multimodal encoding with pre-training tasks designed to capture local and global temporal alignment. The model achieves strong transfer across video-and-language tasks, while broader task extensions and improved pre-training tasks remain future work.

  • HERO uses Cross-modal and Temporal Transformers to learn multimodal fusion and temporal context.
  • Novel pre-training tasks capture temporal alignment between video and language at both local and global levels.
  • HERO exceeds state of the art across multiple video-and-language tasks after pre-training on two large-scale video datasets.
  • The paper introduces two new datasets for text-based video-moment retrieval and video question answering.
  • Future work includes extending HERO to other video-and-language tasks and developing better-designed pre-training tasks.

A.1 Additional Experiments

Additional experiments show that HERO transfers effectively across retrieval, inference, question answering, and captioning settings. Performance gains depend on pre-training, while global frame-level features limit some fine-grained inference cases and short-context captioning settings.

  • HERO achieves state-of-the-art results across TVR, How2R, TVQA, How2QA, VIOLIN, and TVC comparisons.
  • Pre-training significantly improves HERO across TVR, How2R, TVQA, How2QA, VIOLIN, and TVC tasks.
  • HERO learns video-subtitle alignment during pre-training, performing reasonably on How2R without seeing How2R queries.
  • Approximately +2.9% improvement on VIOLIN follows pre-training, but global frame-level features make region-level hypothesis inconsistencies difficult to capture.
  • HERO with pre-training surpasses MMT on TVC across all metrics, although the non-pre-trained model is slightly inferior to the state-of-the-art baseline.
  • TVC’s short average video context contrasts with HERO’s design for long-video representation learning, motivating task-specific or decoder pre-training.

A.2 Qualitative Analysis

Qualitative analyses indicate that HERO’s Video-Subtitle Matching predictions often overlap ground-truth moments and that its Cross-modal Transformer learns diverse attention patterns. The authors identify remaining limits in interpreting these learned representations.

  • VSM predictions largely overlap ground-truth moments on both TV and HowTo100M examples.
  • Speaker information, character emotion, and salient objects such as rubber bands can provide matching clues for VSM.
  • Cross-modal Transformer attention includes vertical, diagonal, block, heterogeneous, and reversed-block patterns.
  • Reversed-block patterns represent cross-modality attention between text and video frames.
  • Vertical patterns usually focus on a specific frame rather than special tokens, differing from earlier observations.
  • More thorough quantitative and qualitative analysis of what video-language pre-training learns is left for future work.

A.3 Downstream Adaptation

HERO is adapted to retrieval, question answering, inference, captioning, and single-channel video tasks through task-specific heads and matching procedures. These adaptations reuse its hierarchical multimodal and temporal representations for local and global reasoning.

  • Downstream adaptation uses end-to-end fine-tuning for retrieval, question answering, inference, and multimodal captioning.
  • Text-based moment retrieval computes local frame-level and global clip-level query-video matching scores using the VSM procedure.
  • Video QA appends each answer candidate to subtitle sentences and temporal inputs, then predicts answer probabilities from QA-aware representations.
  • When available, Video QA training combines answer classification with span prediction, weighted by λ.
  • Video-and-language inference pools query-aware global representations and trains binary classification with cross-entropy loss.
  • A Transformer decoder extends HERO to captioning by attending to representations of the annotated video moment.

A.4 Frames/Subtitles Pre-processing

HERO preprocesses video clips as frame sequences and subtitles as sentence sequences, then aligns subtitle sentences with temporally overlapping visual frames while handling ambiguity and unmatched text.

  • Video clips are sampled into visual-frame sequences, while subtitles are parsed into sentence sequences.
  • Subtitle sentences are aligned with visual frames whose timestamps overlap each sentence’s subtitle timestamp.
  • Because subtitle and frame counts usually differ, one subtitle sentence may correspond to multiple visual frames.
  • When multiple sentences overlap one frame, HERO assigns the frame to the sentence with maximal temporal Intersection over Union to avoid duplication.
  • Unpaired subtitle sentences are concatenated with neighboring sentences to avoid information loss.

A.5 Implementation Details

The implementation combines SlowFast and ResNet-101 visual features into 4352-dimensional frame representations and uses separate six-layer Cross-Modal and three-layer Temporal Transformers.

  • HERO concatenates 2304-dimensional SlowFast and 2048-dimensional ResNet-101 features into 4352-dimensional frame features.SlowFast features use fixed frame rates, while ResNet-101 features are extracted at double the frame rate and max-pooled to clip level.
  • The Cross-Modal Transformer uses 6 layers, hidden size 768, and 12 attention heads.
  • The Temporal Transformer uses 3 layers, hidden size 768, and 12 attention heads.
  • Pre-training uses AdamW with learning rate 3e−5 and weight decay 0.01 on Nvidia V100 GPUs.The best pre-trained model uses 16 V100 GPUs for about 3 weeks.

A.6 Downstream Tasks

The paper evaluates video-language understanding across retrieval, question answering, inference, and captioning, using established datasets alongside the newly collected How2R and How2QA benchmarks.

  • Text-based Video-Moment Retrieval: TVR retrieves relevant clips and localizes moments in multi-channel videos containing video and subtitles.TVR uses the TV dataset, with five queries collected per video clip on average.
  • Video Question Answering: TVQA uses multiple-choice questions about video clips and accompanying subtitles, with seven questions and five answers per clip.
  • Text-based Video-Moment Retrieval: DiDeMo and MSR-VTT evaluate text-based retrieval on single-channel video-only data.DiDeMo contains 10.6K videos and 41.2K sentences, while MSR-VTT contains 200K video clip-caption pairs.
  • Text-based Video-Moment Retrieval: Text-based video-moment retrieval separates video retrieval from moment retrieval and evaluates average recall at K.
  • Video Question Answering: TVQA, How2QA, and VIOLIN are evaluated on full-video question answering, with accuracy as the metric.The considered setting excludes grounded-clip QA and question-driven moment localization because full-video QA is the most challenging setting among the three.
  • How2R and How2QA: How2R annotations require selecting a video segment and describing it, while How2QA reuses those segments for four-option question answering.
  • How2R and How2QA: Most selected How2R and How2QA video segments are shorter than 15 seconds, although lengths range from 5 to more than 30 seconds.
  • How2R and How2QA: How2QA questions span seven leading-word types, with most beginning with “what”, “why” or “when”.
Loading 2005.00200v2…