PaperlayerView exact revision on arXiv

Source-linked AI summary

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, Thomas Brox

arXiv:2011.00597v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Video-text representations must handle multiple semantic granularities and long-range context across modalities. COOT addresses this with hierarchical temporal and contextual modeling plus cross-modal cycle consistency, achieving improved retrieval performance and favorable benchmark results.

  • Problem

    Video and text descriptions contain multiple semantic levels, while video-text learning needs to capture long-range temporal dependencies across those levels.

  • Method

    COOT combines attention-aware feature aggregation, a contextual transformer for local-global interactions, and cross-modal cycle-consistency loss.

  • Results

    COOT achieves state-of-the-art video-text retrieval results, including 16.6% better average R@1 than HSE on ActivityNet-captions with fewer parameters.

  • Takeaways & Limitations

    The ablations indicate that hierarchical components and cross-modal cycle consistency jointly and individually improve retrieval performance.

Abstract

from arXiv · show

Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open-source at https://github.com/gingsi/coot-videotext

1 Introduction

COOT addresses long-range temporal dependencies in video-text learning with a hierarchical architecture that models interactions within and across semantic levels. It combines attention-aware aggregation, contextual attention, and cross-modal cycle consistency to improve semantic alignment.

  • Motivation: COOT models long-range temporal context through hierarchical interactions within and between video and text levels.The hierarchy captures relationships among frames, clips, and videos, alongside words, sentences, and paragraphs.
  • Intra-Level Cooperation: Attention-aware feature aggregation models temporal interactions among low-level entities instead of relying on [CLS] tokens or mean pooling.It assigns greater contribution to important entities in frame or word sequences.
  • Inter-Level Cooperation: A contextual attention module models interactions between local clips or sentences and global video or text context.The module highlights context-relevant semantics and suppresses irrelevant semantics.
  • Cross-Modal Alignment: A cross-modal cycle-consistency loss encourages semantic alignment between vision and text in the shared embedding space.The loss enforces interaction between modalities and consistent representations across domains.
  • Contributions: The proposed architecture combines hierarchical transformers, attention-aware aggregation, contextual attention, and cross-modal alignment as its main contributions.These components target intra-level, inter-level, and cross-modal interactions.

2 Cooperative Hierarchical Transformer

COOT represents video and text hierarchically, combining temporal modeling, attention-aware aggregation, and contextual interactions between local and global semantics. Its contextual transformer uses low-level and high-level entities to produce context-sensitive video and text embeddings.

  • Architecture: COOT models frame/word, clip/sentence, and video/paragraph levels with temporal transformers, attention-aware aggregation, and contextual transformers.The video and text branches are symmetric, and the temporal transformers learn relationships between temporal features.
  • Semantic Alignment: COOT uses cosine-distance-based contrastive losses across clip, sentence, video, paragraph, and global-context levels to organize positive and negative embeddings.The formulation defines embeddings for clips and sentences, constant margins, and negative-sample separation in the joint embedding space.
  • Intra-Level Cooperation: Attention-aware feature aggregation models temporal interactions among low-level entities instead of treating features independently with average or max pooling.The aggregation computes attention vectors from transformed keys and uses them to weight feature vectors before aggregation.
  • Inter-Level Cooperation: The contextual attention module links clips or sentences with global video or text context, highlighting relevant semantics and suppressing irrelevant ones.Its local module models short-term low-level interactions, while its global module uses context to weight local semantic values.
  • Inter-Level Cooperation: The contextual transformer combines the mean of local embeddings with a context-dependent embedding to form the final video representation.The contextual output is produced from interactions between local semantics and a global-context query.

3 Cross-Modal Cycle Consistency

COOT introduces a cross-modal cycle-consistency loss that aligns clip and sentence embeddings through differentiable soft nearest neighbors. The loss penalizes when a sentence or clip fails to return to its original temporal location after cycling across modalities.

  • Cross-Modal Cycle Consistency: The cross-modal cycle-consistency loss aligns clips and sentences by identifying semantically aligned pairs as nearest neighbors in a shared embedding space.It replaces cross-modal attention units used in prior work and connects the video and text modalities.
  • Soft Nearest Neighbors: For a sentence embedding, COOT first computes a soft nearest-neighbor clip using similarity scores and then cycles back to the sentence sequence.Soft nearest neighbors preserve the temporal ordering of sentences while allowing differentiable alignment.
  • Cycle-Consistency Objective: A sentence is cycle consistent when the return location matches its original position; deviations are penalized by the squared distance between source and destination locations.The objective is ||i − µ||2, where i is the source location and µ is the soft destination location.
  • Training Use: The differentiable cycle-consistency loss supports supervised and self-supervised training and includes both text-to-video and video-to-text cycles.In the self-supervised setting, videos are split into clips and paragraphs into sentences before both directional cycles are computed.

4 Experimental Setup

COOT is evaluated on ActivityNet-captions and Youcook2 using retrieval metrics, pretrained text and video features, and ablation settings that isolate its components.

  • Evaluation: Performance is measured with recall at K and Median Rank for video-text and clip-sentence retrieval.Reported recall values include R@1, R@5, and R@10.
  • Feature encoding: Paragraph text is encoded with pretrained BERT-Base outputs from its last two layers, producing 1536-dimensional features.
  • Feature encoding: Video features come from provided ActivityNet representations or concatenated 2D and 3D Youcook2 features, with an alternative HowTo100M-pretrained embedding.The Youcook2 alternatives produce 4096-dimensional or 512-dimensional features.
  • Sampling: The model samples up to 80 frame features per clip and video, using uniform interval sampling during training and center frames during validation.
  • Ablations: The ablation study separately evaluates attention-aware feature aggregation, the Contextual Transformer, and cross-modal cycle-consistency loss.The table also includes reproduced HSE results and a setting without the local-global cross-attention layer.

5 Results

COOT improves video-text retrieval across the reported benchmarks and ablations, while using fewer parameters than HSE; its representations also improve captioning with far fewer input features.

  • Ablations: The cross-modal cycle-consistency loss improves both HSE and COOT, while attention-aware aggregation yields a 7.2% average R@1 improvement over average pooling.
  • ActivityNet-captions: COOT achieves 16.6% better average R@1 than HSE on ActivityNet-captions while using fewer parameters.
  • Youcook2: On Youcook2 without HowTo100M features, COOT outperforms Miech et al. and HGLMM on paragraph-to-video and sentence-to-clip retrieval.
  • Youcook2: With HowTo100M-pretrained features, COOT outperforms MIL-NCE by 16.4% in R@1 for paragraph-to-video retrieval.The authors describe this as complementary to methods focused on large-scale pretraining.
  • Efficiency: COOT has 10.6M parameters, 60% fewer than HSE, and trains in under 3 hours on two GTX1080Ti GPUs.
  • Captioning transfer: Using COOT representations with MART improves captioning over appearance and optical-flow features while providing about 100 times fewer features per video.

6 Related Work

The paper situates COOT among video-language representation, retrieval, captioning, and cycle-consistency research, emphasizing hierarchical temporal modeling and cross-modal alignment.

  • Retrieval: The paper evaluates retrieval on YouCook2 under classification-pretrained and HowTo100M-pretrained feature settings.The supplied caption identifies Table 3 as YouCook2 retrieval results, while the experiment passage specifies the two feature settings.
  • Video and Language: Video-language methods use multimodal signals and temporal structure for joint embeddings, retrieval, captioning, and related alignment tasks.Prior work includes self-supervised pretraining from paired clips and texts, temporal localization, and joint video-text embeddings.
  • Cycle-Consistency: Cycle-consistency provides a transitivity-based training objective, and COOT applies it to align video and text rather than only video sequences.The paper describes this as the first use of cycle-consistency in the video-text domain.
  • Captioning: COOT improves captioning representations over MART baselines, producing descriptions that more closely capture the demonstrated actions in qualitative examples.Examples include tire changing, potato preparation, kneeling, and cutting fish, though some outputs remain imperfect.

7 Conclusions

COOT learns a joint video-text embedding space through hierarchical temporal cooperation and cross-modal cycle consistency, achieving state-of-the-art retrieval and captioning performance on two challenging datasets.

  • Conclusions: COOT aligns similar video and text semantics in a joint embedding space while modeling long-range temporal context across hierarchy levels.Its hierarchy includes interactions between frames and words, local contexts and global context, and clips and sentences.
  • Conclusions: The attention-aware feature aggregation module models interactions between frames and words, while the contextual transformer connects local and global contexts.These components address within-level and between-level interactions in the hierarchical representation.
  • Conclusions: The cross-modal cycle-consistency loss enforces semantic alignment between clips and sentences in the shared representation space.The conclusion reports that the architectural components and cycle-consistency loss jointly and individually improve retrieval performance.
  • Conclusions: COOT achieves state-of-the-art retrieval and captioning performance on two challenging datasets.The conclusion reports both retrieval and captioning outcomes at the paper level.

Broader Impact

The paper presents fundamental research without foreseeable societal consequences and notes possible long-term contributions to video search and organization.

  • Broader Impact: The authors identify no foreseeable societal consequence for this fundamental research.The broader-impact statement explicitly limits the claim to foreseeable consequences.
  • Broader Impact: The authors state that this research line may contribute in the long run to video search and video organisation services.This is presented as a possible long-term contribution rather than a demonstrated application outcome.

A.1 Implementation Details

The implementation uses searched hyperparameters, regularization and optimization choices, dataset-specific preprocessing, embedding normalization, and ablations of text and loss components.

  • Hyperparameters: Hyperparameters are selected using manual search and BOHB, with final configurations summarized for the evaluated models.Table 7 covers ActivityNet-Captions and two YouCook2 feature settings.
  • Optimization: GELU performs best among tested activations, while increasing attention capacity can improve results at the cost of memory and optimization difficulty.The tested alternatives include ReLU, SELU, ELU, and GELU.
  • Optimization: Training uses Adam, RAdam, or SGD, with a Reduce on Plateau schedule that reduces learning rate by 10 after stalled validation performance.Training terminates after 15 epochs without improvement, using retrieval-score sums as validation metrics.
  • Loss Settings: The cross-modal cycle-consistency weight is set to λ = 0.01 for ActivityNet and λ = 0.001 for Youcook2.These values are dataset-specific implementation settings.
  • Regularization: Regularization methods must be carefully balanced to address overfitting, including dropout, weight decay, CMC-loss, and Gaussian noise on frame features.The paper points to Table 7 for the trade-offs among these methods.
  • Preprocessing and Retrieval: ActivityNet clips are expanded to at least 10 frames, and output embeddings are L2-normalized before cosine-similarity retrieval.The preprocessing adds frames at clip boundaries, while retrieval compares normalized video and paragraph embeddings.
  • Ablations: A text ablation finds that BERT outperforms GloVe on both evaluated models and all reported metrics.The comparison is conducted on ActivityNet-Captions validation data.
  • Ablations: A loss ablation evaluates COOT after removing loss components across different base models, including contextual attention and attention-aware feature aggregation variants.The supplied table caption identifies CoT as global attention in the contextual transformer and AF as the aggregation module.

A.2 Ablation Studies

The ablations examine supervision noise, text encoding, alignment losses, and sequence pooling across video-text retrieval settings. Results show robustness to unreliable low-level labels, gains from contextual text and alignment losses, and different pooling preferences by hierarchy level.

  • Low-level supervision: Noise from 0% to 40% consistently decreases performance, but higher noise causes no significant further change while performance remains good.The experiment adds uniform noise to clip boundaries and also removes temporal alignment labels completely.
  • Text encoding: Context-aware BERT features significantly improve performance over fixed GloVe embeddings across the evaluated architectures.The text encoder receives entire paragraphs, allowing BERT to use high-level context.
  • Alignment losses: Removing any or all three alignment losses significantly decreases performance, while clustering losses have a positive impact.The study varies low-level pooling, cross-modal cycle consistency, and global-context attention.
  • Sequence pooling: Attention-aware feature aggregation performs best at the low level, whereas average pooling outperforms it at the high level.Low-level sequences are longer and have stronger temporal relationships, while high-level semantics are more independent.
  • Average pooling: Including padding tokens in the sum while dividing by the non-padding length works well on ActivityNet-Captions but does not reproduce the same large gain on YouCook2.The reported average sequence lengths are 3.6 for ActivityNet-Captions and 7.6 for YouCook2.
  • Qualitative analysis: Qualitative retrieval examples show semantically similar results, and t-SNE visualizations cluster videos around activities and similar content.The qualitative evidence covers ActivityNet-Captions and YouCook2 retrieval examples.
Loading 2011.00597v1…