PaperlayerView exact revision on arXiv

Source-linked AI summary

Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval

Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Hilde Kuehne

arXiv:2112.04446v2cs.CVcs.CLcs.SDeess.AS

TL;DR

The paper addresses how to learn useful video representations from complementary modalities such as vision, audio, and text. It proposes a modality-agnostic fusion transformer trained with combinatorial contrastive loss, improving multimodal embedding performance and achieving new state-of-the-art results.

  • Problem

    Multimodal video representation learning seeks to combine complementary visual, audio, and text signals into useful embedding spaces.

  • Method

    A modality-agnostic fusion transformer jointly processes arbitrary modality combinations, while combinatorial contrastive loss trains on individual and paired inputs without position or modality encodings.

  • Results

    The fusion transformer with combinatorial loss improves multimodal embedding performance and leads to new state-of-the-art results.

  • Takeaways & Limitations

    The learned embedding space can fuse any number of input modalities at test time and accommodate inputs of different lengths.

  • Takeaways & Limitations

    The demonstrated setup considers video, audio, and text, although the method can be extended to more modalities.

Abstract

from arXiv · show

Multi-modal learning from video data has seen increased attention recently as it allows to train semantically meaningful embeddings without human annotation enabling tasks like zero-shot retrieval and classification. In this work, we present a multi-modal, modality agnostic fusion transformer approach that learns to exchange information between multiple modalities, such as video, audio, and text, and integrate them into a joined multi-modal representation to obtain an embedding that aggregates multi-modal temporal information. We propose to train the system with a combinatorial loss on everything at once, single modalities as well as pairs of modalities, explicitly leaving out any add-ons such as position or modality encoding. At test time, the resulting model can process and fuse any number of input modalities. Moreover, the implicit properties of the transformer allow to process inputs of different lengths. To evaluate the proposed approach, we train the model on the large scale HowTo100M dataset and evaluate the resulting embedding space on four challenging benchmark datasets obtaining state-of-the-art results in zero-shot video retrieval and zero-shot video action localization.

1. Introduction

The paper proposes a modality-agnostic fusion transformer and combinatorial contrastive training for learning multi-modal video embeddings. The approach handles arbitrary modality combinations and input lengths, improving zero-shot retrieval and localization performance.

  • The work targets meaningful multi-modal embedding spaces for video, supporting tasks such as zero-shot retrieval and classification without human annotation.
  • The fusion transformer jointly processes any number of modalities, enables cross-modal attention, and projects the combined input into a joint embedding space.
  • The modality-agnostic design supports arbitrary modality combinations and variable-length inputs by sharing self-attention weights across tokens and omitting position or modality encodings.
  • Training on HowTo100M and evaluation on four downstream datasets produced improved performance and new state-of-the-art results in zero-shot video retrieval and action localization.
  • The combinatorial contrastive loss trains on every available combination of input modalities, including individual modalities and modality pairs.

2. Related Work

Prior work uses multi-modal learning across vision-language, vision-audio, zero-shot, generation, and multi-task settings. For video, narrated datasets and transformer-based methods provide related approaches for learning cross-modal representations and correspondences.

  • Multi-modal learning spans vision-language, vision-audio, zero-shot, cross-modal generation, and multi-modal multi-task learning.
  • Video combines multiple modalities and supports learning from large-scale data that would be difficult to annotate manually.
  • HowTo100M introduced a narrated-video dataset for multi-modal learning, later extended with clustering and reconstruction losses.
  • Transformer-based video methods have modeled audio-video correspondences, joint video-text encodings, and temporal or spatial attention.

3. Method

The method uses a modality-agnostic fusion transformer to jointly process arbitrary combinations of video, audio, and text, then projects modality-specific outputs into a shared embedding space. Training combines contrastive losses across single and paired modalities so tokens learn to exchange cross-modal information.

  • Input and embedding objective: The projection f maps single or combined text, video, and audio inputs into d-dimensional embeddings, maximizing similarity for semantically related inputs and minimizing it otherwise.The model supports missing modalities and can be extended beyond the three modalities considered explicitly.
  • Input and embedding objective: Modality-specific backbone features are projected and normalized into token sequences, with padding and attention masks enabling batch processing of variable token counts.The reported training protocol uses fixed-length video clips, although the model technically handles variable-length inputs.
  • Design choices: The method omits positional and modality embeddings, relying on backbone-specific token fingerprints and avoiding positional additions that could hinder longer sequences at inference.The tokens are therefore processed without explicit type or position metadata.
  • Multi-modal fusion transformer: The fusion transformer applies modality-independent keys, queries, values, and transformations so tokens from any included modalities can attend to one another.It uses standard multiheaded self-attention, an MLP, layer normalization, and residual connections; “fusion transformer” refers to this training use rather than a novel block architecture.
  • Multi-modal fusion transformer: The model is trained on all single modalities and modality pairs, producing six representations per sample for text, video, audio, text-video, video-audio, and text-audio inputs.With more modalities, all combinations may become infeasible, motivating random modality dropout as an alternative.
  • Projection to shared embedding space: For each modality included in a fused input, output tokens are pooled, passed through modality-specific projections, normalized, and combined into the final embedding.Normalization aligns vector magnitudes so dot-product similarity depends only on the angle between vectors.
  • Combinatorial loss: The combinatorial loss sums weighted contrastive losses between fused representations of disjoint modality subsets, including single-to-pair relationships.For three modalities, the objective includes the six pairings represented by t-v, v-a, t-a, t-va, v-ta, and a-tv; contrastive losses use Noise Contrastive Estimation with temperature τ and batch size B.

4. Experimental Evaluation

The evaluation tests zero-shot retrieval, action localization, and architectural choices across multiple datasets and modality configurations. Results favor fusion with combinatorial training, while performance depends on dataset alignment and temporal processing strategy.

  • Tasks and metrics: The experiments evaluate zero-shot text-to-video retrieval on YouCook2 and MSR-VTT, plus zero-shot step action localization on CrossTask and Mining YouTube.Retrieval uses R@1, R@5, R@10, and median rank; localization uses sliding-window similarity against action-step names.
  • Zero-shot retrieval: 51.3% R@10 on YouCook2 improves over AVLnet and MCN from 45.2% using the same visual, text, and audio backbones.The method achieves state-of-the-art results over the reported YouCook2 baselines.
  • Zero-shot retrieval: On MSR-VTT, using audio with video is less beneficial, and text-to-video retrieval without audio performs best.The authors attribute this to domain shift: HowTo100M audio is mainly speech, whereas MSR-VTT audio may be less related to textual descriptions.
  • Ablation studies: The fusion transformer with combinatorial loss achieves the best ablation performance, whereas the fusion transformer alone drops slightly below the single-modality transformer.Token fusion is broadly beneficial, and shared-transformer fusion with combinatorial loss outperforms modality summing in the compared single-modality-transformer setup.
  • Ablation studies: Modality-specific projections without a [cls] token outperform the compared projection designs on both datasets.The ablation compares shared final projection and an additional [cls] input token against the proposed design.
  • Ablation studies: Setups without positional embeddings perform almost consistently better, while processing all video input at once improves over max-pooling and uniform sampling.The results also support exploiting local temporal dependencies through shorter clips or video-at-once processing.
  • Attention analysis: Attention analysis finds heads specializing in single-modality fusion and other heads responsible for cross-modal attention.Figure 3 averages query-key attention across 30 heads and 128 video clips, with additional examples for random heads.

5. Limitations and Conclusion

The paper concludes that a modality-agnostic fusion transformer trained with combinatorial losses learns a strong multimodal embedding without positional encoding. However, stronger fusion can reduce generalizability when downstream multimodal data differs from the pretraining distribution.

  • Conclusion: The proposed fusion transformer exchanges information across video, audio, and text to form a joint multimodal representation.Combinatorial training over possible modality combinations supports the learned embedding space without position encoding.
  • Limitations: A clear limitation is reduced generalizability when better fusion is applied to multimodal data acquired differently from the training data.The paper identifies domain adaptation or generalization techniques as a future direction.

A. Additional Experimental Evaluation

Additional experiments replace the standard backbones with CLIP features for video and text in zero-shot text-to-video retrieval.

  • CLIP backbones: CLIP experiments use ViT-B/32 features pretrained on the Wikipedia-based WiT image-text dataset for both video and text.The setup extracts one 512-dimensional feature per second of video and one 512-dimensional feature per word of text.
  • CLIP backbones: The CLIP features are projected into the multimodal embedding space for zero-shot text-to-video retrieval evaluation.The passage describes the backbone substitution and projection setup, but does not provide the resulting retrieval values.

A.2. Action Segmentation

The model is evaluated for zero-shot temporal action segmentation using framewise alignment to ordered text steps, with results reported on CrossTask and Mining YouTube. It marginally improves Mining YouTube segmentation and improves CrossTask step-localization recall by 4% over MCN.

  • A.2. Action Segmentation: Temporal segmentation aligns video frames to text action labels using a similarity matrix and Viterbi decoding.The procedure first converts similarities to class probabilities with temperature-scaled softmax.
  • A.2. Action Segmentation: IoU and IoD measure temporal segmentation, while recall measures step action localization in Table 8.IoU is defined from the intersection and union of ground-truth and predicted actions; IoD is also reported.
  • A.2. Action Segmentation: 4% improvement in CrossTask step action localization recall over the MCN baseline accompanies only a marginal Mining YouTube segmentation boost.The method does not benefit on CrossTask segmentation, so recall is treated as the primary CrossTask metric because annotations may omit steps or misorder them.

A.3. CrossTask Specific Results

CrossTask results vary substantially by task category. The method boosts many cooking-related tasks but does not improve several non-cooking, car-related, or shelving tasks, consistent with the training-data distribution discussed by the authors.

  • A.3. CrossTask Specific Results: Significant gains appear across almost all cooking-related CrossTask categories, including “Make Banana Ice Cream” and “Grill Steak.”These category-level recalls are reported in Table 9.
  • A.3. CrossTask Specific Results: No performance improvement is reported for “Change Tire,” “All Oil to Car,” or “Build a Shelves.”These are identified as non-cooking categories in the CrossTask analysis.
  • A.3. CrossTask Specific Results: The authors associate weaker car-related performance with fewer car-related HowTo100M clips than food-related clips.They report 7.8M car-related clips versus 54.4M food-related clips and frame this as an assumption about the observed pattern.

A.4. Fusion Transformer Ablation

The fusion-transformer ablation examines architectural capacity under resource constraints. The best observed configuration uses one transformer block and 64 attention heads, while the supplied table descriptions identify the evaluated architecture factors.

  • A.4. Fusion Transformer Ablation: The ablation varies transformer blocks, attention heads, and hidden size, while also accounting for training batch size.Hidden size depends linearly on the number of heads, and batch-size entries may specify videos multiplied by sampled clips per video.
  • A.4. Fusion Transformer Ablation: Resource constraints require increasing transformer blocks to be offset by reducing attention heads or training batch size.The authors emphasize that a large batch size is essential for contrastive training.
  • A.4. Fusion Transformer Ablation: 1 transformer block with 64 attention heads is the best-performing tested configuration.The authors state that 64 heads is the maximum fitting within available resources and suggest larger configurations may further improve performance.

A.5. Text-Audio Disentangling and Loss Weighting

The ablations test whether separating ASR-derived text from audio and reweighting the text-visual loss improve HowTo100M training. Both adaptations are reported as beneficial, while RoPE does not benefit the representation.

  • A.5. Text-Audio Disentangling and Loss Weighting: Disentangled text-audio training randomly shifts the audio clip by 4 seconds relative to video and text boundaries.The shift is half of an 8-second clip and is intended to prevent text from being learned merely as audio narration.
  • A.5. Text-Audio Disentangling and Loss Weighting: Setting λ_tv = 1 while other listed loss weights equal 0.1 improves training alongside text-audio disentangling.The weighted setup is compared with aligned and disentangled text-audio training in Table 11.
  • A.5. Text-Audio Disentangling and Loss Weighting: RoPE does not benefit the learned multi-modal representation, despite being tested as a relative positional encoding.The paper also reports that absolute positional embeddings are not beneficial.

B. Qualitative Analysis

The qualitative YouCook2 analysis shows that retrieved clips are semantically related to text queries, including cases where the correct video is absent from the top five. Figure 4 presents these top-five retrieval examples, while the accompanying figures describe the audio backbone and ablation architectures.

  • Qualitative retrieval: Retrieved YouCook2 clips are semantically related to their text queries across the shown zero-shot examples.The correct video need not appear among the top five for the retrieved clips to correspond to the query.
  • Qualitative retrieval: For “bring a large pan of water to boil,” the model retrieves videos showing boiling water in a pot.
  • Model components: The audio backbone uses a trainable residual CNN over log-mel spectrograms and produces approximately 1.5 4096-dimensional features per second.It uses 16 kHz audio, 25 ms Hamming windows, 10 ms stride, and 40 Mel filter bands, and is not pretrained.

C.2. Ablation Architectures

The ablation compares architectures ranging from no transformer to a modality-agnostic fusion transformer trained with combinatorial loss. The experiments use distinct fine-tuning schedules for YouCook2 and MSR-VTT, while overall training takes approximately two days on four V100 GPUs.

  • Architectures: Four ablation architectures compare no transformer, modality-specific transformers, fusion transformers with pairwise losses, and fusion transformers with combinatorial loss.The final architecture uses a modality-agnostic transformer with combinatorial inputs and loss.
  • Training setup: YouCook2 fine-tuning runs for 5 epochs at 1e−5 with batch size 256, whereas MSR-VTT runs for 25 epochs at 5e−5 with batch size 128.All listed λ coefficients are set to 1.
  • Training cost: HowTo100M training takes approximately 2 days on four Nvidia V100 32GB GPUs, while dataset fine-tuning takes less than 30 minutes.
Loading 2112.04446v2…