Source-linked AI summary
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
Jianwei Yang, Yonatan Bisk, Jianfeng Gao
TL;DR
Video-text contrastive learning must capture fine-grained word-video alignment while handling the cost of multimodal fusion with many negatives. TACo addresses these issues through token-aware losses and online cascade hard-negative sampling, and reports consistent improvements across retrieval benchmarks and transfer tasks.
Problem
Existing contrastive pipelines use sentence-level losses that underemphasize content words and face high multimodal-fusion costs that limit large negative sets.
Method
TACo combines token-level contrastive loss on selected content words with cascade sampling of hard negatives before multimodal fusion.
Results
TACo improves text-video retrieval across YouCook2, MSR-VTT, and ActivityNet and achieves better or comparable transfer performance on CrossTask and COIN.
Takeaways & Limitations
TACo provides a parameter-free alternative to conventional contrastive learning for video-text alignment across retrieval, localization, and segmentation settings.
Abstract
from arXiv · showhide
Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning. This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques. The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words. This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words. Second, a cascade sampling method is applied to generate a small set of hard negative examples for efficient loss estimation for multi-modal fusion layers. To validate the effectiveness of TACo, in our experiments we finetune pretrained models for a set of downstream tasks including text-video retrieval (YouCook2, MSR-VTT and ActivityNet), video action step localization (CrossTask), video action segmentation (COIN). The results show that our models attain consistent improvements across different experimental settings over previous methods, setting new state-of-the-art on three public text-video retrieval benchmarks of YouCook2, MSR-VTT and ActivityNet.
1. Introduction
TACo improves video-text alignment by focusing contrastive learning on visually grounded content words and efficiently selecting hard negatives for multimodal fusion. Experiments report improvements across retrieval, localization, and segmentation tasks.
- The pipeline combines sentence-level loss over all negatives, token-level loss on content words, and sentence-level loss over cascade-sampled hard negatives.
- TACo introduces token-aware contrastive learning to improve video-text alignment for large-scale pretraining and downstream tasks.
- The token-aware loss emphasizes content words such as nouns and verbs because they are more likely to align with visual video content than function words.
- Cascade sampling selects a small set of hard negative examples online for training multimodal fusion layers instead of using all negatives or random sampling.
- TACo improves text-video retrieval across YouCook2, MSR-VTT, and ActivityNet, while transferred representations achieve better or comparable performance on CrossTask and COIN.
2. Related work
Prior video-language work spans pretrained transformers, video-text alignment models, and negative-sampling strategies. TACo builds on these directions by combining fine-grained alignment with online hard-negative mining before multimodal fusion.
- Video-language transformers pretrained on large datasets support tasks including retrieval, question answering, captioning, and other downstream applications.
- Earlier approaches explored alignment losses separately, whereas later work combined losses before and after multimodal fusion.
- Video-text alignment methods model motion and temporal coherence using attention, joint representations, multilevel encoders, or fine-grained alignment.
- Random negative sampling is common, while existing hard-negative approaches are limited by multimodal fusion or offline computational costs.
3. Method
TACo combines token-aware contrastive losses with cascade hard-negative sampling to improve video-text alignment while making multimodal fusion training tractable. Its objective jointly optimizes sentence-level, token-level, and fusion-level alignment.
- Framework and objective: TACo uses video, language, and multimodal fusion modules whose representations are trained with three cooperating contrastive losses.The model combines sentence-level and token-level alignment before fusion with a fusion-level loss afterward.
- Token-aware alignment: Token-aware contrastive learning adds token-specific alignment because nouns and verbs are more likely to correspond to visual content than function words.TACo weights selected content words using inverse document frequency and aggregates their strongest video-token similarities.
- Sentence-level alignment: Sentence-level contrastive learning aligns averaged video features with [CLS]-based text features using all K−1 minibatch negatives efficiently.The loss optimizes video and text encoders by projecting samples into an aligned feature space.
- Cascade hard-negative sampling: Cascade sampling selects hard negatives using global and token-level similarities before sending only the most difficult pairs through multimodal fusion.For each text and video, the method selects top K′ aligned negative counterparts, reducing fusion computation while retaining challenging examples.
- Framework and objective: The final training objective minimizes L1 + λtL2 + L3 over video, text, and multimodal parameters, while inference sums all three alignment scores.λt controls the token-level loss and is 0.5 by default.
4. Experimental setup
Experiments evaluate TACo across established video-text retrieval and transfer benchmarks using pretrained language and video representations. The setup varies feature types, encoder depth, token budgets, and pretraining or finetuning protocols.
- Benchmarks: Experiments cover text-video retrieval on YouCook2, MSR-VTT, and ActivityNet, plus action step localization on CrossTask and action segmentation on COIN.The benchmarks test both retrieval and transfer of learned multimodal representations.
- Architectural settings: Ablation studies use one video-encoder self-attention layer and two multimodal-fusion layers, differing from several prior architectures.The paper notes that previous methods use different numbers of self-attention layers.
- Representations: The model uses pretrained BERT-base for language and ResNet-152, I3D-X101, or S3D-HM features for video.The 2D and 3D video features come from ImageNet-, Kinetics-400-, or HowTo100M-pretrained models.
- Input and training settings: YouCook2 and MSR-VTT use maximum video and text token counts of 48 and 30, while ActivityNet uses 256 tokens for each modality.Video features are sampled at dataset-specific rates and extracted with 2D or 3D CNN pipelines.
- Feature comparison: Table 2 compares YouCook2 and MSR-VTT retrieval across feature types, with S3D pretrained on HowTo100M outperforming the other features by a large margin.The comparison isolates the effect of video representation choice in the retrieval setup.
5. Results
TACo improves video-text retrieval across benchmark datasets and evaluation protocols, while its component studies support combining multi-level contrastive losses, cascade hard-negative sampling, and noun-verb token supervision. The pretrained representation also transfers to action localization and segmentation tasks.
- Ablations: Combining video-level losses improves retrieval over either loss alone, cascade hard-negative mining adds further gains, and token-level contrastive loss produces the best ensemble.The five settings progress from individual losses to their combination, cascade sampling, and finally token-level loss.
- Ablations: Noun-plus-verb token supervision performs best, while determiner-plus-adposition supervision is worse than the baseline without token-level contrastive loss.Using nouns or verbs individually improves performance, with nouns slightly better than verbs.
- Text-video retrieval: TACo outperforms prior work across YouCook2, MSR-VTT, and ActivityNet under separate-dataset evaluation, using a simpler contrastive pipeline with a smaller model.The comparison uses the same or similar features as prior work and reports the best performance across all three datasets.
- Zero-shot and finetuned evaluation: TACo significantly outperforms prior methods on zero-shot YouCook2 and slightly on zero-shot MSR-VTT after Howto100M pretraining.The improvement is larger on YouCook2 because its domain is closer to Howto100M; TACo also surpasses MIL-NCE on MSR-VTT with the same video features.
- Other video-related tasks: The pretrained model supports action-step localization on CrossTask and significantly outperforms MIL-NCE and ActBert on COIN action segmentation, with performance comparable to UniVL.CrossTask uses frame-to-action-step similarities and official ordering; COIN evaluates the learned video representation without text.
6. Conclusion
TACo is a parameter-free contrastive learning method for video-text alignment that addresses fine-grained alignment and inefficient multi-modal sampling. It improves text-video retrieval and transfers effectively to action localization and segmentation.
- TACo addresses missing fine-grained alignment and inefficient sampling in conventional video-text contrastive learning pipelines.
- Without extra parameters, TACo achieves promising results on three text-video retrieval benchmarks under varied evaluation protocols.
- TACo representations transfer effectively to action step localization and action segmentation.
A. Tokens of interest
Tokens of interest are nouns and verbs whose inverse-document-frequency weights determine their contribution to token-level contrastive losses. The weights are computed from training captions and normalized within each sentence.
- Tokens tagged as VERB or NOUN are selected as tokens of interest using a Spacy part-of-speech tagger.
- idf(token) = log |D| 1 + |{d ∈D : token ∈d}|.
- A token’s inverse document frequency decreases as its occurrence across the caption corpus becomes more frequent.
- For each sentence, normalized idf values assigned to nouns and verbs weight the token-level contrastive losses.
B. Contribution of three contrastive losses
The ablation compares early-stage, later-stage, and combined contrastive losses using separate alignment scores on YouCook2. The reported analysis evaluates how combining the losses affects retrieval performance.
- The comparison uses text-video retrieval performance on YouCook2.
- The study reports separate alignment scores after training with all three contrastive losses.
- Reference results use either early-stage-only or later-stage-only contrastive learning, while additional rows report separate stages for the combined model.
C. Effect of cascade sampling
Cascade sampling improves later-stage alignment by directing harder negatives to the multi-modal fusion layers. On YouCook2, performance improves beyond using later-stage contrastive learning alone or combining early and later losses without cascade sampling.
- The ablation compares later-stage-only loss, combined early- and later-stage losses, and cascade sampling for later-stage loss.
- Combining early- and later-stage losses slightly improves performance over later-stage contrastive learning alone.
- Cascade sampling further improves later-stage alignment by sending more difficult samples to cross-modal fusion layers.The authors connect this hard-negative mining to more discriminative video-text representations and improved final performance.
- The comparison evaluates YouCook2 using only the later-stage alignment score.
D. Effect of video encoder layers
The study examines how video encoder depth affects text-video retrieval on YouCook2. More encoder layers improve performance, while omitting them limits modeling of long-range temporal dynamics.
- D. Effect of video encoder layers: More video encoder layers significantly improve text-video retrieval performance on YouCook2.The experiments use R-152 and S3D-HM features with varying numbers of video encoder layers.
- D. Effect of video encoder layers: With no video encoder layers, the model performs poorly because it cannot adequately capture long-range temporal dynamics.
- D. Effect of video encoder layers: Adding one video encoder layer produces a significant performance improvement, with further gains as layers increase.
E. Comparing model size and FLOPs
The paper compares model sizes and computational costs across methods using parameter counts and FLOPs. Its largest model has comparable size and FLOPs to competing methods, although some comparison values are estimated.
- E. Comparing model size and FLOPs: The comparison reports model size and computational cost using parameter counts and FLOPs across methods.The table distinguishes multi-modal fusion, self-attention, and cross-modal attention components.
- E. Comparing model size and FLOPs: The largest TACo model has comparable size and FLOPs to other methods.
- E. Comparing model size and FLOPs: Some comparison values are estimated from descriptions in the original papers because prior methods generally did not report FLOPs.MMT is identified as the only prior method discussed as reporting parameter counts.
F. Visualizations
Visualizations show that increasing the token-level alignment weight improves the relevance of top-ranked videos for queries on YouCook2 and MSR-VTT. At weight 0.5, all five displayed results match the queried content in both examples.
- F. Visualizations: Increasing the token-level alignment weight from 0.0 to 0.1 and 0.5 makes the retrieved top-five videos more relevant.The figure displays ranked results from top to bottom for YouCook2 and MSR-VTT queries.
- F. Visualizations: For YouCook2, the top five results at weight 0.5 are all about cutting tomato.At weight 0.0, the third and fifth videos are not about tomato; at 0.1, the fourth moves into third place.
- F. Visualizations: For MSR-VTT, the top five results at weight 0.5 are all about two people talking with each other on a table.At weight 0.0, the last three videos are mismatched, and at 0.1 the fifth is replaced by a better-matched video.
- F. Visualizations: The visualization varies token-level alignment weights while showing ranked videos and their associated descriptions.