Source-linked AI summary
CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, Thomas Brox
TL;DR
Cross-modal contrastive losses do not explicitly preserve similarity within each modality and can treat semantically related samples as negatives. CrossCLR adds intra-modal alignment, prunes influential false negatives, and weights them in training; the resulting embeddings consistently improve retrieval and captioning across the reported settings. The approach assumes meaningful pretrained input embeddings and generalizes beyond video-text.
Problem
Existing cross-modal contrastive learning lacks explicit intra-modal similarity preservation and may use semantically related samples as negatives, limiting the joint embedding.
Method
CrossCLR enforces proximity from the original feature spaces, removes highly connected influential samples from negative sets, and applies connectivity-based loss weighting.
Results
CrossCLR consistently improves cross-modal retrieval and video captioning, extends retrieval state of the art on YouCook2 and LSMDC, and generalizes to other modality pairs.
Takeaways & Limitations
Cross-modal contrastive learning benefits from preserving intra-modal structure and avoiding semantically colliding negatives, beyond standard positive-pair and random-negative design.
Takeaways & Limitations
The method assumes that each input modality already has meaningful embeddings in which semantically related concepts are close.
Abstract
from arXiv · showhide
Contrastive learning allows us to flexibly define powerful losses by contrasting positive pairs from sets of negative samples. Recently, the principle has also been used to learn cross-modal embeddings for video and text, yet without exploiting its full potential. In particular, previous losses do not take the intra-modality similarities into account, which leads to inefficient embeddings, as the same content is mapped to multiple points in the embedding space. With CrossCLR, we present a contrastive loss that fixes this issue. Moreover, we define sets of highly related samples in terms of their input embeddings and exclude them from the negative samples to avoid issues with false negatives. We show that these principles consistently improve the quality of the learned embeddings. The joint embeddings learned with CrossCLR extend the state of the art in video-text retrieval on Youcook2 and LSMDC datasets and in video captioning on Youcook2 dataset by a large margin. We also demonstrate the generality of the concept by learning improved joint embeddings for other pairs of modalities.
1. Introduction
Cross-modal contrastive learning supports video-text applications but must address two limitations: missing intra-modal similarity preservation and false negatives among randomly sampled negatives. CrossCLR addresses both through similarity-preserving alignment, influential-sample handling, and loss weighting.
- Cross-modal learning enables video retrieval, captioning, and visual feature learning from text metadata, but linking modalities introduces challenges absent from visual-only learning.
- Existing cross-modal losses align paired modalities but do not explicitly keep similar samples within each modality close in the joint embedding.This can produce semantically weak embeddings when related same-modality features map far apart.
- Randomly treating all nonmatching pairs as negatives can mark influential, semantically related samples as negatives and push them apart.Influential samples are similar to many others and therefore strongly affect the embedding geometry.
- CrossCLR preserves similarity from the original feature spaces, removes highly connected influential samples from negatives, and increases their loss weight.These measures target intra-modal alignment and false-negative avoidance.
2. Related Work
Prior work spans multimodal representation learning from raw or large-scale data and joint embedding from pretrained expert features. CrossCLR belongs to the latter group and improves cross-modal consistency while addressing semantic collisions in negative sampling.
- Multimodal video representation methods use modalities such as RGB, motion, audio, text, objects, and scene labels, with recent work often using transformer-based models.
- Some video-language methods learn from very large datasets and noisy temporal pairings, including multiple positives sampled from nearby narration windows.
- Another research line learns joint embeddings from pretrained expert features rather than learning representations from scratch.Examples combine expert similarities, mixture-of-experts gating, or time-aware modality aggregation.
- CrossCLR enforces consistency within each modality and uses influential samples to avoid pushing semantically similar representations apart.
3. CrossCLR
CrossCLR modifies cross-modal contrastive learning to align similar samples within each modality and reduce semantic collisions from false negatives. It uses influential-sample connectivity to prune negatives and weight the loss, within a video-text embedding pipeline.
- 3. Cross-modal learning: The method uses video and text feature representations, hierarchical local and global transformers, and minibatch positive and negative pairs.Video features come from pretrained experts and text features from BERT-Base uncased; local embeddings feed a global transformer.
- 3. CrossCLR: Standard contrastive learning can treat semantically overlapping pairs as negatives, pushing related samples apart and failing to keep similar same-modality samples nearby.The paper calls this false-negative problem semantic collision.
- 3.1. Inter-Modality and Intra-Modality Alignment: CrossCLR learns cross-modal embeddings with inter-modality and intra-modality alignment objectives.The objective includes A-to-B, A-to-A, B-to-A, and B-to-B contrastive components.
- 3.2. Avoiding Semantic Collision: CrossCLR identifies influential samples by their connectivity to many samples and uses that connectivity for negative-set pruning and loss weighting.Influential samples are removed from negative sets, while their loss contribution is increased.
- 3. CrossCLR: The learning algorithm maintains a queue of samples, computes similarities and influential-sample statistics, then updates both modality encoders to minimize the combined loss.The queue supports a larger negative collection than a minibatch.
4. Experiments
Experiments evaluate CrossCLR on YouCook2 and LSMDC for retrieval, using pretrained expert features and comparisons against existing losses. CrossCLR consistently improves retrieval, produces more consistent embeddings, and improves state-of-the-art retrieval and captioning results.
- Datasets and Metrics: CrossCLR is evaluated on LSMDC and YouCook2 using modality-to-modality retrieval metrics including Recall@K, median rank, and mean rank.LSMDC contains 118,081 clips from 202 movies, while YouCook2 contains 2,000 videos and approximately 14,000 clips.
- Expert Features: CrossCLR uses pretrained appearance, scene, action, object, and HowTo100M experts, with feature selection varying across datasets and comparisons.LSMDC Table 1 uses action and appearance features, whereas the state-of-the-art comparison uses all listed expert features.
- Ablation Study: All three CrossCLR components improve results individually and jointly, with intra-modality alignment yielding the largest individual gain.The components are intra-modality alignment, proximity weighting, and negative pruning.
- Experiment Results: CrossCLR consistently improves retrieval over previous contrastive learning methods on both YouCook2 and LSMDC under the same architecture and experimental setup.The comparison includes MaxMargin, MILNCE, NT-Xent, CLIP, and DCL.
- Similarity Analysis: CrossCLR produces lower positive-pair variance than MaxMargin and SimCLR, while allowing semantically similar negative pairs to receive positive similarity scores.MaxMargin shifts toward smaller positive-pair means, whereas SimCLR and MaxMargin concentrate negative scores around zero.
- Experiment Results: On LSMDC, CrossCLR shows higher intra-modality consistency than NT-Xent and improves text-to-video retrieval over the state of the art while using fewer modalities than MMT.The qualitative analysis also includes Top3 text-to-video and video-to-video retrieval examples.
5. Conclusion
The paper concludes that CrossCLR improves joint embeddings by preserving meaningful intra-modality proximity and reducing false negatives through influential-sample handling. These principles improve retrieval and extend beyond video-text embeddings to other modality pairs.
- Conclusion: CrossCLR preserves proximity between semantically related inputs within each modality while learning cross-modal joint embeddings.This assumes pretrained input embeddings already place related concepts near one another.
- Conclusion: Influential-sample detection supports negative pruning and loss weighting to address false negatives, both of which consistently improve retrieval performance.The method uses interconnections in the input embedding to identify influential samples.
- Conclusion: The same CrossCLR principles successfully apply to other modality pairs beyond video and text.The conclusion presents this as evidence that the improvements are not limited to video-text embeddings.
- Architecture: The architecture uses local and global transformer stages, with the CrossCLR loss applied at both levels.The first stage produces clip-level features and the second aggregates them into video-level representations.
A. Architecture
The architecture represents aligned video and paragraph content hierarchically, first encoding clips and sentences locally and then aggregating them globally. CrossCLR is applied at both representation levels.
- Architecture: Each video is represented as consecutive clips and each paragraph as sentences, with corresponding clips and sentences treated as temporally aligned.The alignment provides paired local units for cross-modal representation learning.
- Architecture: Pretrained visual encoders extract per-frame video features, while BERT extracts features for each sentence.YouCook2 additionally uses HowTo100M pretrained features for video frames.
- Architecture: A local transformer produces clip- and sentence-level embeddings, and a global transformer aggregates them into video- and paragraph-level representations.The global stage takes the local embeddings as input.
- Architecture: CrossCLR is applied at both local and global levels with the objective Llocal+0.6Lglobal.The local loss uses a queue, while the global loss operates on video and paragraph representations.
B. Hyper-parameters
The paper reports training hyperparameters and visualizes how sample connectivity identifies semantically similar, influential samples in Youcook2 embeddings.
- B. Hyper-parameters: The authors report hyperparameter ranges and final choices for LSMDC and YouCook2 in Table 7.The table covers optimizer, architecture, and CrossCLR-loss settings.
- B. Hyper-parameters: Dense graph connections identify samples that are semantically similar to many other Youcook2 samples.These highly connected samples are treated as influential in the embedding analysis.
- B. Hyper-parameters: Pruning removes samples whose normalized similarity scores exceed a chosen threshold from the negative set.A threshold of γ = 1 means no pruning.
B.2. Effect of Weight Scale
CrossCLR’s weight scale and pruning threshold affect retrieval performance, with moderate settings preferred and γ = 0.9 reported as reasonable for both datasets.
- B.2. Effect of Weight Scale: Weight scales around 0.003–0.005 perform well, whereas overly large scales prevent CrossCLR from converging reliably.The study trains three models for each tested κ value.
- B.2. Effect of Weight Scale: Multiple positives use top-K influential samples to augment the positive sets for the video and text losses.The reported settings are K = 2 and β = 0.15 for Youcook2, and K = 5 and β = 0.2 for LSMDC.
- B.2. Effect of Weight Scale: CrossCLR with multiple positives performs similarly to standard CrossCLR on Youcook2 but improves results on LSMDC.The authors attribute the observed difference to the larger size of LSMDC.
C.2. Impact of different modality combinations:
Modality choice strongly affects representation quality: stronger experts improve performance, action features perform best individually, and object features are limited by detection challenges.
- C.2. Impact of different modality combinations:: Stronger feature experts yield higher performance, with action features best and object features worst in single-modality experiments.The comparison covers different modality experts and their combinations.
- C.2. Impact of different modality combinations:: The study combines multiple modalities by concatenating their features or by feeding both modalities to the network.Table 9 distinguishes joint network input from separately trained models whose output embeddings are concatenated.
- C.2. Impact of different modality combinations:: The paper leaves architecture design for combining multiple modalities to future work.The study does not investigate alternative multimodal fusion architectures.
- C.2. Impact of different modality combinations:: Object features underperform because current object detectors face domain shift, visual variation, and many objects in movie videos.The experiments use Faster R-CNN detections with scores above 0.7.
D. Qualitative Results
Qualitative analyses visualize CrossCLR’s semantic organization and captioning behavior, while its implementation computes cross- and intra-modality logits, prunes negatives, weights losses, and applies cross-entropy.
- D. Qualitative Results: The captioning examples compare randomly selected Youcook2 samples with COOT and MART using green, yellow, and red quality labels.Green denotes correct captions, yellow acceptable but inaccurate captions, and red wrong captions.
- D. Qualitative Results: The qualitative captioning examples include a MART output with repeated soup-related actions and ingredients.The displayed example includes additions, simmering, and repeated references to soup.
- D. Qualitative Results: The pseudocode computes video-to-text, text-to-video, and intra-modality logits before pruning negative samples.It then concatenates positive and negative logits for the two directional losses.
- D. Qualitative Results: CrossCLR weights losses using semantic proximity and optimizes the directional objectives with unreduced cross-entropy.The pseudocode defines separate video-to-text and text-to-video loss terms before averaging them.
- D. Qualitative Results: Text embeddings cluster semantically and align with corresponding visual semantics in a t-SNE visualization of Youcook2.Text embedding distances determine which videos are shown as close in the projected space.