Source-linked AI summary
Object-aware Video-language Pre-training for Retrieval
Alex Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, Mike Zheng Shou
TL;DR
Video-language dual encoders have advanced retrieval but lack regularization for fine-grained semantic associations. OA-Trans incorporates object representations into dual-encoder pretraining, and improves Recall@1 across four benchmarks and three downstream tasks.
Problem
Video-language dual encoders lack regularization for fine-grained semantic associations needed to align visual and textual clues for retrieval.
Source: §1 · p. 1Method
OA-Trans extends dual-encoder pretraining with a masked object anchor image and predicted object tags to enhance fine-grained video-text interaction.
Source: §1 · p. 2, §3.2 · p. 4Results
OA-Trans significantly improves Recall@1 on four benchmarks across three downstream tasks, including MSVD from 46.2% to 51.4%.
Source: §1 · p. 2Takeaways & Limitations
Object information provides a complement to the fine-grained semantic alignment missing from current dual-encoder video-language pretraining.
Source: §5 · p. 8The current implementation relies on an externally provided offline object detector; model-generated boxes remain future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Recently, by introducing large-scale dataset and strong transformer network, video-language pre-training has shown great success especially for retrieval. Yet, existing video-language transformer models do not explicitly fine-grained semantic align. In this work, we present Object-aware Transformers, an object-centric approach that extends video-language transformer to incorporate object representations. The key idea is to leverage the bounding boxes and object tags to guide the training process. We evaluate our model on three standard sub-tasks of video-text matching on four widely used benchmarks. We also provide deep analysis and detailed ablation about the proposed method. We show clear improvement in performance across all tasks and datasets considered, demonstrating the value of a model that incorporates object representations into a video-language architecture. The code will be released at \url{https://github.com/FingerRec/OA-Transformer}.
1. Introduction
Video-language dual encoders have advanced retrieval but lack regularization for fine-grained semantic associations. OA-Trans adds object-guided learning to this framework, using a masked anchor frame and object-aware contrastive training, and improves Recall@1 across the evaluated benchmarks and tasks.
1. Introduction
- Video-language dual encoders lack fine-grained semantic regularization, while existing object-based approaches can require costly whole-dataset extraction and depend on detector quality downstream.The cited extraction burden arises from processing objects across billion-level video frames.Source: §1 · pp. 1–2
- MSVD Recall@1 rises from 46.2% to 51.4%, alongside significant Recall@1 gains on four benchmarks across three downstream tasks.Figure 1(b) compares OA-Trans with state-of-the-art methods across three downstream tasks.Source: Figure 1 · p. 1, §1 · p. 2
- Object boxes overlap salient regions, and predicted tags relate to caption concepts, motivating object-guided masking for video-language pretraining.The examples connect “Music” with “Ratio” and “Laptop” with “Computer.”Source: Figure 1 · p. 1, §1 · p. 1
- OA-Trans adds a single masked object anchor frame and a four-stream object-aware contrastive loss to dual-encoder pretraining while maintaining retrieval efficiency.The four streams are raw video, anchor frame, object tags, and raw text.Source: §1 · p. 2
2. Related Work
Video-language pretraining has shifted toward large-scale end-to-end learning, while object-centric methods have been widely explored in image-language tasks. Bringing object-level features into video-language pretraining remains challenging because of extraction cost and feature-encoder differences.
2.1. Video-Language Pretraining
- Large-scale video-language pretraining moved from offline expert features and pre-extracted video features toward end-to-end methods such as ClipBert and Frozen.The cited methods use large-scale data after HowTo100M and address limitations of smaller-scale datasets and space-time CNN pre-extraction.Source: §2.1 · p. 2
2.2. Object in Vision-Language Tasks
- Object-centric features support image-text alignment, but video-text integration remains unresolved; ActBert requires whole-video object extraction and features from other encoders.The cited computation cost and domain gap between feature encoders limit efficient object-centric video-language pretraining.Source: §2.2 · pp. 2–3
3. Approach
OA-Trans extends a dual-encoder video-language framework with object-centric signals, motivated by compositional video semantics and fine-grained video-text interaction. The approach adds masked-anchor and object-tag streams, then discusses its design advantages and alternatives.
3. Approach
- Object representations motivate the approach because they support compositional semantic understanding and may reduce attention to semantically irrelevant information.The section situates OA-Trans as an extension of the dual-encoder framework.Source: §3 · p. 3
3.1. Dual-encoder Framework
- The baseline uses independent video and text encoders with projection heads; matched pairs are positives, other in-batch pairs are negatives, and the matching loss is symmetric.The projected, normalized video and text embeddings share an embedding space.Source: §3.1 · pp. 3–4
3.2. Object-aware Transformer
- OA-Trans adds a masked anchor-image stream and an object-tag stream, formed from the object frame closest in time to the sampled clip’s center.The masked-image representation averages non-masked patch tokens, while the tag stream represents predicted object categories; object detections are extracted offline from eight uniformly sampled frames.Source: §3.2 · p. 4
- Object-guided masking keeps object-region patches and masks others, with random object dropping and temporal shifts used to reduce overfitting.The method is intended to encourage learning object context and relationships rather than simply modeling scene bias.Source: §3.2 · p. 4
- OAC combines object-tag-to-video and sentence-to-masked-frame alignment instead of directly matching tags to local masked images.The paper motivates this cross-guidance because objects are absent at downstream inference and direct matching may damage global video-text matching.Source: §3.2 · p. 4
- The final objective adds OAC loss to global matching loss, with λ controlling their balance; the authors say this helps learn more discriminative downstream representations.The model encourages both encoders to mine object-centric information.Source: §3.2 · p. 4
3.3. Discussion
- OA-Trans uses one object image during pretraining, reduces downstream effects of noisy objects, and leaves the base vision-encoder architecture unchanged.The authors describe it as plug-and-play with existing video-language pretraining methods.Source: §3.3 · p. 5
- The authors studied pure offline features and joint offline-feature/raw-video modeling, and report their proposed solution as the superior design.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate OA-Trans on video-text retrieval and linear probing, using four downstream benchmarks and ablations of object-aware design choices. OA-Trans improves retrieval over prior methods across the reported benchmarks, while qualitative visualizations and ablations examine how object information contributes.
4.1. Pretraining Datasets; 4.2. Downstream Datasets; 4.3. Setup
- Pretraining uses WebVid2.5M video-text pairs and CC3M image-text data; evaluation covers MSRVTT, DiDeMo, MSVD, and LSMDC, with a ViT-B/16 visual encoder and DistilBERT text encoder.For WebVid2.5M, training samples one object-reference frame and four video frames; downstream fine-tuning samples eight video frames.Source: §4.1 · p. 5, §4.2 · pp. 5–6, §4.3 · p. 6
4.4. Video-Text Retrieval
- OA-Trans beats the fairly compared Frozen baseline by 2.6% on MSRVTT R@1 and exceeds prior methods on both MSVD retrieval settings by at least 5% on R@1.The authors also report that WebVid2.5M-pretrained OA-Trans outperforms methods pretrained on 136M clip-caption pairs.Source: §4.4 · p. 6, Table 2 · p. 6
- On DiDeMo and LSMDC, OA-Trans outperforms previous methods on all reported metrics; the paper also reports better zero-shot results than previous methods.Table 3 covers fine-tuned and zero-shot DiDeMo retrieval, while Table 4 covers LSMDC retrieval.Source: §4.4 · p. 6, Table 3 · p. 6, Table 4 · p. 6
4.5. Linear Probe Evaluation
- Linear probing evaluates pretrained representations without fine-tuning the overall model, addressing concerns that fine-tuning complicates fair comparisons and may hide failures to learn general representations.OA-Trans generalizes well on the reported datasets in this evaluation.Source: §4.5 · pp. 6–7, Table 5 · p. 7
4.6. Qualitative Visualization
- Attention visualizations show OA-Trans focusing on relevant regions for both vocabulary-covered nouns such as “rock” and uncovered nouns such as “food” and “money.”The visualization uses text tokens as queries and spatial patch tokens as keys.Source: §4.6 · p. 7, Figure 4 · p. 7
4.7. Ablation Studies
- Object tags contribute more than object regions in the component ablation, while combining both yields the best result; performance generally rises with more objects before stabilizing beyond 10.Masking probabilities above 0.2 reduce accuracy, which the authors relate to removing too many regions and potentially changing semantics.Source: Table 6 · p. 7, §4.7 · p. 8, Figure 5 · p. 8
- Padding object tags improves retrieval by around 1%, while a two-stream strategy improves R@1 by around 3% in both text-to-video and video-to-text tasks.The authors interpret padding as text augmentation and use Two Stream as the default strategy.Source: §4.7 · p. 8, Table 7 · p. 8
- Using both masked-anchor and raw-video inputs performs best, while masked-anchor-only input exceeds raw-video-only input by around 2.5% on R@10.The authors describe the two streams as complementary and connect their combination to object-region-guided local alignment.Source: §4.7 · p. 8, Table 8 · p. 8
5. Conclusion
Current video-language dual-encoder networks lack fine-grained semantic alignment. OA-Trans uses object bounding boxes and tags to generate a contextualized scene representation, an integration that fits cross-modality transformers.
5. Conclusion
- OA-Trans uses object bounding boxes and tags to generate a contextualized representation of the entire scene.The conclusion presents this as a response to the lack of fine-grained semantic alignment in current video-language dual-encoder networks.Source: §5 · p. 8
- Object regions fit naturally into cross-modality transformers, where they play the same architectural role as uniformly spaced patch tokens.Source: §5 · p. 9
- The current implementation uses an externally provided offline object detector; replacing it with model-generated boxes is left for future work.Source: §5 · p. 9