Source-linked AI summary
ActBERT: Learning Global-Local Video-Text Representations
Linchao Zhu, Yi Yang
TL;DR
ActBERT addresses the challenge of learning fine-grained video-text relations when prior visual-token approaches can lose local information. It jointly models global actions, local regions, and text through a tangled transformer and achieves state-of-the-art performance across five video-text benchmarks.
Problem
Prior video-text pre-training can lose detailed local information during visual-feature clustering, limiting fine-grained video-text relation modeling.
Method
ActBERT jointly models global actions, local regional objects, and text with a tangled transformer and four self-supervised surrogate tasks.
Results
ActBERT achieves state-of-the-art performance with a clear margin across five video-related downstream tasks.
Takeaways & Limitations
ActBERT provides a self-supervised framework for joint video-text representation learning that models both global and local visual cues.
Abstract
from arXiv · showhide
In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperforms the state-of-the-arts, demonstrating its superiority in video-text representation learning.
1 Baidu Research 2 ReLER, University of Technology Sydney
The passage lists contact email addresses associated with the paper.
- linchao.zhu and yi.yang are listed under the UTS.edu.au domain.
- The listed contact addresses use the format linchao.zhu and yi.yang at UTS.edu.au.
- The passage contains contact information rather than research content.
1. Introduction
ActBERT is introduced to learn joint video-text representations from unlabeled instructional videos by explicitly modeling global actions, local objects, and linguistic descriptions. Its pre-trained representations are evaluated across five video-and-language tasks.
- Motivation: ActBERT targets self-supervised joint video-text representation learning from unlabeled instructional videos paired with ASR-derived text.Instructional videos provide widely available video-text relationships for pre-training.
- Motivation: Global human actions are modeled alongside local regional objects because actions capture motion and complex human intention, while prior methods largely treated them like objects.The method extracts verbs from descriptions and trains a 3D convolutional network to produce action embeddings.
- Approach: ActBERT uses a TaNgled Transformer to encode global actions, local regional objects, and linguistic tokens with cross-source interactions.The design extends two-modality transformer approaches by incorporating a global cue to guide local visual and linguistic information.
- Approach: Four surrogate tasks train ActBERT: masked language modeling with global and local visual cues, masked action classification, masked object classification, and cross-modal matching.The framework transfers its pre-trained representations to five downstream video tasks.
- Evaluation: ActBERT is evaluated on video captioning, action segmentation, text-video clip retrieval, action step localization, and video question answering.The paper reports state-of-the-art performance across these downstream tasks.
2. Related Work
Related work spans video-and-language modeling, cross-modal pre-training, and instructional-video learning. Existing approaches often struggle with temporal relations, local regional information, or explicit unified action modeling.
- Video and language: Video-and-language tasks evaluate joint video-text representations through question answering, captioning, retrieval, and grounding.Learning relations between ordered frames and descriptions remains difficult because of temporal information and interactions among multiple objects.
- Cross-modal pre-training: Prior video-text BERT work discretized frame features into visual words but largely ignored local regional information.This limitation motivates richer visual representation modeling.
- Instructional videos: Instructional videos provide complex, multi-domain data, and prior work uses their transcriptions as supervision.ActBERT instead explicitly models human actions and local regions in a unified framework.
3. Model Architecture
ActBERT organizes text, action, and regional object inputs into a unified architecture. Its tangled transformer enables cross-modal interactions, while four pre-training tasks use contextual visual and linguistic cues.
- BERT foundations: BERT supplies the transformer and masked-language-modeling foundations that ActBERT extends to video-text inputs.BERT processes contextualized embeddings with bidirectional self-attention and predicts masked words from surrounding context.
- Input representation: ActBERT sequences linguistic descriptions, action features, and regional features between special [CLS] and [SEP] tokens.Additional separators identify clip boundaries, and action features are extracted after global average pooling.
- Input representation: Regional object features combine Faster R-CNN detection features with spatial-position embeddings.The detector supplies object information under the COCO vocabulary, while the combined representation supports visual-text relation modeling.
- Tangled Transformer: The TaNgled Transformer uses three modality-specific transformers and injects visual information into linguistic processing and linguistic information into visual processing.Its cross-modal attention dynamically selects contextual cues across action, region, and language streams.
- Tangled Transformer: Global action cues guide selection of local linguistic and visual hints rather than treating the two modalities equally.This distinguishes the tangled block from a co-attentional block that directly passes keys and values between modalities.
- Pre-training tasks: ActBERT extends masked language modeling with global and local visual cues and adds masked action classification among its four pre-training tasks.Masked action prediction uses linguistic and object features and can exploit action sequences such as “get into”, “rotate”, and “add”.
4. Experiments
ActBERT’s generalization is evaluated quantitatively across five downstream video-and-language tasks.
- The evaluation covers text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization.
4.1. ActBERT implementation details
ActBERT is pretrained on narrated instructional videos using global action, local regional, and linguistic features extracted with specified visual and textual encoders. Pretraining masks inputs and optimizes a 12-layer transformer model for five epochs.
- ActBERT is pretrained on HowTo100M, a YouTube-derived dataset spanning 23,611 instructional tasks with subtitle narrations.The dataset is action-biased and contains long-tailed object nouns.
- Local regional features come from Faster R-CNN with a ResNet-101 backbone, sampled at 1 FPS with at most five boxes per frame.Regions use a detection confidence threshold of 0.4 and RoI pooling.
- Action features are learned by training a ResNet-3D to classify verbs extracted from text descriptions.The action network is initialized from Kinetics-pretrained weights.
- During pretraining, 15% of input features are masked in a 12-layer transformer with hidden size 768.The linguistic transformer starts from BERT weights, while the other transformers are randomly initialized; training uses Adam at 10−5 for five epochs.
4.2. Results on video-and-text tasks
ActBERT is evaluated across five downstream tasks and multiple datasets, outperforming established baselines in captioning, segmentation, localization, retrieval, and question answering. Results also indicate benefits from regional cues and transfer to settings without linguistic inputs or labeled target data.
- Evaluation setup: The five tasks are evaluated on CrossTask, COIN, YouCook2, MSR-VTT, and LSMDC, with test videos excluded from HowTo100M pretraining.
- Evaluation setup: CrossTask contains 83 tasks and 4.7k videos, while COIN contains 180 tasks, 11,827 videos, and 46,354 annotated segments.
- Video captioning: ActBERT outperforms VideoBERT across all YouCook2 captioning metrics, including a 1.36 improvement on METEOR.The authors attribute the result to modeling global and local video cues.
- Action segmentation: ActBERT significantly outperforms COIN action-segmentation baselines by more than 20% using only visual inputs during fine-tuning.Removing regional information reduces performance, indicating the importance of local cues for dense frame labeling.
- Action step localization: ActBERT improves CrossTask action step localization over TVJE by 7% on average and gains a further 4% with regional cues.The full model also exceeds the supervised baseline reported in the comparison.
- Text-video clip retrieval: For text-video retrieval, ActBERT outperforms TVJE and other baselines on YouCook2 and exceeds TVJE by 1.1% on MSR-VTT R@1 without labeled data.On YouCook2, pretrained ActBERT also surpasses fine-tuned TVJE.
- Video question answering: ActBERT exceeds JSFusion by 2.3% on multiple-choice VideoQA and achieves considerable gains on fill-in-the-blank VideoQA in the movie-domain LSMDC dataset.
5. Conclusion
ActBERT performs self-supervised joint video-text modeling by combining global actions, local visual cues, and linguistic descriptions through a tangled transformer.
- ActBERT performs joint video-text modeling in a self-supervised way.
- The model jointly represents global actions, local regional objects, and linguistic descriptions.
- Quantitative results on five video-text benchmarks demonstrate ActBERT’s effectiveness.