ActBERT: Learning Global-Local Video-Text Representations

• Narrated instructional videos provide abundant, accessible paired data because their speech-recognized text aligns with frames and describes scene objects and salient actions.ActBERT uses text aligned by off-the-shelf automatic speech recognition models.Source: §1 · p. 1
• Sun et al. [35] discretize visual features into visual words, but clustering can lose local details such as interacting objects and human actions, limiting fine-grained video-text relations.Source: §1 · p. 1
• ActBERT jointly models global actions, local regional objects, and text, extracting verbs to train action labels and using regional object features and positions as fine-grained cues.A 3D convolution network predicts action labels, whose features become clip-level action embeddings.Source: §1 · pp. 1–2
• Its TaNgled Transformer block encodes actions, regional objects, and linguistic tokens, using modality-specific transformers and two additional multi-head attention blocks for cross-modal communication.Action features catalyze interactions, guiding visual information into the linguistic transformer and linguistic information into the visual transformers.Source: §1 · p. 2
• ActBERT achieves state-of-the-art performance with a clear margin on video captioning, action segmentation, text-video clip retrieval, action step localization, and video question answering.The paper transfers its pretrained model to all five tasks.Source: §1 · p. 2
• ActBERT is trained with four surrogate tasks: masked language modeling with global and local visual cues, masked action classification, masked object classification, and cross-modal matching.Source: §1 · p. 2

Video and language

• Video-language modeling must capture temporal frame-description relations and spatiotemporal object interactions, while many methods rely on manually generated descriptions and substantial human involvement.The paper instead uses automatically generated ASR descriptions as a scalable source for self-supervised video-text learning.Source: §2 · p. 2

Cross-modal pre-training

• Prior approaches use RNN variants [29] [55] or masked encoder-decoder transformers for dense captioning [54], while video-text BERT [35] largely ignores local regions.Video-audio pretraining [3] [16] uses low-level audio signals and considers video synchronization.Source: §2 · p. 2

Instructional videos

• Instructional videos span diverse domains [6] [1] [53] [27], and prior work often uses their transcriptions as supervision [1] [53] [27]; ActBERT jointly models actions and local regions.The paper says ActBERT improves relation modeling over [27] and is more suitable for unsupervised video-text modeling.Source: §2 · p. 2

3.1. Preliminary

• BERT uses bidirectional self-attention to aggregate contextual features, with position embeddings supplying sequence-order information; its pretraining includes masked-word prediction and next-sentence prediction.Masked language modeling predicts randomly masked words from context, while next-sentence prediction classifies whether two sampled sentences are adjacent in the correct order.Source: §3.1 · p. 3

3.2.1 Input Embeddings

• ActBERT embeds text, global actions, and local regions in one sequence, combining position, segment, token, and visual-feature embeddings.Action features come from a 3D CNN trained on clip verbs; regional features combine detector outputs with spatial-position embeddings.Source: §3.2.1 · pp. 3–4

3.2.2 Tangled Transformer

• ActBERT’s tangled transformer uses separate text, action, and region transformers, with cross-modal attention to select cues and blend linguistic and visual features.It uses the global action representation to guide cue selection, then stacks generated key-value pairs with the original ones so both modalities contribute during encoding.Source: §3.2.2 · p. 4

3.2.3 ActBERT Training

• ActBERT pre-trains with masked language modeling, masked action classification, masked object classification, and cross-modal matching.Visual actions and objects help predict masked words; language and objects inform masked-action prediction, masked-region targets use detector distributions, and matching classifies text-video relevance using positive and negative pairs.Source: §3.2.3 · pp. 5–6

4. Experiments

• ActBERT is quantitatively evaluated on five challenging downstream tasks.The evaluation covers text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization.Source: §4 · p. 6
• The evaluated tasks include text-video clip retrieval, video captioning, and video question answering.These tasks assess ActBERT across retrieval, captioning, and question answering.Source: §4 · p. 6
• The evaluation also covers action segmentation and action step localization.Both action-focused tasks are included among the five downstream evaluations.Source: §4 · p. 6

4.1. ActBERT implementation details

• ActBERT pretraining uses 10 adjacent HowTo100M clips with temporally aligned narration tokens, allowing the model to cover longer context and relations across segments.HowTo100M contains 23,611 tasks and subtitle-derived narrations; clip-text associations can be weak or noisy, though pretraining on the dataset improved downstream-task performance.Source: §4.1 · p. 6
• Local regional features come from Faster R-CNN pretrained on Visual Genome [17] following [22], with a ResNet-101 [9] backbone and 1-FPS extraction.Each frame has at most five RoI-pooled boxes above a 0.4 confidence threshold; visual-stream transformer blocks use hidden size 1024 and eight attention heads.Source: §4.1 · p. 6
• Action features are learned by training a ResNet-3D [41] classifier on text-derived verbs, initialized from Kinetics [12] and fed 32-frame, four-second clips.Training uses 8-FPS sampling, 224×224 inputs, softmax loss, learning rate 0.001, batch size 16, and one million iterations; globally averaged features represent clips.Source: §4.1 · p. 6
• ActBERT masks 15% of input features and uses 12 transformer layers with 768 hidden units, initializing the language transformer from BERT pretrained on BookCorpus [58] and English Wikipedia.The other two transformers are randomly initialized; Adam optimization uses a learning rate of 10^-5 for five epochs.Source: §4.1 · p. 6

4.2.1 Datasets

• The five evaluations cover action step localization on CrossTask, action segmentation on COIN, retrieval and captioning on YouCook2, retrieval and multiple-choice VideoQA on MSR-VTT, and fill-in-the-blank VideoQA on LSMDC.The evaluated datasets include cooking, instructional, general web-video, and movie data; test videos from these datasets are removed during HowTo100M pretraining.Source: §4.2 · p. 6, §4.2.1 · pp. 6–7

4.2.2 Video captioning

• ActBERT improves YouCook2 video captioning METEOR by 1.36 over VideoBERT.The comparison uses a pre-trained action transformer without the regional object transformer and reports BLEU, METEOR, and ROUGE on the validation set.Source: §4.2.2 · p. 7, Table 1 · p. 7

4.2.3 Action segmentation

• ActBERT significantly outperforms COIN action-segmentation baselines with more than 20% improvements.Removing regional information reduces performance; fine-tuning uses visual inputs without text descriptions.Source: §4.2.3 · p. 7, Table 2 · p. 7

4.2.4 Action step localization

• ActBERT’s average improvement over TVJE on CrossTask action step localization is 7%, and its full model further improves performance by 4%.ActBERT without region cues also substantially outperforms TVJE, while the full model benefits from regional information.Source: §4.2.4 · p. 7, Table 3 · p. 8

4.2.5 Text-video clip retrieval

• ActBERT outperforms TVJE and other baselines on YouCook2 retrieval, exceeding TVJE +FT, and improves MSR-VTT R@1 by 1.1% over TVJE without labeled data.The evaluation uses Recall@1, Recall@5, Recall@10, and median rank.Source: §4.2.5 · p. 8, Table 4 · p. 8

4.2.6 Video question answering.

• ActBERT outperforms JSFusion by 2.3% on MSR-VTT multiple-choice VideoQA and achieves considerable gains on LSMDC fill-in-the-blank VideoQA.The LSMDC result is reported as public-test accuracy and indicates generalization to movie-domain videos; no exact gain is given in the passage.Source: §4.2.6 · pp. 8–9, Table 5 · p. 8, Table 6 · p. 8

5. Conclusion

• ActBERT models global and local visual cues for fine-grained visual-linguistic relation learning from unlabeled video-text data.It takes global actions, local regional objects, and linguistic descriptions as input.Source: §5 · p. 9
• A novel tangled transformer enhances communication among global actions, local regional objects, and linguistic descriptions.Source: §5 · p. 9
• Quantitative results on five video-text benchmarks demonstrate ActBERT’s effectiveness.The authors identify video action recognition and detection, and more powerful video and text modeling modules, as future directions.Source: §5 · p. 9