PaperlayerView exact revision on arXiv

Source-linked AI summary

SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning

Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, Lijuan Wang

arXiv:2111.13196v4cs.CV

TL;DR

Video captioning traditionally relies on offline-extracted features from fixed-rate frames and mismatched image or video understanding tasks. SwinBERT directly processes raw video frames with an end-to-end Transformer and learnable sparse attention, outperforming prior methods across five benchmarks, often by large margins.

  • Problem

    Video captioning commonly uses offline-extracted dense features from fixed-rate frames, with a discrepancy between feature-extractor training and the downstream captioning task.

  • Method

    SwinBERT is an end-to-end pure Transformer that directly takes raw video frames, supports variable-length frame sequences, and uses a learnable Sparse Attention Mask to model long-range video sequences.

  • Results

    SwinBERT outperforms previous state-of-the-art methods across five video captioning benchmarks, including CIDEr gains of +25.4 on MSVD and +55.4 on YouCook2.

  • Takeaways & Limitations

    The Sparse Attention Mask regularizes training, improves long-range video sequence modeling, and provides further performance improvements across all five datasets.

Abstract

from arXiv · show

The canonical approach to video captioning dictates a caption generation model to learn from offline-extracted dense video features. These feature extractors usually operate on video frames sampled at a fixed frame rate and are often trained on image/video understanding tasks, without adaption to video captioning data. In this work, we present SwinBERT, an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description. Instead of leveraging multiple 2D/3D feature extractors, our method adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates. Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames as opposed to previous successes with sparsely sampled video frames for video-and-language understanding tasks (e.g., video question answering). Moreover, to avoid the inherent redundancy in consecutive video frames, we propose adaptively learning a sparse attention mask and optimizing it for task-specific performance improvement through better long-range video sequence modeling. Through extensive experiments on 5 video captioning datasets, we show that SwinBERT achieves across-the-board performance improvements over previous methods, often by a large margin. The learned sparse attention masks in addition push the limit to new state of the arts, and can be transferred between different video lengths and between different datasets. Code is available at https://github.com/microsoft/SwinBERT

1. Introduction

SwinBERT addresses video captioning with an end-to-end Transformer that directly processes video frames, replacing offline-extracted 2D/3D features. Across five benchmarks, it combines dense frame sampling with learnable sparse attention and substantially improves captioning performance.

  • Video captioning describes a video's visual content in natural language while modeling spatial-temporal dynamics and visual-textual relationships.
  • SwinBERT is an end-to-end fully Transformer-based model that directly takes raw video frames as inputs for caption generation.It uses a video Transformer instead of offline-extracted 2D/3D features.
  • Unlike prior fixed-frame-rate approaches, its video Transformer learns from variable-length frame sequences, and experiments favor denser sampling for captioning.The reported comparison uses 64 frames for SwinBERT versus 16 for CLIPBERT.
  • SwinBERT introduces a learnable Sparse Attention Mask to reduce redundancy in consecutive frames and improve long-range video sequence modeling.The mask is presented as a regularizer and is quantitatively validated for caption generation.
  • +25.4 CIDEr on MSVD, +55.4 on YouCook2, +0.9 on MSRVTT, +5.9 on TVC, and +14.9 on VATEX versus prior state-of-the-art methods.These are absolute CIDEr improvements across five video captioning benchmarks.

2. Related Work

Video captioning builds on fixed visual representations and transformer-based video understanding, while related video-language models target retrieval or other tasks. The paper positions video transformers as an unexplored direction for caption generation.

  • Video captioning methods commonly refine fixed frame features with an encoder and generate descriptions through a language decoder.
  • The proposed framework combines VidSwin video tokens with a multimodal transformer encoder and masked language modeling for caption generation.
  • Video transformers model spatial-temporal representations for video understanding, but prior work focused mainly on action recognition rather than captioning.
  • Pure-transformer video-language models such as Frozen-in-Time and CLIP4Clip focused on retrieval and cannot be directly applied to video captioning.

3. Method

SwinBERT directly processes raw video frames with VidSwin and a multimodal transformer encoder to generate captions. It supports variable-length inputs and uses a learnable sparse attention mask to reduce redundant video-token connections during end-to-end training.

  • 3.1. Model Architecture: SwinBERT uses VidSwin to encode raw-frame spatial-temporal representations, then a multimodal transformer encoder generates the caption.
  • 3.1. Model Architecture: The architecture tokenizes VidSwin grid features into video tokens before feeding them to the multimodal transformer for caption generation.
  • 3.1. Model Architecture: Variable-length video sequences are supported, and longer densely sampled inputs can improve caption performance measured by CIDEr scores.
  • 3.1. Model Architecture: The multimodal encoder performs sequence-to-sequence generation using textual and visual inputs with causal self-attention for caption tokens.
  • 3.2. Learning with Sparse Attention Mask: Longer videos provide more information but increase attention cost, while consecutive frames introduce redundancy that motivates the learnable Sparse Attention Mask.
  • 3.2. Learning with Sparse Attention Mask: The attention mask constrains video-token interactions while leaving text-token attention unrestricted to preserve access to visual details.
  • 3.2. Learning with Sparse Attention Mask: The sparsity regularizer uses λ and mask activations V_i,j to encourage stronger important-token relationships and fewer meaningless connections.
  • 3.2. Learning with Sparse Attention Mask: Sigmoid activations produce continuous mask values between 0 and 1, with a threshold of 0.5 yielding a binary mask.

4. Experiments

Experiments evaluate SWINBERT across five video-captioning benchmarks, frame-sampling settings, sparse-attention variants, longer sequences, transfer settings, and qualitative examples. SWINBERT consistently improves captioning performance, with sparse attention supporting long-range modeling while reducing attention redundancy.

  • Main Results: SWINBERT achieves better performance than three multimodal methods on TVC using video frames alone.TVC captions include visual events and supplementary subtitle information, while VALUE, HERO, and MMT use visual and subtitle inputs.
  • Main Results: 109.0 CIDEr on YouCook2 is +55.4 higher than VALUE and +44.0 higher than ActBERT.These comparisons concern visual-only approaches on YouCook2.
  • Frame Sampling: Increasing sampled frames from 2 to 64 consistently improves CIDEr on MSRVTT and VATEX.The experiment disables the sparse attention mask and evaluates T = {2, 4, 8, 16, 32, 64} frames.
  • Sparse Attention: The proposed sparsity constraint improves CIDEr by +2.8 on MSRVTT and +0.5 on VATEX over the ablation baselines.The comparison includes no learnable mask and a learnable mask without sparsity constraints.
  • Sparse Attention: Learnable sparse attention outperforms full and heuristic window attention, while improving CIDEr across sequence lengths and producing new state-of-the-art results on all five benchmarks.Spatial Window and Temporal Window degrade performance relative to Full Attention, whereas the learnable mask improves over both.
  • Transferability: Sparse attention transfers from 32 to 64 frames with comparable or better CIDEr, and transferring across datasets improves CIDEr on both target datasets.Temporal linear interpolation supports frame-rate transfer, while VATEX pretraining followed by fine-tuning improves MSRVTT and MSVD.
  • Training Behavior: More than 95% of attention-mask elements become zero without stopping CIDEr from increasing during training.This indicates that the sparsity constraint reduces non-zero attention elements while captioning optimization continues.

5. Conclusion

SwinBERT combines an end-to-end fully Transformer-based video-captioning architecture with adaptively learned sparse attention. Across five benchmark datasets, it outperforms previous state-of-the-art methods by a large margin, while future work targets large-scale video-language pre-training.

  • SwinBERT is an end-to-end fully Transformer-based architecture for video captioning with adaptively learned sparse attention.
  • Across 5 popular benchmark datasets, SwinBERT achieves better performance than previous state-of-the-art methods by a large margin.
  • Large-scale video-language pre-training is identified as future work to further enhance captioning performance.

A. Analysis of Different Video Backbones

SwinBERT generalizes across video backbones, with sparse attention consistently improving performance. However, TimeSformer produces longer video-token sequences, increasing language-model memory costs and limiting scalability to longer inputs.

  • SwinBERT often outperforms recent state-of-the-art methods using either TimeSformer or Video Swin Transformer as the video backbone.Adding sparse attention consistently improves performance across the considered backbones, and stronger backbones yield better captioning performance.
  • TimeSformer generates longer video-token sequences than Video Swin Transformer, creating extra language-model memory costs from quadratic complexity.GPU memory constraints limited rebuttal-period TimeSformer training to 8 frames per clip, while Video Swin offers a favorable memory-accuracy trade-off.

B. Influence of Pre-Training on Backbone

SwinBERT remains competitive when compared under matched or different pre-training conditions. Its performance is consistently improved by the sparse-attention regularization across tested hyperparameter choices.

  • With the same SlowFast backbone, SwinBERT achieves better performance than VALUE.
  • Despite using less pre-training data than VALUE, SwinBERT achieves better caption performance.VALUE uses CLIP-ViT and SlowFast backbones pre-trained on 400M image-text pairs and Kinetics-400, whereas SwinBERT uses ImageNet and K400/600 pre-training.
  • SwinBERT gives consistent improvements across different choices of the sparsity regularization hyperparameter λ.

D. Additional Qualitative Results

Qualitative examples show that SwinBERT generates semantically correct or reasonable captions across cooking, open-domain, descriptive, and event-focused video datasets. The examples include fine-grained object, action, and movement recognition.

  • On cooking videos, SwinBERT produces captions that are semantically correct even when their wording differs from the ground truth.For example, “Place the basil on the pizza” is considered semantically correct for “Place basil leaves on top of the pizza.”
  • On MSRVTT, SwinBERT recognizes varied actions including giving a speech, applying makeup, and playing golf.
  • On VATEX, SwinBERT recognizes fine-grained objects from multiple viewpoints and generates semantically reasonable captions.Examples include drum set, paper airplane, high chair, and curling iron.
  • On MSVD, SwinBERT correctly recognizes video events and detailed movements such as dancing and playing a flute.

E. Additional Training Details

The model uses end-to-end training with a pretrained Video Swin Transformer and multimodal transformer, alongside standardized frame preprocessing and task-specific training settings. Additional evaluations cover backbone choices, pretraining data, and dataset splits, while the paper notes accessibility benefits and deployment limitations.

  • Model Training: Both VidSwin and the multimodal transformer are trained end-to-end, with VidSwin initialized from Kinetics-600 pretrained weights.The multimodal transformer has 12 layers and hidden size 512.
  • Input Preprocessing: Video frames are resized to a 224-pixel shorter side, randomly cropped during training, and center-cropped during inference.Training uses the same crop location for every frame in a video.
  • Optimization: Task-specific training epochs and learning rates are selected based on validation-set performance because datasets differ in scale and domain.
  • Broader Impact and Limitations: Video captioning may improve accessibility for low-vision and blind users, but generated captions remain drafts because performance is sensitive to training-data distributions.The authors caution that out-of-distribution videos and incorrect semantics or identities can produce undesirable predictions requiring further editing.

G. Additional Results on Different Splits

The additional results report validation, test, and private-test evaluations and discuss how computational constraints limit conclusions about the best input length. Sparse attention improves the baseline across λ settings, while dataset characteristics shape the relative importance of temporal and spatial features.

  • Different Splits: Validation, test, and private-test split results are reported, with private-test scores obtained from the VALUE leaderboard evaluation server.
  • Input Length: 64-frame inputs give the best observed performance, while 128-frame training is inefficient because GPU memory constraints force batch size 1.The authors report that more frames appear beneficial, but empirically conclude only that 64 frames perform best.
  • Sparse Attention: Sparse attention consistently improves the baseline across different choices of λ.
  • Dataset Characteristics: VATEX captions emphasize detailed actions requiring more temporal features, whereas MSVD captions focus more on scenes and objects requiring stronger spatial features.
  • Qualitative Results: Qualitative examples are provided for YouCook2, MSRVTT, VATEX, and MSVD.
Loading 2111.13196v4…