PaperlayerView exact revision on arXiv

Source-linked AI summary

AIM: Adapting Image Models for Efficient Video Action Recognition

Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, Mu Li

arXiv:2302.03024v1cs.CV

TL;DR

Full finetuning of video models is costly, while image-pretrained models lack temporal reasoning for video tasks. AIM freezes the image model and adds lightweight adapters for spatial and temporal adaptation, achieving comparable or better results with substantially lower tuning costs across four benchmarks. The method is simple and applicable to different image-pretrained models, but its temporal modeling can be insufficient for temporally challenging videos.

  • Problem

    Full finetuning of video models is computationally expensive, while directly using image-pretrained models for video is limited by their lack of temporal reasoning.

  • Method

    AIM freezes a pre-trained image transformer and adds lightweight adapters for spatial, temporal, and joint adaptation during video finetuning.

  • Results

    AIM achieves comparable or higher performance than full-finetuned state-of-the-art methods on four video action-recognition benchmarks, including 87.5% on K400 with 38M tunable parameters.

  • Takeaways & Limitations

    AIM is simple, cost-effective to train, and generally applicable to different image-pretrained models.

  • Takeaways & Limitations

    Reusing image-pretrained self-attention for temporal modeling may be insufficient for temporally challenging videos such as SSv2.

Abstract

from arXiv · show

Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, full finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation to gradually equip an image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters on four video action recognition benchmarks. Thanks to its simplicity, our method is also generally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage is \url{https://adapt-image-models.github.io/}.

1 INTRODUCTION

AIM addresses the cost of full video-model finetuning by freezing a pre-trained image transformer and adding lightweight adapters for spatial and temporal reasoning. It reports competitive or better accuracy with substantially lower tuning and training costs.

  • Full finetuning of video models can require substantial computation and memory, motivating parameter-efficient alternatives.One cited example required 1200 Tesla V100 GPU hours to train.
  • Directly adapting image models to video remains less explored because image models lack temporal reasoning capability.
  • AIM freezes the pre-trained image model and adds lightweight adapters for spatial, temporal, and joint adaptation.The method progressively equips image representations with spatiotemporal reasoning while updating only the added adapters.
  • 50% lower memory footprint and 42% lower training time are reported on Swin-B compared with VideoSwin.
  • 87.5% accuracy on K400 is achieved with 38M tunable parameters, while matching or exceeding full-finetuned state-of-the-art methods across four benchmarks.
  • 9% absolute accuracy improvement over TimeSformer is reported when using 1% of the training data.

2 RELATED WORK

Related work spans image-pretrained initialization for video recognition, temporal extensions or inflation into video models, and parameter-efficient finetuning developed first in NLP and later applied to vision.

  • Image pre-trained models: Image-pretrained vision transformers provide initialization for downstream transfer learning and achieve strong image-recognition performance.
  • Video action recognition: Video action-recognition methods commonly extend image models with temporal modules or inflate them into video models.
  • Parameter-efficient finetuning: Parameter-efficient finetuning reduces trainable parameters and computation while targeting performance comparable to or better than full finetuning.The approach originated in NLP as increasingly large language models made full finetuning less feasible and was subsequently studied in computer vision.

3 METHODOLOGY

AIM adapts frozen image-based ViT blocks for video by progressively adding spatial, temporal, and joint adapters, enabling spatiotemporal reasoning with lightweight tuning.

  • Preliminary: AIM starts from a ViT that represents images as patch sequences processed by transformer blocks containing self-attention, MLP, normalization, and skip connections.The final class tokens provide global visual representations for classification.
  • Preliminary: Video modeling requires both frame-level appearance modeling and temporal reasoning across frames, whereas a space-only baseline processes frames independently and averages their class tokens.Space-time models add temporal modules but typically require costly full finetuning.
  • Spatial adaptation: Spatial adaptation inserts a bottleneck Adapter after self-attention, updates only the Adapters, and achieves comparable performance to a fully finetuned space-only baseline.The frozen image model thereby adapts its spatial features to video data, although spatial adaptation alone lacks temporal modeling.
  • Temporal adaptation: Temporal adaptation reuses the frozen image model’s self-attention across the frame dimension, then adds an Adapter to adapt the resulting temporal features.The reused temporal and spatial attention layers share weights but operate on different input dimensions, avoiding additional attention parameters.
  • Joint adaptation: Joint adaptation adds another Adapter in parallel with the MLP to jointly tune spatiotemporal representations after sequential spatial and temporal adaptation.The final prediction averages the class tokens from input frames before classification.

4 EXPERIMENTS

Experiments evaluate AIM on four video action-recognition benchmarks and show that lightweight adaptation progressively adds spatiotemporal reasoning while retaining competitive performance with fewer tunable parameters. AIM also improves efficiency, data efficiency, and performance across several benchmark settings, though it trails some fully finetuned video models on the temporal-heavy SSv2 dataset.

  • Datasets: AIM is evaluated on K400, K700, SSv2, and Diving-48, with SSv2 requiring stronger temporal modeling and Diving-48 designed to resist reliance on static representations.The four benchmarks cover broad action recognition, temporal reasoning, and fine-grained actions.
  • Effectiveness of Components: Spatial adaptation raises SSv2 accuracy from 15.1% to 36.7% while tuning 3.7M rather than 86M parameters, matching the full-finetuned space-only model at 36.2%.The component study compares frozen space-only, full-finetuned space-only, and full-finetuned space-time baselines.
  • Effectiveness of Components: Switching from IN-21K to CLIP pre-training improves accuracy from 62.0% to 66.4%, showing that AIM can benefit from stronger image-pretrained representations.The reported comparison uses the same adaptation approach with different image-pretrained backbones.
  • State-of-the-Art Comparisons: On K400, AIM ViT-B/16 reaches 83.9% top-1 accuracy with 606 GFLOPs, while AIM ViT-L/14 reaches 87.5% using 38M tunable parameters.With 16 frames, AIM also outperforms MTV-L while using fewer computations.
  • State-of-the-Art Comparisons: AIM remains competitive across K700, SSv2, and Diving-48, including 70.6% versus EVL's 66.7% on SSv2 and 90.6% top-1 accuracy on Diving-48.On K700, AIM ViT-B/16 outperforms MTV-L and MViTv2-B, while AIM ViT-L/14 is comparable to MaskFeat despite less demanding inputs.
  • Limitations: On SSv2, AIM falls behind some fully finetuned video models because image pre-training and reused self-attention may not capture complicated temporal information fully.The paper identifies SSv2 as a temporal-heavy dataset and suggests additional temporal adaptation for such settings.

5 DISCUSSION

AIM transfers effectively across pretrained image backbones while improving data efficiency, reducing training costs, and exposing trade-offs in adapter placement and capacity.

  • Different Pre-trained Models: AIM outperforms full-finetuned TimeSformer with both IN-21K and CLIP pretrained ViT-B weights, while achieving comparable performance to VideoSwin with Swin-B.These comparisons use matching backbones and related attention structures where applicable.
  • Data Efficiency: 8.9% absolute accuracy improvement over TimeSformer occurs when AIM uses only 1% of K400 training data.AIM outperforms full-finetuned TimeSformer across all tested data amounts, with larger advantages as data decreases.
  • Training Cost: 50% lower memory cost and 42% shorter training time are achieved versus VideoSwin on the Swin-B backbone.Compared with TimeSformer, AIM reduces memory cost by 30% and training time by 25%.
  • Position of Adapters: Adapters placed in the bottom six transformer blocks perform much worse than placements in upper, distributed, or all blocks.All compared variants use the same number of tunable parameters.
  • Bottleneck Ratio of Adapters: A larger adapter bottleneck ratio generally improves performance, but performance plateaus beyond 0.25; a ratio of 0.0625 still reaches 83.3% top-1 accuracy.Increasing the ratio also increases the number of tunable parameters.

6 CONCLUSION

AIM equips frozen pretrained image models with spatiotemporal reasoning through lightweight spatial, temporal, and joint adapters. It reduces training cost while achieving comparable or better performance across four benchmarks, though temporal modeling remains limited for challenging videos.

  • 6 CONCLUSION: AIM introduces spatial, temporal, and joint adaptation while updating only newly added adapters.The method is designed to transfer pretrained image models efficiently to video action recognition.
  • 6 CONCLUSION: AIM achieves comparable or better performance than prior methods on four video action recognition benchmarks with substantially lower training cost.The method is described as simple and generally applicable to different image pretrained models.
  • 6 CONCLUSION: Reusing spatial attention for temporal modeling may be insufficient for temporally challenging videos.The paper suggests future reuse of pretrained text or audio model weights for video temporal modeling.

A IMPLEMENTATION DETAILS

The implementation adds spatial, temporal, and joint adapters throughout ViT blocks, initializes them to preserve the pretrained model initially, and varies training schedules and frame sampling across experiments.

  • A IMPLEMENTATION DETAILS: Spatial, temporal, and joint adapters are added in every ViT block, with bottleneck ratio 0.25 and scaling factor 0.5.The first adapter fully connected layer is randomly initialized, while the second is initialized to zero.
  • A IMPLEMENTATION DETAILS: Zero-initializing the second adapter fully connected layer keeps the adapted model close to the pretrained model at training start.This initialization is used alongside random initialization of the first fully connected layer.
  • A IMPLEMENTATION DETAILS: The implementation uses 50 training epochs with AdamW and stronger augmentations including label smoothing, RandAugment, and random erasing.An additional adapter is placed before T-MSA to enhance temporal modeling.
  • A IMPLEMENTATION DETAILS: Experiments uniformly sample 8, 16, or 32 frames, while inference samples one temporal clip.These experiments use the stated adapter configuration and a 50-epoch schedule.

B VISUALIZATION

Attention visualizations compare frozen, spatially adapted, temporally adapted, and fully finetuned models to show how temporal adaptation changes attention toward motion-relevant regions.

  • B VISUALIZATION: The visualization compares a frozen space-only model, SA, SA plus TA, and full-finetuned TimeSformer.These are the four model variants shown in the attention-map analysis.
  • B VISUALIZATION: For Brush Painting, adding SA concentrates attention on the brush, while TA further focuses it on the brush-painting region.The frozen model’s attention is scattered and includes unrelated areas.
  • B VISUALIZATION: For Something falling like a rock, frozen and SA models attend to the object but fail to model its movement, whereas TA learns relationships across frames.Correct recognition requires modeling how the object moves through the input frames.

C PER-CLASS ANALYSIS

AIM’s performance varies by the temporal complexity of SSv2 actions: it does better on ordinary motion but worse when distinguishing subtle directional differences.

  • AIM performs better on normal action classes with decent motion but worse on classes differing only in subtle motion.Examples include distinguishing “Pulling something from left to right” from its right-to-left counterpart.
  • The authors conjecture that reusing image-pretrained self-attention for temporal modeling cannot fully capture complicated temporal information in some SSv2 classes.Most AIM parameters remain frozen, potentially limiting capacity for complex temporal information.
  • AIM falls behind some state-of-the-art full-finetuned video models on the temporal-heavy SSv2 dataset.

D MORE COMPARISONS OF TRAINING COST

AIM reduces training costs relative to full-finetuned video models, including large backbones, while tuning fewer parameters and gaining larger improvements when training data is scarce.

  • 14.3G and 13.7G GPU memory are sufficient for AIM with ViT-L and Swin-L, respectively, whereas TimeSformer requires 21.2G and VideoSwin cannot fit on an 8 Tesla V100 32G server.
  • Tuning fewer parameters may benefit communication-efficient and privacy-preserving federated learning because tunable parameters are communicated between servers and clients.
  • Table 9 compares AIM with full-finetuned models on training memory cost, while Table 10 compares AIM and EVL under different pre-training datasets.
  • AIM obtains larger accuracy improvements over the full-finetuned baseline when only a small amount of training data is available.

E COMPARISON TO EVL UNDER DIFFERENT PRE-TRAINED DATASETS

Across IN-21K and CLIP pre-training, AIM consistently outperforms EVL while using considerably fewer tunable parameters; its ViT implementation adds lightweight spatial, temporal, and joint adapters.

  • AIM consistently outperforms EVL under both IN-21K and CLIP pre-training while using considerably fewer tunable parameters.
  • AIM is presented as simple to implement through PyTorch-style pseudocode for an adapted ViT block.
  • AIM’s adapted ViT block contains spatial, temporal, and MLP adapters alongside the original attention, normalization, and MLP layers.
  • Temporal adaptation rearranges tokens by frames, applies the temporal adapter to normalized self-attention outputs, and adds the result residually.
  • Spatial adaptation applies an adapter to normalized self-attention outputs and adds the adapted representation residually.
  • Joint adaptation adds a scaled MLP adapter output to the original MLP transformation after normalization.
Loading 2302.03024v1…