PaperlayerView exact revision on arXiv

Source-linked AI summary

ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning

Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, Hongsheng Li

arXiv:2206.13559v3cs.CV

TL;DR

Full fine-tuning is costly, while pre-trained image models lack temporal knowledge for video understanding. The paper proposes ST-Adapter for parameter-efficient image-to-video transfer, achieving performance comparable to or better than full fine-tuning and state-of-the-art video methods with far fewer updated parameters.

  • Problem

    Parameter-efficient transfer from large pre-trained image models to video tasks remains limited because image models lack temporal knowledge and video models are costly to build and train.

  • Method

    ST-Adapter adds a lightweight bottleneck with feature reduction, spatio-temporal modeling, and feature recovery to a pre-trained image model.

  • Results

    ST-Adapter performs on par with full fine-tuning on K400 and SSv2 while updating 7.2M versus 121.57M parameters.

  • Takeaways & Limitations

    ST-Adapter enables large pre-trained image models to support video action recognition with substantially greater parameter efficiency than full fine-tuning.

  • Takeaways & Limitations

    SA+TS is incompatible with Linear Probing because channel shifting alters the original model behavior while freezing the backbone.

Abstract

from arXiv · show

Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus on downstream tasks from the same modality (e.g., image understanding) of the pre-trained model. This creates a limit because in some specific modalities, (e.g., video understanding) such a strong pre-trained model with sufficient knowledge is less or not available. In this work, we investigate such a novel cross-modality transfer learning setting, namely parameter-efficient image-to-video transfer learning. To solve this problem, we propose a new Spatio-Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning per video task. With a built-in spatio-temporal reasoning capability in a compact design, ST-Adapter enables a pre-trained image model without temporal knowledge to reason about dynamic video content at a small (~8%) per-task parameter cost, requiring approximately 20 times fewer updated parameters compared to previous work. Extensive experiments on video action recognition tasks show that our ST-Adapter can match or even outperform the strong full fine-tuning strategy and state-of-the-art video models, whilst enjoying the advantage of parameter efficiency. The code and model are available at https://github.com/linziyi96/st-adapter

1 Introduction

The paper studies parameter-efficient transfer from large pre-trained image models to video tasks, where full fine-tuning is costly and image models lack temporal reasoning. It introduces ST-Adapter and benchmarks image-to-video adaptation against efficient and fully fine-tuned video approaches.

  • Full fine-tuning of increasingly large foundation models for each downstream task becomes prohibitively expensive in training cost and model storage.
  • Video understanding is especially costly and challenging because pre-trained image models lack the temporal reasoning needed to model dynamic content.
  • The paper benchmarks multiple image-to-video fine-tuning strategies and state-of-the-art video models using CLIP- and ImageNet-21K-pre-trained ViT models.
  • ST-Adapter uses a compact bottleneck that reduces feature dimensions, performs spatial-temporal modeling, and restores feature dimensions.
  • Experiments show ST-Adapter outperforms efficient alternatives, full fine-tuning, and state-of-the-art video methods on action recognition datasets.

2 Related Work

Related work develops efficient adaptation methods primarily for language and image or vision-language tasks, while video action recognition research emphasizes architectures capable of modeling temporal information.

  • NLP parameter-efficient transfer methods commonly add task-specific adapters between layers and update only those lightweight modules.
  • Vision transfer methods apply prompt or adapter-style tuning mainly to image recognition and vision-language tasks, often focusing on CLIP’s text encoder.
  • Video action recognition has progressed from CNNs to Transformers across datasets such as Kinetics and Something-Something.

3 Methodology

The methodology begins with temporal aggregation and spatio-temporal attention baselines, then adapts the NLP Adapter concept into ST-Adapter for efficient spatial-temporal reasoning in video.

  • Temporal aggregation: Temporal aggregation averages per-frame class-token representations to produce a compact clip representation for classification.
  • Spatio-temporal attention: Spatio-temporal attention methods add temporal attention to spatial modeling, but representative video ViTs generally require full per-task fine-tuning.
  • Adapter preliminaries: The standard Adapter applies down-projection, nonlinear activation, up-projection, and residual addition to transform layer features efficiently.
  • Spatio-Temporal Adapter (ST-Adapter): ST-Adapter extends the Adapter with a depth-wise 3D convolution between bottlenecks to provide spatio-temporal reasoning in compressed features.
  • Spatio-Temporal Adapter (ST-Adapter): The down-projected features are reshaped into temporal and spatial dimensions before depth-wise convolution, enabling layer-wise spatio-temporal modeling.
  • Integration: ST-Adapter modules can be integrated at different Transformer positions, with a single module providing decent empirical performance.

4 Experiments

Experiments evaluate image-to-video transfer on three action-recognition datasets using ViT-B/16 backbones, multiple fine-tuning baselines, and two pre-training strategies. ST-Adapter generally preserves or improves accuracy while reducing parameter, training-time, and data requirements.

  • Experimental setup: Experiments cover Kinetics-400, Something-Something-v2, and Epic-Kitchens-100 using ViT-B/16 with CLIP or ImageNet-21K pre-training.K400 emphasizes spatial appearance, whereas SSv2 contains richer temporal information; EK100 evaluates egocentric verb and noun recognition.
  • Experimental setup: The benchmark compares full, partial, and temporal fine-tuning with linear probing, adapters, prompt tuning, attention pooling, and temporally augmented ViTs.The compared approaches differ in which parameters are updated and whether they incorporate temporal modeling beyond average pooling.
  • Main results: 82.0 vs. 81.7 for K400 and 66.3 vs. 66.1 for SSv2 show ST-Adapter performing on par with Full Fine-tuning while updating 7.2M vs. 121.57M parameters.For CLIP-pre-trained models, ST-Adapter significantly outperforms the other efficient fine-tuning methods.
  • Main results: 86.7% top-1 accuracy on K400 establishes a new state-of-the-art for the largest ViT-L model with ST-Adapter.Across K400, SSv2, and EK100, the adapted image foundation model achieves results comparable to or better than previous task-specific methods.
  • Main results: ST-Adapter consistently improves egocentric-video results and can train directly on target egocentric video, unlike transformer approaches requiring intermediate Kinetics fine-tuning.Without ST-Adapter, direct CLIP-to-egocentric adaptation is more sensitive to hyperparameter settings.
  • Efficiency and ablations: At lower training budgets, full fine-tuning accuracy drops significantly faster, while ST-Adapters retain stronger performance; their advantage also grows as labeled data shrinks.Ablations further find temporal span most sensitive to kernel shape, and deeper adapters more beneficial than shallower ones.

5 Conclusions

The paper presents ST-Adapter as a lightweight approach for parameter-efficient image-to-video transfer learning, with reported efficiency and performance benefits.

  • 5 Conclusions: Kernel size is denoted as kT × kH × kW for time, height, and width.Table 7 organizes experiments by kernel shape.
  • 5 Conclusions: ST-Adapter enables parameter-efficient image-to-video transfer learning with a lightweight, easy-to-implement design.The method uses standard primitive operators and is intended to support usability and deployment.
  • 5 Conclusions: 20 times fewer updated parameters are reported while matching or surpassing full fine-tuning and fully trained state-of-the-art video models.The conclusion also reports faster training and lower computing-resource consumption.

Checklist

The checklist reports that the paper addresses contributions and scope, reproduction details, compute resources, and asset citations, while omitting several ethics and reporting items.

  • Checklist: The paper reports that its main claims accurately reflect the paper’s contributions and scope.The checklist marks this item as Yes.
  • Checklist: The paper reports no limitations discussion, no error bars, and no discussion of potential negative societal impacts.These checklist items are marked No.
  • Checklist: The paper reports code, training details, compute resources, and citations for existing assets, while not reporting asset licenses.The checklist marks reproduction materials, training details, compute reporting, and creator citations as Yes, but licensing as No.
  • Checklist: The paper reports no consent, personally identifiable information, or offensive-content discussion, and marks human-subject risk and compensation items as not applicable.These checklist entries concern the use or curation of data and human-subject research.

A Appendix

The appendix documents implementation and baseline settings, evaluates ST-Adapter across foundation models, backbones, datasets, training budgets, inference speed, and attention behavior.

  • Implementation details: Experiments use PyTorch, a configuration from Table 8, simpler augmentation than end-to-end fine-tuning, and a 20% held-out validation set for brief hyperparameter tuning.The implementation details describe the general experimental setup.
  • Baseline implementation details: Full fine-tuning updates CLIP-initialized layers with 1/100 learning rate and weight decay, while partial fine-tuning updates only the last Transformer block.The baseline configuration reports 1/100 as the best tested scaling for CLIP models.
  • Experiments on additional backbone architectures: ST-Adapter results are also provided for Swin-B models, although the sampling settings are not directly comparable with those in the referenced work.The appendix describes the comparison as indicative within a reasonable range.
  • Inference Speed: ST-Adapter introduces only a small inference-speed overhead, performing slightly below TimeSformer space only in the reported test.Latency is measured at batch size 1 and throughput at batch size 32.
  • Additional datasets and visualization: On UCF-101 and HMDB-51, ST-Adapter reaches similar top performance to recent state-of-the-art competitors, including a large-margin comparison with VideoPrompt.The appendix also visualizes greater attention to action-related regions after adding ST-Adapters, whereas unadapted CLIP is distracted by background.
Loading 2206.13559v3…