PaperlayerView exact revision on arXiv

Source-linked AI summary

BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning

Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H. Li, Ge Li

arXiv:2309.15785v2cs.CV

TL;DR

Video dialogue requires temporal modeling, but adapting image-based systems faces compute costs and limited video resources. BT-Adapter adds a temporal branch to pretrained image models, and the paper reports state-of-the-art zero-shot and video-conversation performance, including stronger average conversation results without video instruction tuning.

  • Problem

    Image-to-video adaptation lacks an effective low-cost temporal modeling option: full visual-encoder tuning is costly, while parameter-efficient alternatives struggle to capture spatiotemporal features.

    Source: §1 · p. 1

  • Method

    BT-Adapter adds a branching temporal module alongside a pretrained image model, using asymmetric tube masking and tailored alignment objectives while keeping the backbone frozen.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    BT-Adapter achieves state-of-the-art zero-shot video-text retrieval and fine-tuned video-conversation results, while its untuned version exceeds fine-tuned SOTAs on average.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A pretrained BT-Adapter can integrate with image-conversation models using the same CLIP version to support video conversation without video instructions.

    Source: §1 · p. 2

    BT-Adapter may still disrupt pretrained knowledge compared with fully concatenation-based modeling in VideoChatGPT.

    Source: §11 · p. 13

Abstract

from arXiv · show

The recent progress in Large Language Models (LLM) has spurred various advancements in image-language conversation agents, while how to build a proficient video-based dialogue system is still under exploration. Considering the extensive scale of LLM and visual backbone, minimal GPU memory is left for facilitating effective temporal modeling, which is crucial for comprehending and providing feedback on videos. To this end, we propose Branching Temporal Adapter (BT-Adapter), a novel method for extending image-language pretrained models into the video domain. Specifically, BT-Adapter serves as a plug-and-use temporal modeling branch alongside the pretrained visual encoder, which is tuned while keeping the backbone frozen. Just pretrained once, BT-Adapter can be seamlessly integrated into all image conversation models using this version of CLIP, enabling video conversations without the need for video instructions. Besides, we develop a unique asymmetric token masking strategy inside the branch with tailor-made training tasks for BT-Adapter, facilitating faster convergence and better results. Thanks to BT-Adapter, we are able to empower existing multimodal dialogue models with strong video understanding capabilities without incurring excessive GPU costs. Without bells and whistles, BT-Adapter achieves (1) state-of-the-art zero-shot results on various video tasks using thousands of fewer GPU hours. (2) better performance than current video chatbots without any video instruction tuning. (3) state-of-the-art results of video chatting using video instruction tuning, outperforming previous SOTAs by a large margin.

1. Introduction

BT-Adapter adds a temporal modeling branch to pretrained image-conversation models to address the cost and temporal-capacity trade-offs of video dialogue. The paper reports strong video-task and video-conversation results, including gains without video instruction tuning.

1. Introduction

  • Video dialogue is costly because multiple frames add to an already demanding LLM-and-image-encoder system, while strong video-pretrained encoders and comparably scaled video-text instruction data are lacking.The introduction frames adapting image-centric dialogue models as an alternative that can leverage image-model knowledge.Source: §1 · p. 1
  • Full visual-encoder tuning can increase GPU demands, whereas parameter-efficient temporal methods keep the encoder frozen but may struggle to capture crucial spatiotemporal features.The introduction describes this as an efficiency-effectiveness trade-off in CLIP-based temporal modeling.Source: §1 · pp. 1–2
  • BT-Adapter reports state-of-the-art zero-shot video-text retrieval and beats prior SOTAs across video-conversation benchmarks after fine-tuning; untuned BT-Adapter beats fine-tuned SOTAs on average.Figure 1 compares zero-shot MSRVTT Recall@1 with pretraining GPU hours and provides a quantitative comparison of video conversations.Source: Figure 1 · p. 2, §1 · p. 2
  • BT-Adapter adds a branching spatial-temporal module alongside the pretrained image model, preserving its forward progression while providing temporal modeling.The branch is pretrained with a CLIP version and can integrate with image-conversation models using that version without video instructions.Source: §1 · p. 2
  • An asymmetric tube-masking mechanism and two tailored training objectives are presented as ways to reduce computation, accelerate convergence, and improve outcomes.The paper also lists comprehensive studies of temporal-modeling methods for video dialogue among its contributions.Source: §1 · p. 2

2. Related Work

Prior temporal-modeling approaches trade temporal capability against compute efficiency and preservation of pretrained knowledge. This section presents BT-Adapter’s branching design, compares temporal-modeling strategies, and describes asymmetric masking and alignment objectives for pretraining.

2. Related Work

  • Prior work includes joint, interpolate, and concatenate temporal modeling; BT-Adapter instead branches from the backbone, and the authors introduce zero-shot image-to-video conversation without video instruction tuning.The authors distinguish their focus from STAN’s, which had not explored zero-shot image-to-video transfer or video conversations.Source: §2 · p. 3

3. Methodology

  • The methodology extends pretrained image-language models such as CLIP and LLaVA with temporal modeling while aiming to preserve their pretrained knowledge.The paper focuses its implementation on CLIP, which is used as the visual encoder in the cited image-dialogue models.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1. Branching Temporal Adapter

  • BT-Adapter is a continuous branch alongside frozen CLIP, gathering same-position patch tokens across frames for temporal attention while CLIP preserves spatial and aligned features.The branch and CLIP outputs are combined for the final representation.Source: §3.1 · pp. 3–4

3.2. Temporal Modeling for Video Conversation

  • BT-Adapter balances parameter efficiency, multimodal affinity, and temporal sensitivity, and has a clear advantage over the other temporal-modeling methods evaluated for video conversation.The comparison uses video instruction tuning, keeps the LLM frozen, and includes no video-text pretraining.Source: §3.2 · p. 5, Table 1 · p. 5

3.3. Pretraining with Asymmetric Masking

  • Asymmetric tubular masking leaves CLIP tokens unchanged and masks same-position branch tokens across frames; a ratio ρ ≥70 reduces the computational budget by at least half.The strategy retains pretraining knowledge from CLIP while reducing spatiotemporal redundancy.Source: §3.3 · p. 5
  • MBTA aligns unmasked branch tokens with CLIP’s last-layer tokens, while MBCA aligns branch patch tokens with text embeddings.The authors use patch tokens for cross-modal alignment because downstream conversation and generation typically use patch tokens as input.Source: §3.3 · pp. 5–6

4. Experiments

Experiments evaluate BT-Adapter on traditional video tasks and video conversation, including zero-shot and instruction-tuned settings. Results cover retrieval, dialogue, component ablations, and a qualitative example.

4.1. Experiment Settings

  • Evaluations cover zero-shot retrieval and recognition across seven benchmarks, plus zero-shot and instruction-tuned video dialogue; downstream tasks use openai-CLIP-L/14 with four adapter layers by default.The default pretraining mask ratio is 70%; instruction tuning uses VideoChatGPT-100K.Source: §4.1 · p. 6

4.2. Quantitative evaluation

  • Without video instruction tuning, BT-Adapter outperforms instruction-tuned video dialogue methods on average; instruction tuning further increases its advantage.Its performance margin is notably larger on ActivityNet, highlighting strength on long video sequences.Source: §4.2 · pp. 6–7
  • BT-Adapter is competitive across retrieval datasets, best on most metrics, and uses 130× fewer GPU hours than UMT, 560× fewer than TVTSv2, and 2687× fewer than InterVideo.It slightly trails UMT on DiDeMo R@1 and Singularity on LSMDC R@1, while exceeding them by more than 5% on MSRVTT and ActivityNet.Source: §4.2 · p. 7, Table 5 · p. 7

4.3. Ablation Study

  • Branching separate-ST modeling improves on separate-ST layers, and adding multi-level selective backbone–branch combination yields further gains across all benchmark datasets.The structural ablation evaluates zero-shot retrieval R@1 on MSRVTT and DiDeMo and zero-shot video conversation.Source: §4.3 · p. 7, Table 6 · p. 7
  • In BT-Adapter training-objective ablations at a 70% mask rate, both MBTA and MBCA improve downstream results, with MBCA showing more substantial progress; zero-shot retrieval is reported using R@1.Source: §4.3 · p. 7, Table 7 · p. 7
  • Performance gains plateau at about four to six branching layers; results remain stable at masking rates up to 70%, while higher rates reduce training time.The authors therefore use a 70% masking rate.Source: §4.3 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

4.4. Qualitative Results

  • In a qualitative example, BT-Adapter-LLaVA gives a more informative and accurate answer to a video-related question than VideoChatGPT.The paper notes that additional dialogue examples appear in the appendix.Source: §4.4 · p. 8, Figure 5 · p. 8

5. Conclusion

The conclusion presents BT-Adapter as a parameter-efficient approach to image-to-video adaptation and video conversation. It combines branching temporal modeling with asymmetric masking and two training objectives.

5. Conclusion

  • BT-Adapter is a branching separate-ST network for temporal modeling, paired with asymmetric masking and two training objectives.The paper reports that it can be integrated with pretrained image conversation models for video dialogue without manual video instruction tuning.Source: §5 · p. 8
  • The authors report superiority over other temporal modeling methods for video conversation and state-of-the-art results across two zero-shot video tasks and video conversations.They also report significantly lower computation costs.Source: §5 · p. 8
  • The conclusion frames the contribution as parameter-efficient yet effective image-to-video adaptation and video conversation.The work was supported by the National Natural Science Foundation of China (No. 62172021).Source: §5 · p. 8

6. Datasets

The paper describes datasets and evaluation settings for video retrieval, action recognition, video-text pretraining, and video conversation.

6. Datasets

  • Zero-shot text-to-video retrieval uses MSRVTT, DiDemo, LSMDC, and ActivityNet, with frame counts and token-length settings specified for each benchmark.MSRVTT uses the 1K-A split; ActivityNet evaluation focuses on video-paragraph retrieval using the val1 split.Source: §6 · p. 11
  • Zero-shot action recognition uses Kinetics-400, HMDB-51, and UCF-101, with the action tag itself as the textual query rather than a template.The frame number and maximum token length are set to 16 and 12, respectively.Source: §6 · p. 11
  • BT-Adapter is pretrained on WebVid2M, a stock-footage dataset with about 2.5 million video-caption pairs and 12,000 video hours.Pretraining uses eight frames and a maximum token length of 32.Source: §6 · p. 11
  • Video conversation evaluation includes the VideoChatGPT benchmark and zero-shot QA on MSRVTT-QA, MSVD-QA, and ActivityNet-QA, with GPT-based relative scores on a 1-to-5 scale.The conversation benchmark assesses five generative-response aspects, while the QA evaluation uses GPT as an evaluation assistor.Source: §6 · p. 11

7. Implementation Details

Implementation uses PyTorch and mmaction2.0, with configuration settings reported in Table 8.

7. Implementation Details

  • The implementation uses PyTorch and mmaction2.0, and the authors describe their data augmentation as simpler than that of other methods.Configuration settings are listed in Table 8, except where alternate configurations are specified.Source: §7 · p. 11

8. Zero-Shot Results on Action Recognition

BT-Adapter achieves state-of-the-art zero-shot video recognition among CLIP-based methods, including outperforming InternVideo. The reported recognition results cover HMDB, UCF, and K400.

8. Zero-Shot Results on Action Recognition

  • BT-Adapter achieves state-of-the-art zero-shot video recognition among CLIP-based methods, outperforming InternVideo.The paper attributes this to video encoding and spatiotemporal modeling effectiveness.Source: §8 · p. 11
  • BT-Adapter contributes to video-only recognition despite being pretrained solely on video-language datasets.Source: §8 · p. 11
  • Table 9 reports zero-shot video recognition results on HMDB, UCF, and K400.Source: Table 9 · p. 12

9. Experimental Comparison With Similar Methods

Compared with ST-Adapter and STAN, BT-Adapter combines efficiency and effectiveness, outperforming both in the reported zero-shot retrieval and video-conversation comparisons. STAN’s results vary with whether CLIP is frozen.

9. Experimental Comparison With Similar Methods

  • BT-Adapter is both efficient and effective, outperforming ST-Adapter and both STAN settings across the reported aspects.The comparison covers zero-shot MSRVTT retrieval and video conversation after one epoch of pretraining on MSRVTT.Source: §9 · p. 12
  • ST-Adapter performs suboptimally across metrics; the authors suggest its inserted 3-D convolutions may rapidly degrade pretrained multimodal knowledge.Source: §9 · p. 12
  • With open CLIP, STAN performs well on zero-shot retrieval but worse on video conversation and requires longer pretraining; freezing CLIP improves all metrics.Frozen-CLIP STAN still falls short of BT-Adapter in all aspects.Source: §9 · p. 12

10. More Ablation Results

The ablations favor zero initialization for temporal projection and weighted selection for combining backbone and branch outputs. Additional figures show dialogue examples involving action sequences and actions in specific frames.

10. More Ablation Results

  • Zero initialization for temporal projection substantially outperforms random initialization and no projection.The authors suggest that building temporal reasoning from scratch mitigates adverse effects on the pretrained spatial prior, making zero initialization better suited to image-to-video transfer.Source: §10 · p. 12, Table 11 · p. 13
  • Weighted selection produces the most favorable backbone-branch combination results among addition, weighted selection, and concatenation with linear projection.The authors suggest that layers and samples require different amounts of information from the backbone and branch.Source: §10 · p. 12, Table 11 · p. 13
  • Figure 6 presents video-dialogue results concerning sequences of actions.Source: Figure 6 · p. 13
  • Figure 7 presents video-dialogue results concerning actions in a specific video frame.Source: Figure 7 · p. 13

11. More Qualitative Results

The qualitative examples cover temporally sensitive dialogue scenarios, including unusual actions and complex scenes in a three-minute video. The paper also reports a frame-text recognition challenge and broader risks from LLM-generated content.

11. More Qualitative Results

  • Figures 6–9 illustrate diverse video-dialogue scenarios where temporal sensitivity matters, including unusual actions and complex actions and scenes in a three-minute video.Source: §11 · p. 13, Figure 8 · p. 13, Figure 9 · p. 13
  • Figure 10 is identified as a bad case; the accompanying discussion says BT-Adapter struggles to recognize text content within frames.The authors note that the method may disrupt pretrained knowledge relative to VideoChatGPT’s fully concatenation-based modeling.Source: Figure 10 · p. 13, §11 · p. 13

12. Broader Impact

  • LLM-generated content may inherit training-data biases, including negative societal implications, or be inaccurate and non-factual, potentially undermining trust in online information.Source: §12 · p. 14

Related papers

Loading 2309.15785v2…