PaperlayerView exact revision on arXiv

Source-linked AI summary

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi

arXiv:2402.03746v3cs.CV

TL;DR

VLMM alignment is challenged by the limited volume and quality of multimodal instruction-tuning data, which is associated with responses that are not properly grounded in visual content. VLM-RLAIF uses AI-generated preference feedback with video-description context and expanded SFT training. The authors report that it outperforms previous models across multiple video benchmarks.

  • Problem

    Multimodal video–text alignment is challenged by instruction-tuning data that are less abundant and comprehensive than text-only data, and VLMM responses may be insufficiently grounded in visual content.

    Source: §1 · pp. 1–2

  • Method

    VLM-RLAIF aligns video and text through VLMM-generated preference feedback, using detailed video descriptions as context and expanding SFT data with a curriculum strategy.

    Source: §1 · p. 2, §3.1 · p. 3, §5 · p. 9

  • Results

    VLM-RLAIF outperforms previous models across multiple multimodal video-text understanding benchmarks, with the authors describing the pattern as evidence of generalization across tasks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reported benchmark results support the authors’ conclusion that VLM-RLAIF generalizes across multiple video-text understanding tasks.

    Source: §5 · p. 9

    VLM-RLAIF’s effectiveness depends substantially on the quality of the AI model’s generated responses, motivating further work on synthetic-data quality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs). The previous approaches for VLMMs involved Supervised Fine-Tuning (SFT) with instruction-tuned datasets, integrating LLM with visual encoders, and adding additional learnable modules. Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data compared to text-only data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF), providing self-preference feedback to refine itself and facilitating the alignment of video and text modalities. In specific, we propose context-aware reward modeling by providing detailed video descriptions as context during the generation of preference feedback in order to enrich the understanding of video content. Demonstrating enhanced performance across diverse video benchmarks, our multimodal RLAIF approach, VLM-RLAIF, outperforms existing approaches, including the SFT model. We commit to open-sourcing our code, models, and datasets to foster further research in this area.

1 Introduction

Video–text alignment in VLMMs is challenged by the limited volume and quality of multimodal instruction-tuning data. VLM-RLAIF uses AI-generated preference feedback, context-aware reward modeling, and expanded SFT training, and the paper reports superior performance across video benchmarks.

1 Introduction

  • Prior VLMM approaches combine LLMs with visual encoders and additional learnable parameters, then use multimodal instruction-tuning data for supervised fine-tuning.These models extend LLM capabilities to video understanding, video question answering, and instruction-following.Source: §1 · p. 1
  • Multimodal instruction-tuning data are less abundant and comprehensive than text-only data, and this challenge is associated with responses insufficiently grounded in visual content.The SFT-model responses are illustrated in Figure 2.Source: §1 · pp. 1–2
  • VLM-RLAIF uses the VLMM to generate preference feedback on its responses, applying RLAIF to align video and text with less human intervention than RLHF.Context-aware reward modeling supplies detailed video descriptions during preference-feedback generation to improve clarity about video content.Source: §1 · p. 2, Figure 2 · p. 2
  • The approach also expands SFT instruction-tuning data and uses a curriculum strategy to improve video–text alignment.The expanded data include human-labeled video question answering and object-centric multimodal instruction-tuning examples.Source: §1 · p. 2
  • VLM-RLAIF outperforms state-of-the-art VLMMs across a wide array of video benchmarks, including VLM-SFT.Figure 1 summarizes performance across VQA, video generation, text-to-video retrieval, and action recognition; detailed comparisons appear in Tables 1–3.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Related work describes multimodal models built by connecting modality encoders to LLMs and fine-tuning them with instruction-following data. It also contrasts human-preference-based RLHF with RLAIF, which uses LLMs to assign preferences.

Multimodal large model

  • Multimodal models extend LLMs to image, video, and 3D point-cloud comprehension, among other modalities.The stated goal is to enable LLMs to understand various modalities.Source: §2 · p. 2
  • Common approaches extract modality representations with pretrained encoders, project them into the LLM token space, and apply supervised fine-tuning on modality-specific instruction data.The passage names LLaVA, Video-LLaVA, and Point-LLM as examples of this approach.Source: §2 · pp. 2–3

Reinforcement learning from feedback

3 VLM-RLAIF Framework

VLM-RLAIF aligns video and text by using the video model to generate preference feedback and then optimizing a policy with a learned reward model. Context-rich preference selection and a two-stage SFT curriculum are central parts of the framework.

3 VLM-RLAIF Framework

  • VLM-RLAIF uses AI feedback to align video and text: an SFT model generates preferences, a reward model learns from them, and reinforcement learning optimizes the policy.The policy starts from VLM-SFT and is optimized with PPO against the trained reward model.Source: §3 · p. 3

3.1 Context-Aware Reward Modeling

  • The SFT judge compares responses using a narrative of detailed descriptions generated for video segments of up to 20 frames, alongside the video and question.The segment captions are concatenated and supplied with instruction inputs and a guiding prompt to collect context-aware preferences.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • The reward model learns from preference pairs with cross-entropy loss, assigns scalar rewards to prompt-response inputs, and guides policy training; the authors report 7B VLM-RLAIF surpasses 13B VLM-SFT.The reward model is initialized from VLM-SFT with its final linear layer removed; the authors use 13B VLM-SFT to train it.Source: §3 · p. 3, §3.1 · p. 4

3.2 Two-stage Curriculum SFT

  • SFT data are augmented with human-annotated video question-answering datasets and object-centric instruction data derived from detailed video narratives.The Video Localized Narratives are transformed into object-centric instruction-tune data.Source: §3.2 · p. 4
  • The two-stage curriculum sorts SFT examples by answer length, training first on shorter-answer “easy” tasks and then on longer-answer “hard” tasks.Answer length is used as the criterion for task difficulty.Source: §3.2 · pp. 4–5

4 Experiments

VLM-RLAIF outperforms prior approaches, including VLM-SFT, across video-generation, VideoQA, retrieval, and action-recognition evaluations. Analyses report gains from SFT data augmentation, curriculum learning, larger preference sets, and context-aware labeling, with qualitative examples illustrating better-grounded responses.

4.1 Experimental Setup

  • Training combines 80k open-source video-text instructions, 67k video-QA examples, and 180k generated object-centric examples; each SFT curriculum stage runs for one epoch.The video input uses 50 uniformly sampled frames.Source: §4.1 · p. 5

4.2 Quantitative Analysis

  • VLM-RLAIF performs on par with GPT-4V on the video-based generative benchmark and significantly outperforms prior approaches, including VLM-SFT, across three zero-shot VideoQA benchmarks.The authors note that GPT-4V requires much more computational resources, making that comparison unfair.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 7
  • VLM-RLAIF outperforms other methods, including VLM-SFT, on both zero-shot text-to-video retrieval datasets and both zero-shot action-recognition datasets.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 Detailed Analysis

  • Adding instruction-following datasets and applying curriculum learning each improve VLM-SFT performance in the authors’ ablation study.Table 4 compares the dataset additions and curriculum-learning condition.Source: §4.3 · p. 7, Table 4 · p. 7
  • Performance increases monotonically as preference data grows from 10k to 40k.Figure 5 tracks five video-generation metrics covering correctness, explanation detail, contextuality, temporal understanding, and consistency.Source: Figure 5 · p. 8, §4.3 · p. 8
  • Context information improves preference-labeling performance, and using three video clips with aggregated captions improves it further.The authors conjecture that detailed segment captions enrich contextual clarity for reward modeling.Source: Table 5 · p. 7, §4.3 · p. 8
  • Larger SFT models improve video-generation performance; scaling the reward model from 7B to 13B further improves results except for MSR-VTT text-to-video retrieval R@5.A 7B reward model yields a 5–12% improvement in zero-shot VideoQA.Source: §4.3 · pp. 8–9, Table 7 · p. 9, Table 8 · p. 9

4.4 Qualitative Analysis

  • Two qualitative examples show VLM-RLAIF producing more accurate and better-grounded video-understanding responses than VLM-SFT.The examples highlight a more detailed recognition of an athlete’s attire and better grounding to visual input.Source: Figure 6 · p. 8, §4.4 · p. 9

5 Conclusion

The paper proposes VLM-RLAIF, combining AI feedback with context-aware reward modeling, expanded instruction-tuning data, and curriculum-based training. It reports significant gains over previous models on multiple multimodal video-text benchmarks.

5 Conclusion

  • VLM-RLAIF uses reinforcement learning from AI feedback and context-aware reward modeling to support AI feedback for self-improvement.The authors also expand the SFT instruction-tuning dataset and use a curriculum-based training approach.Source: §5 · p. 9
  • VLM-RLAIF significantly outperforms previous models on multiple multimodal video-text understanding benchmarks, with the authors suggesting generalization across tasks.The framework combines AI feedback with context-aware reward modeling, expanded instruction-tuning data, and curriculum-based training.Source: §5 · p. 9
  • The curriculum-based approach is described as particularly effective for gradually learning complex video-text relationships.The conclusion presents this alongside expanded instruction-tuning data as part of the training approach.Source: §5 · p. 9

Limitations

The approach relies on AI-generated feedback, so its effectiveness depends on the quality of the model’s generated responses. Evaluation spans several VLMM benchmark types but does not include temporal reasoning tasks.

Limitations

  • VLM-RLAIF’s effectiveness depends largely on the quality of the AI model’s generated responses used to synthesize feedback.The authors call for further research to improve synthetic-data quality and establish a more reliable RLAIF system.Source: Limitations · p. 9
  • Evaluation covers videoQA, video-based generation, retrieval, and recognition, but not temporal reasoning tasks identified as necessary for real-world application.The authors identify applying their method to temporal reasoning as future work.Source: Limitations · p. 9

A Details About Object-Centric Instruction-Tune Data

The paper creates object-centric instruction-tuning examples by converting detailed video narratives into question-and-answer pairs grounded in objects and their surroundings.

A Details About Object-Centric Instruction-Tune Data

  • Video Localized Narratives provide object appearance, action, and tracked-coordinate descriptions that serve as source material for instruction-tuning data.The authors use the dataset’s ground-truth captions to preserve rich video semantics.Source: A Details About Object-Centric… · p. 14
  • ChatGPT is prompted with the narratives to generate question-and-answer pairs that turn object descriptions into instruction-tuning examples.The questions and answers are intended to be anchored in visual specifics of each object and its surroundings.Source: A Details About Object-Centric… · p. 14
  • The resulting examples target details such as objects’ appearance and actions within video scenes.The paper illustrates the approach with questions about the number, appearance, and actions of parrots.Source: A Details About Object-Centric… · p. 14

B Input Prompts for Reward Modeling

The appendix presents prompts used to generate object-centric instruction data and to support context-aware reward modeling. It also shows qualitative examples of model answers and reward scores.

B Input Prompts for Reward Modeling

  • Three prompt types support context-aware reward modeling: generating responses, selecting a preference between responses, and producing a reward score.The appendix associates these functions with Figures 8, 9, and 10, respectively.Source: B Input Prompts for Reward Modeling · p. 14, Figure 8 · p. 16, Figure 9 · p. 16
  • The object-centric data prompt instructs ChatGPT-3.5-turbo to create five conversational question-and-answer pairs from video narratives.Its example questions address scene details such as who throws an egg, where the people are, and what they wear.Source: Figure 7 · p. 15, B Input Prompts for Reward Modeling · p. 15
  • The VLM-RLAIF examples show responses described as more accurately grounded in visual input than VLM-SFT responses.The figure caption marks accurate responses in blue and less accurate VLM-SFT responses in red.Source: Figure 11 · p. 17
  • In the reward examples, the trained reward model assigns a higher reward to the answer described as more accurately grounded in the video.The figure compares two VLM-SFT answers to the same question.Source: Figure 12 · p. 18
Loading 2402.03746v3…