PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward

Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, Yiming Yang

arXiv:2404.01258v2cs.CVcs.AI

TL;DR

Assessing whether video-instruction responses are factually grounded remains challenging, particularly when detecting hallucinations. This paper uses detailed video captions as evidence for language-model reward scoring and reports an 8.1% accuracy improvement over the SFT model.

  • Problem

    A robust reward system for video-conditioned responses remains difficult to develop, especially for distinguishing preferred answers from less preferred ones when generated content may contain hallucinations.

    Source: §1 · p. 1

  • Method

    The paper uses detailed video captions as supporting evidence for language models to score video QA responses and generate cost-efficient reward data for DPO.

    Source: §3.3 · p. 4

  • Results

    8.1% accuracy improvement over the SFT model is reported for LLAVA-HOUND-DPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Caption-based reward data offers a cost-efficient alternative for DPO where GPT-4V's cost, speed, and limited accessibility hinder scalability.

    Source: §3.3 · p. 4

    The reward process depends on high-quality video captions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has not been conclusively established. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying this tailored reward through DPO significantly improves the performance of video LMMs on video QA tasks.

1 Introduction

The paper introduces caption-based language-model rewards for evaluating video QA responses and applying DPO, supported by a large video-caption dataset. It reports an 8.1% accuracy improvement over the SFT counterpart.

1 Introduction

  • Although RL and DPO have helped LLMs generate more honest, helpful, and harmless content, multimodal alignment remains limited by reward assessment challenges and scarce cross-modal data (Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2023; Sun et al., 2023b; Rafailov et al., 2024; Chen et al., 2024b; Hosseini et al., 2024; Liu et al., 2023b; Sun et al., 2023a).The introduction identifies hallucinations in video-grounded responses as a consequence of this data scarcity.Source: §1 · p. 1
  • Human preference collection is costly and labor-intensive: LLaVA-RLHF (Sun et al., 2023a) collected 10k evaluated instances for $3000.Video inputs further challenge scalable preference distillation because they require analyzing multiple frames.Source: §1 · p. 1
  • Detailed video captions act as a proxy for video content, letting a language model assess response accuracy and hallucinations and produce chain-of-thought feedback plus a numerical reward.The proposed cost-effective mechanism is intended to evaluate video instruction-following responses and support preference optimization.Source: §1 · pp. 1–2
  • 8.1% accuracy improvement over the SFT counterpart is reported for LLAVA-HOUND-DPO, marking the first successful DPO application in video LMMs and SOTA video-QA performance.The paper says this improvement is achieved by using language-model feedback as the DPO reward.Source: §1 · p. 2
  • SHAREGPTVIDEO provides 900k detailed captions covering temporal dynamics, world knowledge, object attributes, and spatial relationships.The dataset was developed with a prompting technique using GPT-4V and is presented as a resource for LMM training and video-understanding research.Source: §1 · p. 2
  • The caption-proxy reward is reported to align well with evaluations from GPT-4V-generated rewards, which are more powerful but costlier.The comparison concerns evaluations derived from the GPT-4V model-generated rewards.Source: §1 · p. 2

2 Related Work

The related work situates the approach among video-capable LMMs, video-text datasets, and preference modeling, highlighting detailed video captions as evidence for DPO-based video LMM alignment.

2.1 Large Multi-Modal Models

2.2 Video-text Datasets

  • Video-text datasets often offer brief captions, while alternatives include ASR text with alignment limitations and human-written instructions limited to ActivityNet and 100k pairs.The paper uses GPT-4V with crafted prompts to produce detailed video captions as a community resource for LMM training.Source: §2.2 · p. 2

2.3 Preference Modeling for LMMs

3 Method

The method uses detailed video captions as evidence for language-model reward scoring, then applies DPO to improve factual alignment in video instruction following. It builds caption and instruction datasets before generating preference pairs for training.

3.1 Prompting GPT-4V Model for Detailed Video Caption Distillation

  • The caption collection combines 400k WebVid, 450k VIDAL, and 50k ActivityNet videos; GPT-4V captions each from ten uniformly extracted frames.The caption prompt covers temporal dynamics, world knowledge, object attributes, spatial relationships, and aesthetic assessments.Source: §3.1 · p. 3

3.2 SFT with Generated Video Instruction Data from Detailed Caption

  • ChatGPT generates three question-answer pairs per caption from 20k ActivityNet, 30k WebVid, and 30k VIDAL captions, producing 240k SFT instruction examples.Using detailed captions for generation is intended to keep the instructions factually consistent with video content.Source: §3.2 · p. 4

3.3 DPO with Feedback from Language Model as Reward

  • ChatGPT scores six SFT-generated responses for each of 20k sampled questions (120k total), using caption evidence and a 1–5 factuality and quality scale.Its inputs include the question, ground-truth answer, and model prediction alongside the detailed caption.Source: §3.3 · p. 4
  • Answers scoring at least 3 become positive examples and those below 3 negative; excluding uniform-score cases yields approximately 17k DPO training instances.The policy and reference models both start from SFT weights, with β set to 0.1.Source: §3.3 · pp. 4–5
  • Caption-based reward assignment costs under $20, compared with $3,000 for 10k human preference labels and $30 per million tokens for GPT-4V labeling.The paper also describes GPT-4V reward labeling as slower and less scalable for multi-frame video inputs. The cited prior works are Sun et al. (2023a) and Li et al. (2023d).Source: §3.3 · pp. 4–5

4 Assessment of Evaluator with GPT-4V Caption as Evidence

The section compares caption-based reward scoring with direct GPT-4V evaluation of video frames and finds moderate score correlation and substantial preference agreement, cautiously supporting the benchmark’s applicability to video QA evaluation.

4 Assessment of Evaluator with GPT-4V Caption as Evidence

  • The comparison used the same scoring guidelines and evaluated 1,200 sampled examples with direct GPT-4V video-frame input versus caption-based reward scoring.The sample comprised 200 videos each from WebVid, VIDAL, and ActivityNet, with one question and two SFT-model predictions per video; Azure API filtering left 196, 151, and 143 videos, respectively, with both answers evaluated.Source: §4 · p. 5
  • PCC was 0.47 (p < 0.01), more than 75% of ChatGPT scores fell within one standard deviation of GPT-4V scores, and preference agreement exceeded 70%.The reported standard deviation was σ = 1.31; preference agreement excludes ties.Source: §4 · p. 5
  • Average scores were 2.9 for ChatGPT and 3.5 for GPT-4V, indicating GPT-4V tended to give slightly more positive evaluations.Source: §4 · p. 5

5 Experimental Results

LLAVA-HOUND-DPO improves over its SFT baseline on existing video QA benchmarks, while caption-supported evaluation and pretraining analyses examine generalization, training choices, and captioning performance.

5.1 Model Architecture, Image Data Mix-up and Training Pipelines

  • The Video-LLaVA-based model is trained through caption pretraining, image-and-video instruction tuning, and DPO using 17k preference examples.Caption pretraining uses 650k image and 900k video captions; SFT uses 600k image and 240k video instructions, and DPO runs for 3 epochs at 5e-7 learning rate.Source: §5.1 · p. 6

5.2 Existing Benchmark Evaluation

  • In two examples, LLAVA-HOUND-DPO gives a more accurate inference, correctly identifying a video subject that LLAVA-HOUND-SFT and Video-LLaVA miss, while correcting an SFT hallucination.The examples also expose benchmark questions that may be grammatically incorrect or nonsensical and single-word answers that hinder evaluation of helpfulness and hallucination detection.Source: §5.2 · p. 7

5.3 Proposed Benchmark Evaluation with GPT-4V Caption as Supporting Evidence

  • Video-LLaVA performs best among baseline models on the proposed in-domain and out-of-domain caption-supported QA evaluation.The evaluation uses gpt-3.5-turbo-0301 and counts scores ≥3 as correct; the long-form questions are intended to assess relevance, accuracy, clarity, and completeness.Source: §5.3 · p. 7, Table 2 · p. 8, Table 3 · p. 8
  • Image-only [8] achieves 65.97% average in-domain accuracy versus SFT’s 66.06%, but out-of-domain accuracy is 49.32% versus 56.50%.[9] and [6] differ by 0.2% in average accuracy; the authors attribute [6]’s larger gain over [5] to large-scale video-caption pretraining, given the similar SFT-data quality.Source: §5.3 · p. 8
  • Unfreezing the MLP during DPO lowers performance while remaining above SFT, and a 3e-7 learning rate lowers performance relative to 5e-7.Source: §5.3 · p. 8
  • Self-play [11] yields accuracy 3% below SFT, suggesting it may be less effective for video LMM alignment than introducing a reward model.The comparison is with [6], and the self-play method uses ground-truth answers as preferred and model-generated responses as dispreferred.Source: §5.3 · pp. 8–9
  • Out-of-domain accuracy peaks around 2.5 epochs (350 steps); DPO answer generation reaches 68.29%, versus a ranker plateau of approximately 62% with 64 candidates.Performance improves consistently during epochs 0–2; at temperature 1.0, SFT accuracy is 43.3%, compared with 57.8% under greedy decoding.Source: Figure 5 · p. 9, §5.3 · p. 9

5.4 Analysis on Video Captioning Ability from Pre-training

  • Adding more distilled caption data improves video-captioning performance across in-domain and out-of-domain datasets, although a gap from GPT-4V remains.The initial checkpoint uses 90k captions (10% of 900k total); the authors report that static-scene WebVid needs less data than dynamic, diverse VIDAL.Source: §5.4 · p. 9, Figure 6 · p. 10

6 Conclusion

The paper proposes a cost-effective reward system that uses detailed captions as proxies for video content. Its reward scores align with GPT-4V evaluation metrics, and using the reward in DPO yields SOTA performance on video QA tasks.

  • The caption-based reward mechanism enhances DPO training, resulting in SOTA performance on video QA tasks.Source: §6 · p. 10
  • The reward scores are well-aligned with GPT-4V evaluation metrics.Source: §6 · p. 10
  • The proposed cost-effective reward system uses detailed captions as proxies for video content.Source: §6 · p. 10

7 Reproducibility Statement

The authors plan to release materials to support reproducibility, including video data, model weights, and training and testing code.

  • They plan to release distilled video captions with their corresponding frames.Source: §7 · p. 10
  • The planned release includes pre-trained, SFT, and DPO model weights.Source: §7 · p. 10
  • They also plan to release code for training and testing with existing and proposed benchmarks.Source: §7 · p. 10

A Effect of ChatGPT Version on Official Benchmark Evaluation

ChatGPT versions produce significantly different absolute scores on zero-shot video QA benchmarks, although model rankings by average accuracy remain consistent within each version. The study recommends explicitly naming the ChatGPT version and uses gpt-3.5-turbo-0613 for comparison because it most closely aligns with Video-LLaVA.

A Effect of ChatGPT Version on Official Benchmark Evaluation

B Evaluation of Captioning Ability from pre-training … F Prompts for GPT-4V and ChatGPT Queries

The supplied appendix passages document DPO accuracy over training epochs, compare direct answer generation with reward ranking, and show prompts and examples used for evaluation. They specify a score threshold for accuracy but provide no numeric accuracy values in the figure captions.

F Prompts for GPT-4V and ChatGPT Queries

  • Figure 8 presents dataset examples annotated by GPT-4V.Source: Figure 8 · p. 15
  • Figures 9 and 10 compare QA testing in an existing benchmark with testing in the proposed new benchmark.The captions identify the comparison scope but do not report its outcomes.Source: Figure 9 · p. 15, Figure 10 · p. 16
  • Test accuracy for in-domain and out-of-domain video QA peaks at approximately epoch 2.5, equivalent to 350 training steps.The figure reports a consistent trend across both settings.Source: Figure 11 · p. 16
  • Using the DPO model directly for answer generation outperforms using it as a ranker for SFT-generated candidates at temperature 1.0.In the ranker setup, the DPO model assigns reward scores to candidate answers from the SFT model.Source: Figure 12 · p. 17
  • The appendix includes a GPT-4V evaluation prompt for video captioning and a ChatGPT prompt for instruction generation.The captions identify the prompt topics without describing their contents.Source: Figure 13 · p. 17, Figure 14 · p. 18
  • The ChatGPT QA evaluation prompt uses a detailed caption, question, ground-truth answer, and prediction; scores ≥3 count as correct for accuracy.The prompt generates an assessment of prediction quality and a score based on predefined criteria.Source: Figure 15 · p. 19
  • The GPT-4V QA evaluation prompt uses video frames, a question, and a prediction, with scores ≥3 counted as correct for accuracy.It is used to assess the quality of the ChatGPT evaluation prompt in Figure 15.Source: Figure 16 · p. 20

Related papers

Loading 2404.01258v2…