Source-linked AI summary
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, Yiming Yang
TL;DR
Video instruction-following needs scalable rewards that can detect factual errors and hallucinations, but direct video evaluation and human preference collection are costly. The paper uses detailed captions as proxy evidence for language-model reward scoring and applies those rewards in DPO. The resulting reward aligns with GPT-4V evaluation, while DPO improves video LMM performance, including an 8.1% accuracy gain over SFT.
Problem
Video instruction-following lacks a scalable reward system that reliably distinguishes factual responses from hallucinations, while human and direct video-model evaluation are costly.
Method
The paper generates detailed video captions and uses them as supporting evidence for language-model scoring and DPO preference optimization.
Results
8.1% accuracy improvement over the SFT counterpart is achieved, and the caption-based reward is well-aligned with GPT-4V evaluation.
Takeaways & Limitations
Caption-supported language-model rewards provide a cost-effective basis for improving video LMM alignment and video QA performance through DPO.
Abstract
from arXiv · showhide
Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has not been conclusively established. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying this tailored reward through DPO significantly improves the performance of video LMMs on video QA tasks.
1 Introduction
The paper targets the difficulty of scaling reliable rewards for video instruction following, especially when responses contain hallucinations. It introduces detailed captions as evidence for language-model scoring and applies the resulting reward to DPO.
- Video instruction-following alignment remains difficult because reward systems must distinguish preferred responses from hallucinated or less-preferred answers.
- Human preference collection is costly, while video reward distillation is difficult to scale because evaluating videos requires analyzing multiple frames.
- Detailed video captions provide proxy evidence that lets a language model assess response accuracy, detect hallucinations, and produce explanatory feedback with a numerical reward.
- 900k detailed captions in SHAREGPTVIDEO support the reward mechanism and cover temporal dynamics, world knowledge, object attributes, and spatial relationships.
- 8.1% accuracy improvement over the SFT counterpart is achieved by LLAVA-HOUND-DPO using the caption-based reward.
- The paper presents DPO with language-model feedback as a way to improve alignment of video LMMs and video QA performance.
2 Related Work
Prior work adapts language-model alignment and preference modeling to multimodal settings, but video datasets and reward methods remain constrained by caption quality, domain coverage, and scaling costs. This work extends caption-supported preference optimization to video LMM alignment.
- Video LMMs adapt LLM backbones for multimodal instruction following, and this work uses the Video-LLaVA backbone with DPO-based preference modeling.
- Existing video-text datasets often use brief captions or keywords, while human-created video instructions can be costly and limited in domain and scale.
- Earlier preference-modeling work used captions or AI feedback to improve factuality and video LMM performance.
- Image-focused studies applied DPO to GPT-4V-derived rewards or ChatGPT-generated preference pairs informed by detailed descriptions.
- This paper extends DPO to video LMM alignment by using detailed captions as factual evidence for reward modeling.
3 Method
The method builds detailed video captions, uses them to create instruction data and caption-supported rewards, then trains DPO on selected positive and negative responses. It also compares caption-based evaluation with GPT-4V-based evaluation.
- Method workflow: The workflow constructs captions, generates video instruction data for SFT, and feeds captions into DPO to refine factual alignment.
- Caption distillation: 900k videos from WebVid, VIDAL, and ActivityNet are represented by ten uniformly sampled frames that GPT-4V converts into detailed captions.
- Instruction data: ChatGPT generates three question-answer pairs per sampled caption, producing 240k instruction examples for SFT.
- Reward generation: GPT-4V’s cost, speed, and accessibility limitations motivate generating DPO rewards from detailed captions as supporting evidence.
- Reward generation: ChatGPT evaluates a question, ground-truth answer, prediction, and caption, producing explanatory feedback and a 1-to-5 factual-alignment reward.
- DPO construction: Responses scored at least 3 become positive examples and responses below 3 become negative examples, yielding approximately 17k DPO training instances.
- Cost comparison: The caption-based reward method costs less than $20 under the stated pricing model, compared with $3,000 for 10k human preference data points.
4 Assessment of Evaluator with GPT-4V Caption as Evidence
The caption-based reward is compared with GPT-4V evaluation using the same scoring guidelines, showing moderate alignment and substantial preference agreement across sampled video QA examples.
- Both evaluators follow the same scoring guidelines, while GPT-4V directly receives video frames, the question, and the model prediction.
- The comparison samples 200 videos each from WebVid, VIDAL, and ActivityNet, with one question and two SFT-model predictions per video.
- PCC 0.47 (p < 0.01) indicates a moderate positive correlation between ChatGPT and GPT-4V reward scores.
- More than 70% preference agreement was observed between ChatGPT and GPT-4V after excluding ties.
- Table 1 reports zero-shot video QA performance across models, with interpretation cautioned because of evaluation challenges and mixed result provenance.
5 Experimental Results
Experiments evaluate LLAVA-HOUND across standard and caption-supported video QA benchmarks, compare DPO generation with ranking, and analyze training and ablation effects. DPO improves average accuracy over SFT, while benchmark examples expose evaluation limitations and caption-supported QA broadens assessment.
- Benchmark results: LLAVA-HOUND-DPO reaches 70.75% average accuracy versus 62.65% for LLAVA-HOUND-SFT, an 8.1% improvement across the evaluated benchmarks.It also outperforms VLM-RLAIF in the reported comparison.
- Existing benchmark evaluation: Figure 4 examples show DPO correcting hallucinated or incorrect inferences while exposing grammatical questions and single-word answers in existing benchmarks.The paper proposes caption-derived long-form questions to evaluate relevance, accuracy, clarity, and completeness more fully.
- Caption-supported benchmark evaluation: In-domain evaluation reports 65.97% for an image-instruction-only baseline and 66.06% for LLAVA-HOUND-SFT, while out-of-domain accuracy drops to 49.32% and 56.50%, respectively.The authors suggest video QA training could potentially enhance generalization.
- Ablation analysis: A smaller learning rate of 3e-7 decreases performance relative to 5e-7, while unfreezing the MLP also reduces performance but remains above the SFT baseline.These ablations indicate sensitivity to DPO training choices.
- Training analysis: DPO performance improves through roughly two epochs and peaks around 2.5 epochs, corresponding to 350 training steps, on out-of-domain video QA.The reported DPO training configuration uses three epochs and approximately 420 training steps.
- Generator versus ranker: Direct DPO generation achieves 68.29% accuracy, exceeding DPO ranking of up to approximately 62% with 64 candidates and SFT decoding at 43.3% temperature-based accuracy.Greedy SFT decoding reaches 57.8%, and ranking performance plateaus as candidates increase.
6 Conclusion
The study proposes a cost-effective reward system using detailed video captions as proxies for video content, and reports alignment with GPT-4V evaluations plus improved DPO video QA performance.
- The caption-based reward scores align well with GPT-4V evaluation metrics and improve DPO training for video QA.The reported outcome is state-of-the-art performance on video QA tasks.
7 Reproducibility Statement
The authors plan to release captions with corresponding frames, model weights, and code for training and testing.
- The release includes distilled video captions paired with corresponding frames.
- The release includes pretrained, SFT, and DPO model weights.
- The release includes code for training and testing on existing and proposed benchmarks.
A Effect of ChatGPT Version on Official Benchmark Evaluation
ChatGPT versions produce different absolute evaluation scores, while model rankings under the same version remain relatively stable; the study therefore emphasizes specifying the evaluator version.
- ChatGPT versions produce significant variations in absolute scores on zero-shot video QA benchmarks.
- Prior studies may reach inaccurate evaluation conclusions when they omit the ChatGPT version used.
- The study uses gpt-3.5-turbo-0613 as the comparison benchmark because its performance aligns most closely with Video-LLaVA.
- Relative model rankings remain comparatively consistent when models are evaluated with the same ChatGPT version.
E Additional DPO Results
The additional results describe evaluation prompts and DPO behavior across video QA settings, including peak test performance near 350 training steps and stronger direct generation than ranking.
- Test-set performance peaks at approximately epoch 2.5, equivalent to 350 training steps, in both in-domain and out-of-domain video QA.
- The DPO model performs better when used directly for answer generation than when used as a ranker for SFT-generated candidates.
- The video-caption evaluator assesses coverage and hallucinations in LMM-generated captions from video frames.
- The GPT-4V evaluation prompt for video captioning is included as part of the additional evaluation materials.
- The ChatGPT QA prompt combines a detailed caption, question, ground-truth answer, and model prediction to produce an assessment and score.Scores of at least 3 count as correct for accuracy calculation.
- The GPT-4V QA evaluator uses video frames, a question, and a model prediction, with scores of at least 3 counted as correct.This evaluation assesses the quality of the ChatGPT-based evaluation.