PaperlayerView exact revision on arXiv

Source-linked AI summary

LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment

Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang, Cheng Jin, Hao Li

arXiv:2412.04814v3cs.CV

TL;DR

Text-to-video models remain difficult to align with subjective human preferences, and existing rating-based video assessment overlooks the reasoning behind judgments. LIFT builds a rationale-bearing feedback dataset and reward model, then uses the learned reward for fine-tuning; its CogVideoX-2B case study outperforms CogVideoX-5B across all 16 VBench metrics.

  • Problem

    T2V alignment with subjective human preferences is difficult, while existing video assessors overlook evaluation reasoning and video-based feedback for alignment remains unexplored.

    Source: §1 · pp. 1–2

  • Method

    LIFT collects 10K human ratings with rationales, trains LIFT-CRITIC to learn a reward function, and fine-tunes T2V models using reward-weighted likelihood maximization.

    Source: §1 · p. 2

  • Results

    CogVideoX-2B fine-tuned with LIFT outperforms CogVideoX-5B across all 16 VBench metrics.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The CogVideoX-2B case study demonstrates the effectiveness of LIFT for aligning T2V models with human expectations.

    Source: §5 · p. 8

    The dataset covers three dimensions, a limited set of text categories, and good/normal/bad feedback scores, constraining the diversity of human feedback.

    Source: D. Limitations and Future Works · p. 14

Abstract

from arXiv · show

Recent advances in text-to-video (T2V) generative models have shown impressive capabilities. However, these models are still inadequate in aligning synthesized videos with human preferences (e.g., accurately reflecting text descriptions), which is particularly difficult to address, as human preferences are subjective and challenging to formalize as objective functions. Existing studies train video quality assessment models that rely on human-annotated ratings for video evaluation but overlook the reasoning behind evaluations, limiting their ability to capture nuanced human criteria. Moreover, aligning T2V model using video-based human feedback remains unexplored. Therefore, this paper proposes LiFT, the first method designed to leverage human feedback for T2V model alignment. Specifically, we first construct a Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k human annotations, each including a score and its corresponding rationale. Based on this, we train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment, measuring the alignment between given videos and human expectations. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood. As a case study, we apply our pipeline to CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B across all 16 metrics, highlighting the potential of human feedback in improving the alignment and quality of synthesized videos.

1. Introduction

LIFT targets the difficulty of aligning text-to-video generation with subjective human preferences by combining reason-bearing human feedback, a learned reward model, and reward-weighted fine-tuning. Its CogVideoX-2B case study reports performance above CogVideoX-5B across all 16 VBench metrics.

1. Introduction

  • T2V models still produce artifacts, semantic misalignment, and unnatural motion, while subjective preferences are difficult to formalize as objective functions.These challenges make incorporating human preferences into T2V models difficult.Source: §1 · p. 1
  • Video quality assessors trained on human ratings overlook evaluation reasoning, and video-based human feedback for T2V alignment remains unexplored.The text identifies T2VQA [17] and VideoScore [13] as rating-based assessment approaches.Source: §1 · pp. 1–2
  • The proposed pipeline uses human annotations to train a reward model that supplies the signal for fine-tuning T2V outputs toward human expectations.The authors position the dataset's rationales as a way to capture nuanced evaluation criteria beyond numerical scores.Source: §1 · p. 2
  • LIFT collects 10K annotations with ratings and rationales across semantic consistency, motion smoothness, and video fidelity, then trains LIFT-CRITIC to predict scores and capture reasons.The learned reward is used to fine-tune the T2V model through reward-weighted likelihood maximization.Source: §1 · p. 2
  • CogVideoX-2B fine-tuned with LIFT outperforms CogVideoX-5B across all 16 VBench metrics.The paper presents this as a case study of the pipeline.Source: §1 · p. 2

2. Related Work

Prior T2V work includes diffusion-model development and reward-based alignment, but image-based feedback misses temporal video information and rating-based video assessment omits reasoning. LIFT is presented as a reward model that evaluates synthesized videos with ratings and reasons.

2. Related Work

  • Diffusion-based T2V pipelines adapt pretrained T2I models or jointly train on image and video data, while newer approaches address spatiotemporal resolution.The passage says these models remain inadequate at aligning synthesized videos with human preferences.Source: §2 · p. 2
  • Image reward models used in T2V training overlook temporal dynamics, while T2VQA [17] and VideoScore [13] assess human-rated videos but focus on scores rather than evaluation reasoning.The authors propose a reward model that provides both ratings and corresponding reasons.Source: §2 · p. 2
  • Visual generative-model alignment has used reinforcement learning and reward fine-tuning, but video-based human feedback has not been fully leveraged for T2V alignment.The paper frames its pipeline as an exploration of fine-tuning T2V models with human preferences.Source: §2 · p. 2

3. LIFT

LIFT collects video ratings with rationales, trains a reward model to capture human preferences, and uses that reward to fine-tune a text-to-video model. Its dataset spans diverse categories, and its alignment methods balance reward weighting with realistic video-text training data.

3.1. Overview

  • LIFT’s three-stage pipeline collects human feedback, learns a reward function, and fine-tunes a text-to-video model to better align with human expectations.The stages are human feedback collection, reward function learning, and T2V model alignment.Source: §3.1 · p. 3

3.2. LIFT-HRA: Human Feedback Collection

  • 10K human-feedback video quality question-answer pairs form LIFT-HRA, combining scores with rationales across semantic consistency, motion smoothness, and video fidelity.Prompts are built from diverse subjects, scenes, and actions, then used to generate multiple videos for annotation.Source: §3.2 · pp. 3–4
  • Most LIFT-HRA videos are rated Bad, followed by Good and Normal; feedback varies across categories, reflecting different evaluation challenges.The dataset covers diverse categories, with multiple video types and varying sample counts within categories.Source: §3.2 · p. 4

3.3. LIFT-CRITIC: Reward Function Learning

  • LIFT-CRITIC fine-tunes a pretrained vision-language model to generate qualitative scores and explanations, rather than predicting scores through a regression head.Detailed reasoning is included in answer options so the model predicts multi-level scores and justifications.Source: §3.3 · p. 4

3.4. T2V Model Alignment

  • Reward-weighted learning increases the likelihood of higher-reward synthesized videos, while a real video-text dataset regularizes training against weak temporal consistency.The real-data term is intended to ground frame-to-frame dynamics; the paper sets its balancing hyperparameter λ to 1.Source: §3.4 · p. 5
  • Rejection sampling keeps synthesized samples rated Good in every dimension, trading efficiency for reduced-data overfitting risk compared with reward-weighted learning.Reward-weighted learning uses all samples but is more computationally intensive; rejection sampling filters the synthesized dataset to high-quality samples.Source: §3.4 · pp. 5–6

4. Experiments

Experiments evaluate LiFT with benchmark, qualitative, and human comparisons, then test reward-learning choices, reward-model scale, and reasoning annotations. The reported results favor the fine-tuned model and show benefits from larger reward models and rationale-based learning.

4.1. Experimental Setup

  • The setup fine-tunes CogVideoX-2B and LIFT-CRITIC using human-feedback data, with additional real video samples included for reward learning and alignment.The setup uses about 10K LIFT-HRA pairs, 1K real videos for reward-model training, 40K synthesized videos, and about 20K real video-text pairs for alignment.Source: §4.1 · p. 6

4.2. Results

  • LIFT-CRITIC-40B improves nearly all evaluation metrics, including Subject Consistency and Motion Smoothness, and exceeds CogVideoX-5B in Imaging Quality and Multiple Objects.Qualitative comparisons also report better semantic consistency, motion smoothness, and video fidelity, including a successful camera transition and a clearer farmer face.Source: §4.2 · pp. 6–7, Figure 5 · p. 7

4.3. Human Evaluation

  • In pairwise human evaluation on 120 prompts, CogVideoX-2B-LiFT has an advantage over CogVideoX-2B and CogVideoX-5B.The reported figure shows positive-vote percentages and highlights cases with at least three of six positive votes.Source: §4.3 · p. 7, Figure 6 · p. 7

4.4. Ablation Studies

  • Rejection sampling reduces fine-tuning data while outperforming CogVideoX-2B and remaining competitive with CogVideoX-5B, though it is slightly less effective than reward-weighted learning.Only videos rated “Good” across all three evaluation dimensions are retained.Source: §4.4 · pp. 7–8, Figure 8 · p. 8, Figure 7 · p. 8
  • LIFT-CRITIC-40B yields consistent improvements over the 13B reward model across all metrics, and larger reward models improve reward-model evaluation accuracy.The paper attributes the alignment improvements to better capture of nuanced human preferences.Source: §4.4 · p. 8
  • For LIFT-CRITIC-40B, adding reasoning raises average accuracy from 87.06% to 93.55%.The comparison holds model size constant and evaluates reward learning with versus without reason-based annotations.Source: §4.4 · p. 8

5. Conclusion

The conclusion presents LiFT as a human-feedback pipeline for T2V alignment and reports that its CogVideoX-2B case study surpasses CogVideoX-5B across 16 metrics. The accompanying appendices describe prompt generation, annotators, evaluation guidance, and the annotation interface.

5. Conclusion

  • LiFT combines human ratings and reasons in LIFT-HRA, trains LIFT-CRITIC to learn a reward function, and uses it to align a T2V model.In the CogVideoX-2B case study, the fine-tuned model outperforms CogVideoX-5B across all 16 metrics.Source: §5 · p. 8

A.1. Video-Text Dataset Generation

  • Prompts combine randomly selected subjects, places, and actions, then an LLM refines them into detailed descriptions for video generation.The selection categories include humans, animals, places, and simple or complex actions.Source: §A.1 · p. 11

A.2. Annotator Management

  • Ten expert annotators, all undergraduate or graduate students, were recruited; they were English-proficient and some were compensated.Some annotators later became coauthors.Source: §A.2 · p. 11

A.3. Annotation Guideline

  • Annotators score semantic consistency, motion smoothness, and video fidelity as Good, Normal, or Bad, with written reasons for each dimension.Guidelines provide examples and direct annotators to look up unfamiliar terms or skip confusing or obscure prompts.Source: §A.3 · p. 11

A.4. Annotation Interface

  • The web interface places the video beside its text prompt and collects a score and written reason for each evaluation dimension.The prompt appears above the response box.Source: §A.4 · p. 11

A.5. Data Correction

The data-correction procedure uses coarse filtering, iterative model-assisted refinement with human review, and final integration of reviewed re-annotations.

A.5. Data Correction

  • Coarse filtering removes obvious annotation errors, irrelevant feedback, and samples with unclear or ambiguous videos or prompts.The filtering checks consistency with the predefined evaluation dimensions and scoring guidelines.Source: §A.5 · pp. 11–12
  • Iterative refinement alternates reward-model annotation and human review when model-generated ratings or rationales disagree substantially with the original annotations.The corrected second half is used to retrain the model, which is then applied to the first half.Source: §A.5 · p. 12
  • Final integration applies the final reward model to data removed during coarse filtering and reintegrates newly annotated samples that pass human review.The process aims to identify and rectify inconsistencies, ambiguities, and errors.Source: §A.5 · pp. 11–12

B.1. Implementation Details

Implementation details specify reward-model training configurations and the evaluation benchmarks used for T2V generation.

B.1. Implementation Details

  • LIFT-CRITIC-13B and 40B are trained for 20 epochs with different batch sizes and LoRA settings; vision towers and their projectors are frozen while LLM linear layers are fine-tuned.Training uses AdamW with a 1e−5 base learning rate, a cosine schedule with 3e−2 warmup, and eight NVIDIA H100 GPUs.Source: §B.1 · p. 12
  • T2V evaluation uses VBench’s disentangled quality dimensions, while the reward model is also assessed on VideoScore across visual quality, temporal consistency, dynamic degree, text-to-video alignment, and factual consistency.VBench prompts are optimized using Qwen2.5-72B-Instruct.Source: §B.1 · p. 12, Table 4 · p. 12

B.2. Evaluation on VideoScore Benchmark

The section compares CogVideoX-2B, CogVideoX-2B-LiFT, and CogVideoX-5B using VideoScore and reports the results in Tab. 4.

B.2. Evaluation on VideoScore Benchmark

  • The VideoScore comparison evaluates CogVideoX-2B, CogVideoX-2B-LiFT, and CogVideoX-5B, with Tab. 4 reported as further evidence of LiFT’s effectiveness.The passage gives no metric values or specific ranking.Source: §B.2 · p. 12

B.3. Qualitative Results of LIFT-CRITIC

The section reports qualitative evaluations of LiFT-Critic, a T2V-Turbo application, and additional model comparisons; it also describes the scope of a video-assessment table.

B.3. Qualitative Results of LIFT-CRITIC

  • Qualitative results in Figs. 12 and 13 are presented to demonstrate LiFT-Critic’s nuanced evaluations and alignment with human preferences.The passage does not state the specific outcomes shown in those figures.Source: §B.3 · p. 12
  • Table 5 reports video-assessment metrics, with the first seven measuring quality and the remaining metrics measuring semantics; “RM” denotes “Reward Model”.Source: Table 5 · p. 13

B.4. Reward-weighted Learning on T2V-Turbo

  • The authors apply LiFT to T2V-Turbo as a robustness test and report the results in Table 5.No specific outcome is stated in the supplied passage.Source: §B.4 · p. 13

B.5. More Qualitative Comparison

  • Additional qualitative comparisons are provided in Figs. 14 and 15.Source: §B.5 · p. 13

B.6. Human Evaluation

The human evaluation compares three CogVideoX variants, and LiFT-Critic’s rankings closely match human evaluations across all three dimensions.

B.6. Human Evaluation

  • The human evaluation compares CogVideoX-2B, CogVideoX-5B, and CogVideoX-2B-LiFT using a web-based interface.Source: §B.6 · p. 13
  • LiFT-Critic’s rankings are highly consistent with human evaluations across all three dimensions when comparing CogVideoX-2B, CogVideoX-5B, and CogVideoX-2B-LiFT.The passage presents this consistency as validation that the reward model captures human preferences and provides reliable, interpretable feedback for alignment.Source: §B.6 · p. 13

B.7. Ablation Study of Real Video-Text Dataset

The ablation reports declines in three temporal-quality metrics when Dreal is excluded; the societal-impact discussion flags potential bias risks and calls for continued scrutiny.

B.7. Ablation Study of Real Video-Text Dataset

  • Dreal is incorporated to address limitations of synthesized-only data, whose videos often have low temporal consistency that hinders coherent subject alignment and smooth motion.Source: §B.7 · p. 13
  • Excluding Dreal significantly reduces subject consistency, background consistency, and temporal flickering scores, underscoring real data’s role in temporal quality.The authors say Dreal grounds the model in realistic frame-to-frame dynamics and supports semantic and temporal fidelity.Source: §B.7 · p. 13, Figure 10 · p. 13

C. Societal Impacts

  • The societal-impact discussion warns that annotation biases may propagate into the reward model and lead to unintended or unfair outcomes, despite curation and transparency measures.The authors call for continuous scrutiny to support ethical and equitable deployment.Source: C. Societal Impacts · pp. 13–14

D. Limitations and Future Works

The paper identifies limits in dataset coverage and optimization strategy, and reports that the work followed ethical guidelines with no known ethical concerns.

D. Limitations and Future Works

  • The dataset covers three dimensions, four text categories, and good/normal/bad feedback, motivating broader dimensions, categories, and richer annotation formats.Source: D. Limitations and Future Works · p. 14
  • The work uses reward-weighted likelihood maximization, while RLHF is proposed as an alternative that may support online sampling and KL-regularization.Source: D. Limitations and Future Works · p. 14

E. Ethical Statement

  • The authors report that, to their knowledge, the study raises no ethical concerns and that experiments followed established ethical guidelines.Source: E. Ethical Statement · p. 14

Related papers

Loading 2412.04814v3…