PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback

Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, Sherry Yang

arXiv:2412.02617v2cs.LGcs.AIcs.CV

TL;DR

Text-to-video models struggle with realistic dynamic object interactions, motivating feedback-based alignment. The paper derives a unified view of RL-finetuning and uses binary VLM feedback focused on object dynamics. AI feedback produces the strongest interaction-scene improvements across AI, human, and quality-metric evaluations, while online extension remains limited by VLM cost and other bottlenecks.

  • Problem

    Text-to-video models struggle with dynamic object interactions, unrealistic movements, and physics violations, motivating investigation of feedback for improving object dynamics.

  • Method

    The paper unifies offline RL-finetuning objectives and uses binary VLM feedback that evaluates video coherence, physical accuracy, task completion, and inconsistencies.

  • Results

    AI feedback drives the most significant interaction-scene improvements compared with popular video-quality metrics, as confirmed by AI, human, and quality-metric evaluations.

  • Takeaways & Limitations

    VLM feedback is especially useful for improving multi-object interactions, new-object appearances, and spatial relationships in dynamic scenes.

  • Takeaways & Limitations

    The study focuses on offline and iterative RL-finetuning because online VLM querying is costly, text-video reward training lacks a standard recipe, and policy-gradient methods are unstable.

Abstract

from arXiv · show

Large text-to-video models hold immense potential for a wide range of downstream applications. However, they struggle to accurately depict dynamic object interactions, often resulting in unrealistic movements and frequent violations of real-world physics. One solution inspired by large language models is to align generated outputs with desired outcomes using external feedback. In this work, we investigate the use of feedback to enhance the quality of object dynamics in text-to-video models. We aim to answer a critical question: what types of feedback, paired with which specific self-improvement algorithms, can most effectively overcome movement misalignment and realistic object interactions? We first point out that offline RL-finetuning algorithms for text-to-video models can be equivalent as derived from a unified probabilistic objective. This perspective highlights that there is no algorithmically dominant method in principle; rather, we should care about the property of reward and data. While human feedback is less scalable, vision-language models could notice the video scenes as humans do. We then propose leveraging vision-language models to provide perceptual feedback specifically tailored to object dynamics in videos. Compared to popular video quality metrics measuring alignment or dynamics, the experiments demonstrate that our approach with binary AI feedback drives the most significant improvements in the quality of interaction scenes in video, as confirmed by AI, human, and quality metric evaluations. Notably, we observe substantial gains when using signals from vision language models, particularly in scenarios involving complex interactions between multiple objects and realistic depictions of objects falling.

1 Introduction

Text-to-video models still struggle with realistic, instruction-following object dynamics, motivating external feedback for improving generated interactions. The paper studies feedback types and RL-based self-improvement, finding that VLM-based AI feedback yields the strongest interaction-scene improvements.

  • Current text-to-video models generate unrealistic objects or movements, static scenes, and ignored behaviors that hinder practical use.
  • External feedback is investigated as a scalable alternative for reducing hallucination and improving video events when video-model scaling and detailed labeling remain difficult.
  • RWR and DPO can arise from a shared probabilistic objective, so reward and data properties matter more than a theoretically dominant offline algorithm.
  • The paper evaluates metric-based alignment, human-preference, and dynamics feedback, while proposing VLM feedback focused on object interactions.
  • VLM-based AI feedback produces the most significant interaction-scene improvements, confirmed by AI, human, and quality-metric evaluations.

2 Preliminaries

The paper formulates DDPM video generation as a finite-horizon MDP and introduces an optimality event to derive a unified RL-finetuning objective. Existing algorithms can then be recovered through policy projection or preference-modeling assumptions.

  • Denoising Diffusion Probabilistic Models: DDPM generates videos through a parameterized reverse denoising process from Gaussian noise, with a forward process that progressively adds noise.
  • Probabilistic Inference View of RL-Finetuning Objectives: The denoising process is represented as a T-horizon MDP whose states contain noisy video and conditioning, actions choose earlier denoised states, and reward arrives at the terminal video.
  • Probabilistic Inference View of RL-Finetuning Objectives: A binary optimality event O indicates whether the generated video is optimal, enabling decomposition of log p(O = 1 | c) with a variational distribution.
  • Probabilistic Inference View of RL-Finetuning Objectives: Assuming optimality likelihood is proportional to exp(β^-1r(x0, c)) yields an explicit unified RL-finetuning objective.
  • Probabilistic Inference View of RL-Finetuning Objectives: Existing algorithms are recovered from the unified objective through expectation-maximization policy projection or Bradley-Terry preference assumptions.

3 RL-Finetuning with Feedback

This section develops a unified view of RL-finetuning objectives, compares metric-based rewards, and proposes binary VLM feedback for improving dynamic video interactions.

  • 3.1 Connection to Practical Algorithms: Offline RL-finetuning methods can be recovered from a unified KL-regularized objective, so reward and data properties matter more than inherent algorithmic superiority.The framework connects practical approaches through probabilistic inference and covers forward- and reverse-projection variants, including RWR and DPO.
  • 3.2 Metric-based Reward for RL-Finetuning: Metric-based rewards evaluate framewise alignment, human-preference proxies, or optical flow, with linear reward shaping used to control gradient scale.The compared metrics are CLIP, HPSv2, PickScore, and Optical Flow.
  • 3.3 AI Feedback from Vision-Language Models: RL-finetuning with RWR and DPO improves most evaluated metrics over pretrained models or SFT and generalizes better to unseen prompts.The result is summarized in Table 1 for independently optimized reward-algorithm combinations.
  • 3.3 AI Feedback from Vision-Language Models: AI feedback with RWR or DPO achieves the best overall quality compared with metric-based rewards, across Gemini, GPT, human, and many VBench evaluations.The comparison targets consistency and smoothness as well as preference-based quality.
  • 3.3 AI Feedback from Vision-Language Models: VLM feedback evaluates prompt-video pairs for coherence, physical accuracy, task completion, and inconsistencies, returning a binary accept-or-reject label.Gemini-1.5-Pro is used mainly for generation and evaluation, while GPT-4o tests generalization.
  • 3.3 AI Feedback from Vision-Language Models: VLMs classify true videos as preferable to generated videos 90.3% of the time, supporting their use as scalable proxies for human supervision.Preliminary evaluations also found that VLMs recognized scene events such as successful or failed bottle grasping.

4 Experiments

The experiments evaluate RL-finetuning algorithms and feedback choices for challenging object movements, finding that AI feedback substantially improves dynamic-scene quality across evaluations. Results also distinguish algorithm trade-offs, movement categories, iterative behavior, human alignment, and practical deployment settings.

  • 4.1 A Set of Challenging Object Movements: The benchmark targets challenging object movements including object removal, multiple objects, deformable objects, and directional movement because generated videos often ignore behaviors or violate physical laws.Using Something-Something-V2, it selects 32 prompts per category from the validation split: 24 for training and 8 for testing; RL-finetuning uses 120 training and 40 test prompts, larger and more diverse than prior work.
  • 4.2 RL-Finetuning Works Better than SFT: RL-finetuning improves most reward metrics and generalizes better than SFT, while DPO outperforms RWR on 7 of 10 metrics.The comparison suggests that the KL-regularization direction creates meaningful practical differences despite the shared objective perspective.
  • 4.3 VLMs Enhance Dynamic Scene Quality: RL-finetuning with Gemini AI feedback achieves the strongest overall dynamic-scene improvements, including +3.8% Gemini, +2.4% GPT, and +11.8% human-evaluation gains for RWR-AIF.DPO-AIF achieves +3.5%, +1.8%, and +14.7% absolute gains, while Gemini feedback also improves four VBench scores, especially consistency and smoothness.
  • 4.5 Takeaways for Algorithm Choices: Iterative RWR-AIF keeps improving from 52.66% to 56.41% to 58.83%, while DPO-AIF saturates at 56.25%; RWR is therefore suited to online improvement but risks offline overfitting.For offline-only or generalization-focused settings, the recommendations favor reverse-BT-projection; forward-EM-projection is favored for iterative optimization within a prompt set.
  • 4.3 VLMs Enhance Dynamic Scene Quality: Improvements are largest for multiple-object and falling-down scenes, whereas deformable and directional movements are initially easier and multi-step interactions remain difficult.A drawer task requiring opening, picking up a bottle opener, and replacing it remains stuck at the first step after finetuning.
  • 4.4 Connection to Human Evaluation: VLM preference correlates most strongly with human preference among automated feedback methods, reaching Pearson R = 0.746 with p ≤0.01.Metric-based DPO can over-optimize its target metric while reducing the proportion of videos accepted by VLM evaluation.

5 Discussion and Limitation

The paper compares AI feedback with differentiable video metrics and frames offline iterative RL-finetuning as a practical response to current feedback and optimization constraints.

  • VLM-based AI feedback is contrasted with differentiable rewards as a way to assess and improve dynamic video scenes.
  • The study focuses on offline and iterative RL-finetuning because online VLM queries are costly, text-video reward training lacks a standard recipe, and policy gradients can be unstable.

6 Related Works

Related work spans RL for image and video generation, AI feedback, and object-interaction modeling. This paper distinguishes itself by optimizing dynamic object movement with offline feedback and evaluating beyond visual style.

  • RL for Text-to-Image Generation: Prior text-to-image RL methods optimize objectives such as compressibility, aesthetics, or human preference, whereas this work targets the more complex temporal dynamics of text-to-video.
  • RL for Text-to-Video Generation: Existing text-to-video RL work mainly uses metric-based rewards for aesthetic or visual quality objectives, while this paper studies feedback for object movement using offline learning.
  • RL for Text-to-Video Generation: Compared with concurrent DPO approaches, this work analyzes DPO–RWR equivalence and evaluates AI feedback with AI, human, and VBench measures rather than VBench alone.
  • AI Feedback for LLMs: Long-context VLMs provide feedback on physical plausibility, consistency, and instruction following, offering automated signals for dynamic-scene generation.
  • Modeling Object Interaction in Video Generation: Unlike prior object-interaction methods evaluated mainly with automated metrics, this work combines VLM and human evaluation and shows RL-finetuning can improve content beyond visual style.

7 Conclusion

The paper examines design choices for improving dynamic scenes and concludes that iterative RL-finetuning with VLM feedback performs best among the evaluated reward choices.

  • 7 Conclusion: Iterative RL-finetuning with VLM-based AI feedback enhances dynamic-scene quality better than metric-based rewards.The method may mitigate multi-step interactions, new-object appearance, and spatial-relationship failures.

A Details of Model Training

The experiments use Something-Something-V2 dynamic-object prompts, evaluate videos with an eight-frame VLM review protocol, and apply offline RL-finetuning with labeled generated samples.

  • A Details of Model Training: The training setup uses a 1.6B-parameter base video model and a 1.4B-parameter super-resolution model, with RL applied only to the base model.Something-Something-V2 supplies 160K training and 8.5K validation prompt-video pairs; 15,360 generated samples are collected for finetuning.
  • B Example of Something-Something-V2: The benchmark covers five challenging movement principles: object removal, multiple objects, deformable objects, directional movement, and falling down.
  • Object Removal (Train/Test): Object-removal prompts test revealing, extracting, uncovering, burying, or emptying objects across training and evaluation examples.
  • Multiple Objects (Train/Test): Multiple-object prompts test placement, stacking, pushing, separation, proximity, passage, and object-fit interactions.
  • Deformable Object (Train/Test): Deformable-object prompts include tearing, spreading, squeezing, twisting, unfolding, stuffing, and stacking flexible materials.
  • Directional Movement and Falling Down: Directional and falling-down prompts evaluate controlled movement, orientation changes, dropping, tipping, sliding, and objects falling from surfaces.
  • D Prompt for AI Feedback from VLMs: The VLM reviewer examines eight consecutive images for coherence, physical accuracy, task completion, and object inconsistencies before accepting or rejecting the sequence.
  • E Pseudo Algorithm for RL-Finetuning from Feedback: The offline RL pipeline pretrains a video model, generates finetuning samples, labels them with VLM or metric rewards, and optimizes RWR or DPO objectives.

F Failure Mode of SoTA Open Text-to-Video Models

State-of-the-art open text-to-video models often produce visually strong videos but fail to represent prompted object movements and complex dynamic interactions. These failures include missing actions such as falling, tearing, and dropping, as well as unrealistic scene dynamics.

  • Open models such as VideoCrafter, CogVideoX, and Wan 2.1 remain insufficient for generating realistic dynamic scenes.Preliminary experiments report weak dynamic-scene generation despite strong overall model quality.
  • Product models such as Sora can generate seemingly good videos but still produce implausible dynamics, including objects unnaturally appearing in a scene.
  • VideoCrafter fails to depict prompted movements including objects falling off and tearing.

G Performance Gap before and after RL-Finetuning

RL-finetuning with reward feedback improves dynamic object interactions while largely preserving pretrained video quality. Generated examples compare pretrained, RWR-AIF, and DPO-AIF models, illustrating the resulting changes in interaction quality.

  • G Performance Gap before and after RL-Finetuning: Reward-based RL-finetuning improves complex dynamic object interactions compared with SFT while maintaining pretrained performance within a reasonable range.The comparison uses AI feedback, human feedback, and VBench evaluations; conventional FVD, FID, and IS slightly decrease while CLIP improves.
  • H Example of Generated Video: Generated-video examples compare pretrained models with RWR-AIF and DPO-AIF outputs.

I Failure Mode of Over-Optimization

Metric-based reward optimization can over-optimize visual scores without improving video dynamics, while RL-finetuned models still fail on several classes of complex scene interaction. These failures involve multi-step actions, newly appearing objects, and three-dimensional spatial relationships.

  • I Failure Mode of Over-Optimization: DPO optimization of metrics such as HPSv2 or PickScore can improve metric values while making generated videos visually worse to humans.The reported over-optimization issue is illustrated by videos whose frame tone becomes dark without dynamic improvement.
  • J Failure Mode of Dynamic Scene Generation: RL-finetuned models still struggle with multi-step interactions, new-object appearance, and spatial three-dimensional relationships.Examples include stuffing a bottle opener into a drawer, placing pencils on a blanket, and dropping a wallet behind a vase.

K Extended Analysis of Correlation between Human Evaluation and Automated Feedback

AI preference from vision-language models correlates most positively with human preference among the evaluated automated feedback signals, supporting its use as a proxy for human evaluation. However, this alignment remains weak, reflecting the complexity of video-quality judgment.

  • VLM-based AI preference has the most significant positive correlation with human preference among the evaluated feedback signals.The analysis averages each metric across algorithm-feedback combinations and compares Pearson correlations with human preference.
  • VLM judgments do not fully align with humans because video evaluation combines criteria including alignment, aesthetics, smoothness, consistency, spatial relationships, and temporal dependencies.The paper therefore calls for improved VLM quality judgments calibrated with human perception.
  • Across 160 prompts, AI preference shows only a weak positive correlation with human preference, with R = 0.231 and p ≤0.01.Other automated feedback signals show no notable correlations in the reported comparison.
Loading 2412.02617v2…