PaperlayerView exact revision on arXiv

Source-linked AI summary

VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos

Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao

arXiv:2505.23693v1cs.CVcs.AIcs.CL

TL;DR

MLLM evaluation has largely focused on natural videos, leaving their interpretation of AIGC videos underexplored. VF-EVAL benchmarks four feedback and reasoning tasks, and experiments find that frontier models still struggle while human-preference alignment may benefit video generation.

  • Problem

    Most existing video assessments focus on natural videos, leaving MLLM capabilities for interpreting AIGC videos underexplored.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    VF-EVAL evaluates AIGC-video feedback through coherence validation, error awareness, error type detection, and reasoning evaluation.

    Source: §1 · pp. 1–2

  • Results

    Frontier MLLMs face significant challenges on AIGC-video tasks, while RePrompt suggests aligning MLLM feedback with human preferences can make feedback more valuable and trustworthy.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Aligning MLLM feedback with human preferences can potentially enhance the quality and coherence of AI-generated videos.

    Source: §1 · p. 2

    The RePrompt pipeline uses textual human feedback without specifying the positions of errors, limiting feedback granularity.

    Source: Limitations · p. 9

Abstract

from arXiv · show

MLLMs have been widely studied for video question answering recently. However, most existing assessments focus on natural videos, overlooking synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in video generation rely on MLLMs to evaluate the quality of generated videos, but the capabilities of MLLMs on interpreting AIGC videos remain largely underexplored. To address this, we propose a new benchmark, VF-Eval, which introduces four tasks-coherence validation, error awareness, error type detection, and reasoning evaluation-to comprehensively evaluate the abilities of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that even the best-performing model, GPT-4.1, struggles to achieve consistently good performance across all tasks. This highlights the challenging nature of our benchmark. Additionally, to investigate the practical applications of VF-Eval in improving video generation, we conduct an experiment, RePrompt, demonstrating that aligning MLLMs more closely with human feedback can benefit video generation.

1 Introduction

The introduction identifies gaps in MLLM feedback for AIGC videos and introduces VF-EVAL to assess feedback and reasoning across four tasks. Experiments find persistent challenges across models, while RePrompt suggests human-preference alignment may improve generated-video quality and coherence.

1 Introduction

  • MLLMs are increasingly used in video generation for quality assessment and to provide feedback intended to improve generated content.Source: §1 · p. 1
  • Prior MLLM feedback for AIGC videos can lack precision, while implicit quality scores may miss the nuances of video quality.The introduction links this challenge to synthetic visual characteristics that differ from traditional videos.Source: §1 · p. 1
  • VF-EVAL assesses feedback through coherence validation, error awareness, and error type detection, plus reasoning evaluation spanning six subtasks.The reasoning subtasks cover spatial and temporal reasoning, action and object reasoning, counting, and information synopsis.Source: §1 · pp. 1–2
  • Across 13 frontier MLLMs, models struggle on AIGC tasks; auxiliary methods can improve feedback accuracy, and open-source models are competitive and further improvable.The introduction also reports that aligning MLLM feedback with human preferences can potentially enhance video quality and coherence.Source: §1 · p. 2
  • RePrompt compares MLLM and human feedback for video-generation prompts; aligning MLLM feedback with human preferences can potentially improve quality and coherence.The reported benefit is framed as potential rather than guaranteed.Source: §1 · p. 2

2 Related works

Prior video-understanding benchmarks largely use natural videos, leaving MLLM reasoning on AIGC videos underexplored. Existing video-quality evaluation work ranges from quantitative computer-vision scores to MLLM assessments, while VF-EVAL focuses on detailed AIGC feedback and error diagnosis.

2.1 Video Understanding Benchmark

  • Existing video-understanding datasets typically use natural videos, leaving MLLM reasoning on AIGC videos underexplored; VF-EVAL addresses this with four tasks.Source: §2.1 · p. 3

2.2 Evaluation of AIGC Video Generation

  • Traditional computer-vision quality assessments provide quantitative scores but can miss divergence from human preferences and areas needing improvement (Tu et al., 2021; Ging et al., 2024; Huang et al., 2024; Fan et al., 2024; Liu et al., 2023).Source: §2.2 · p. 3
  • Prior MLLM assessments offer more interpretable quality evaluation (Wu et al., 2024a; Meng et al., 2024; Wang et al., 2024c), whereas VF-EVAL benchmarks detailed AIGC feedback, including quality-issue diagnosis and specific error identification.Source: §2.2 · p. 3

3 VF-EVAL Benchmark

VF-EVAL structures evaluation of MLLM feedback on AIGC videos around four tasks, with dedicated data construction and human validation.

3.1 Task Formulation

  • Coherence Validation tests prompt-video alignment and prompt revision, while Error Awareness detects general errors and Error Type Detection identifies errors in greater detail.Coherence Validation uses open-ended questions; Error Awareness uses yes-or-no questions, and Error Type Detection primarily uses multiple-choice questions.Source: §3.1 · pp. 3–4
  • Reasoning Evaluation covers six sub-tasks—spatial and temporal, action and object reasoning, counting, and information synopsis—with an LLM judging responses against human answers.The benchmark uses GPT-4.1-mini for this response evaluation.Source: §3.1 · p. 4

3.2 Dataset Construction Guidelines

  • Dataset guidelines target broad scenario coverage, knowledge-intensive questions, and reasoning problems, including distractors generated by GPT-4o and checked by human reviewers.The team uses 1,000 prompts generated by GPT-4o and validates them with human experts.Source: §3.2 · pp. 4–5

3.3 Dataset Construction

  • The dataset combines videos generated by proprietary and open-source models with AIGC videos collected from Lavie and OpenSora in Videophy's training split.Generation sources include Pika, Kling, Pixeldance, Gen-3, and T2V-turbo-v2.Source: §3.3 · p. 5
  • Human annotators identify errors across video quality, commonsense and physical violations, and morality; reviewed MLLM-generated distractors complete multiple-choice items, while open-ended items cover coherence and reasoning.Yes-or-no questions are converted from multiple-choice questions, with “Yes” set as the correct answer to investigate bias toward perceiving videos as normal.Source: §3.3 · p. 5

3.4 VF-EVAL Data Analysis

  • VF-EVAL contains 9,740 question-answer pairs: 1,826 Yes-Or-No, 5,932 Multiple-choice, and 1,982 Open-Ended; its videos are longer than those in existing works.The statistics are reported across question types and data splits.Source: §3.4 · p. 5, Table 2 · p. 6
  • Three annotators with top interagreement scores reviewed every pair after construction, correcting 2,395; the authors cite the low revision percentage as evidence of quality.A second annotator also checks annotations during construction.Source: §3.4 · p. 6

4 Experiments

Across 13 frontier MLLMs, VF-Eval reveals substantial performance gaps, including for the best overall model, GPT-4.1. Performance varies by task, and the experiments also describe model settings and dataset-quality checks.

4.1 Experiment Settings

  • The evaluation covers seven open-source model series and two proprietary series, using Chain-of-Thought prompting and image inputs for models without native video support.For those models, visual input is limited by the maximum number of images fitting their context window.Source: §4.1 · p. 6

4.2 Main Findings

  • Without visual data, performance drops significantly, particularly on Coherence Validation and Reasoning Evaluation.Error Awareness performs worse than random guessing, which the authors interpret as models tending to perceive videos as normal.Source: Figure 3 · p. 6, §4.2 · p. 6
  • GPT-4.1, the best overall model, remains far from human behavior on every sub-task, while overall performance varies across models.The authors also report a large gap between the best proprietary and best open-source models, alongside potential for open-source models to become competitive through fine-tuning.Source: §4.2 · p. 6, Table 3 · p. 7
  • Task-specific results vary: models do better on Error Awareness “Quality” than “CP” but underperform on Error Type Detection, including morality violations.The authors attribute the difference to limited video-quality-assessment knowledge and difficulty detecting subtle “CP” violations.Source: §4.2 · p. 7
  • AIGC-video blur, sudden appearances or disappearances, and abrupt structural changes challenge detail capture and commonsense reasoning.In Coherence Validation, models may identify prompt-video misalignment yet offer only simple expansions rather than better generation prompts.Source: §4.2 · p. 7

5 Analysis

The analysis finds that human-revised prompts produce videos that outperform originals, with gains differing across evaluation aspects. It also examines reasoning-task variation and recurring error types in MLLM feedback.

5.1 Reasoning abilities analysis

  • GPT models and InternVL3-38B are stronger on Information Synopsis and Object Reasoning, while models perform worse on Spatial and Temporal Reasoning.The authors suggest stronger performance may reflect greater incorporated knowledge.Source: §5.1 · pp. 7–8

5.2 Error Analysis

  • Error analysis identifies prompt-generation misconceptions, reliance on textual cues, missed critical details, and over-reliance on commonsense.For example, models may choose randomly when distractors also violate physical laws, or infer a single ball from commonsense despite video evidence.Source: §5.2 · p. 8

5.3 REPROMPT Analysis

  • REPROMPT tests human-in-the-loop feedback by having annotators revise LLM-generated prompts using video content and regenerate the videos.Human judges compare each regenerated video against its original on subject consistency, background consistency, aesthetic appeal, and overall image quality.Source: §5.3 · p. 8
  • Human-revised prompts consistently outperform originals, with the clearest gains in subject consistency and aesthetics and more modest gains in visual quality and background consistency.The experiment compares 300 regenerated videos with originals using pairwise human judgments across four aspects.Source: Table 4 · p. 8, §5.3 · p. 8

6 Conclusion

Frontier MLLMs struggle with VF-Eval tasks involving AIGC videos, while the conclusion reports greater feedback value when MLLMs align better with human preferences.

6 Conclusion

  • AIGC-video temporal inconsistencies, unrealistic object behavior, and visual-textual misalignment challenge MLLM feedback, while human-preference alignment can make that feedback more valuable and trustworthy.The conclusion also suggests computer-vision techniques and in-context learning can further enhance feedback precision.Source: §6 · p. 8

Limitations

The study identifies three scope and design limitations: it considers only text-to-video generation, uses textual human feedback without error locations, and excludes cross-modal videos. Annotator biographies are presented in Table 5.

Limitations

  • Only text-to-video models are considered, leaving error cases in image-generated videos outside this study’s scope.The authors note that image-generated videos may exhibit other types of error cases.Source: Limitations · p. 9
  • The RePrompt pipeline uses textual human feedback without error locations, limiting feedback granularity.Source: Limitations · p. 9
  • Cross-modal videos are excluded, leaving scenarios involving audio information from some video-generation models unrepresented.Source: Limitations · p. 9

A.1 Annotator Biography

A.2 Data Annotation and Validation

The annotation workflow uses Label Studio, a separate reviewer for each video, and rechecks for annotators below an 80% agreement score. Its five tasks cover coherence validation, error awareness and type detection, and reasoning evaluation.

A.2 Data Annotation and Validation

  • Each video receives a separate review, and annotators with an Annotator Inter Agreement score below 80% must have their annotations rechecked.Source: §A.2 · p. 12

A.3 Annotation Interface and Guideline

  • In Label Studio’s five-task interface, annotators first complete a 10-video trial; Q1 checks video-prompt misalignment, Q2–Q4 select and locate error types, and Q5 requires a reasoning question.Source: §A.3 · p. 13, Figure 5 · p. 13

B Experiment Setup

The experiment setup documents model configurations and evaluation prompts, with response extraction for differently formatted outputs. RePrompt details define its video-quality metrics and describe the win-rate calculation inputs.

B.1 MLLM Model Configuration

  • Table 6 lists VF-EVAL model details, including default input frames selected from 2, 4, 8, or 16 based on context window.Source: Table 6 · p. 14, §B.1 · p. 14

B.2 Prompts for Evaluation

[EXTRACT_YESNO_RESPONSE_PROMPT]]

[EXTRACT_OPENEND_RESPONSE_PROMPT]

[CV_SHOT_PROMPT]

  • The coherence-validation shot prompt asks the model to answer a question and put its final answer on the last line in an “Answer:” format.Source: [CV_SHOT_PROMPT] · p. 16

B.3 REPROMPT Details

  • RePrompt evaluates subject consistency, background consistency, aesthetic quality, and image quality; its win-rate calculation uses human-preferred pairs and total evaluated pairs.Source: §B.3 · p. 18

C.1 Examples of Reasoning Tasks

The reasoning-evaluation examples span six task types, from spatial and temporal reasoning to synopsis. The examples include questions about object movement and change, alongside sample video interpretations.

C.1 Examples of Reasoning Tasks

  • The benchmark covers spatial and temporal reasoning, action and object reasoning, counting, and information synopsis.Spatial reasoning concerns object positions; temporal reasoning detects anomalies; action and object reasoning assess actions and object properties; counting targets sudden appearances or disappearances; synopsis summarizes themes or infers background.Source: §C.1 · p. 19
  • Illustrative questions ask where an astronaut walks after jumping off the moon and how coffee changes within a cup.Source: §C.1 · p. 19
  • Sample responses describe coffee expanding and receiving more coffee while its amount remains the same, and a flying ball sinking into a mountain.Source: §C.1 · p. 19
  • Figure 13 provides detailed examples of reasoning tasks.Source: Figure 13 · p. 19

C.2 Error Analysis

The error analysis groups Qwen2-VL-72B-Instruct’s AIGC-video feedback errors into four types, spanning prompt misconceptions, missed visual details, and reliance on prior or textual information.

C.2 Error Analysis

  • The model may add information and constraints to the original prompt instead of understanding how to create a good video prompt.This is categorized as misconception of video creation.Source: §C.2 · p. 20, Figure 14 · p. 20
  • The model may rely on prior knowledge rather than observing details in the video.This is categorized as excessive dependence on textual cues.Source: §C.2 · p. 20, Figure 15 · p. 20
  • The model may answer questions while overlooking critical details in AIGC videos.Source: §C.2 · p. 20, Figure 16 · p. 21
  • The model may over-rely on textual information.This is categorized as over-reliance on commonsense knowledge.Source: §C.2 · p. 20, Figure 17 · p. 21

Related papers

Loading 2505.23693v1…