PaperlayerView exact revision on arXiv

Source-linked AI summary

FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling

Peiyuan Zhang, Xiangyu Zhao, Hongbo Liu, Xiaoxing Hu, Mingxin Liu, Shuran Ma, Yunhang Shen, Jian Hu, Haihan Gao, Haoyu Cao, Xue Yang

arXiv:2608.21839v1cs.CV

TL;DR

Reliable text-to-video reward modeling requires efficient evaluations backed by dependable supervision, but existing holistic judges lack explicitly verifiable, sample-specific evaluation contracts. FIRM-Video applies dimension-specific checklists, evidence verification, and verified-score aggregation to construct supervision and train reward models. The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench and strongest Best-of-8 VBench performance across three generators.

  • Problem

    Existing text-to-video judges lack sample-specific, explicitly verifiable evaluation contracts, while efficient end-to-end reward models depend heavily on supervision quality.

  • Method

    FIRM-Video constructs dimension-specific checklists, verifies criteria against temporal visual evidence, aggregates verified decisions, and transforms them into natural-language supervision.

  • Results

    The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench and the highest VBench Total, Quality, and Semantic Scores in Best-of-8 sampling across three generators.

  • Takeaways & Limitations

    Checklist-driven supervision supports accurate absolute assessment and effective candidate ranking for text-to-video evaluation and selection.

  • Takeaways & Limitations

    The models have not yet been used to directly optimize text-to-video generators through reinforcement learning or preference optimization.

Abstract

from arXiv · show

Reliable reward models are essential for text-to-video evaluation and alignment. However, the trade-off between evaluation accuracy and inference efficiency places high demands on the quality of training supervision. Existing approaches often rely on holistic judges with fixed rubrics or open-ended reasoning, leading to incomplete inspection, unfaithful justification, and entangled attribution. We introduce FIRM-Video, a unified checklist-driven data construction framework based on a check-before-score principle: construct dimension-specific checklists, verify each criterion against temporal visual evidence, and aggregate only verified decisions. For Instruction Following, FIRM-Video decomposes prompts into weighted atomic requirements; for World Coherence, it constructs prompt-calibrated, target-specific checks grounded in visible entities and actions; and for Perceptual Quality, it applies a generic taxonomy of visual defects. The verified criteria and scores are further transformed into natural-language analyses for end-to-end reward modeling. Subsequently, we construct FIRM-Video-90K with 88,044 dimension-specific instances from 29,348 videos, and introduce FIRM-Video-Bench with 750 point-wise human annotations across 250 videos. The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench while consistently delivering the highest VBench Total, Quality, and Semantic Scores in Best-of-8 sampling across three video generators.

1 Introduction

FIRM-Video addresses unreliable and inefficient text-to-video evaluation by constructing verifiable, dimension-specific supervision under a check-before-score framework. It yields large-scale training data, an expert benchmark, and reward models with strong human alignment and selection performance.

  • Motivation: Reliable evaluation must assess instruction following, world coherence, and perceptual quality, yet efficient end-to-end reward models depend heavily on reliable training supervision.Existing holistic judges lack sample-specific, explicitly verifiable evaluation contracts and can miss prompt-specific or temporal evidence.
  • Method: FIRM-Video constructs dimension-specific checklists, verifies each criterion against temporal visual evidence, and aggregates only verified decisions.Instruction Following uses weighted atomic requirements; World Coherence uses prompt-calibrated target checks; Perceptual Quality uses a visual-defect taxonomy.
  • Datasets: 88,044 dimension-specific reasoning-and-score instances were constructed from 29,348 videos and 3,012 unique prompts in FIRM-Video-90K.The dataset is generated by applying the checklist-driven pipeline to diverse generated videos.
  • Results: The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench and the highest VBench Total, Quality, and Semantic Scores with N = 8 across three generators.It reduces overall MAE from 1.33 to 0.78 and improves Total Score by 0.27–1.01 points versus the strongest competing selector.
  • Datasets: FIRM-Video-Bench contains 750 fine-grained human annotations across 250 videos covering Instruction Following, World Coherence, and Perceptual Quality.The benchmark provides a unified testbed for agreement between reward models and human judgments.

2 Related Work

Text-to-video generation has advanced through diffusion, DiT, and flow-matching architectures, while reward modeling evolved from fixed metrics and human-labeled functions to vision-language-model judges. Recent generative judges provide rationales but can require costly supervision and produce difficult-to-verify explanations.

  • Text-to-Video Generation Models: Text-to-video generation progressed from U-Net diffusion models to larger DiT and flow-matching architectures, improving visual quality, temporal consistency, and instruction following.Open and proprietary systems expanded access and continued gains in resolution, motion realism, long-range coherence, and controllability.
  • Reward Modeling: Early visual-generation reward studies used fixed metrics such as FID and CLIP Score, which often showed limited agreement with human judgments.Later methods learned reward functions from point-wise human annotations or pair-wise preferences using regression or ranking objectives.
  • VLM-Based Reward Modeling: Vision-language-model reward models progressed from scalar or multi-dimensional scores toward generative assessments and rationales.VideoScore2 generates detailed evaluations but relies on costly human supervision and subjective, model-expanded explanations that are difficult to verify.

3 Methodology

FIRM-Video constructs dimension-specific checklists, verifies criteria against video evidence, and aggregates verified decisions into scores and rationales for efficient end-to-end reward modeling.

  • Data Collection: The dataset combines human preference sources into approximately 30K samples covering videos from over 20 generation models.The retained data include about 3K pairs from Text2Video-Human Preferences and 27K from VideoFeedback2.
  • Instruction Following: Instruction Following decomposes prompts into weighted atomic requirements, verifies each against temporally ordered frames, and maps their importance-weighted satisfaction to a five-point rating.Unverifiable requirements are treated as unsatisfied, and each verified requirement includes brief visual evidence.
  • World Coherence: World Coherence grounds entities and actions without the prompt, constructs target-specific checks, and uses weighted clean ratios to provide partial credit for partially affected targets.The prompt is introduced during checklist construction to distinguish intentional fantasy or stylization from genuine violations.
  • Perceptual Quality: Perceptual Quality uses a manually curated generic checklist covering defects such as blur, instability, flickering, compression artifacts, and other visual distortions.The evaluator directly produces a checklist-based natural-language rationale for this dimension.
  • Reward Model Training: The offline pipelines transform verified dimension-specific outputs into rationales and scores, which FIRM-Video-8B predicts in one forward pass without explicit checklist construction.This distills checklist-based supervision into an efficient end-to-end reward model.

4 Experiments

Experiments evaluate human agreement, out-of-domain preference ranking, and best-of-N selection. FIRM-Video-8B performs best on the main benchmark and consistently improves VBench selection across generators.

  • FIRM-Video-Bench: FIRM-Video-8B reduces overall FIRM-Video-Bench MAE from 1.33 to 0.78 relative to its Qwen3-VL base model.It also achieves strict accuracy of 0.42 and relaxed accuracy of 0.84, with the best overall MAE among evaluated models.
  • FIRM-Video-Bench: FIRM-Video-8B achieves the lowest World Coherence MAE while remaining highly competitive on Instruction Following and Perceptual Quality.Both FIRM-Video variants substantially improve over their corresponding base models.
  • MJ-Bench-Video: On MJ-Bench-Video, the InternVL3-based model ranks first on Alignment and Consistency & Coherence and second on Overall, while the Qwen3-VL-based model ranks second on Consistency & Coherence and Fitness.The benchmark measures pair-wise preference accuracy across Alignment, Consistency & Coherence, Fitness, and Overall.
  • Best-of-N Sampling: FIRM-Video-8B achieves the best VBench Total, Quality, and Semantic Scores across LaVie-Base, CogVideoX-2B, and Wan2.1-T2V-1.3B.Against the strongest competitor, Total Score improves by 0.27, 1.01, and 0.66 across the three generators, while Semantic Score improves by 1.24–2.11.
  • Scaling with the Number of Samples: With LaVie-Base and N ≥4, FIRM-Video-8B performs best as the candidate pool grows, increasing from 80.36 to 80.97.At N = 16, it exceeds random sampling by 1.03 points and Qwen3-VL-8B by 0.33 points.

5 Limitations

The work validates FIRM-Video through human-aligned evaluation and best-of-N sampling, but does not directly use the reward models to optimize text-to-video generators.

  • Scope Boundary: The study does not directly optimize text-to-video generators with FIRM-Video-8B, leaving reinforcement learning or preference optimization for future work.Its validation focuses on human-aligned evaluation and best-of-N sampling.

6 Conclusion

FIRM-Video applies a check-before-score framework to construct dimension-specific checklists, verify criteria against temporal evidence, and aggregate verified decisions for T2V reward modeling. The resulting supervision improves agreement with human judgments and supports strong candidate ranking across three T2V generators.

  • FIRM-Video constructs checklists for Instruction Following, World Coherence, and Perceptual Quality, verifies each criterion against temporal visual evidence, and aggregates only verified decisions.
  • FIRM-Video-90K and FIRM-Video-Bench provide the framework’s synthesized supervision and expert-annotated evaluation resources.
  • Reward models trained on FIRM-Video-90K substantially improve agreement with human judgments over their base models.
  • The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench and the strongest best-of-N sampling performance across three T2V generators on VBench.

A Implementation Details of the Data Pipeline

The implementation pipeline converts prompt or video-grounded targets into weighted verification checklists, computes normalized scores, and maps them to five-point ratings. IF uses verified requirement satisfaction, while WC uses target-level clean ratios and violation checks.

  • Instruction Following: Instruction Following decomposes prompts into atomic visual requirements with importance scores, excludes audio-only requirements, and shares each prompt’s checklist across generated videos.
  • Instruction Following: IF normalizes requirement weights, treats unverifiable requirements as unsatisfied, and computes a weighted score before rounding and five-point rating conversion.
  • World Coherence: World Coherence grounds entity and action targets, assigns target importance, and converts each target into verification questions inheriting that importance.
  • World Coherence: WC uses violation indicators to compute target clean ratios and an importance-weighted score, providing partial credit when only some target aspects are affected.
  • Score Mapping: Continuous checklist scores are rounded and mapped to five-point ratings using empirically determined thresholds, with an IF rating of 5 requiring all positively weighted requirements to be satisfied.

A.4 Ablation of Score Aggregation

The ablation compares importance-weighted aggregation with uniform averaging while holding checklists, verification results, and score mapping fixed. Importance weighting improves selected IF metrics and improves all reported WC metrics, including lowering WC MAE from 0.86 to 0.80.

  • The ablation keeps checklists, verification results, and score mapping fixed while comparing importance-weighted aggregation with uniform averaging.
  • Instruction Following: The uniform IF baseline assigns equal weight to all requirements, whereas FIRM-Video uses importance-weighted aggregation.
  • World Coherence: The uniform WC baseline averages target clean ratios, whereas FIRM-Video uses importance-weighted aggregation.
  • World Coherence: WC MAE decreases from 0.86 to 0.80 with importance weighting, while accuracy, relaxed accuracy, and SRCC improve.
  • Instruction Following: Importance weighting improves IF stability and relaxed accuracy while preserving IF MAE and SRCC.

B Statistical Analysis of FIRM-Video-90K and FIRM-Video-Bench

The paper identifies FIRM-Video-90K and FIRM-Video-Bench as the two datasets whose score distributions are summarized in the statistical analysis.

  • Table 5 summarizes the statistics of FIRM-Video-90K and FIRM-Video-Bench.
  • The score distributions of FIRM-Video-90K and FIRM-Video-Bench are presented in Table 5.

C Best-of-N Sampling and Results on VBench

VBench candidates are selected separately for each FIRM-Video reward dimension using the candidate with the highest corresponding reward. The mapping distinguishes VBench’s Video-Quality and Video–Condition Consistency groups.

  • For each VBench dimension, the candidate with the highest corresponding reward is selected.
  • Table 6 maps VBench dimensions to FIRM-Video reward dimensions, including Video-Quality and Video–Condition Consistency groups.

C.2 Detailed Results

The detailed-results materials define FIRM-Video’s three evaluation dimensions, specify five-point human annotation and JSON-scored model outputs, and report VBench results across quality and condition-consistency subdimensions. Qualitative examples illustrate FIRM-Video-8B evaluations across samples with varying quality.

  • Detailed Results: Tables 7 and 8 report Best-of-8 results for VBench’s video-quality and video–condition consistency subdimensions.
  • Evaluation Dimensions: Instruction following measures how accurately and completely videos satisfy requested subjects, attributes, actions, scenes, styles, viewpoints, and temporal order.
  • Evaluation Dimensions: Visual quality covers resolution, clarity, sharpness, brightness, color stability, flicker, compression artifacts, and other low-level distortions.
  • Evaluation Dimensions: World consistency evaluates violations of everyday knowledge, common sense, or basic physical laws, including temporal, motion, interaction, and structural problems.
  • Human Evaluation: Human annotators independently rate instruction following, perceptual quality, and world coherence on a five-point scale after reviewing the video, an eight-frame overview, and its prompt.
  • Qualitative Examples: FIRM-Video-8B evaluation examples visualize quantitative analyses of samples with varying quality.
Loading 2608.21839v1…