PaperlayerView exact revision on arXiv

Source-linked AI summary

VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects

Xiangbo Gao, Sicong Jiang, Bangya Liu, Xinghao Chen, Minglai Yang, Siyuan Yang, Mingyang Wu, Jiongze Yu, Qi Zheng, Haozhi Wang, Jiayi Zhang, Jie Yang, Zihan Wang, Qing Yin, Zhengzhong Tu

arXiv:2604.16272v2cs.CVcs.AIcs.CL

TL;DR

Video-editing research lacks large human-annotated resources with complete editing examples and evaluators specialized to instruction following, rendering quality, and edit locality. The paper introduces VEFX-Dataset, VEFX-Reward, and VEFX-Bench to address these gaps, and reports stronger human-judgment alignment for its evaluator alongside persistent separations among visual plausibility, instruction following, and content preservation.

  • Problem

    Video-editing evaluation lacks complete human-annotated editing triplets and dedicated automatic assessment of instruction following, rendering quality, and preservation of unedited content.

  • Method

    The paper builds a 5,049-example dataset, trains VEFX-Reward on source video, instruction, and edited video inputs, and releases a 300-pair benchmark.

  • Results

    VEFX-Reward consistently outperforms generic VLM judges and prior reward-model baselines on IQA/VQA metrics and group-wise preference evaluation.

  • Takeaways & Limitations

    Current systems often achieve plausible rendering without reliably satisfying instructions or preserving non-target content, supporting multidimensional video-editing evaluation.

  • Takeaways & Limitations

    The benchmark aggregates dimensions with a weighted geometric mean that assigns IF twice the weight of RQ and EE.

Abstract

from arXiv · show

As AI-assisted video creation becomes increasingly practical, instruction-guided video editing has become essential for refining generated or captured footage to meet professional requirements. Yet the field still lacks both a large-scale human-annotated dataset with complete editing examples and a standardized evaluator for comparing editing systems. Existing resources are limited by small scale, missing edited outputs, or the absence of human quality labels, while current evaluation often relies on expensive manual inspection or generic vision-language model judges that are not specialized for editing quality. We introduce VEFX-Dataset, a human-annotated dataset containing 5,049 video editing examples across 9 major editing categories and 32 subcategories, each labeled along three decoupled dimensions: Instruction Following, Rendering Quality, and Edit Exclusivity. Building on VEFX-Dataset, we propose VEFX-Reward, a reward model designed specifically for video editing quality assessment. VEFX-Reward jointly processes the source video, the editing instruction, and the edited video, and predicts per-dimension quality scores via ordinal regression. We further release VEFX-Bench, a benchmark of 300 curated video-prompt pairs for standardized comparison of editing systems. Experiments show that VEFX-Reward aligns more strongly with human judgments than generic VLM judges and prior reward models on both standard IQA/VQA metrics and group-wise preference evaluation. Using VEFX-Reward as an evaluator, we benchmark representative commercial and open-source video editing systems, revealing a persistent gap between visual plausibility, instruction following, and edit locality in current models. Our project page is https://xiangbogaobarry.github.io/VEFX-Bench/.

1 Introduction

VEFX-Bench addresses unresolved video-editing evaluation gaps with human-annotated data, a dedicated multidimensional reward model, and a standardized benchmark. Its evaluation reveals that current systems can produce plausible visuals without reliably following instructions or preserving non-target content.

  • Motivation: Video-editing evaluation must separately assess instruction execution, visual coherence, and preservation of unchanged content.Existing evaluation lacks complete source–instruction–edited triplets with fine-grained human labels and often relies on manual inspection or generic VLM judges.
  • Contributions: VEFX-Dataset contains 5,049 human-annotated examples spanning 9 categories and 32 subcategories, scored on IF, RQ, and EE.The examples are generated by commercial systems, open-source models, and agentic editing pipelines, using a 4-point rubric.
  • Contributions: VEFX-Bench provides 300 curated video-prompt pairs for standardized comparison of editing systems.VEFX-Reward is used to evaluate representative commercial and open-source systems under the same multidimensional protocol.
  • Contributions: VEFX-Reward jointly processes the source video, editing instruction, and edited result to predict per-dimension quality scores with ordinal regression.The three dimensions are Instruction Following, Rendering Quality, and Edit Exclusivity.
  • Results: VEFX-Reward aligns more strongly with human judgments than generic VLM judges and prior reward-model baselines across IQA/VQA metrics and group-wise preference evaluation.The benchmark analysis also exposes task-dependent strengths and persistent weaknesses in instruction following and edit locality.

2 Related Work

Prior work advances instruction-guided video editing and develops broad or modality-specific evaluation tools, but existing metrics and benchmarks do not fully capture editing-specific quality. Reward modeling has similarly progressed from image generation and editing toward video, motivating a dedicated video-editing evaluator.

  • Instruction-guided video editing: Instruction-guided video editing extends image-editing ideas into the temporal domain and increasingly uses video-native diffusion or flow-matching architectures.Representative research models include VACE and UniVideo.
  • Evaluation: CLIP score, SSIM, and LPIPS capture narrow aspects of video quality rather than instruction fidelity, temporal consistency, or unintended edits.General video-generation suites such as VBench and VBench++ are not designed to jointly compare source and edited videos.
  • Reward modeling: Visual reward modeling developed from image generation to image editing, with methods such as EditReward learning multidimensional human-preference signals.The related-work passage indicates that this research direction is extending into video.

3 VEFX-Dataset and VEFX-Bench

VEFX-Dataset and VEFX-Bench provide human-annotated, multi-dimensional resources for evaluating video editing across diverse tasks, systems, and content. Their analyses show reliable labels, distinct quality dimensions, and substantial variation in editing difficulty.

  • Dataset and benchmark: 5,049 editing examples cover 9 major categories and 32 subcategories, with 300 curated video-prompt pairs released for standardized system comparison.The dataset uses diverse commercial, open-source, and agentic editing systems, while the benchmark provides a standardized evaluation protocol.
  • Dataset and benchmark: VEFX-Dataset is the only compared resource combining edited outputs, human annotation, and multi-dimensional quality labels.These properties support reward-model training by distinguishing different editing failure modes.
  • Annotation protocol: The annotation protocol scores Instruction Following, Rendering Quality, and Edit Exclusivity independently on a 4-point scale.The dimensions respectively assess semantic execution, visual and temporal quality, and unintended changes outside the target region.
  • Annotation reliability: Within-1 agreement reaches 93.5% for IF, 97.2% for RQ, and 91.7% for EE on the 550-example cross-check subset.Exact agreement is 75.2%, 87.2%, and 72.2%, respectively, indicating stronger alignment for rendering quality than for borderline semantic and locality judgments.
  • Dataset statistics and analysis: 41.2% of samples score 1 on IF, whereas 78.6% score 3 or 4 on RQ, showing that visual plausibility is often easier than semantic correctness.Frequent patterns such as (1, 4, 4) demonstrate plausible outputs that fail the instruction.
  • Dataset statistics and analysis: Camera Angle Editing is hardest on IF at 1.76, while Style Editing reaches 2.87 and Visual Effect Editing reaches the highest overall mean at 2.93.RQ stays between 3.00 and 3.39 across task types, while EE ranges from 2.22 in Creative Editing to 3.06 in Instance Motion Editing.
  • Dataset statistics and analysis: IF–RQ, IF–EE, and RQ–EE correlations are 0.241, 0.195, and 0.327, supporting three-axis labels rather than a single scalar score.The dataset also spans diverse scene types and resolutions, including 1920×1080, 3840×2160, and portrait 1080×1920 videos.

4 VEFX-Reward: Human-Aligned Video Editing Reward Model

VEFX-Reward is a dedicated video-editing reward model that jointly compares source video, instruction, and edited result. It predicts separate ordinal scores for instruction following, rendering quality, and edit exclusivity.

  • Model design: VEFX-Reward jointly processes the original video, editing instruction, and edited video to assess video-editing quality.This distinguishes it from reward models targeting image editing or video generation alone.
  • Problem formulation: The model predicts Instruction Following, Rendering Quality, and Edit Exclusivity scores on the ordinal scale {1, 2, 3, 4}.These dimensions separately measure semantic execution, visual and temporal consistency, and unintended changes beyond the target region.
  • Model design: VEFX-Reward-4B and VEFX-Reward-32B use three dimension-specific reward tokens with a shared reward head for joint multimodal reasoning.The tokens query IF, RQ, and EE while preserving one shared backbone.
  • Ordinal regression: Ordinal regression models each dimension through ordered threshold decisions rather than unconstrained scalar regression.The reward head predicts whether each score exceeds 1, 2, and 3, using binary cross-entropy with an ordinal constraint.
  • Ordinal regression: At inference, ordered probabilities are converted into a continuous score on [1] [4] using their expected value.This soft prediction is used in all reported experiments.
  • Training and data processing: Training uses 4,200 examples and evaluation uses 849 test examples, with paired videos sampled uniformly at 4 FPS.Frame resolution is capped at 399,360 pixels while native aspect ratios are preserved.

5 Experiments

The experiments evaluate VEFX-Reward against generic judges and prior reward models using global human-score agreement and within-group preference consistency. VEFX-Reward-32B performs best overall, while the results show complementary scale effects across IF, RQ, and EE.

  • Evaluation Setup: VEFX-Reward is evaluated against VLM-as-a-Judge systems, EditReward, and VE-Bench using standard IQA/VQA metrics and group-wise preferences.The evaluation reports SRCC, KRCC, PLCC, and RMSE, then tests ordering among candidate edits sharing the same source video and instruction.
  • Overall Results: 0.780 SRCC, 0.616 KRCC, 0.790 PLCC, and 0.475 RMSE are achieved by VEFX-Reward-32B on the human overall score.VEFX-Reward-4B follows with 0.760 SRCC, 0.595 KRCC, 0.771 PLCC, and 0.493 RMSE; EditReward and VE-Bench perform worse overall.
  • Dimension-wise Behavior: VEFX-Reward-32B is strongest on IF and EE, whereas VEFX-Reward-4B is slightly stronger on RQ across the four standard metrics.The results suggest that larger scale mainly benefits instruction faithfulness and edit exclusivity, while rendering-quality prediction is already close to saturation at 4B scale.
  • Baseline Comparison: VEFX-Reward maintains more consistent performance across dimensions and metrics than strong VLM judges and previous reward models.EditReward has negative RQ correlations and no dedicated EE head, while VE-Bench predicts only a single scalar score.
  • Scatter-plot Analysis: VEFX-Reward-32B shows the tightest monotonic relationship with human overall scores, while EditReward is nonlinear and VE-Bench has greater dispersion.The scatter-plot pattern is consistent with the quantitative overall results.
  • Group-wise Preference Evaluation: 0.872 Pairwise Accuracy is achieved by VEFX-Reward-32B, with VEFX-Reward-4B close at 0.863 for within-group human preference ordering.Pairwise Accuracy ignores score-scale mismatch because it evaluates only relative ordering within candidate groups.
  • Ablation and Design Choices: Ordinal regression, 4 FPS sampling, and approximately 400K pixels per frame are selected as the strongest development configuration.The design choices align with ordered 1–4 labels, temporal coverage, and preservation of subtle editing cues without clear higher-resolution gains.

6 Benchmarking Existing Video Editing Models

VEFX-Bench uses VEFX-Reward-32B to compare ten commercial and open-source video editing systems across instruction following, rendering quality, and edit exclusivity. The benchmark reveals strong task and dimension imbalances, with GeoAgg emphasizing weaknesses rather than allowing strong dimensions to compensate fully.

  • Benchmark Setup: Ten representative commercial and open-source video editing models are scored with VEFX-Reward-32B on the same 1–4 scale.The benchmark reports soft expected predictions for IF, RQ, and EE, using Overall (GeoAgg) as its primary ranking metric.
  • Aggregation: Overall (GeoAgg) is a weighted geometric aggregate designed to reduce compensation across dimensions and penalize weak instruction following more strongly.The benchmark sets (α, β, γ) = (2, 1, 1), giving IF twice the weight of RQ and EE.
  • Coverage Adjustment: Incomplete benchmark coverage is treated as missing data and adjusted using inverse-propensity weighting under observed-item-covariate assumptions.Coverage-adjusted estimates are used for IF, RQ, EE, and Overall (Mean), while Overall (GeoAgg) is averaged from per-sample predictions.
  • Overall Rankings: Kling o3 omni ranks first under Overall (GeoAgg), followed by Kling o1, Runway Gen-4.5, and Seedance 2.0.The top systems combine strong IF and RQ with competitive EE; Seedance 2.0 remains weaker on IF despite strong RQ and EE.
  • Key Findings: Modern systems often produce visually plausible videos, but reliable instruction following and locality preservation distinguish the strongest editing models.Among open-source systems, UniVideo is stronger than VACE and remains competitive with several commercial systems, especially on EE.
  • Score Distributions: The top commercial systems have high score medians but substantial prompt-level variance, while RQ is generally higher and more concentrated than IF.The gap between Overall (Mean) and Overall (GeoAgg) is most visible for models with unbalanced dimensions.
  • Task-wise Profiles: Kling o3 omni and Kling o1 show broad task coverage, while Grok Imagine is strong on style, instance, and visual-effect editing but weaker on camera control.Runway Gen-4.5 and Seedance 2.0 are more balanced but slightly lower overall.

7 Conclusion

VEFX-Dataset, VEFX-Reward, and VEFX-Bench provide human-annotated data, automated evaluation, and standardized comparison for video editing. Results show that task-specific, multi-dimensional evaluation better exposes gaps between rendering quality, instruction following, and content preservation.

  • VEFX-Dataset contains 5,049 human-annotated examples labeled for Instruction Following, Rendering Quality, and Edit Exclusivity.
  • VEFX-Reward consistently outperforms generic VLM judges and prior reward-model baselines on IQA/VQA metrics and group-wise preference evaluation.
  • Current systems often produce plausible rendering without reliably satisfying instructions or preserving non-target content.
  • The released resources support benchmarking, model selection, and reward-driven optimization for video editing.

A Additional Training Details for VEFX-Reward

VEFX-Reward is trained on a stratified 4,200-example split and tested on 849 examples. Training uses aligned video sampling and a two-stage optimization schedule.

  • VEFX-Reward uses 4,200 training examples and 849 test examples, with splits stratified across editing categories and pipelines.
  • Original and edited videos are uniformly sampled at 4 FPS with aligned temporal indices, approximately 632 × 632 maximum resolution, and a 32,768-token sequence limit.
  • Training freezes pretrained parameters for 1 epoch, then fine-tunes language-side components and the reward head for 49 epochs while keeping the vision tower frozen.

B Additional Experimental Details

The appendix specifies reward-model variants, evaluation procedures, editing-system construction, and annotation criteria. Human agreement is strongest for rendering quality and weakest for edit exclusivity, although all dimensions show broad consistency.

  • Evaluation Setup: VEFX-Reward-4B and VEFX-Reward-32B apply the same reward-model design to Qwen3-VL backbones at 4B and 32B scales.
  • Evaluation Setup: Both variants and VLM judges use the same 849-example test split and score Instruction Following, Rendering Quality, and Edit Exclusivity on the human 1–4 rubric.
  • Evaluation Setup: External baselines use native outputs: EditReward averages two heads, whereas VE-Bench produces one scalar and lacks a dedicated Edit Exclusivity head.
  • Editing Systems: The dataset combines commercial end-to-end editors with category-specific pipelines using segmentation, pose or camera controls, first-frame editing, and temporal propagation.
  • Annotation Dimensions: Instruction Following measures semantic execution, Rendering Quality measures visual and temporal fidelity, and Edit Exclusivity measures preservation of unrelated regions.
  • Annotation Reliability: 87.2% exact agreement for Rendering Quality exceeds 75.2% for Instruction Following and 72.2% for Edit Exclusivity, while within-1 agreement reaches 97.2%, 93.5%, and 91.7%, respectively.

F.1 Task Type Difficulty Ranking

Task difficulty varies substantially by editing type and subcategory. Camera Angle editing is hardest for semantic execution, while high cross-pipeline variance identifies videos with especially informative reward-learning signals.

  • Task-Level Difficulty: Camera Angle editing is hardest for Instruction Following because it requires geometric and 3D scene reasoning that current systems handle poorly.
  • Task-Level Difficulty: Style Editing is easiest for Instruction Following but has relatively low Edit Exclusivity, reflecting tension between global transformation and locality preservation.
  • Dimension Comparison: Rendering Quality varies less across task types than Instruction Following or Edit Exclusivity, indicating that visual plausibility is easier than precise semantic execution.
  • Video-Level Difficulty: High-variance videos provide stronger reward-model preference signals because different pipelines succeed or fail on the same input.

G Per-Category Detailed Results

The six-system evaluation reveals that instruction following and edit exclusivity separate models more strongly than rendering quality. Category-level heatmaps show strong performance on several attribute, style, and instance edits, while camera edits remain difficult.

  • The supplementary heatmaps report mean VEFX-Reward-32B scores by dimension across the nine main editing categories.Figures 16–18 cover Instruction Following, Rendering Quality, and Edit Exclusivity, respectively.
  • Instruction Following varies most across categories and is the main source of model separation.
  • Grok Imagine and Kling Omni are strongest on many attribute, style, and instance-editing tasks.
  • Camera-angle and camera-motion edits remain difficult for nearly all systems.
  • Rendering Quality is comparatively stable across categories, suggesting visually plausible outputs are easier to produce than semantically correct ones.
  • Edit Exclusivity exposes the sharpest locality gap among the evaluated systems.

H.1 Definitions of Standard IQA/VQA Metrics

This section defines standard correlation and error metrics used to compare model predictions with human labels, including calibration before PLCC and RMSE. Supplementary plots provide distributional views of the six-model evaluation, while the ethics discussion notes a potential reward-gaming risk.

  • Definitions of Standard IQA/VQA Metrics: SRCC measures monotonic agreement between predicted scores and human labels using rank differences.
  • Definitions of Standard IQA/VQA Metrics: Kendall’s τ-b accounts for ties through concordant, discordant, and tied prediction or human ranking pairs.
  • Definitions of Standard IQA/VQA Metrics: PLCC measures linear agreement after calibration, whereas RMSE measures calibrated absolute deviation.PLCC uses calibrated model predictions and human scores; RMSE is described as an error measure after calibration.
  • Definitions of Standard IQA/VQA Metrics: A four-parameter logistic mapping calibrates raw model scores before PLCC and RMSE computation.The parameters β1, . . . , β4 are fitted by non-linear least squares on the evaluation set.
  • Figures 19–21 compare reward-model scales and visualize score distributions across models and dimensions.Figure 19 compares 4B and 32B predictions across IF, RQ, EE, and Overall; Figures 20–21 show score spread and density.
  • Scaling from 4B to 32B mainly improves IF, EE, and Overall, while RQ remains relatively similar across scales.The 32B predictions form visibly tighter trends around human annotations for IF, EE, and Overall.
Loading 2604.16272v2…