VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal

• Existing benchmarks lack paired edited videos and use precise segmentation masks unlike user-provided masks, while traditional metrics can diverge from human evaluation.Global averaging can dilute localized masked-region defects, allowing artifacted outputs to receive higher SSIM scores.Source: §1 · p. 2
• VORD provides paired edited videos and graffiti masks across diverse data types, scenarios, object categories, association effects, video lengths, and frame rates.The first-frame masks are manually painted and later masks are generated by a segmentation model.Source: §1 · pp. 3–4
• rMPAF combines image removal and trained video generation to create realistic paired videos with physical motion and alignment in non-edited regions.These pairs address the scarcity of real-world removal videos and the limited scene diversity and motion patterns of static-camera or rendered videos.Source: §1 · pp. 3–4
• VOR-MDSM evaluates object removal completeness, mask-background consistency, and logical plausibility using a VLM supervised by human ratings.It frames evaluation as question answering and uses temporal-spatial video information to produce scores aligned with perception.Source: §1 · pp. 3–4
• Experiments show VOR-MDSM complements existing evaluation methods and correlates more strongly with human perceptual judgments.Source: §1 · p. 4

2.1 Video Object Removal Models

• VOR models have progressed from 3D convolutional networks and optical flow to transformers and diffusion, with some methods targeting associated effects or interactions.ProPainter [63] combines optical flow and transformers; VOID [37] focuses on interaction-aware removal.Source: §2.1 · p. 4

2.2 Video Object Removal Evaluation Datasets

• Existing evaluation datasets often lack paired edited videos or rely on a single generation paradigm, limiting data diversity and robust evaluation.VORD combines model-generated, tool-rendered, and camera-captured pairs across varied objects, scenes, and association effects.Source: §2.2 · pp. 4–5

2.3 Video Object Removal Assessment

• PSNR, SSIM, and LPIPS [62] measure pixel or structural similarity and depend on reference images, while perceptual VLM approaches have not been optimized specifically for human-aligned VOR assessment.VOID [37] uses prompts with closed-source VLMs; VOR-Bench introduces a locally deployable model supervised by human annotations.Source: §2.3 · p. 5

3.1 Dataset Construction

• VORD comprises 150 video pairs across three equally represented data types, incorporates graffiti masks, and covers 14 visual types; its model-generated data had the lowest values on three metrics, below CineTechBench’s corresponding values.The 150-pair scale was selected to balance scenario diversity, statistical significance, and inference efficiency.Source: §3.1 · pp. 5–6

3.2 Realistic Motion-capable Paired-video Acquisition Framework

• rMPAF generates paired videos by removing objects from source-video endpoints and interpolating intermediate frames with a fine-tuned Wan-2.1-FLF2V model.Its five stages are source collection, aesthetic filtering, endpoint removal, frame interpolation, and final selection.Source: §3.2 · p. 7, Figure 3 · p. 7
• rMPAF produces unified-style paired videos that better fit real scenarios and offer more diverse shot trajectories than tool-rendered data, at lower cost than camera capture.The framework uses shared first- and last-frame anchors and the same fine-tuned interpolation model for original and removed videos; samples with temporal inconsistency, removal failures, visual artifacts, or invalid masks are manually filtered.Source: §3.2 · p. 7

3.3 Evaluation Metrics

• The evaluation dimensions are object removal completeness, mask-background consistency, and logical plausibility, with temporal failures incorporated into the latter two rather than scored separately.Mask-background consistency covers blending details such as color, texture, material, and patterns; logical plausibility considers semantic relation, physical laws, and common sense.Source: §3.3 · p. 8

3.4 VOR-MDSM

• VOR-MDSM scores original and edited video pairs across three dimensions, using human ratings to fine-tune a VLM without requiring removal ground truth.Fifteen vision experts rated 3,299 edited pairs across the dimensions; each pair received five independent ratings, averaged for training supervision.Source: §3.4 · p. 9

4.2 Evaluation Results

• Traditional-metric rankings diverge from VOR-MDSM and qualitative comparisons: ProPainter scores higher traditionally, while ROSE scores better on VOR-MDSM.The authors therefore report rankings based on VOR-MDSM, noting that pixel-level similarity alone does not capture all VOR-specific quality factors.Source: §4.2 · pp. 10–11
• ROSE achieves the best overall performance across the three VOR-MDSM dimensions, followed by EffectErase and VOID.The table also includes background-preservation measures; qualitative results describe ROSE as producing the cleanest output.Source: §4.2 · pp. 10–11, Table 3 · p. 11

4.3 Ablation Study

• Human annotations show high consistency, with ICC(2,5) values of 0.8931, 0.8913, and 0.8576 for ORC, MBC, and LP, respectively.Median standard deviations are below 1.0 for all three dimensions; MBC has the highest low-score proportion at 51.3%.Source: §4.3 · pp. 11–12
• VOR-MDSM achieves r, ρ, and τ > 0.9 on all three dimensions and outperforms VideoLLaMA3, Qwen3-VL-8B, and Gemini-3.1-pro.At the method level, Spearman correlations with human rankings are 0.382 for aggregated traditional metrics, 0.697 for ReMOVE, and 0.976 for VOR-MDSM.Source: §4.3 · p. 12
• The double-input training format is reported to perform best among the four tested input formats.The formats include temporal concatenation, spatial concatenation, triple input, and double input; the comparison uses correlations with human ratings.Source: §4.3 · p. 12, Figure 9 · p. 13, Table 4 · p. 13
• VORD’s 150-pair and VORD-L’s 300-pair evaluations produce the same four model tiers, supporting the authors’ conclusion that 150 pairs suffice to separate performance reliably.VORD-L increases the camera-captured data proportion to 50%.Source: §4.3 · p. 14
• With finely segmented masks, mask-background consistency improves and logical plausibility improves for all models except FGT, while removal completeness remains largely unchanged.VOID is excluded because its mandatory four-value mask prevents a fair comparison.Source: §4.3 · p. 14
• Cross-dataset validation reports high correlations with human ratings on DAVIS and ROSE-Bench, while VOR-MDSM averages 0.99 seconds per video under full-video inputs.Traditional-metric rankings vary substantially across benchmarks; VOR-MDSM takes less time than VideoLLaMA3 (1.22 seconds) and the three traditional metrics (21.50 seconds).Source: §4.3 · p. 14, Table 5 · p. 14

5 Conclusion

• VOR-Bench evaluates video-removal models through VORD, rMPAF, and VOR-MDSM, addressing limitations in existing evaluation sets and metrics.VORD includes paired edited videos, diverse data types, and graffiti masks; rMPAF generates paired edited videos, and VOR-MDSM assesses three dimensions.Source: §5 · p. 14
• VOR-MDSM assesses object removal completeness, mask-background consistency, and logical plausibility with a human perception-driven multimodal evaluator.The conclusion describes VORD’s data as better aligned with practical application scenarios.Source: §5 · p. 14
• VOR-Bench provides references for the performance improvement of video-removal models.Source: §5 · p. 14

6.1 Detailed Information of Vision Experts

• Fifteen visual experts with video-editing experience and relevant professional backgrounds underwent training before evaluation.They had previously participated in subjective evaluations of text-to-image and text-to-video models.Source: §6.1 · p. 20

6.2 Detailed Information of Scoring Criteria

• Object removal completeness scores range from 1 to 5 according to removal percentage, with score 5 reserved for 100% completion.The listed percentage bands progress from 0–50% for score 1, through 90–100% for score 4, to 100% for score 5.Source: §6.2 · pp. 20–21, Figure 10 · p. 20
• Mask-background consistency scores whether the removed area blends with its background, from large holes and artifacts at score 1 to perfect blending at score 5.Intermediate levels distinguish poor transitions, occasional seams or texture inconsistencies, and slight boundary or texture differences.Source: §6.2 · p. 21, Figure 11 · p. 21
• Logical plausibility scores range from hallucinated objects or obvious deviations at the low end to no physical-law or scene-logic issues at score 5.Intermediate scores capture slight logical issues and subtler issues visible on closer inspection.Source: §6.2 · p. 21, Figure 12 · p. 22

7.1 Metric-Induced Model Rankings

• VOR-MDSM’s induced model ranking has a substantially higher Spearman correlation with human preference than rankings from traditional metrics and ReMOVE.Source: §7.1 · p. 22, Table 6 · p. 23

7.2 Traditional Metrics Across Benchmarks

• Traditional-metric model rankings vary substantially between DAVIS and ROSE-Bench, while VORD’s VOR-MDSM cross-axis standard deviations are relatively small.The authors interpret the ranking variation as sensitivity to evaluation references and the small standard deviations as balanced cross-axis evaluation.Source: §7.2 · p. 22, Table 7 · p. 23

8 More Visualization Samples

• Corresponding video files are provided in the attachments, organized by those three data types.Source: §8 · p. 23
• Figure 13 shows model results across model-generated, camera-captured, and tool-rendered data.Source: Figure 13 · p. 24