Source-linked AI summary
GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Zhangqi Jiang, Zheng Sun, Xianfang Zeng, Yufeng Yang, Xuanyang Zhang, Yongliang Wu, Wei Cheng, Gang Yu, Xu Yang, Bihan Wen
TL;DR
Existing image-editing benchmarks have narrow task coverage and standard evaluation methods inadequately capture visual consistency. The paper introduces GEditBench v2, PVC-Judge, and VCReward-Bench to provide broader, pairwise, human-aligned assessment; its benchmark rankings correlate strongly with human Arena rankings, while the benchmark remains limited to single-image editing and constrained in size.
Problem
Existing benchmarks restrict task coverage, while evaluation approaches face reproducibility, cost, and human-alignment limitations.
Method
The paper combines a 23-task benchmark with open-set instructions, an open-source pairwise visual-consistency judge, expert preference evaluation, and BT/Elo ranking.
Results
ρ=0.929, p<2e-7 Spearman correlation between Overall Elo and human Arena rankings validates alignment of the automated evaluation ecosystem with human preferences.
Takeaways & Limitations
GEditBench v2 provides a human-aligned evaluation ecosystem for comparing instruction-based image-editing models across diverse and open-set scenarios.
Takeaways & Limitations
The benchmark excludes multi-image editing because current open-source VLMs show substantial performance gaps from proprietary models in multi-image understanding.
Abstract
from arXiv · showhide
Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.
1 INTRODUCTION
Instruction-based image editing has advanced toward complex natural-language modifications, but evaluation still struggles to measure visual consistency and broad real-world generalization. The paper addresses these gaps with GEditBench v2, PVC-Judge, and VCReward-Bench.
- Modern editing models execute complex visual modifications from natural-language instructions.
- Absolute VLM scoring is difficult to reproduce, creates accuracy-cost trade-offs, and aligns poorly with human pairwise preferences.
- Existing benchmarks mainly use closed predefined task categories, limiting evaluation of open real-world generalization.
- GEditBench v2 extends evaluation to open-set, out-of-distribution instructions, while PVC-Judge provides pairwise visual-consistency assessment.
- VCReward-Bench contains 3,506 expert-annotated preference pairs for measuring assessment models’ alignment with human visual-consistency judgments.
2 RELATED WORK
Prior work spans evolving image-editing architectures and increasingly automated benchmarks, but evaluation remains fragmented across costly human judgments and heterogeneous metrics.
- Image-editing research progressed from modular text-guided pipelines to unified free-form generative architectures.
- Early diffusion editing models demonstrated feasibility with synthetic supervision but struggled with complex reasoning.
- Benchmarking shifted from costly human evaluation toward automated L1, CLIP, and DINO metrics.
- Combining disparate automated metrics can produce fragmented and inconsistent assessments.
3 GEDITBENCH V2
GEditBench v2 broadens image-editing evaluation through diverse task coverage, open-set real-world instructions, and multidimensional pairwise assessment. Its design emphasizes human alignment, realistic generalization, and practical evaluation constraints.
- Benchmark construction: GEditBench v2 organizes editing into local, global, reference, and hybrid categories covering 19 established tasks.
- Evaluation metrics: Pairwise comparison consistently achieves higher agreement with human judgments than pointwise scoring across all three dimensions.
- Benchmark construction: The benchmark adds in-image text translation, chart editing, and enhancement with low-level restoration and photo-rescue tasks.
- Benchmark construction: Open-Set Editing contains 100 trending real-world instructions outside predefined taxonomies to test instruction generalization.
- Benchmark construction: The benchmark uses real-world user editing instances, substitutes original uploads for privacy, and focuses on single-image tasks.
- Evaluation metrics: Evaluation measures instruction following, visual quality, and visual consistency, including preservation of non-target regions.
- Evaluation metrics: Pairwise training is intended to improve out-of-distribution generalization without losing discriminative power, with one-time O(n^2) setup followed by O(n) comparisons for new models.
4 PAIRWISE VISUAL CONSISTENCY JUDGE
PVC-Judge is developed through candidate generation, region-decoupled preference construction, and pairwise evaluation strategies tailored to local and global edits. The pipeline uses diverse generated outputs and task-specific consistency signals to train the final evaluator.
- PVC-Judge development comprises candidate image generation, two preference data construction pipelines, training, and VCReward-Bench evaluation.
- Candidate image generation: The candidate pipeline curates prompts and generates edited outputs from seven image-editing models, producing approximately 180k candidate images.
- Candidate image generation: Candidate prompt-image pairs are sampled per task for semantic diversity, with an ablation varying the number of pairs from 500 to 3,000.
- Preference data construction: The object-centric pipeline localizes edited entities, partitions edit and non-edit regions, and applies region-specific metrics to preserve background fidelity while assessing subject identity.
- Preference data construction: The human-centric pipeline decomposes face identity, body appearance, and hair appearance, excluding the edited attribute while measuring remaining attributes with specialized models.
- Preference data construction: Preference pairs use task-wise normalized primary metrics, threshold-based winner-loser selection, auxiliary validation, and Pareto filtering; global tasks use VLM pairwise assessment.
- Training details: P=6 is selected as the balance between supervision density and computation cost, and PVC-Judge is fine-tuned from Qwen3-VL-8B-Instruct with LoRA.
5 EXPERIMENTS
The experiments evaluate PVC-Judge against open- and closed-source assessment models and benchmark 16 image-editing systems. PVC-Judge shows strong human-aligned visual-consistency evaluation, while the overall ranking exposes trade-offs between instruction following and consistency.
- Meta-evaluation: PVC-Judge is compared with Qwen-3-VL-8B-Instruct, GPT-5.1, Gemini 3 Pro, EditScore, and EditReward using pairwise evaluation prompts.
- Meta-evaluation: PVC-Judge consistently outperforms its base model across tasks and benchmarks and surpasses GPT-5.1 on average despite its 8B size.
- Main results: The OpenEdit leaderboard ranks 16 editing models using Overall, Instruction Following, Visual Quality, and Visual Consistency Elo scores derived from pairwise comparisons.
- Main results: ρ=0.929, p<2e-7 for the correlation between Overall Elo and human-annotated Arena rankings.
- Main results: Nano Banana Pro ranks first, Seedream 4.5 follows closely, and FLUX.2 [klein] 9B is the open-source champion.
- Main results: High visual-consistency scores can reflect under-editing, motivating joint evaluation of instruction following and visual consistency rather than consistency alone.
6 CONCLUSION
The paper presents a unified ecosystem combining a broad benchmark, a pairwise visual-consistency evaluator, and a meta-benchmark for evaluator assessment. It targets complex open-set editing and human-aligned evaluation.
- GEditBench v2 is a 23-task benchmark extending image-editing evaluation to complex open-set, real-world scenarios.
- PVC-Judge is an open-source pairwise visual-consistency evaluator trained with two region-decoupled preference data synthesis pipelines.
- VCReward-Bench evaluates image-editing assessment models for visual consistency.
A LIMITATIONS
The authors identify limitations involving benchmark scale and reliance on pretrained foundation models. These constraints affect sample diversity and may introduce inherited biases into preference data.
- Computational resources and long inference times constrain benchmark size, potentially reducing sample diversity within individual editing tasks.
- Preference-pair construction relies on pretrained models such as SAM, CLIP, and DINOv3, which may introduce biases into the resulting dataset.
B.1 DETAILED TASKS EXPLANATION
GEditBench v2 organizes image editing evaluation into 23 tasks across local, global, reference, hybrid, and open-set categories, while pairwise prompts and region-specific metrics support evaluation of instruction following, visual quality, and visual consistency.
- Task taxonomy: 23 editing tasks are organized into five categories: Local, Global, Reference, Hybrid, and Open-set.The taxonomy covers both basic manipulation and advanced instruction understanding.
- Local Editing: Local Editing evaluates spatially restricted modifications to specific regions or objects across 12 tasks.Examples include subject addition, removal, replacement, and size adjustment.
- Global Editing: Global Editing evaluates holistic changes to an image’s visual atmosphere or structural layout across six tasks.Tasks include background, style, tone, enhancement, camera-motion, and line-to-image transformations.
- Reference Editing: Reference Editing tests whether models transfer identities or styles from an external guiding image across character, object, and style reference tasks.Object Reference requires preserving an object’s exact physical details across novel scenes and states.
- Hybrid and Open-Set Editing: Hybrid prompts combine 3–5 predefined tasks, whereas Open-Set prompts use mixed or loosely specified intents to test generalization beyond fixed categories.The open-set subset contains 100 diverse prompts collected from public online sources.
- Evaluation protocol: The evaluation protocol uses pairwise prompts across instruction following, visual quality, and visual consistency, adopting Decide-Only because it is faster without sacrificing comparable Qwen3-VL-8B-Instruct accuracy.Across 890 testing pairs, average evaluation time was 133.93s for Decide-Only versus 411.84s for Decide-Before-Reason.
C.3 TRAINING HYPER-PARAMETERS
The supplied passages identify the training hyper-parameter table and show the preference-dataset distribution, alongside the custom annotation interface used for benchmark construction.
- Training configuration: Table 5 summarizes the hyper-parameters for the LoRA model trained during the experiments.
- Preference dataset: Figure 16 shows the preference dataset’s allocation across 16 editing tasks for training PVC-Judge.
- Annotation interface: Figure 17 shows screenshots of the custom-built annotation interface for VCReward-Bench.
D ANNOTATION PROTOCOL FOR VCREWARD-BENCH
VCReward-Bench is built through multi-model candidate generation, expert pairwise annotation across four dimensions, and Pareto-style filtering that isolates visual-consistency differences.
- Candidate generation: A diverse ensemble of open-source and proprietary models generates candidate images to capture varied editing behaviors, artifacts, and failure modes.
- Expert annotation: Experts compare candidate pairs on Instruction Following, Visual Quality, Visual Consistency, and Overall preference.The interface separates decisions by dimension and offers Prefer A, Both Good, Both Bad, and Prefer B.
- Pair filtering: Pareto-style filtering retains pairs where one image is strictly preferred in visual consistency and non-inferior on all remaining dimensions.This removes trivial pairs and focuses the benchmark on informative visual-consistency differences.
- Dataset outcome: 3,508 testing preference pairs spanning 21 tasks resulted from the generation, annotation, and filtering pipeline.
E.1 FULL NUMERICAL RESULTS OF META-EVALUATION
PVC-Judge consistently leads open-source assessment models on visual-consistency meta-evaluation across most complex tasks and performs comparably to large closed-source models.
- Meta-evaluation results: PVC-Judge achieves state-of-the-art results among open-source assessment models across most complex visual-consistency tasks.The reported comparisons cover EditReward-Bench and VCReward-Bench.
- Meta-evaluation results: PVC-Judge demonstrates assessment capabilities comparable to large closed-source proprietary models.
E.2 QUALITATIVE ANALYSIS
Qualitative analyses reveal persistent weaknesses in open-set instruction interpretation, spatial-relation editing, and fine-grained detail preservation. These results expose limitations across both open-source and closed-source models, including identity failures on small faces.
- Open-Set Editing: Open-source models often miss implicit constraints in open-set prompts, producing missing, misaligned, or only partially compliant edits.These failures indicate difficulty generalizing beyond predefined editing categories.
- Overall Findings: The qualitative findings show that current models still struggle to fuse complex textual cues with visual generation.This supports evaluating editing beyond narrowly predefined tasks to reveal real-world limitations.
- Relation Change: Open-source models frequently simplify or ignore spatial instructions instead of preserving complex inter-object dependencies.Observed errors include disjointed object addition or removal and partial relational edits.
- Fine-Grained Detail: Open-source models often distort small or background subjects, while closed-source models improve structure but can lose identity on small faces.The analysis identifies fine-grained detail fidelity as a persistent challenge for practical deployment.