PaperlayerView exact revision on arXiv

Source-linked AI summary

StableI2I: Spotting Unintended Changes in Image-to-Image Transition

Jiayang Li, Shuo Cao, Xiaohui Li, Zhizhen Zhang, Kaiwen Zhu, Yule Duan, Yu Qiao, Jian Zhang, Yihao Liu

arXiv:2605.04453v1cs.CVcs.AI

TL;DR

I2I evaluations often emphasize instruction following and visual quality while overlooking whether outputs preserve input content. StableI2I assesses fidelity and pre–post consistency, and the authors report that it more accurately evaluates these dimensions.

  • Problem

    Current I2I evaluations mainly assess instruction following and output aesthetics or perceptual quality, while largely ignoring whether editing or restoration preserves input-image fidelity.

    Source: §1 · p. 1

  • Method

    StableI2I evaluates I2I fidelity across semantic, structural, and appearance dimensions, while StableI2I-Bench tests MLLMs on fidelity assessment.

    Source: §1 · p. 2

  • Results

    StableI2I more accurately assesses content fidelity and consistency, while its benchmark poses a substantial challenge to existing MLLMs.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    StableI2I-Bench provides a tool for measuring perceptual ability beyond single-image understanding under multi-image consistency constraints.

    Source: §6 · p. 8

    Compared with general-purpose MLLMs, StableI2I is less robust to template variations involving additional prior knowledge or more complex instruction structures.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

In most real-world image-to-image (I2I) scenarios, existing evaluations primarily focus on instruction following and the perceptual quality or aesthetics of the generated images. However, they largely fail to assess whether the output image preserves the semantic correspondence and spatial structure of the input image. To address this limitation, we propose StableI2I, a unified and dynamic evaluation framework that explicitly measures content fidelity and pre--post consistency across a wide range of I2I tasks without requiring reference images, including image editing and image restoration. In addition, we construct StableI2I-Bench, a benchmark designed to systematically evaluate the accuracy of MLLMs on such fidelity and consistency assessment tasks. Extensive experimental results demonstrate that StableI2I provides accurate, fine-grained, and interpretable evaluations of content fidelity and consistency, with strong correlations to human subjective judgments. Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.

1. Introduction

StableI2I addresses unintended changes in image-to-image outputs by evaluating semantic and pixel-level consistency between inputs and outputs. The paper also introduces a benchmark and data-construction pipeline for fidelity assessment.

1. Introduction

  • Sampling randomness can cause content drift, while evaluations focused on instruction following, aesthetics, or perceptual quality can overlook unintended changes to input content.Such changes can include repainting image regions or removing objects, with potential consequences in high-stakes applications.Source: §1 · p. 1
  • Figure 1 shows ImgEdit-Judge failing to detect substantial repainting, whereas StableI2I more accurately assesses content fidelity and consistency (Ye et al., 2025).The paper attributes this limitation to MLLMs’ insensitivity to fine-grained pixel-level and structural information.Source: Figure 1 · p. 1, §1 · pp. 1–2
  • Rigid mask-based comparisons can misjudge valid global effects of edits and do not apply to restoration tasks (Ryu et al., 2025).The paper argues that evaluation must interpret instructions and image content to produce context-dependent analyses.Source: §1 · p. 2
  • StableI2I jointly evaluates semantic and pixel-level consistency, conditioning on the instruction and selectively attending to regions and attributes that should remain consistent.It defines Structure Level, Semantic Level, and Low-level Appearance as complementary fidelity dimensions.Source: §1 · p. 2
  • StableI2I-Bench uses formatted question–answer pairs to assess MLLMs’ high-level semantic reasoning and low-level visual perception across three fidelity dimensions.Source: §1 · p. 2
  • A multi-stage, multi-task data-construction pipeline improves data diversity and model robustness, while error amplification targets long-tail subtle consistency violations.Source: §1 · p. 2

2. Related Works

Existing I2I quality assessment either depends on often-unavailable ground truth or overlooks source-conditioned semantic consistency. MLLM-based assessment remains coarse-grained and often misses low-level variations and diagnostic depth.

2.1. Quality Assessment for I2I Transition

  • FR metrics require often-unavailable ground truths, while standard NR methods focus on absolute aesthetics or perceptual quality and miss source-image semantic consistency.This motivates source-conditioned evaluation of content fidelity and structural preservation without ground truth.Source: §2.1 · p. 2

2.2. MLLM-based I2I Transition Assessment

  • Prior MLLM-based I2I assessments use general-purpose prompting or GPT-4o-prior distillation, but are often coarse-grained and miss pixel-level variation and diagnostic depth.The literature motivates fidelity-centric, instruction-aware evaluation of both semantic and perceptual consistency.Source: §2.2 · p. 2

3. From Data to Benchmark and Training

The section describes how StableI2I data is generated and annotated, how StableI2I-Bench is defined, and how the training corpus combines consistency-evaluation tasks with data for broader visual understanding. It also identifies limits in realistic editing data and pixel-level perception that motivate additional training data.

3.1. Data Construction Pipeline

  • Restoration samples are error-amplified by guiding a restoration model with deliberately corrupted semantic descriptions, while editing results are generated from varied instructions and generative models.The restoration pipeline applies random degradations to natural images, uses GPT-5 to describe original content and perturb those descriptions, then uses the corrupted descriptions to guide restoration.Source: §3.1 · p. 3
  • Fidelity annotations cover semantic changes, structural or texture misalignment, and low-level appearance defects, using semi-automatic labels for synthesized restoration data and manual labels for real-world restoration and editing data.Synthesized restoration annotations start with GPT-5 API labeling based on known semantic corruption, followed by human filtering and correction.Source: §3.1 · p. 3

3.2. StableI2I-Bench: Benchmark Definition

  • StableI2I-Bench tests whether open- and closed-source models can use prompts to judge pre–post consistency, using 1,000 human-annotated image pairs for each of three fidelity dimensions.Each formatted prompt includes input and output images, the I2I control instruction, dimension-specific background knowledge, an output specification, and structured answers.Source: §3.2 · p. 3

3.3. StableI2I-Train: Training Corpus Construction

  • Given the limited capacity of its relatively small 8B-parameter MLLM, StableI2I uses fixed task templates for Binary & Type QA and Open-ended QA.Binary & Type QA produces concise standardized outputs, while Open-ended QA provides natural-language error descriptions.Source: §3.3 · p. 4
  • Free-form Descriptive data preserves visual perception and description abilities, drawing from ShareGPT4V and CapRL across natural and AIGC images, tables, and descriptive and multiple-choice QA styles.The data is introduced as a multi-task descriptive QA dataset.Source: §3.3 · p. 4
  • Because free-form outputs are difficult to constrain with structured rewards, human-annotated descriptions are converted into deterministic multiple-choice QA for finer-grained content understanding and analysis.The training framework uses reinforcement learning, which makes fixed output formats and coarse-grained content correctness easier to evaluate reliably.Source: §3.3 · p. 4
  • Realistic editing errors are difficult to construct and annotate at scale, and Texture-Aware Enhancement Data is added to address insufficient pixel-level perception in the ViT encoder.The paper points to a multi-stage training scheme to expand data scale and enhance model capability.Source: §3.3 · p. 4

4. StableI2I

StableI2I trains Qwen3-VL-8B-Instruct through supervised fine-tuning and GRPO, then expands its training data with filtered annotations from unlabeled I2I tasks. A final fine-tuning stage combines original and newly generated data to improve perceptual understanding and generalization.

StableI2I

  • Training first uses supervised fine-tuning on Binary & Type QA, Multiple-choice QA, and Free-form Descriptive data, then GRPO to improve generalization.The rewards are defined separately for multiple-choice and binary-answer tasks.Source: §4 · p. 5
  • For multiple-choice tasks, predictions outside the ground-truth option set receive zero reward; otherwise, reward scales with the fraction of ground-truth options selected.The maximum multiple-choice reward is M.Source: §4 · p. 5
  • For binary-answer tasks, the predicted answer must exactly match ground truth; Yes requires both problem sets empty, while No requires both non-empty.The reward also uses a penalty parameter α for false-positive predictions.Source: §4 · p. 5
  • The reinforcement-learning-trained model annotates unlabeled data across I2I tasks, GPT-5 filters obvious errors, and GPT-5 generates think and problem fields for open-ended QA.The open-ended QA data combines original Binary & Type QA with newly annotated Binary & Type QA.Source: §4 · p. 5
  • Final full fine-tuning of Qwen3-VL-8B-Instruct on the mixture of original and newly annotated data yields the final model, with stronger perceptual understanding and improved generalization.The overall training pipeline is shown in Fig. 5.Source: §4 · p. 5

5. Experiments

The experiments evaluate StableI2I against multimodal models and use it to assess image-editing and restoration systems. They find that structural fidelity remains challenging, while multiple-choice QA and data augmentation improve StableI2I’s assessment capability.

5.1. Evaluation Results on StableI2I-Bench

  • After training and fine-tuning, StableI2I outperforms existing state-of-the-art vision models on structure-level fidelity assessment, where mainstream models generally perform poorly.Gemini-3-pro achieves the best overall performance among mainstream models; semantic-level assessment is generally stronger, while structure-level assessment demands pixel-level alignment and can be difficult when outputs repaint global structure.Source: Table 1 · p. 6, §5.1 · p. 6
  • Seven volunteers’ judgments of 50 outputs from Bagel, Nano-Banana, and GPT-Image-1 were largely consistent with StableI2I’s assessments.The examples were sampled from ImgEdit-Bench, and the model used StableI2I-Bench response templates.Source: Figure 6 · p. 6, §5.1 · p. 6
  • StableI2I is less robust to template variation, while general-purpose models’ performance drops across all three fidelity dimensions when benchmark-provided structured priors are removed.Both evaluation settings use the same image pairs and control instruction, but StableI2I uses a specialized fixed template; additional prior knowledge or complex prompts can weaken its robustness.Source: §5.1 · p. 6

5.2. Model Performance on I2I Tasks Assessed by StableI2I

  • On editing and restoration benchmarks, Bagel and the Qwen series show superior fidelity, while most models score better on semantics than structure; Flux.1-dev and GPT-Image-1 show severe repainting.StableI2I scores the proportion of samples judged to have no detected fidelity issue in each dimension; repainting largely discards the original structural layout and yields extremely low structure-level scores.Source: Table 2 · p. 7, §5.2 · p. 7
  • Qualitative evaluations show all three information-drift types in real-world tasks, with unintended content repainting the most critical issue; analysis focuses on semantic and low-level appearance errors.Structural errors are treated more coarsely because they involve global changes with limited categories.Source: Figure 7 · p. 7, §5.2 · pp. 7–8, Figure 8 · p. 8

5.3. Ablation Study

  • Multiple-choice QA turns open-ended content descriptions into fixed-choice questions and substantially improves perceptual capability under identical parameter settings and training steps.The ablation compares adding multiple-choice questions in neither, one, or both training stages.Source: Table 3 · p. 8, §5.3 · p. 8
  • StableI2I’s augmented-data retraining achieves substantially better overall performance than Dev.1 while largely retaining Dev.2’s gains; RL improves semantic accuracy but reduced data diversity degrades other categories.The final model combines Dev.2’s augmented data with the first-stage data and retrains the base model using SFT.Source: Table 4 · p. 8, §5.3 · p. 8

6. Conclusion

StableI2I evaluates I2I fidelity at semantic and pixel levels, while StableI2I-Bench tests MLLMs’ multi-image consistency assessment. The conclusion also identifies an appendix for detailed visualizations and failure-case analysis.

6. Conclusion

  • StableI2I is presented as the first framework to systematically evaluate I2I fidelity from semantic and pixel-level perspectives.It assesses whether generative models preserve critical visual information and provides StableI2I-Bench for evaluating MLLMs under multi-image consistency constraints.Source: §6 · p. 8
  • By requiring semantic and pixel-level consistency across multiple images, StableI2I-Bench challenges existing MLLMs and measures perceptual ability beyond single-image understanding.Source: §6 · p. 8
  • The authors believe StableI2I can improve I2I generation quality, leading to more faithful, realistic, and perceptually consistent outputs.Source: §6 · pp. 8–9

Limitations

  • For detailed subjective visualizations and analysis of specific failure cases, the conclusion refers readers to Appendix B.4.Source: §6 · p. 9

Impact Statement … A.4. Detailed Description of StableI2I-Train

StableI2I assesses semantic, structural, and low-level fidelity in image-to-image transitions without reference images, while StableI2I-Bench provides human-annotated examples and evaluation templates. The supplementary material details the data construction and annotation workflows and cautions that consistency judgments require ongoing bias audits.

Impact Statement

  • StableI2I targets unintended semantic and structural drift and measures fidelity across semantic, structural, and low-level appearance dimensions without reference images.The framework is intended as a diagnostic tool for high-stakes domains such as medical imaging and remote sensing; the authors call for ongoing audits of potential bias in how consistency is defined.Source: Impact Statement · p. 9

A. Dataset

  • The dataset appendix details data sources, construction models, human annotation workflows, and prompts used during data generation.Source: A. Dataset · p. 12

A.1. Data Construction and Statistics

  • Data construction draws on ImageNet, Unsplash, DIV2K, Alchemist, and ArtiMuse, resizing images so their maximum side is at most 1344 pixels.For restoration, the pipeline uses ESRGAN degradation, text-guided restoration models, and second-stage enhancement; the Low-level Dataset samples 500 public-dataset images. Editing uses four generation models and GPT-5 prompts organized around add, replace, and remove.Source: §A.1 · p. 12

A.2. Human Annotation and Annotator Training

  • Annotation used a cross-annotation and review protocol, with 10 annotators per task working for seven consecutive days after video-based training.Annotators checked GPT-5 judgments and degradation levels for restoration, while editing annotations also included error types and optional bounding boxes.Source: §A.2 · p. 13
  • 94% overall annotation pass accuracy was achieved, with 6,722 human-annotated restoration samples and 4,839 image-editing samples.The annotation company was selected after pilot accuracies of 0.852, 0.725, and 0.700; the highest-scoring company conducted full-scale annotation.Source: §A.2 · p. 13

A.3. Detailed Description of StableI2I-Bench

  • StableI2I-Bench uses human-annotated samples and has no overlap with Stable-Train; its appendix presents evaluation templates and representative examples.Source: §A.3 · p. 13

A.3.1. STABLEI2I-BENCH EVALUATION TEMPLATES

  • The fixed benchmark prompt supplies before-and-after images, the task instruction, prerequisite knowledge, and an output format; each of three dimensions contains 1,000 pairs, with no-issue cases at most 50%.Semantic evaluation flags unintended additions, removals, or replacements; structural evaluation checks misalignment and repainting outside targeted regions; low-level evaluation checks unintended blur, noise, color cast, exposure degradation, or artifacts while ignoring effects explicitly intended by the task.Source: §A.3.1 · pp. 13–15

A.3.2. STABLEI2I-BENCH EXAMPLES

  • The three illustrative benchmark examples label unintended repainting, semantic replacement, and blur as the respective ground-truth issues.Source: §A.3.2 · pp. 16–18

A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN · A.4.2. INPUT PROMPT TEMPLATE OF STABLEI2I-TRAIN · B. Supplementary Experimental Results

The supplement describes StableI2I-Train’s QA construction, synthetic data and training-data composition, then specifies fixed prompts for evaluating structural consistency, semantic drift and low-level appearance changes. The supplied passages contain no material from section B.

A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN

  • Multiple-choice QA separates error-type identification from identifying affected objects or regions, with subtype questions also including unaffected objects.Type questions enumerate error types plus a none-of-the-above option and allow multiple correct choices; subtype questions jointly require image understanding and question generation.Source: §A.4.1 · p. 18
  • Synthetic training data pairs nearly identical crops to teach fine-grained pixel alignment and includes degraded images spanning blur, noise, compression artifacts and color distortions.Texture-Aware Enhancement Data uses crops with a 95–98% cropping ratio; the degraded subset targets low-level appearance shifts.Source: §A.4.1 · pp. 19–20
  • Image Identity asks whether two identical images differ, using COCO images and the synthetic-data pipelines to strengthen pixel-level correspondence perception.The task is an additional training category beyond editing and restoration.Source: Figure 10 · p. 20, §A.4.1 · p. 21
  • 174,866 free-form descriptive samples are reported, alongside Binary & Type, Open-ended and Multiple-choice QA training volumes and an overall SFT composition figure.The supplied passages identify Tables 7–9 and Figure 10 but do not provide their cell values or proportions.Source: Table 7 · p. 20, Table 8 · p. 20, Table 9 · p. 20

A.4.2. INPUT PROMPT TEMPLATE OF STABLEI2I-TRAIN

  • Fixed prompt formats cover Semantic, Structure and Low-level Appearance dimensions for Binary & Type QA, while Open-ended QA covers Semantic and Low-level Appearance.Structure Level is excluded from Open-ended QA because global changes do not admit more fine-grained textual descriptions.Source: §A.4.2 · p. 21
  • Structure prompts judge consistency in texture-preserving regions and classify misalignment or repainting, while semantic prompts detect additions, deletions or modifications outside the intended edit.Semantic prompts mark any potential inconsistency as “No”; structure prompts also support a simple identity judgment when no task type is given.Source: §A.4.2 · p. 21
  • Low-level prompts check preserved regions for degradation, color cast or artifacts, ignoring changes requested by the task and reporting only out-of-scope violations.The open-ended format requires a two-stage scope and preservation analysis before problem reporting, with only supplied issue keys allowed.Source: §A.4.2 · pp. 22–23

B.1. Supplementary Results of StableI2I and MLLMs on StableI2I-Bench

Under identical, less informative prompting, StableI2I outperforms general-purpose MLLMs across all three evaluation dimensions. Removing instructional priors most harms MLLMs’ semantic-level judgments, while structure-level performance changes little.

B.1. Supplementary Results of StableI2I and MLLMs on StableI2I-Bench

  • StableI2I uses a fixed, unified inference template, whereas StableI2I-Bench gives general MLLMs richer instructional priors; a controlled comparison evaluates MLLMs with StableI2I’s template.This setting isolates the prompt-template effect and assesses performance under identical prompting conditions.Source: §B.1 · p. 23
  • Even with StableI2I’s less informative prompt, StableI2I outperforms all general-purpose MLLMs across the three evaluation dimensions.The controlled comparison attributes the advantage to task-specific training and stronger pixel-level perceptual and alignment capabilities, not a more favorable prompt.Source: Table 10 · p. 23, §B.1 · p. 24
  • Removing the benchmark’s additional instructional priors most reduces MLLM performance on Semantic Level, while Structure Level remains largely unchanged or improves slightly.The authors attribute the limited structural change to current models’ difficulty judging structure, which additional priors do not substantially resolve.Source: §B.1 · p. 23
  • The benchmark’s instruction-rich template makes evaluation feasible for general-purpose MLLMs rather than unfairly advantaging StableI2I; removing it degrades MLLM results while StableI2I remains robust.The findings support an intrinsic capability gap and the need for task-specialized fidelity assessment, while highlighting current MLLMs’ limitations in fine-grained I2I evaluation.Source: §B.1 · p. 24

B.2. Ablation study with ImgEdit-Judge

StableI2I outperforms ImgEdit-Judge in fidelity assessment on two benchmarks. The study evaluates ImgEdit-Judge using Physical & Detail Integrity and sampled model-output comparisons against human preference judgments.

B.2. Ablation study with ImgEdit-Judge

  • The study compares evaluation results with human preference judgments using 50 randomly sampled pairs per benchmark and ImgEdit-Judge’s Physical & Detail Integrity dimension.The candidate pool included Qwen-Image-Edit-2511 (Wu et al., 2025a), Nano-Banana, GPT-Image-1, and Omnigen2 (Wu et al., 2025b).Source: §B.2 · p. 24
  • StableI2I achieves better performance than ImgEdit-Judge in fidelity assessment on ImgEdit-Bench and GEdit-Bench.The comparison accuracy results are reported in Table 11.Source: Table 11 · p. 24, §B.2 · p. 24

B.3. Supplementary Results of StableI2I in Real-World I2I Evaluation Settings

Supplementary examples show StableI2I evaluation results across three benchmarks and illustrate detailed model descriptions of detected errors.

B.3. Supplementary Results of StableI2I in Real-World I2I Evaluation Settings

  • Subjective evaluation examples cover ImgEdit-Bench (Fig. 12), GEdit-Bench (Fig. 13), and the Low-level Dataset (Fig. 14).Source: §B.3 · p. 24
  • For erroneous cases, the model accurately identifies and describes error types and corresponding target objects along Semantic Level and Low-level Appearance dimensions.Fig. 15 presents detailed descriptions, supplementing the short-form answers shown in the preceding results.Source: §B.3 · p. 24
  • In each Fig. 15 example pair, the left image is the input and the right image is the output.Source: §B.3 · p. 24

B.4. Demonstration of Model Limitations

StableI2I can misclassify valid edits when the task’s intended semantic changes conflict with its judgments of repainting, content loss, or pixel-level alignment. The authors attribute these errors partly to task coverage and model capacity, and propose broader task coverage and selective abstention as possible remedies.

Demonstration of Model Limitations

  • The model wrongly flags valid style transfer as repainting, may mistake object extraction for lost content, and penalizes pixel misalignment from correct human-pose edits.The paper describes these as failure cases that humans would consider correct, including the illustrated style-transfer, human-motion, and object-extraction tasks.Source: §B.4 · pp. 24–25
  • The authors trace errors mainly to conflicts between the actual I2I processing direction and the model’s judgment, and propose broader task coverage or selective abstention.They suggest expanding the range of I2I tasks to better cover complex editing semantics; abstaining may be reasonable in ambiguous cases.Source: §B.4 · p. 24
  • The authors also attribute some failures to model capacity and parameter scale, noting that closed-source large models usually adapt more flexibly in these scenarios.They suggest that multimodal large models could progress further if they achieved pixel-level perceptual sensitivity comparable to StableI2I.Source: §B.4 · p. 25

Related papers

Loading 2605.04453v1…