PaperlayerView exact revision on arXiv

Source-linked AI summary

StableI2I: Spotting Unintended Changes in Image-to-Image Transition

Jiayang Li, Shuo Cao, Xiaohui Li, Zhizhen Zhang, Kaiwen Zhu, Yule Duan, Yu Qiao, Jian Zhang, Yihao Liu

arXiv:2605.04453v1cs.CVcs.AI

TL;DR

Existing I2I evaluations largely overlook whether outputs preserve the input’s semantic content and spatial structure. StableI2I addresses this with fidelity-focused evaluation and StableI2I-Bench, achieving more accurate assessment of semantic and pixel-level consistency.

  • Problem

    Current I2I evaluations emphasize instruction following, aesthetics, or perceptual quality while largely ignoring unintended semantic and spatial changes from input to output.

  • Method

    StableI2I evaluates I2I fidelity through instruction-conditioned regional and attribute attention across structure, semantic, and low-level appearance dimensions, supported by StableI2I-Bench.

  • Results

    StableI2I more accurately assesses content fidelity and consistency than existing metrics and achieves substantially better overall performance after data augmentation.

  • Takeaways & Limitations

    StableI2I provides systematic semantic- and pixel-level fidelity assessment, while StableI2I-Bench evaluates MLLMs under multi-image consistency constraints.

  • Takeaways & Limitations

    StableI2I is less robust than general-purpose MLLMs to template variations involving additional prior knowledge or complex instruction structures.

Abstract

from arXiv · show

In most real-world image-to-image (I2I) scenarios, existing evaluations primarily focus on instruction following and the perceptual quality or aesthetics of the generated images. However, they largely fail to assess whether the output image preserves the semantic correspondence and spatial structure of the input image. To address this limitation, we propose StableI2I, a unified and dynamic evaluation framework that explicitly measures content fidelity and pre--post consistency across a wide range of I2I tasks without requiring reference images, including image editing and image restoration. In addition, we construct StableI2I-Bench, a benchmark designed to systematically evaluate the accuracy of MLLMs on such fidelity and consistency assessment tasks. Extensive experimental results demonstrate that StableI2I provides accurate, fine-grained, and interpretable evaluations of content fidelity and consistency, with strong correlations to human subjective judgments. Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.

1. Introduction

StableI2I addresses unintended content drift in image-to-image outputs, which existing evaluations largely overlook while emphasizing instruction following, aesthetics, or perceptual quality. It jointly evaluates semantic and pixel-level consistency and introduces StableI2I-Bench for systematic fidelity assessment across three dimensions.

  • Motivation: Generative-model sampling randomness can cause substantial information drift between generated outputs and input images, including in state-of-the-art systems such as Nano-Banana.This motivates methods for evaluating and calibrating content drift.
  • Limitations of Existing Evaluation: Current I2I evaluations mainly assess instruction following, output aesthetics, or perceptual quality while overlooking whether editing and restoration outputs remain faithful to their inputs.Existing metrics may score images highly despite unintended texture and semantic changes, such as unnecessary repainting of sky and sandy areas.
  • Limitations of Existing Evaluation: Mask-based consistency checks can reject valid global variations, including illumination and shadows that are causally induced by an edit.Replacing an object with a tree can plausibly change the shadow beneath it, making rigid separation-based comparisons inadequate.
  • StableI2I: StableI2I jointly considers semantic and pixel-level consistency to better judge semantic content and pixel-level details between input and output images.The model is fidelity-oriented and designed for I2I evaluation.
  • StableI2I-Bench: StableI2I-Bench systematically evaluates modern MLLMs across Structure Level, Semantic Level, and Low-level Appearance fidelity dimensions.StableI2I conditions on the input instruction and selectively attends to regions and attributes that must remain consistent.

2. Related Works

Prior image-transition quality assessment uses Full-Reference and No-Reference paradigms, but I2I evaluation also requires semantic consistency alongside aesthetic quality and remains underexplored. Existing approaches include general-purpose MLLMs prompted through engineering or trained through distillation.

  • Quality Assessment Models: Natural image-transition quality models are conventionally categorized as Full-Reference and No-Reference paradigms.Full-Reference metrics rely on ground truths that are often unavailable, whereas standard No-Reference methods predominantly assess absolute aesthetics or perceptual quality.
  • I2I Evaluation: I2I transition evaluation requires a multi-dimensional perspective covering semantic consistency and aesthetic quality, yet this domain remains largely underexplored.The passage characterizes this gap as a limitation of prior evaluation practice.
  • I2I Evaluation: Prior I2I works primarily use general-purpose MLLMs through prompt engineering or distillation methods.Prompt-engineering examples include MagicBrush and CompBench, while ImgEdit is described as using distillation.

3. From Data to Benchmark and Training

StableI2I constructs error-amplified I2I data and StableI2I-Bench to evaluate semantic, structural, and low-level appearance fidelity. Its training data combines fixed-format evaluation tasks with descriptive and multiple-choice QA, while addressing limited editing-data scale and pixel-level perception.

  • Data Construction: StableI2I’s data pipeline generates restoration errors through random degradations and controlled semantic perturbations, then annotates semantic, structural, and low-level appearance fidelity dimensions.The three dimensions cover unintended semantic additions, deletions, or modifications; structural misalignment and repainting; and degradations such as noise, blur, color shift, or artifacts.
  • Benchmark Construction: StableI2I-Bench samples 1,000 human-annotated image pairs per fidelity dimension and evaluates models with formatted prompts containing paired images, instructions, dimension knowledge, and structured answers.The benchmark targets whether open-source and closed-source models can correctly judge I2I pre–post consistency rather than merely follow evaluation prompts.
  • Training Data: Because StableI2I fine-tunes a relatively small 8B-parameter MLLM, training uses fixed task templates for Binary & Type QA, Open-ended QA, Multiple-choice QA, and Free-form Descriptive data.Free-form Descriptive data preserves visual perception and description across natural images, AIGC images, tables, and multiple QA styles.
  • Training Data: Multiple-choice conversion makes human descriptions deterministic and reward-compatible, improving fine-grained content understanding when free-form outputs are difficult to constrain with structured rewards.This design supports reliable evaluation beyond fixed formats and coarse-grained content correctness.
  • Training Limitations: The training design remains limited by the difficulty of constructing diverse, realistic large-scale editing errors and by insufficient pixel-level perceptual capacity in the ViT encoder.Texture-Aware Enhancement Data is introduced to strengthen encoder perception at the pixel level, while later multi-stage training expands data scale and model capability.

5. Experiments

Experiments evaluate mainstream multimodal models on StableI2I-Bench and use StableI2I to assess image editing and restoration systems. Results show strong semantic-level performance but persistent structural inconsistency, while StableI2I aligns with human judgments and outperforms existing vision models after training.

  • 5.1. Model Performance on StableI2I-Bench: Removing structured priors from the benchmark template significantly reduces general-purpose MLLM performance across all three fidelity dimensions.The specialized StableI2I and benchmark settings use identical image pairs and I2I control instructions but different inference templates.
  • 5.1. Model Performance on StableI2I-Bench: Gemini-3-pro achieves the best overall performance among mainstream models, which generally perform strongest on the Semantic Level and weaker on structural fidelity.Models can often detect prominent low-level degradations, but struggle when globally consistent semantic content accompanies altered structure.
  • 5.1. Model Performance on StableI2I-Bench: After training and fine-tuning, StableI2I outperforms existing state-of-the-art vision models on I2I fidelity assessment.The result indicates substantial room for improving current visual models’ fidelity assessment capabilities.
  • 5.1. Model Performance on StableI2I-Bench: StableI2I evaluation outputs are largely consistent with human judgments in a study involving seven volunteers and 50 generated images.The images were sampled from Bagel, Nano-Banana, and GPT-Image-1 outputs on ImgEdit-Bench.
  • 5.2. Model Performance on I2I Tasks Assessed by StableI2I: Bagel and the Qwen series show superior fidelity, while most models preserve semantics better than fine-grained structure across editing and restoration tasks.Flux.1-dev and GPT-Image-1 exhibit severe content repainting, producing extremely low Structure Level scores.
  • 5.2. Model Performance on I2I Tasks Assessed by StableI2I: Unintended content repainting is the most critical information-drift issue observed in real-world image editing and restoration evaluations.StableI2I’s qualitative results show that all three types of information drift can occur in these scenarios.

6. Conclusion

StableI2I systematically evaluates fidelity in image-to-image tasks from semantic and pixel-level perspectives, while StableI2I-Bench benchmarks MLLM multi-image consistency assessment. The framework aims to support more faithful, realistic, and perceptually consistent outputs, with detailed failure-case analysis deferred to Appendix B.4.

  • Contributions: StableI2I is the first framework to systematically evaluate I2I fidelity from both semantic and pixel-level perspectives.It assesses whether generative models preserve critical visual information.
  • Contributions: StableI2I-Bench provides a precise benchmark for evaluating MLLMs under multi-image consistency.The benchmark targets fidelity assessment across multiple images.
  • Impact: StableI2I is intended to improve I2I generation quality toward more faithful, realistic, and perceptually consistent outputs.This conclusion states the framework’s expected practical impact on generated images.
  • Limitations: Detailed subjective visualizations and analysis of specific failure cases are provided in Appendix B.4.The conclusion directs readers to the appendix for this analysis.

Impact Statement · A. Dataset

StableI2I evaluates unintended semantic and structural drift in image-to-image transitions through reference-free content-fidelity measurements. The dataset section supplements this framework with details on data sources, construction models, annotation, and generation prompts.

  • Impact Statement: StableI2I is a unified evaluation framework for improving the reliability of image-to-image transitions.It is designed to detect unintended semantic and structural drift.
  • Impact Statement: The framework measures content fidelity without requiring reference images.Its methodology covers semantic, structural, and low-level appearance dimensions.
  • Impact Statement: StableI2I provides a diagnostic tool for detecting unintended drift in image-to-image outputs.The paper frames this capability as supporting more reliable I2I transitions.
  • Impact Statement: The framework is intended for high-stakes applications such as medical imaging and remote sensing.These applications motivate the need to preserve information during image-to-image transitions.
  • A. Dataset: The dataset section describes the data sources used in the benchmark’s construction.It presents supplementary information about how the data were sourced.
  • A. Dataset: The dataset section documents the models used for data construction.These models are part of the supplementary dataset-building details.
  • A. Dataset: The dataset section explains the human annotation workflow.The workflow is included among the supplementary details accompanying the dataset.
  • A. Dataset: The dataset section records the prompts adopted for data generation.These prompts are described as part of the supplementary data-generation process.

A.1. Data Construction and Statistics

StableI2I-Bench is constructed from diverse image sources and task-specific pipelines for restoration and editing. The data are size-constrained, synthesized with multiple models, and statistically documented in Table 6.

  • Data sources and preprocessing: The dataset draws images from ImageNet, Unsplash, DIV2K, Alchemist, and ArtiMuse, then resizes images exceeding a 1344-pixel maximum side length while preserving aspect ratio.The exact image counts from each source are reported in Table 6.
  • Image restoration: Restoration data use ESRGAN degradation followed by text-guided restoration with SeeSR, SUPIR, or OSEDiff, with some outputs enhanced again using SwinIR or ESRGAN.The second-stage enhancement increases the diversity of generated restoration samples.
  • Image editing: Editing samples are generated with Qwen-Image-Edit-2509, OmniGen, SD3, and GPT-Image-1 using GPT-5-crafted instructions covering add, replace, and remove edits.The instructions integrate visual information from each image and produce concise edits; first-stage sample counts are reported in Table 6.
  • Dataset statistics: Table 6 reports dataset usage and annotation statistics, including intact images, successfully synthesized restoration and editing samples, GPT-5-labeled restoration samples, and a human subset.The human subset is randomly sampled from GPT-5–annotated data, while restoration and editing counts refer to samples successfully produced by their respective pipelines.

A.2. Human Annotation and Annotator Training … A.4. Detailed Description of StableI2I-Train

StableI2I-Bench is built from human annotations and evaluates I2I transitions across semantic, structural, and low-level appearance dimensions using fixed, task-conditioned comparison templates. Its annotation protocol combines pilot-based company selection, trained annotators, cross-annotation review, and explicit inconsistency labels.

  • A.2. Human Annotation and Annotator Training: The highest-accuracy annotation company, achieving 0.852 in pilot annotation, was selected for full-scale annotation after 100 samples per task were evaluated.The other two companies achieved 0.725 and 0.700, respectively.
  • A.2. Human Annotation and Annotator Training: The dataset contains 15,000 restoration samples from GPT-5 coarse annotations and 4,839 image-editing samples, with each task handled by 10 annotators over seven working days.Annotators received video-based training and used cross-annotation with a review phase.
  • A.2. Human Annotation and Annotator Training: Restoration annotation judged GPT-5 correctness and three-level degradation, whereas editing annotation added error types and optional bounding-box localization before the same degradation labeling.The resulting set included 6,722 human-annotated restoration samples and 4,839 editing samples, with 94% overall annotation pass accuracy at acceptance.
  • A.3.1. STABLEI2I-BENCH EVALUATION TEMPLATES: StableI2I-Bench uses fixed prompts containing two input images, the I2I instruction, prerequisite knowledge, and the required output format across three evaluation dimensions.The dimensions are Semantic Level, Structure Level, and Low-level Appearance, each containing 1000 image pairs.
  • A.3.1. STABLEI2I-BENCH EVALUATION TEMPLATES: Semantic evaluation detects unintended additions, removals, or replacements in regions expected to remain unchanged, while ignoring low-level differences and plausible edit side effects.For NULL prompts, semantic identity is expected and any semantic difference is inconsistent.
  • A.3.2. STABLEI2I-BENCH EXAMPLES: Structure evaluation checks unintended misalignment and repainting in non-targeted regions, including geometric distortion, spatial shifts, texture re-rendering, and altered surface details.Physically plausible consequences of the intended edit are excluded, and the output records dominant issues in valid JSON.
  • A.3.2. STABLEI2I-BENCH EXAMPLES: Low-level appearance evaluation targets unintended blur, noise, color cast, exposure degradation, and artifacts while excluding intended effects and high-level semantic or structural changes.A randomly selected example with an outfit matte-finish instruction received the ground-truth output {"answer": "No", "problem": ["blur"]}.
  • A.3.2. STABLEI2I-BENCH EXAMPLES: Illustrative benchmark outputs label structural repainting and semantic replacement as inconsistent, with ground-truth decisions {"answer": "No", "problem": ["repainting"]} and {"answer": "No", "problem": ["replace"]}.The examples cover structure-level cardigan editing and semantic-level image restoration.

A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN · A.4.2. INPUT PROMPT TEMPLATE OF STABLEI2I-TRAIN · B. Supplementary Experimental Results

The supplementary sections detail StableI2I-Train’s data synthesis, multiple-choice question construction, task-specific prompt templates, and the training-data composition. They also define preservation-focused reasoning procedures for identifying semantic, structural, and low-level inconsistencies between pre- and post-processing images.

  • A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN: Multiple-choice QA separates error-type identification from identifying the affected object or region.Type questions identify the error category, whereas Subtype questions identify the object or region where the error occurs.
  • A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN: Subtype questions jointly require image understanding and question generation by contrasting incorrectly affected objects with unaffected image content.The options include objects incorrectly affected and objects present but remaining unaffected.
  • A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN: Texture-Aware Enhancement Data use 95–98% random crops to preserve global semantics while emphasizing subtle pixel-level structural differences.This construction encourages fine-grained pixel alignment and correspondence; Degraded Image Data cover blur, noise, compression artifacts, and color distortions.
  • A.4.1. SUPPLEMENTARY DETAILS ON DATA SYNTHESIS AND SCALE OF STABLEI2I-TRAIN: Image Identity adds a task for detecting differences between identical images and uses COCO data processed through the texture-aware and degraded-image pipelines.The task strengthens fine-grained pixel-level correspondence perception across multiple images.
  • A.4.2. INPUT PROMPT TEMPLATE OF STABLEI2I-TRAIN: Fixed prompt templates organize Binary & Type QA across Semantic Level, Structure Level, and Low-level Appearance, while Open-ended QA uses Semantic Level and Low-level Appearance.Structure Level is omitted from Open-ended QA because it represents global changes without finer-grained textual descriptions.
  • A.4.2. INPUT PROMPT TEMPLATE OF STABLEI2I-TRAIN: Structure-level prompts assess whether texture-consistent regions remain unchanged and classify major inconsistencies as structural misalignment or texture repainting.They instruct evaluation of unchanged areas and overall texture and color consistency in restoration tasks.
  • A.4.2. INPUT PROMPT TEMPLATE OF STABLEI2I-TRAIN: Semantic prompts identify additions, deletions, or modifications in regions that should remain unchanged, using a single-line JSON decision format.The prompt compares semantic content in the post-processed image with the pre-processed image.

B.1. Supplementary Results of StableI2I and MLLMs on StableI2I-Bench

A controlled evaluation using StableI2I’s prompt template shows that its advantage over general-purpose MLLMs persists without instruction-rich prompting. The results attribute this gap to task-specific capabilities rather than prompt engineering.

  • Prompt Templates: StableI2I uses a fixed task template, whereas StableI2I-Bench provides richer instructional priors to help general-purpose MLLMs perform the evaluation.The differing templates reflect StableI2I’s task-specific training and the benchmark’s need to make evaluation feasible for general-purpose models.
  • Controlled Comparison: A controlled experiment evaluates MLLMs with StableI2I’s template to isolate prompt-template effects and compare intrinsic performance under identical prompting conditions.The comparative results are reported in Table 10.
  • Prompt Effects: Removing additional instructional priors causes the largest MLLM performance drop on Semantic Level, while Structure Level remains largely unchanged and slightly improves.The authors attribute this contrast to the inherent difficulty of judging Structure Level for current models.
  • Performance: Even under less informative prompting, StableI2I consistently outperforms all general-purpose MLLMs across all three evaluation dimensions.The authors attribute this robustness to task-specific training and enhanced pixel-level perceptual and alignment capabilities.
  • Fairness of Prompting: The controlled comparison shows that instruction-rich benchmark prompts support MLLM feasibility rather than unfairly benefiting StableI2I, while StableI2I remains robust without them.Removing the auxiliary prior substantially degrades MLLM performance but preserves StableI2I’s strong performance.
  • Implications: Overall, the performance gap reflects intrinsic capability differences, validating task-specialized fidelity assessment and exposing limitations of general-purpose MLLMs in fine-grained I2I evaluation.The findings rule out prompt engineering as the primary confounding explanation for StableI2I’s advantage.

B.2. Ablation study with ImgEdit-Judge

The ablation study compares ImgEdit-Judge with StableI2I by testing whether their fidelity assessments align with human preference judgments across ImgEdit-Bench and GEdit-Bench.

  • Experimental setup: Because the models use different evaluation dimensions, the study compares their outputs rather than directly assessing single-image results.The comparison evaluates whether model-based results are consistent with human preference judgments.
  • Experimental setup: ImgEdit-Judge is evaluated using the Physical & Detail Integrity dimension.
  • Experimental setup: Experiments are conducted on ImgEdit-Bench and GEdit-Bench.

B.3. Supplementary Results of StableI2I in Real-World I2I Evaluation Settings

Supplementary results illustrate StableI2I’s evaluation performance across real-world I2I settings, including editing and low-level datasets, with detailed descriptions of erroneous cases. The model accurately identifies error types and corresponding target objects across Semantic Level and Low-level Appearance dimensions.

  • Supplementary evaluation examples: Additional subjective examples demonstrate StableI2I’s evaluation performance on ImgEdit-Bench, GEdit-Bench, and the Low-level Dataset.The examples are presented in Fig. 12, Fig. 13, and Fig. 14, respectively.
  • Error analysis: For erroneous cases, the model generates detailed descriptions along the Semantic Level and Low-level Appearance dimensions.These descriptions supplement the model’s short-form answers to evaluation questions.
  • Error analysis: The model accurately identifies and describes both error types and their corresponding target objects.Each example pair contains the input image on the left and the output image on the right.

B.4. Demonstration of Model Limitations

StableI2I fails on several valid I2I edits, especially when processing directions overlap or conflict with its judgment direction. These failures reflect limitations in handling complex editing semantics, model capacity, and parameter scale.

  • Observed failure cases: StableI2I incorrectly flags valid style transfer as problematic repainting and may judge object extraction as loss of non-target content.The documented failure cases also include human motion and object extraction, although humans consider all four tasks correct.
  • Error sources and remedies: The model’s errors mainly arise when the actual I2I processing direction overlaps or conflicts with its judgment direction.The authors propose expanding the task scope to cover such cases and allowing selective abstention in ambiguous situations.
  • Model limitations: Model capacity and parameter scale also contribute to these limitations, while closed-source large models generally adapt more flexibly in such scenarios.The authors suggest that multimodal large models could improve further if they achieved StableI2I-level pixel sensitivity.
  • Detailed failure examples: The detailed examples show false detections of replacement, removal, and color-cast problems in regions that should remain preserved.Examples include changed background, shirt, missing boats and bridges, removed costume or sky fragments, and an introduced cool color cast.
Loading 2605.04453v1…