PaperlayerView exact revision on arXiv

Source-linked AI summary

PICABench: How Far Are We from Physically Realistic Image Editing?

Yuandong Pu, Le Zhuo, Songhao Han, Jinbo Xing, Kaiwen Zhu, Shuo Cao, Bin Fu, Si Liu, Hongsheng Li, Yu Qiao, Wenlong Zhang, Xi Chen, Yihao Liu

arXiv:2510.17681v3cs.CVcs.AI

TL;DR

Image-editing benchmarks and models often emphasize instruction completion while overlooking physical effects that contribute to realism. The paper introduces a physics-focused benchmark, a region-grounded evaluation protocol, and video-derived training data. Current models remain far from physically realistic editing, while fine-tuning on PICA-100K improves physical consistency.

  • Problem

    Existing models and benchmarks focus mainly on instruction completion or semantic fidelity, leaving physical effects in common edits insufficiently evaluated.

    Source: §1 · p. 1

  • Method

    The paper develops PICABench across three physical dimensions and eight sub-dimensions, PICAEval with targeted region-grounded questions, and the video-derived PICA-100K dataset.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §3.3 · p. 6

  • Results

    Current models remain far from physically realistic edits, while PICA-100K fine-tuning improves physical consistency; Flux.1-Kontext gains +1.71% in overall accuracy over the base model.

    Source: §4.2 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The benchmark, evaluation protocol, and video-based supervision offer a foundation for progress toward physics-aware image editing.

    Source: §6 · p. 10

    The current framework supports only single-image inputs, not multi-image or multi-condition contexts.

    Source: §5 · p. 10

Abstract

from arXiv · show

Image editing has achieved remarkable progress recently. Modern editing models could already follow complex instructions to manipulate the original content. However, beyond completing the editing instructions, the accompanying physical effects are the key to the generation realism. For example, removing an object should also remove its shadow, reflections, and interactions with nearby objects. Unfortunately, existing models and benchmarks mainly focus on instruction completion but overlook these physical effects. So, at this moment, how far are we from physically realistic image editing? To answer this, we introduce PICABench, which systematically evaluates physical realism across eight sub-dimension (spanning optics, mechanics, and state transitions) for most of the common editing operations (add, remove, attribute change, etc.). We further propose the PICAEval, a reliable evaluation protocol that uses VLM-as-a-judge with per-case, region-level human annotations and questions. Beyond benchmarking, we also explore effective solutions by learning physics from videos and construct a training dataset PICA-100K. After evaluating most of the mainstream models, we observe that physical realism remains a challenging problem with large rooms to explore. We hope that our benchmark and proposed solutions can serve as a foundation for future work moving from naive content editing toward physically consistent realism.

1 INTRODUCTION

Image-editing benchmarks often emphasize semantic fidelity while overlooking physical effects, leaving the field’s progress toward physically realistic edits unclear. PICABench, PICAEval, and PICA-100K address this gap through a physics-focused benchmark, region-grounded evaluation, and video-derived training data.

1 INTRODUCTION

  • Physical realism remains challenging: instruction-aligned edits can still produce unharmonized lighting, deformation, or implausible state transitions.The paper identifies correct physical effects, including interactions with lighting, shadows, and object support, as part of editing realism.Source: §1 · p. 1, Figure 1 · p. 2
  • PICABench organizes physical realism into three dimensions and eight checkable sub-dimensions, covering common editing operations and physical effects.Optics covers light propagation, reflection, refraction, and light-source effects; mechanics covers deformation and causality; state transition covers global and local changes.Source: §1 · pp. 1–3
  • PICAEval uses targeted per-example questions and human-annotated key regions to assess physical correctness, yielding high agreement with human assessments.The questions align with physical sub-dimensions and direct attention to evidence such as reflection surfaces and contact interfaces.Source: §1 · p. 2
  • PICA-100K is a synthetic dataset of 100k editing examples constructed from videos, and fine-tuning on it improves physical realism without sacrificing semantic quality.Its pipeline uses a text-to-image model as a scene renderer and an image-to-video model as a state-transition simulator, then extracts coherent editing pairs and recalibrates instructions.Source: §1 · pp. 2–3
  • Current models still struggle to produce physically realistic edits across common operations, despite PICABench distinguishing models by physical awareness.The paper benchmarks 11 open- and closed-source models across diverse architectures and scales.Source: §1 · p. 3

2 RELATED WORK

Instruction-based editing models have improved instruction alignment and visual quality, but many prioritize semantic and perceptual results over physical constraints. Existing evaluation approaches can miss physically implausible edits, motivating physics-aware, region-grounded assessment.

2.1 INSTRUCTION-BASED IMAGE EDITING MODELS

  • Despite progress in instruction alignment and visual fidelity, editing models often neglect physical constraints, producing unrealistic shadows, refractions, or deformations.Unified frameworks support diverse tasks, while closed-source systems show strong user-intent alignment and visual fidelity.Source: §2.1 · p. 3

2.2 INSTRUCTION-BASED IMAGE EDITING BENCHMARKS

  • VLM-as-a-Judge benchmarks assess instruction adherence and perceptual quality, but can overlook implausible physical effects or hallucinate judgments.The paper introduces PICAEval as a region-grounded, QA-based metric for localized, interpretable physical-consistency assessment.Source: §2.2 · p. 3

3 METHOD

PICABench evaluates physical realism across three dimensions and eight checkable sub-dimensions, and PICAEval assesses edits through localized physical-plausibility questions. The paper also constructs PICA-100K, a synthetic video-derived dataset for training physics-aware editors.

3.1 PICABENCH

  • PICABench covers 900 editing samples across Optics, Mechanics, and State Transition, divided into eight sub-dimensions with concrete criteria.Its taxonomy includes optical effects, mechanical plausibility, and realistic state transitions.Source: §3.1 · p. 4

3.2 DATA CURATION

  • The curation pipeline pairs natural images with physics-relevant instructions, expanded into superficial, intermediate, and explicit prompts and manually reviewed.The prompt levels probe intrinsic physical priors, provide physical-rule rationales, or specify expected edit outcomes.Source: §3.2 · pp. 4–5

3.3 PICAEVAL

  • PICAEval uses human-annotated regions and localized questions to assess physical plausibility, with spatial grounding, decomposition, and concrete checks intended to improve evaluation.Each entry’s question set is manually reviewed; the score is the proportion of answers exactly matching reference labels.Source: §3.3 · pp. 5–6

3.4 STRONG BASELINE: LEARNING PHYSICAL REALISM FROM VIDEOS

  • PICA-100K uses generated scenes and video-synthesized physical transformations to produce controllable training supervision, avoiding the expense and labor of large real-world datasets.The pipeline uses text-to-image rendering, motion-oriented edit instructions, and video models; first and last frames form image pairs labeled by GPT-5.Source: §3.4 · pp. 6–7
  • PICA-100K contains 105,085 instruction-based editing samples across eight physics categories; experiments report that its pipeline generates high-quality data and significantly enhances model performance on physics-aware image-editing tasks.The work targets physical realism and controllable generation that keeps non-edited regions stable.Source: §3.4 · p. 7

4 EXPERIMENT

Across the tested models, physical realism remains far from saturated, while video-derived fine-tuning improves results on two editing backbones. The authors also report that PICAEval aligns with human judgments better than its baseline.

4.1 EVALUATION DETAILS

  • Evaluation uses default model settings and superficial prompts; PICAEval crops annotated edit regions before VLM assessment, while consistency is measured outside the edited area.The cropped regions are resized to a maximum long side of 1024 pixels, and results use GPT-5 and Qwen2.5-VL–72B evaluators.Source: §4.1 · pp. 8–9

4.2 BENCHMARK RESULTS

  • Several closed-source models exceed 60 overall accuracy, but all open-source models remain below 60, leaving substantial headroom for physically realistic editing.The passage reports this comparison for the benchmark results overall.Source: §4.2 · p. 9
  • Unified open-source architectures consistently underperform dedicated editing models, while accuracy increases with prompt detail and consistency can decrease as a trade-off.The authors speculate that weak internalized physics may limit gains from intermediate prompts.Source: §4.2 · p. 9
  • 58.29%→62.13% overall accuracy for Qwen-Image-Edit after PICA-100K fine-tuning, with consistency rising from 19.43dB to 21.91dB.For Flux.1-Kontext, fine-tuning improves overall accuracy by +1.71% and consistency from 24.57dB to 25.23dB.Source: §4.2 · p. 9
  • MIRA400K-trained models perform worse in overall accuracy than the proposed synthetic-data pipeline’s results.The comparison uses a real-video-based dataset constructed from 400K edited images.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 VALIDITY OF PICAEVAL

  • PICAEval correlates more highly with human Elo judgments than the baseline, which omits edit-region annotations.The authors attribute the stronger alignment to per-case, region-level annotations and designed questions mitigating VLM hallucinations.Source: §4.3 · pp. 9–10

5 LIMITATIONS AND FUTURE DIRECTIONS

The framework has limitations in dataset scale and generation complexity, training strategy, and supported input conditioning.

5 LIMITATIONS AND FUTURE DIRECTIONS

  • PICA-100K uses a relatively simple generation pipeline and remains limited in scale.The authors describe the dataset as effective despite these constraints.Source: §5 · p. 10
  • Training uses supervised finetuning (SFT), which brings modest gains but may underuse the data’s potential.Source: §5 · p. 10
  • The current framework supports only single-image inputs, not multi-image or multi-condition contexts.The authors aim to extend it to more expressive conditioning formats.Source: §5 · p. 10

6 CONCLUSION

The paper presents PICABench and PICAEval for assessing physical realism, reports that current models remain far from physically realistic edits, and finds that fine-tuning on PICA-100K significantly boosts physical consistency.

6 CONCLUSION

  • Current image-editing models remain far from producing physically realistic edits.Source: §6 · p. 10
  • PICABench evaluates physical realism, while PICAEval provides a region-grounded, QA-based metric for fine-grained assessment.Source: §6 · p. 10
  • Fine-tuning on the video-derived synthetic dataset PICA-100K significantly boosts physical consistency.Source: §6 · p. 10

A.1 TASK DEFINITION

The task definitions organize physical consistency into optics, mechanics, and state transitions, each specified through concrete criteria for judging edited images.

A.1.1 OPTICS

  • Optics checks shadow geometry, view-dependent reflections, coherent refraction, and whether introduced light sources fit the scene’s illumination.Criteria include shadow direction, length, softness, and occlusion; preserved edge continuity through transparent media; and color casts, penumbra, and brightness falloff for new light sources.Source: §A.1.1 · p. 14

A.1.2 MECHANICS

  • Mechanics checks whether deformations respect material properties and whether contacts and supports remain stable under gravity.Rigid objects should not bend plastically, while edited objects should not float, interpenetrate, or rest in unstable equilibria.Source: §A.1.2 · p. 14

A.1.3 STATE TRANSITION

  • State transitions check that global changes propagate across the scene and local changes integrate with nearby context and causal cues.Examples include consistent illumination changes for day-to-night transitions and local light spill and reflections from flames.Source: §A.1.3 · p. 14

A.2 MORE BENCHMARK RESULTS

Across the reported results, most models score below 60 on PICABench, while more informative prompts and fine-tuning are associated with improved performance.

A.2 MORE BENCHMARK RESULTS

  • Most models score below 60 on PICABench-Superficial, indicating a substantial gap in physics-aware image generation.The reported pattern is similar to the one in Table 1.Source: §A.2 · p. 14, Table 4 · p. 15
  • The reported benchmark results include intermediate-prompt evaluations by GPT-5 and Qwen2.5-VL-72B, and explicit-prompt evaluations by GPT-5.Table 5 covers intermediate prompts with GPT-5; Table 7 covers intermediate prompts with Qwen2.5-VL-72B; Table 8 covers explicit prompts with GPT-5.Source: §A.2 · pp. 14–15, Table 5 · p. 15
  • Performance improves with more informative prompts, and the fine-tuned model consistently outperforms the baseline.Source: §A.2 · p. 15

A.3 DETAILS OF BENCHMARK METRICS

The metrics pair VLM answer accuracy with image consistency measured over non-edited regions. Results tables report accuracy and consistency alongside selected physical sub-dimensions.

A.3 DETAILS OF BENCHMARK METRICS

  • Accuracy is defined over annotated question-answer pairs using VLM predictions, reference answers, and an indicator function.N is the number of annotated QA pairs; the indicator tests whether the predicted and reference answers match.Source: §A.3 · p. 15
  • PICABench tables report Accuracy (%) and Consistency (dB) with selected physical sub-dimension scores.The captions identify Light propagation, Light Source Effects, Global State Transition, and Local State Transition among the reported categories.Source: Table 6 · p. 16, Table 7 · p. 16, Table 8 · p. 17
  • Consistency uses PSNR over non-edited pixels, comparing each source image with its edited image and averaging scores across samples.The metric description identifies the non-edited region as the area used for the per-sample consistency score.Source: §A.3 · pp. 16–17
  • The PSNR definition uses MAX for the maximum pixel value, such as 255 in an 8-bit image.The passage gives the example value for 8-bit images.Source: §A.3 · p. 17

A.4 DETAILED HUMAN EVALUATION PROTOCOL

The human evaluation compares model outputs in randomized pairwise trials, aggregates preferences with Elo ratings, and repeats rating updates to reduce order effects.

A.4 DETAILED HUMAN EVALUATION PROTOCOL

  • Each human-evaluation trial presents a reference image and two model outputs under one fixed instruction, asking which output better matches it.The A/B order is randomized, and annotators are vetted volunteers.Source: §A.4 · p. 17
  • The study evaluates 9 models at three difficulty levels, sampling 50 items per level and collecting 27,000 votes per split.The sampled items are stratified over the physics-law taxonomy; each item yields 36 model-pair comparisons, each judged by 5 annotators.Source: §A.4 · p. 17
  • Elo aggregates pairwise preferences by comparing a model's vote ratio with its expected win probability.The rating update uses a vote-count adjustment and a base step size K=24.Source: §A.4 · p. 17
  • The comparison stream is shuffled and Elo updates are rerun for 50 rounds to reduce order effects and improve stability.The final model Elo score is computed from these repeated updates.Source: §A.4 · p. 18

A.5 MORE VISUALIZATION

These appendices visualize model behavior on PICABench, show prompts and examples used in benchmark construction, and illustrate PICA-100K examples and prompt designs.

A.5 MORE VISUALIZATION

  • Across visualized PICABench cases, most models make superficial edits that miss the physics or fail to understand the instruction, while only a few, including the fine-tuned Flux.1-Kontext model, are physically plausible in most cases.Bagel can reason to the correct result in its chain-of-thought yet fail to execute it in the generated image.Source: §A.5 · p. 18

A.6 PROMPTS FOR PICABENCH

  • The appendix identifies separate prompts for generating PICABench edit instructions and evaluation question-answer pairs.Figure 16 is used for edit-instruction prompts, and Figure 17 for QA-generation prompts.Source: §A.6 · p. 18

B.1 EXAMPLE OF PICA-100K

  • PICA-100K examples focus on manifestations of physical laws, including a superficial instruction to remove a yellow chair.The examples are presented as source-and-edited pairs.Source: §B.1 · pp. 18–19

B.2 PROMPTS FOR PICA-100K

  • PICA-100K prompt examples span optics, mechanics, and state transitions, with figures showing superficial prompts for individual physical laws.The appendix also identifies prompts for generating videos from images and prompts for generating PICA-100K edit instructions.Source: §B.2 · p. 19, Figure 10 · p. 21, Figure 11 · p. 22
Loading 2510.17681v3…