Source-linked AI summary
Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency
M. Moein Esfahani, Sepehr Salem, Mohammed Alser, Vince Calhoun
TL;DR
Physical inconsistencies in edited images can be difficult to evaluate because they are sparse and real images rarely provide labels for the violated rule or relevant region. The paper introduces a paired-counterfactual benchmark with region and language supervision, then tests diagnostic models under controlled shifts. LLaVA-1.5-7B category macro-F1 falls from 64.0% on the standard split to 40.8% under intervention shift, indicating reliance on construction-specific cues alongside useful category learning.
Problem
Physical inconsistencies can be sparse and local, while real images rarely provide ground-truth labels for the violated physical rule or relevant evidence region.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The results support benchmark learnability but do not establish general physical reasoning, and performance remains sensitive to how violations are constructed.
Source: §7 · p. 8Removed contact shadows can leave boundary or halo cues, and auxiliary-render composites may alter more than the intended relation despite mitigation.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Modern image editing models can satisfy a text instruction while breaking the physics of the edited scene. A new object may cast no shadow, a mirror may fail to reflect visible geometry, or an object may float above a surface that should support it. We study physical plausibility diagnosis, detecting whether an edited image violates scene physics, naming the violation type, localizing the affected region, and explaining the failure in language. We introduce a counterfactual benchmark whose controlled synthetic component uses Mitsuba~3 to generate 5,500 images from 500 scene families. Each family contains one clean image and ten matched violations involving shadows, reflection, support, surface response, and occlusion. The renderer pipeline provides category labels, affected-region masks and boxes, scene metadata, and explanation targets. We use LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL3.5-8B as diagnostic baselines rather than proposed methods. On a 1,650-image synthetic test set, the adapted baselines reach 64.0--67.8\% category macro-F1 on standard held-out scenes. For LLaVA-1.5-7B, category macro-F1 falls from 64.0\% on the standard split to 40.8\% under intervention shift. This gap shows that high in-distribution accuracy partly reflects cues tied to rendering and counterfactual construction.
1 Introduction
Physical inconsistencies can be visually subtle, and realistic appearance alone does not establish that scene relations are physically consistent. The benchmark uses controlled counterfactuals to evaluate whether models identify and localize violations rather than rely on rendering-specific cues.
1 Introduction
- The benchmark tests whether models identify physical violations and localize affected regions instead of relying on surface appearance alone.Relevant constraints include shadows, reflection, support and contact, surface response, and depth ordering.Source: §1 · pp. 1–2
- Paired counterfactual scenes provide category labels, boxes, masks, and physical metadata while holding the remaining scene configuration fixed.This setup enables tests of whether models recognize physical constraints or exploit renderer- and intervention-specific shortcuts.Source: §1 · pp. 1–2
- The benchmark separates held-out-scene recognition from appearance and violation-generation shifts, and uses PICABench for evaluation on physics-aware edited images.PICABench provides region-grounded annotations and questions across optical, mechanical, and state-transition effects.Source: §1 · p. 2
- The study introduces 500 scene families and 5,500 images with renderer-derived supervision, alongside diagnostic baselines rather than a new model or training method.The listed baselines include CLIP, LLaVA-1.5, Qwen2.5-VL, and InternVL3.5.Source: §1 · p. 3
2 Related Work
Prior work evaluates broad physical reasoning, image-editing realism, and manipulation detection, but the paper focuses on controlled physical inconsistencies in static images. Its paired counterfactuals provide exact intervention metadata and spatial supervision for testing transfer beyond the rendering procedure.
2 Related Work
- Broad physical-reasoning benchmarks motivate dedicated evaluation, while this work targets static-image inconsistencies with paired counterfactuals and spatial grounding.Related work also covers spatial plausibility and physical implausibility in generated videos.Source: §2 · p. 3
- Physics-aware editing benchmarks show that instruction following and perceptual quality alone are insufficient to evaluate edited-image realism.PICABench uses region-grounded questions; Re-Edit and PhyEditBench provide other reasoning-aware or physics-aware editing evaluations.Source: §2 · p. 3
- Manipulation detection may localize changes, but this task tests a named physical constraint and grounds the region affected by a controlled intervention.The paper distinguishes physical-consistency diagnosis from authenticity detection.Source: §2 · pp. 3–4
- Paired counterfactual rendering supplies exact labels and spatial supervision, while separate appearance and intervention shifts test whether cues transfer beyond the construction procedure.A change in the intervention mechanism is the stricter test of recognizing the intended physical relation.Source: §2 · p. 4
3 Task Definition
The task evaluates whether visible evidence in a static RGB image is consistent with specified, image-observable physical constraints. For inconsistent images, it requires identifying the violated constraint and localizing the affected region, without claiming to determine all physical possibility.
3 Task Definition
- The model output comprises a physical-consistency label, predicted constraint, normalized affected-region box, and natural-language explanation.Clean images receive the consistent label and no violation category.Source: §3 · p. 4
- The benchmark covers visible constraints involving illumination and shadows, reflection, support and contact, surface response, and depth ordering.It evaluates consistency with the physical assumptions defined for each benchmark group, not possibility under every unobserved scene configuration.Source: §3 · p. 4
- Ten fine-grained labels span five physical families, and some closely related label pairs are not assumed to be independent.Examples include wrong shadow direction versus shadow illumination inconsistency, and floating object versus impossible occlusion.Source: §3 · p. 4
- Affected-region targets mark an anomalous introduced cue or the expected location of a missing cue, but may omit contextual evidence needed for diagnosis.The paper distinguishes affected-region localization from complete evidence grounding.Source: §3 · p. 4
4 Counterfactual Physical Consistency Benchmark
The benchmark pairs clean scenes with controlled physical violations and evaluates whether models generalize beyond the appearance and construction procedures used to create them. Renderer-derived annotations localize intervention-affected regions, which are not necessarily complete evidence for diagnosis.
Controlled scene construction
- The controlled subset contains 500 scene families, each with one clean image and ten counterfactuals; every family contributes all 11 labels.Scene families are grouped when constructing partitions, and the scored set uses a category-specific clean reference where preconditions differ.Source: §4 · pp. 4–5, Table 1 · p. 5
Matched physical counterfactuals
- Ten violation categories alter shadows, reflections, support, surface response, or occlusion while preserving unrelated scene parameters as closely as possible.Interventions change scene parameters or composite auxiliary renders; examples include lifted objects, changed illumination, and replaced reflected content.Source: §4 · pp. 4–5
Appearance and intervention shifts
- Appearance shift holds out geometries, materials, and lighting; intervention shift holds out violation-construction mechanisms while matching scene appearance.Both shifts keep scene families disjoint and categories balanced, and match affected-region area and counterfactual effect size across mechanisms.Source: §4 · p. 5
- Composited examples may retain halos, color changes, or other editing cues, so intervention-shift results and audits are required but cannot prove every shortcut is removed.The study audits difference images and reports performance by construction mechanism.Source: §4 · p. 5
Renderer-derived annotations
- The pipeline derives labels, scene and intervention metadata, explanations, affected-region masks, and tight boxes using geometry passes, render differences, or known compositing regions.The affected region marks pixels changed by construction, not necessarily all diagnostic evidence; for a floating object, the air gap may matter too.Source: §4 · p. 5
5 Benchmark Models and Adaptation
Three existing vision-language models are adapted as diagnostic baselines, not introduced as a new architecture or learning method. Training targets structured diagnoses and boxes, with synthetic-only and mixed-source variants evaluated separately.
Model baselines and adaptation
- LLaVA-1.5-7B, Qwen2.5-VL-7B-Instruct, and InternVL3.5-8B are evaluated as baselines, with each pretrained model also adapted using LoRA.The models provide different visual encoders, image-resolution policies, and language backbones at a similar scale.Source: §5 · p. 6
Training targets
- Training examples pair an image with an instruction, and targets specify consistency, the physical constraint, the affected region, and an explanation.Prompt tokens are excluded from the training loss.Source: §5 · p. 6
- The spatial output is a normalized (xmin, ymin, xmax, ymax) box; renderer masks define and validate the target but are not model outputs.Source: §5 · p. 6
Training variants
- The synthetic-only variant tests zero-shot PICABench transfer, while the mixed-source variant additionally adapts on PICA-20K and is evaluated separately.Both variants use the same response format for their training targets.Source: §5 · p. 6
6 Experiments
Across distribution shifts, LLaVA-1.5-7B loses more category macro-F1 when violation-generation mechanisms change than when scene appearance changes. Standard held-out results show adapted models can learn the labels, but that score alone does not establish physical reasoning.
Setup
- The evaluation separates unseen-scene recognition, appearance shift, intervention shift, and native PICABench questions, with synthetic-only transfer and PICA-20K adaptation reported separately.The combined synthetic test has 1,650 images from 150 unseen scene families, split evenly across the three synthetic suites.Source: §6 · p. 6
Baselines
- The baseline comparison includes majority-class, zero-shot CLIP, zero-shot LLaVA, and three LoRA-adapted VLM probes.The majority baseline is a trivial lower-bound reference under class imbalance; CLIP uses category-description similarity without task-specific training.Source: §6 · p. 6
Generalization beyond held-out scenes
- 23.2 category-F1 points are lost under intervention shift, compared with 11.6 points under appearance shift for LLaVA-1.5-7B.The larger intervention gap indicates substantial sensitivity to how a violation is constructed.Source: Table 3 · p. 7, §6 · p. 7
Standard held-out results
- 67.8% category macro-F1 is the strongest standard held-out result, achieved by adapted Qwen2.5-VL; this result alone does not show physical reasoning.Zero-shot CLIP and LLaVA perform poorly on the 11-class task, and zero-shot LLaVA has 2.0% clean recall.Source: §6 · p. 7, Table 2 · p. 7
Localization and explanation evaluation
- Box IoU measures localization of the intervention-associated region, not complete causal evidence grounding; explanation correctness is rated independently by three raters.Raters use a three-level rubric, with substantial agreement (Fleiss’ κ = 0.78).Source: §6 · p. 7
7 Discussion, Limitations, and Ethics
Adapted models fit several rendered violation labels but lose accuracy under controlled changes, so the results support benchmark learnability rather than general physical reasoning. The discussion also identifies limits in localization targets and synthetic cues, and cautions against treating violation scores as evidence of malicious editing.
7 Discussion, Limitations, and Ethics
- Models fit several rendered violation labels but do not retain that accuracy under controlled changes; direct visual evidence is easier than closely related sub-types.The authors characterize this as evidence of benchmark learnability, not general physical reasoning.Source: §7 · p. 8
- 23.2 points: LLaVA’s category macro-F1 gap under intervention shift shows that adaptation still learns cues tied to counterfactual construction, not just physical categories.Text-image similarity alone reaches 1.9% category macro-F1 on the standard split; PICA-20K adaptation improves external box IoU by 12.1 points and QA correctness by 10.5 points, but both remain below the synthetic standard split.Source: §7 · p. 8
- Affected-region masks provide reproducible localization targets, but may omit cues required for causal diagnosis; PICABench’s broader taxonomy is not interchangeable with the synthetic labels.Explanations are evaluated separately from intervention-associated localization.Source: §7 · p. 8
- Paired construction and area matching reduce, but do not eliminate, boundary or halo cues and unintended changes in some auxiliary-render composites.The authors make intervention-shift results and mechanism-level breakdowns central because of these residual risks.Source: §7 · p. 8
- A high violation score is not evidence that an image was forged, generated, or maliciously edited; the system is positioned for pipeline diagnosis and dataset auditing.It is not an authenticity oracle.Source: §7 · p. 8
8 Conclusion
The conclusion presents a counterfactual benchmark and finds that adapted models learn useful physical categories while remaining sensitive to construction-specific cues. The related-work discussion distinguishes this physical-relation benchmark from methods focused on authenticity or manipulation.
8 Conclusion
- LLaVA’s category macro-F1 drops from 64.0% to 40.8% when the intervention mechanism changes, indicating reliance on construction-specific cues alongside useful category learning.The conclusion describes external adaptation as helpful but says a large transfer gap remains.Source: §8 · p. 8
- The benchmark provides 5,500 clean and violated images from 500 scene families, with category, region, metadata, and language supervision.The conclusion frames the study as a counterfactual benchmark for grounded physical consistency.Source: §8 · p. 8
- Future work on internal representations, causal interventions on model features, and explanation faithfulness lies outside this study.Source: §8 · p. 8
A Additional Related Work
- Unlike authenticity or manipulation methods, this benchmark labels the physical relation changed by a controlled intervention.Source: A Additional Related Work · p. 10
C Baseline and Evaluation Details
The baseline and evaluation details describe the rendering conditions, model adaptation setup, released records, and human reference annotations. They also identify the pilot results table as a per-category recall and F1 breakdown.
C Baseline and Evaluation Details
- Figure 2 varies object geometry, camera view, lighting, and scene appearance across clean and violated examples, while recording visible artifacts such as halos.The panel supports visual auditing rather than hiding hard boundaries or unrelated color changes.Source: Figure 2 · p. 11
- Rendered scenes use a point emitter with weak constant environment lighting, 30°–70° light elevation, eight azimuth directions, and a 38° horizontal camera field of view.Images use 150 samples per pixel, path depth 8, and a box reconstruction filter.Source: C Baseline and Evaluation Details · p. 11
- The model probe freezes the visual encoder and applies LoRA to language-attention projections, using a shared response schema, split, stopping rule, and parser.LLaVA training uses rank 32, alpha 64, dropout 0.05, and three epochs; the passage also specifies optimizer and hardware settings.Source: C Baseline and Evaluation Details · p. 11
- PICA-20K records retain image paths, instructions, physics category and law, operation, difficulty, grounded question–answer pairs, and edit-area boxes.The selection file reports category counts and source identities to support reconstruction without leakage.Source: C Baseline and Evaluation Details · p. 11
- Table 5 reports per-category recall and F1 for the 1,650-image development pilot, while the human reference subset contains 220 images with 20 examples per label.Three raters provide binary decisions, fine categories, affected-region boxes, and short reasons; agreement is assessed for labels and boxes.Source: Table 5 · p. 12, C Baseline and Evaluation Details · p. 12
D Development-Pilot Category Analysis
The development pilot used an earlier in-distribution pool to expose fine-category stress cases. Its confusion matrix highlights two close category pairs that motivate operational definitions.
D Development-Pilot Category Analysis
- The earlier LLaVA-1.5-7B + LoRA development pool contains 1,650 in-distribution images and is not part of the headline comparison.It is retained to reveal intended fine-category stress cases.Source: D Development-Pilot Category Analysis · p. 12
- Shadow–illumination inconsistency is absorbed by wrong shadow direction in the development-pilot confusion matrix.The close pair motivates operational definitions in Table 4.Source: Figure 3 · p. 13
- Impossible occlusion is absorbed by floating object in the development-pilot confusion matrix.The close pair motivates operational definitions in Table 4.Source: Figure 3 · p. 13