Source-linked AI summary
TransPhy: Visual In-Context Learning for Physically Grounded Image Editing
Siyi Xie, Xuanke Shi, Jinsheng Quan, Haoran Tang, Zukai Chen, Lei Yang, Quan Wang
TL;DR
Physically grounded VICL must infer a demonstrated transformation, adapt it to a query’s physical context, and preserve irrelevant content, beyond appearance-level relation transfer. TransPhy decomposes this into physical-rule induction and transition-aligned rendering, and experiments report improved transfer, physical plausibility, and unseen-rule generalization.
Problem
Existing VICL methods provide limited support for transformations whose outcomes depend on material, geometry, interactions, and environmental conditions.
Method
TransPhy predicts the demonstrated rule and query-specific target state, then renders the result with token-wise mixture-of-experts adaptation guided by localized transition cues.
Results
TransPhy improves novel-instance transfer and unseen-rule generalization, producing more complete and physically plausible transformations with competitive perceptual quality.
Takeaways & Limitations
PhysVICL-74 provides 74 rules, 5,240 source–target pairs, and nearly 75K contexts for evaluating novel-instance transfer and unseen-rule generalization.
Takeaways & Limitations
The benchmark does not assess numerical dynamics or simulator-level physical accuracy.
Abstract
from arXiv · showhide
Visual demonstrations provide a natural interface for specifying image transformations that are difficult to describe exhaustively with text. However, existing visual in-context learning (VICL) methods primarily focus on appearance-level relation transfer and provide limited support for physically grounded transformations, whose outcomes depend on material properties, geometry, object interactions, and environmental conditions. Given a source--target exemplar pair and a query image, physically grounded VICL requires a model to infer the demonstrated transformation, adapt its effects to the query-specific scene context, and preserve rule-irrelevant content. We introduce PhysVICL-74, comprising 74 physically grounded transformation rules and 5,240 source--target image pairs that form nearly 75K training and evaluation contexts. Its benchmark split separately evaluates novel-instance transfer and unseen-rule generalization. We further propose TransPhy, a framework that decomposes physically grounded VICL into physical-rule induction and transition-aligned rendering. TransPhy first predicts the demonstrated rule and an explicit query-specific target-state description, and then synthesizes the target image through token-wise mixture-of-experts adaptation, with expert routing guided by localized transition cues. Experiments show that TransPhy improves physical-rule adherence, query consistency, and unseen-rule generalization over existing visual in-context editing methods.
1 Introduction
Physically grounded VICL asks models to infer transformations from visual exemplars, adapt them to query-specific physical context, and preserve unaffected content. TransPhy addresses this with explicit rule induction and transition-aligned rendering, while PhysVICL-74 evaluates transfer and generalization.
- Physically grounded transformations depend on material, geometry, interactions, and environment, so exemplar differences cannot simply be copied across queries.
- Existing VICL methods often encode exemplar relations globally, missing heterogeneous effects such as volume changes during melting.A model may generate a liquid puddle while leaving the solid volume unchanged.
- PhysVICL-74 contains 74 transformation rules, 5,240 source–target pairs, and nearly 75K contexts for novel-instance transfer and unseen-rule evaluation.
- TransPhy separates transferable-rule interpretation from query-conditioned realization through textual target-state descriptions and token-wise mixture-of-experts routing.
- TransPhy improves novel-instance transfer and unseen-rule generalization, producing more complete and physically plausible transformations with competitive perceptual quality.
2 Related Work
Related work establishes VICL for transferring visual relations and unified multimodal models for joint understanding and generation, but physically grounded editing remains difficult. TransPhy motivates explicit physical transformation interpretation and query-specific routing to avoid superficial exemplar copying.
- VICL image-editing methods transfer exemplar transformations through optimization, attention or feature manipulation, and learned transfer mechanisms.
- Physics-aware benchmarks study physical properties, relations, dynamics, and broader visual or knowledge-based reasoning, while editing methods address physical transitions and interactions.
- Unified multimodal models combine visual understanding and generation, providing a basis for comparing exemplars, reasoning about transformations, and generating edits.
- Direct generation may transfer superficial exemplar differences or incidental visual content, motivating explicit transformation interpretation and transition-aligned token-wise routing.
3 Task and Dataset Construction
PhysVICL-74 defines physically grounded transformations by their dependence on material, geometry, interactions, or environment and organizes 74 rules into three mechanism-based families. Its construction uses mined concepts, deduplication, ambiguity filtering, and human verification.
- Physically grounded VICL transfers an implicit transformation from an exemplar to a query while preserving rule-irrelevant content.
- The benchmark represents each target as one human-validated plausible realization because multiple physically plausible outcomes may exist.
- The 74 rules comprise Scene-Level, Object-Level, and Matter-Level families spanning scene conditions, mechanical responses, and material-state changes.
- Rule construction mines concepts and seed images from prior resources, then merges semantic duplicates and removes ambiguous or instance-specific edits.
- Dedicated human annotators review rule correctness, physical plausibility, and preservation of rule-irrelevant content, with failed samples regenerated.
4 Methodology
TransPhy decomposes physically grounded visual in-context editing into explicit rule induction and query-conditioned rendering. It combines semantic intermediates, token-wise expert adaptation, and transition-guided routing to preserve scene content while modeling spatially heterogeneous effects.
- Problem formulation: The task requires separating a transferable physical rule from exemplar-specific appearance, then adapting that rule to the query’s structure and context.The output should preserve the query’s identity, spatial layout, and rule-irrelevant content.
- Progressive physical rule induction and rendering: TransPhy’s understanding pathway predicts an explicit rule and query-specific target-state description before generation.These textual intermediates specify what transfers and how the transformation should manifest in the query.
- Progressive physical rule induction and rendering: Token-wise MoE-LoRA lets different spatial tokens invoke different low-rank rendering experts for transformed objects, interaction regions, secondary effects, and preserved content.This addresses the spatial heterogeneity that standard LoRA cannot model because it applies one low-rank update to every spatial token.
- Fine-grained transition alignment: ViT-derived feature differences provide localized transition evidence that guides router alignment toward transition-responsive positions.Feature differences are less sensitive than raw pixel differences to color shifts, generation noise, and local texture variations.
- Fine-grained transition alignment: STC aligns where expert responses should vary without assigning a predefined semantic identity to individual experts, while load balancing discourages expert collapse.The generation objective learns expert rendering behavior, STC provides spatial sensitivity, and the load-balancing objective encourages non-collapsed specialization.
5 Experiments
Experiments evaluate TransPhy on novel instances of seen rules and transformations unseen during training, using complementary metrics and qualitative comparisons. TransPhy shows stronger quantitative and qualitative performance across physical transition fidelity, content preservation, plausibility, and perceptual fidelity.
- Experimental Setup: PhysVICL-74 evaluates novel instances of 57 training-seen rules, 17 entirely unseen rules, and 20 sampled Relation252K rules.The benchmark uses disjoint exemplar pairs and queries for the novel-instance split.
- Experimental Setup: The evaluation compares TransPhy with FLUX.1-Fill-dev and BAGEL-MoE on seen rules, adding RelationAdapter, VisualCloze, and LoRWeB for unseen rules.BAGEL-MoE matches the backbone, adapters, data, and inference settings but removes STC-based router alignment.
- Quantitative Evaluation: TransPhy increases Object-Level TA from 3.03 to 3.39 and Matter-Level CP from 3.23 to 3.67 over BAGEL-MoE on seen transformations.TA, CP, and RP are GPT-5.6 scores, while CLIP-D and LPIPS are objective metrics.
- Quantitative Evaluation: On unseen transformations, TransPhy outperforms matched BAGEL-MoE on every metric in both settings and ranks first in six of ten metric–setting combinations.The results cover transition fidelity, query preservation, physical plausibility, and perceptual fidelity.
- Qualitative Evaluation: Qualitatively, TransPhy preserves query geometry on seen rules and transfers material, representation, and lighting effects with limited background changes on unseen rules.Competing methods are described as capturing coarse appearances, under-applying effects, altering backgrounds, or leaving queries nearly unchanged.
6 Ablation Studies
Ablations show that staged rule understanding and transition-token alignment are central to TransPhy, while moderate token selection balances precision and recall. Expert routing varies across layers and produces distinct localized effects, supporting non-redundant specialization.
- Training Design: Removing Stage 1 causes the largest drops in TA from 3.25 to 1.78 and RP from 3.36 to 1.87.The ablation conditions generation on a single fixed instruction instead of dedicated rule-understanding training.
- Training Design: Removing STC alignment reduces RP to 2.95 and increases LPIPS from 0.31 to 0.37.This indicates that expert alignment improves plausible and spatially faithful rendering.
- Expert Count: Increasing the expert count from 4 to 16 yields modest gains, with E = 8 giving the lowest LPIPS and E = 16 performing best on TA, CP, and RP.The reported analysis suggests the bottleneck may lie in the base model rather than expert capacity.
- Token Selection: ρ = 15% achieves the highest transition-token F1 of 67.05, balancing the precision favored by ρ = 5% against the noisy recall of ρ = 30%.Candidates are selected by ViT cosine difference and refined through merging and feature similarity.
- Expert Routing: Layer-wise routing varies across layers, while forced experts produce distinct localized effects, providing evidence of emergent non-redundant expert specialization.The comparison routes all generation tokens through one expert versus natural routing.
7 Conclusion
The paper formulates physically grounded VICL, introduces PhysVICL-74 and TransPhy, and reports improved performance and unseen-rule generalization. It positions these contributions as support for future research on visual rule induction and context-aware image editing.
- Contributions: PhysVICL-74 covers 74 transformation rules and approximately 75K exemplar–query contexts with novel-instance and unseen-rule evaluation protocols.The benchmark is designed for physically grounded transformations.
- Contributions: TransPhy combines physical-rule induction with transition-aligned token-wise expert adaptation for fine-grained, physically plausible synthesis.The framework targets transformation transfer from exemplars to new queries.
- Findings: Experiments report strong performance on physically grounded transformations and improved generalization to unseen rules over existing VICL methods.The conclusion frames these results as support for future study of visual rule induction and context-aware image editing.