Source-linked AI summary
Alterbute: Editing Intrinsic Attributes of Objects in Images
Tal Reiss, Daniel Winter, Matan Cohen, Alex Rav-Acha, Yael Pritch, Ariel Shamir, Yedid Hoshen
TL;DR
Alterbute addresses the difficulty of editing intrinsic object attributes while preserving identity and scene context. It combines relaxed diffusion training with VNE-based supervision and constrained inference, achieving state-of-the-art intrinsic attribute editing. The method also supports correlated multi-attribute edits when attributes are not in conflict.
Problem
Intrinsic attribute editing must change color, texture, material, or shape while preserving identity, but existing methods are either too coarse or too restrictive, and aligned training pairs are virtually nonexistent.
Method
Alterbute trains a diffusion model on intrinsic and extrinsic changes using identity references, attribute prompts, and scene context, then constrains inference with the original background and mask while using VNEs for supervision.
Results
Alterbute achieves state-of-the-art results in identity-preserving intrinsic attribute editing with a single unified model and can jointly edit non-conflicting intrinsic attributes.
Takeaways & Limitations
VNEs provide identity categories that preserve identity-defining features while allowing natural intrinsic variation for scalable supervision.
Takeaways & Limitations
Reshaping rigid objects remains challenging because geometry is often correlated with identity-defining features, sometimes producing unrealistic or unsuccessful shapes.
Abstract
from arXiv · showhide
We introduce Alterbute, a diffusion-based method for editing an object's intrinsic attributes in an image. We allow changing color, texture, material, and even the shape of an object, while preserving its perceived identity and scene context. Existing approaches either rely on unsupervised priors that often fail to preserve identity or use overly restrictive supervision that prevents meaningful intrinsic variations. Our method relies on: (i) a relaxed training objective that allows the model to change both intrinsic and extrinsic attributes conditioned on an identity reference image, a textual prompt describing the target intrinsic attributes, and a background image and object mask defining the extrinsic context. At inference, we restrict extrinsic changes by reusing the original background and object mask, thereby ensuring that only the desired intrinsic attributes are altered; (ii) Visual Named Entities (VNEs) - fine-grained visual identity categories (e.g., ''Porsche 911 Carrera'') that group objects sharing identity-defining features while allowing variation in intrinsic attributes. We use a vision-language model to automatically extract VNE labels and intrinsic attribute descriptions from a large public image dataset, enabling scalable, identity-preserving supervision. Alterbute outperforms existing methods on identity-preserving object intrinsic attribute editing.
1. Introduction
Alterbute addresses the challenge of changing intrinsic object attributes while preserving perceived identity and scene context. It combines a relaxed training objective with Visual Named Entities (VNEs) to enable scalable identity-preserving supervision.
- Intrinsic editing changes color, texture, material, or shape while attempting to preserve object identity, unlike extrinsic editing of pose, lighting, or background.
- Identity definitions trade off editability against preservation: coarse categories permit excessive variation, whereas instance-level identity permits almost none.
- Existing methods typically preserve either a coarse identity through unsupervised diffusion priors or an overly restrictive identity through subject-driven supervision.
- Alterbute relaxes training to learn intrinsic and extrinsic edits, then constrains inference by reusing the original extrinsic context.
- VNEs group visually similar instances sharing a common name while allowing intrinsic variation, providing a middle-ground representation of object identity.
- A vision-language model extracts VNE labels and intrinsic attribute descriptions from a large public dataset, producing scalable training supervision.
- Alterbute edits all intrinsic attributes with one model while preserving identity, achieving state-of-the-art results on challenging intrinsic attribute edits.
- The authors disclose that Gemini supported automated data curation and evaluation judging, but not model training or inference.
2. Related Works
Prior diffusion-based editing methods support broad image manipulation but struggle with intrinsic attribute changes and identity preservation. Alterbute targets this gap with reference-based, VNE-guided intrinsic editing.
- Diffusion models support high-fidelity text-to-image generation, with extensions for inpainting, style transfer, and image-to-image editing.
- Instruction-based methods offer prompt-driven edits but struggle with intrinsic attribute manipulation.
- Personalization methods can generate high-fidelity images but require per-object optimization and do not support intrinsic edits.
- Alterbute differs by using a tuning-free approach with an SDXL backbone and VNE-based identity-preserving attribute editing.
- Other methods address selected properties such as albedo, roughness, material, or texture, rather than the full intrinsic-attribute scope.
3. Method
Alterbute is a diffusion-based approach for editing an object's intrinsic attributes while preserving its identity and extrinsic scene context. It combines relaxed supervised training, Visual Named Entities, and controlled inference conditioning.
- Alterbute targets edits to color, texture, material, or shape while preserving object identity and extrinsic scene properties.
- The method relaxes training to permit both intrinsic and extrinsic edits, making supervised learning feasible despite scarce paired images with fixed scenes.At inference, the model reuses the original background and object mask to constrain extrinsic changes.
- Visual Named Entities are fine-grained identity categories that balance editability with identity preservation between coarse categories and instance-level identifiers.Examples include “Porsche 911 Carrera” and “iPhone 16 Pro”; VNEs group visually similar objects while allowing intrinsic and extrinsic variation.
- Gemini assigns VNE labels and intrinsic attribute descriptions to OpenImages objects, producing scalable clusters and textual training prompts without manual labeling.Unlabeled instances are filtered out, and extracted attributes include color, texture, material, and shape.
- The model fine-tunes a pretrained latent diffusion UNet conditioned on a masked reference object, an attribute prompt, and a background image with a binary mask.The reference image is sampled from the same VNE cluster, while the network predicts the denoised target image using a standard diffusion L2 loss.
- Qualitative comparisons show Alterbute applying target attributes while maintaining object identity and scene context, whereas baselines often fail at editing or identity preservation.
4. Experiments
Alterbute is evaluated on intrinsic attribute editing using qualitative comparisons, a 30-object benchmark, user preferences, VNE analysis, and conditioning and identity-definition ablations. Results indicate strong identity-preserving edits across attributes, while VNE-based references and grid conditioning support controlled editing.
- Evaluation setup: The evaluation uses a dedicated set of 30 objects paired with multiple intrinsic-attribute prompts because no standard benchmark exists.The set includes popular objects and underrepresented categories such as furniture and vehicles.
- Qualitative comparisons: Alterbute modifies requested intrinsic attributes while preserving object identity and scene context, outperforming general-purpose editors qualitatively.It is also reported as the only compared method capable of identity-preserving object reshaping.
- VNE analysis: The automated pipeline produces 69,744 VNE clusters containing 1,079,442 labeled images from approximately 9 million OpenImages images and 16 million object boxes.Cluster sizes are heavy-tailed, with most clusters small and a few containing thousands of instances.
- Conditioning ablation: Channel-wise identity conditioning collapses to near-identity outputs, showing that spatial grid concatenation and cross-image self-attention are structurally necessary for identity transfer.Without cross-image self-attention, the model fails to apply requested attribute edits.
- Identity ablation: VNE-based identity references provide identity-preserving yet attribute-diverse supervision, whereas DINOv2, instance retrieval, and in-place references often damage editability or identity conditioning.DINOv2 may group identity-distinct objects, while instance retrieval often lacks intrinsic variation.
5. Discussion & Limitations
Alterbute supports targeted and multi-attribute intrinsic edits, but its flexibility has practical boundaries. Coarse masks can introduce background artifacts, rigid-object reshaping can produce unrealistic geometries, VNE labeling may be biased, and the benchmark remains limited in scale.
- Editing control: At inference, a single key-value prompt edits one specified intrinsic attribute while unspecified attributes are preserved from the reference.Training randomly omits prompts in 10% of examples to support this targeted control.
- Multi-attribute editing: Alterbute can edit multiple intrinsic attributes simultaneously when their natural dependencies do not create contradictory combinations.The model captures attribute correlations from training data, such as material-color incompatibilities.
- Masking limitation: Coarse bounding-box masks improve reshaping flexibility but may cause slight background inconsistencies within the masked region.Pre-removing the object is suggested as a remedy because it supplies a clean background without masking.
- Shape limitation: Rigid-object reshaping remains challenging because geometry is often correlated with identity-defining features, sometimes producing unrealistic or unintended shapes.Generated geometry may also fail to reflect the intended change.
- Annotation limitation: Gemini-based VNE labeling may introduce annotation bias because categories prominent in its training distribution can receive better coverage.High-confidence filtering mitigates spurious labels, and the prompting pipeline is replicable.
- Benchmark limitation: The evaluation benchmark covers 30 objects and 100 editing cases, but its limited scale leaves expansion to more objects, combinations, and scenarios as future work.The benchmark was designed to cover all four attribute types and long-tail categories.
6. Conclusion
Alterbute edits color, texture, material, and shape while preserving object identity and scene context. Its relaxed training objective and automatically extracted Visual Named Entities provide scalable supervision, and the unified model achieves state-of-the-art intrinsic attribute editing.
- Alterbute edits color, texture, material, and shape while preserving object identity and scene context.
- The method combines relaxed training over intrinsic and extrinsic changes with inference constrained to intrinsic edits.
- Visual Named Entities are automatically extracted visual identity categories whose intrinsic variation supports supervision.
- A single unified model achieves state-of-the-art results in intrinsic attribute editing.
Impact Statement
Alterbute is evaluated against open-source general-purpose and specialized editors, with statistical testing and safeguards noted for responsible use.
- Impact and safety: The method targets non-human objects and is not designed or trained for human faces or biometric features.This limits applicability to identity fraud or deepfake generation.
- Evaluation setup: Alterbute is compared with general-purpose and specialized intrinsic attribute editors using publicly available models and recommended settings.No baseline was fine-tuned on task-specific data.
- Compared methods: The baseline set includes FlowEdit, InstructPix2Pix, OmniGen, UltraEdit, Diptych, MaterialFusion, and MimicBrush.These methods cover text-driven editing, unified diffusion, fine-grained editing, inpainting, material transfer, and localized attribute transfer.
- Statistical analysis: The study reports statistical significance testing for the evaluation results.Table 3 uses a binomial statistical test, although the supplied passage does not include the p-value.
B. Quantitative Evaluation
The quantitative evaluation combines user preferences, VLM judgments, conventional identity and text-alignment metrics, and task-specific assessment to evaluate intrinsic attribute editing.
- User study: Participants strongly preferred Alterbute over all competing baselines, with 3,320 total responses collected.The study produced 500 ratings per general-purpose baseline and 410 per attribute-specific method.
- VLM evaluation: VLM-based evaluation used Gemini, GPT-4o, and Claude 3.7 Sonnet on the same image pairs and question as the user study.Five ratings per sample yielded 3,320 total comparison decisions.
- Metrics: Conventional evaluation measures identity preservation with DINO and CLIP-I and target attribute alignment with CLIP-T.Identity preservation uses average pairwise cosine similarity between the edited object and identity reference image.
- Metrics: Single conventional metrics can be misleading because failed edits may preserve similarity while identity-discarding outputs may achieve high CLIP-T.The paper therefore combines qualitative comparisons, user studies, and VLM assessments for a more task-aligned evaluation.
C.1. Comparison with Commercial Systems
Alterbute is compared with two concurrent commercial editing systems using the VLM-based evaluation protocol.
- Commercial comparison: Alterbute is on par with Qwen-image-editing and clearly outperforms FluxKontext.The comparison uses VLM-based evaluation and reports win rates in Table 5.
- Commercial comparison: The results indicate competitiveness with state-of-the-art commercial systems despite Alterbute being a fully transparent academic method.This conclusion is stated by the authors alongside the commercial-system comparison.
C.2. Per-Attribute Breakdown
Alterbute’s win rates remain consistent across color, texture, material, and shape editing, with shape edits performing best against baselines.
- Per-attribute results: Performance is consistent across all four intrinsic attribute categories.The per-attribute breakdown reports win rates for all four categories in Table 6.
- Per-attribute results: Shape edits achieve the highest win rates among the evaluated attribute types.The authors suggest this may reflect that shape changes are most challenging for baselines.
C.3. Compute Efficiency
Alterbute remains effective when training is shortened to 50K steps, retaining strong win rates against all baselines. Table 7 evaluates this compute-efficiency trade-off using win rate (%).
- C.3. Compute Efficiency: 75–78% win rate: Alterbute still wins comparisons against all baselines at 50K training steps.This represents a drop of only ∼7 percentage points from the full 100K-step model.
- C.3. Compute Efficiency: 57–61% preference: VLM evaluators only marginally prefer the 100K-step model over the 50K-step model.The comparison is directly between the two Alterbute training budgets.
- C.3. Compute Efficiency: A halved training budget has far less impact than the quality gap between Alterbute and the baselines.This comparison is reported alongside the 50K-step win rates and the 100K-versus-50K preference study.