Source-linked AI summary
CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing
Yucheng Wang, Zedong Wang, Yuetong Wu, Yue Ma, Dan Xu
TL;DR
Unified image editors can mishandle heterogeneous, conflicting conditions because static fusion does not adapt computation to each modality. CARE-Edit introduces timestep- and condition-aware routing across specialized experts, and reports strong performance across contextual editing tasks while reducing task interference; its scope remains limited by additional hyperparameters and a finite expert set.
Problem
Fixed backbones and static multimodal fusion cannot adaptively allocate capacity to heterogeneous conditions, contributing to boundary bleeding, identity or style drift, and inconsistent multi-condition behavior.
Method
CARE-Edit uses a latent-attention router with sparse top-K selection to dispatch tokens among Text, Mask, Reference, and Base experts, complemented by Mask Repaint and Latent Mixture.
Results
CARE-Edit reports strong results across erasure, replacement, text-driven editing, and localized style transfer, improving faithfulness, boundary cleanliness, and identity/style preservation over unified editors and swapstyle pipelines.
Takeaways & Limitations
Dynamic, condition-aware experts are reported to resolve multi-condition conflicts and reduce task interference in versatile image editing.
Takeaways & Limitations
CARE-Edit introduces additional hyperparameters such as top-K, and its expert set does not yet cover all broader edit types.
Abstract
from arXiv · showhide
Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs photometric). In particular, prevalent ControlNet and OmniControl variants combine multiple conditioning signals (e.g., text, mask, reference) via static concatenation or additive adapters which cannot dynamically prioritize or suppress conflicting modalities, thus resulting in artifacts like color bleeding across mask boundaries, identity or style drift, and unpredictable behavior under multi-condition inputs. To address this, we propose Condition-Aware Routing of Experts (CARE-Edit) that aligns model computation with specific editing competencies. At its core, a lightweight latent-attention router assigns encoded diffusion tokens to four specialized experts--Text, Mask, Reference, and Base--based on multi-modal conditions and diffusion timesteps: (i) a Mask Repaint module first refines coarse user-defined masks for precise spatial guidance; (ii) the router applies sparse top-K selection to dynamically allocate computation to the most relevant experts; (iii) a Latent Mixture module subsequently fuses expert outputs, coherently integrating semantic, spatial, and stylistic information to the base images. Experiments validate CARE-Edit's strong performance on contextual editing tasks, including erasure, replacement, text-driven edits, and style transfer. Empirical analysis further reveals task-specific behavior of specialized experts, showcasing the importance of dynamic, condition-aware processing to mitigate multi-condition conflicts.
1. Introduction
CARE-Edit addresses conflicts in unified image editing by dynamically routing multimodal conditions to specialized experts rather than processing all edits through a fixed shared pathway. Its modules and progressive training support diverse tasks, with reported gains in faithfulness, boundary cleanliness, and identity/style preservation.
- Fixed backbones and static condition fusion struggle with heterogeneous edits and conflicting signals, causing artifacts such as boundary color bleeding, identity or style drift, and inconsistent behavior.
- CARE-Edit routes tokens dynamically to Text, Mask, Reference, and Base experts using prompts, mask statistics, reference features, and diffusion timesteps.Sparse top-K routing enables token-wise and timestep-aware prioritization, while a persistent shared expert stabilizes training.
- Mask Repaint refines user-defined masks, Latent Mixture fuses expert outputs, and Routing Select activates only the most relevant experts for each token.
- Progressive training moves from single-task data to complex multi-task samples, allowing experts to develop specialization while mitigating mode collapse.
- CARE-Edit is evaluated on instruction-based and subject-driven tasks including erasure, replacement, text-driven edits, and localized style transfer.
- Reported results improve edit faithfulness, boundary cleanliness, and identity/style preservation over unified editors and swapstyle pipelines.Training dynamics and task-expert relationship analyses support condition-aware processing as a way to reduce task interference.
2. Related Work
Prior image-editing methods commonly specialize in single-signal instruction or subject conditioning, while sparse diffusion MoE work motivates routed computation. CARE-Edit differs by using heterogeneous, timestep-aware experts to address conflicts among multiple editing conditions.
- Instruction-based Editing: Instruction-based editing methods cover global refinement, semantic steering, instruction following, localized control, and recent unified pipelines, but often struggle with conflicting multi-source constraints.
- Subject-driven Editing: Subject-driven methods learn subject or style concepts through embeddings or adapters, but risk overfitting and unintended edits outside target regions.
- Mixture-of-Experts for Image Editing: Sparse mixture-of-experts models scale capacity through routed specialization, and timestep-aware token routing has been shown effective in diffusion MoE models.
- Mixture-of-Experts for Image Editing: Unlike homogeneous expert applications, CARE-Edit uses heterogeneous Text, Mask, Reference, and Base experts for multimodal editing conflicts.
- Mixture-of-Experts for Image Editing: The router selectively activates experts along the denoising trajectory to allocate model capacity for different contextual-editing conditions.
3. Methodology
CARE-Edit unifies multimodal image editing with condition-aware routing over heterogeneous experts inside a diffusion transformer. Mask refinement, sparse token routing, expert fusion, and lightweight fine-tuning coordinate semantic, spatial, reference, and global editing demands.
- Multimodal representation: CARE-Edit encodes text, base image, reference image, and mask inputs into a shared latent token sequence for unified multimodal editing.The sequence is propagated through a DiT backbone, with text, image, and mask encoders producing modality-specific latent representations.
- Expert fusion and optimization: Latent Mixture fuses active expert outputs with per-token and per-timestep awareness, while only lightweight adapters, routing, and fusion components are trainable.LoRA-style fine-tuning adapts the pretrained DiT while keeping pretrained encoders frozen.
- Specialized experts: Four heterogeneous experts specialize in semantic synthesis, spatial boundary refinement, identity/style transfer, and global background coherence.The Text, Mask, Reference, and Base experts correspond to these distinct editing competencies.
- Condition-aware routing: A token-wise router combines local token content with global task context to assign condition-dependent expert probabilities during denoising.The router uses token keys, a task-condition query, annealed temperature, and exponential-moving-average logit smoothing to stabilize selection.
- Condition-aware routing: Top-K routing activates only the most relevant experts for each token, while residual aggregation and shared routing preserve stable expert influence.The method sets K to 3 and uses convex residual fusion plus a shared-expert fraction to support computational efficiency and representation continuity.
- Mask Repaint: Mask Repaint refines user-defined masks at every diffusion step using current latent and reference features to align soft boundaries with object contours.The latent-space update promotes smooth transitions between edited and preserved regions without explicit pixel-wise supervision.
4. Experiments
CARE-Edit is evaluated across instruction-based and subject-driven benchmarks, where it generally outperforms unified editors and shows specialized expert behavior. Ablations indicate that routing, Latent Mixture, and Mask Repaint contribute to performance.
- Experimental Setup: CARE-Edit is evaluated on EMU-Edit, MagicBrush, and DreamBench++, covering instruction-based and subject-driven contextual editing.The evaluation includes erasure, replacement, text-driven edits, localized style transfer, and multi-object settings.
- Instruction-based Editing: With approximately 120K training data, CARE-Edit achieves the best CLIPim, CLIPout, and DINO scores on EMU-Edit.On MagicBrush, it obtains the highest CLIPout and DINO scores while maintaining a competitive L1.
- Instruction-based Editing: CARE-Edit produces cleaner, more instruction-faithful edits with sharper boundaries and fewer artifacts than competing editors.These qualitative results visually confirm the reported quantitative comparisons.
- Subject-driven Contextual Editing: CARE-Edit achieves the best performance across all metrics on DreamBench++ in both single- and multiple-object settings.It slightly but consistently outperforms OmniGen2, preserving subject identity and structure during complex contextual changes.
- Ablation Studies: Removing expert routing causes a large performance drop, while disabling Latent Mixture or Mask Repaint also degrades results.The ablation isolates routing, expert aggregation, and mask refinement as important components for diverse editing behaviors and precise edits.
- Ablation Studies: K = 3 yields optimal routing results, while the small differences among K = 2, 3, and 4 indicate robustness.Using fewer experts may underexpress the model, whereas more experts slightly hurts performance because specialization is reduced.
- Task-Expert Analysis: The Base Expert remains active across tasks, while Mask specializes in structure-aware edits and Reference is heavily activated for style transfer.These patterns show distinct task demands being met by corresponding expert specialization.
5. Conclusion and Discussion
CARE-Edit uses heterogeneous experts and efficient routing to support versatile, high-fidelity multimodal editing while addressing multi-condition conflicts. The authors identify hyperparameter overhead and limited expert coverage as remaining boundaries.
- Conclusion: CARE-Edit addresses multi-condition conflicts through heterogeneous experts and efficient routing for versatile, high-fidelity image editing.The framework improves controllability through masks and references with modest overhead.
- Limitations and Future Work: CARE-Edit introduces additional hyperparameters such as top-K and currently covers most common, rather than all possible, edit types and modalities.Future work considers dynamic expert loading or expansion to broaden coverage.
Supplementary Material
The supplementary material provides dataset and implementation details, extended qualitative comparisons, and additional empirical analysis of specialized experts.
- Appendix A: Dataset and Implementation Details: Appendix A documents training-dataset construction, mask-aware image pairs, public-dataset comparisons, architectures, optimization, schedules, and hyperparameters.It covers both dataset and implementation specifications.
- Appendix B: Extended Qualitative Comparisons: Appendix B adds qualitative comparisons for instruction-based and subject-driven editing, including object removal, replacement, and style transfer.The appendix includes per-category and per-edit-type samples and discussion of CARE-Edit’s behavior.
- Appendix C: Additional Empirical Analysis: Appendix C visualizes latent attention maps to analyze how condition-aware routing and specialized experts contribute across diffusion timesteps.The analysis covers reference-guided subject preservation and mask-aware control.
A.1. Training Dataset
CARE-Edit’s training data combines diverse editing sources with mask-aware, background-consistent synthetic pairs and controllable prompt templates. The resulting corpus supports subject-centric training and multi-object evaluation with substantially fewer samples than several baselines.
- CARE-Edit collects data for instruction-based editing, object removal and replacement, and style transfer from multiple high-quality sources.
- Subjects200K provides precise foreground masks and clean white-background reference images for spatially grounded contextual editing.
- A GPT-based pipeline generates background-consistent image pairs with diverse foregrounds, fine masks, and coarse bounding-box spatial priors.
- Scene-level templates preserve subject identity while varying environments, lighting, camera views, or atmosphere across five descriptions per category.
- Approximately 120K training triplets support DreamBench++ multiple-object results that outperform OmniControl, UNO, and OmniGen2 across all reported metrics.The passage attributes this data efficiency to mask-aware, subject-centric curriculum and curated multi-paired construction.
A.2. Implementation Details
CARE-Edit is implemented with frozen FLUX.1-dev backbone weights, LoRA-based condition modules, regularized diffusion training, and benchmark-specific evaluation across instruction-based editing tasks. Qualitative comparisons report better preservation and localization than several strong baselines.
- CARE-Edit uses FLUX.1-dev as its backbone and applies condition-aware LoRA modules while keeping the original backbone weights frozen.
- The training objective combines diffusion reconstruction with load-balancing, mask-boundary, and mixture-smoothness regularizers.The objective is LCARE = Ldiff + λload Lload + λmask Lmask + λmix Lmix.
- The implementation uses task-specific scene prompts that vary background conditions while preserving each subject’s core identity.
- Evaluation covers EMU-Edit, MagicBrush, and DreamBench++ for object, region-based, and personalized subject-driven editing.
- Qualitative comparisons show stronger instruction following, fewer spurious background changes, and sharper localized boundaries than the compared methods.
B.2. Subject-driven Contextual Image Editing
Subject-driven contextual editing requires both identity preservation and plausible placement of reference objects in new scenes. CARE-Edit uses coarse masks to disambiguate location and scale, producing contextually plausible compositions while preserving subject properties.
- DreamBench++ subject-driven editing evaluates identity preservation while composing people, pets, or products into complex new contexts.
- Text prompts alone do not uniquely specify the size or placement of inserted reference objects in the base image.
- User-provided masks supply intended location and approximate size, resolving spatial ambiguity even when the masks are coarse.
- Representative contextual edits produce plausible object geometry and scale while preserving the main subject’s identity, pose, and global lighting.
B.3. More Results on Diverse Editing Tasks
CARE-Edit produces qualitative results across removal, addition, replacement, and style transfer. The examples emphasize clean local edits, controlled object placement, preserved scene structure, and reduced detail loss.
- Object Removal: CARE-Edit removes relatively large masked objects without obvious seams or blur while leaving unedited regions nearly unchanged.
- Addition and Replacement: For addition and replacement, coarse masks control edited-object scale and placement while text instructions guide scene-compatible content.
- Style Transfer: For style transfer, CARE-Edit preserves scene structure and object boundaries while avoiding severe detail loss.
- Across the showcased tasks, CARE-Edit handles removal, addition, replacement, and stylistic changes.
C. Analysis of Expert Lattent Attention Maps
CARE-Edit’s experts develop distinct attention patterns during training, with Base preserving global structure, Mask specializing in edit regions, and Reference injecting appearance cues. These complementary roles support condition-aware workload distribution and reduce interference in multi-condition editing.
- Expert Specialization During Training: The analysis visualizes Base, Mask, and Reference attention maps at T = 0, T = 30K, T = 70K, and T = 100K to examine their evolving specialization.These visualizations are used to test whether experts acquire distinct complementary roles rather than collapsing into a uniform average.
- Expert Specialization During Training: The Base Expert maintains spatially widespread activation throughout training, preserving the original image’s structure, lighting, and layout.Its global coherence allows other experts to focus on differential edits while unedited regions remain consistent.
- Expert Specialization During Training: The Mask Expert evolves from diffuse attention to fine-grained boundaries aligned with intended edit objects by T = 100K.It concentrates within and around editing regions while suppressing background influence, supporting geometric restructuring without leakage.
- Expert Specialization During Training: The Reference Expert develops semantically sparse activation in regions requiring texture synthesis, material changes, or photometric adjustment.Its activation grows as the model maps reference-image features into the generated latent space and remains orthogonal to Base activity.
- Expert Specialization During Training: CARE-Edit distributes responsibilities across experts: Mask handles where to edit, Reference handles appearance, and Base maintains global image consistency.The analysis presents this division as evidence that condition-aware routing mitigates interference from multiple inputs.