Source-linked AI summary
ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling
Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, Jingren Zhou
TL;DR
ACE++ targets the limited universality of instruction-based image generation and editing while preserving the generative priors of pretrained text-to-image models. It introduces LCU++ and a two-stage training scheme, yielding an all-in-one model and lightweight vertical models with reported qualitative gains in image quality and prompt following.
Problem
Universal image editing models lag behind task-specific methods, while existing versatile image creators can have unsatisfactory generated-image aesthetics.
Method
ACE++ improves LCU into LCU++, extends it to arbitrary editing and generation tasks, and uses two-stage training to adapt pretrained text-to-image diffusion models efficiently.
Results
Qualitative analyses report superior generated-image quality and prompt-following ability, with models covering general editing and generation plus several vertical scenarios.
Takeaways & Limitations
ACE++ provides both an all-in-one editing and generation model and lightweight models intended to support downstream applications across multiple scenarios.
Abstract
from arXiv · showhide
We report ACE++, an instruction-based diffusion framework that tackles various image generation and editing tasks. Inspired by the input format for the inpainting task proposed by FLUX.1-Fill-dev, we improve the Long-context Condition Unit (LCU) introduced in ACE and extend this input paradigm to any editing and generation tasks. To take full advantage of image generative priors, we develop a two-stage training scheme to minimize the efforts of finetuning powerful text-to-image diffusion models like FLUX.1-dev. In the first stage, we pre-train the model using task data with the 0-ref tasks from the text-to-image model. There are many models in the community based on the post-training of text-to-image foundational models that meet this training paradigm of the first stage. For example, FLUX.1-Fill-dev deals primarily with painting tasks and can be used as an initialization to accelerate the training process. In the second stage, we finetune the above model to support the general instructions using all tasks defined in ACE. To promote the widespread application of ACE++ in different scenarios, we provide a comprehensive set of models that cover both full finetuning and lightweight finetuning, while considering general applicability and applicability in vertical scenarios. The qualitative analysis showcases the superiority of ACE++ in terms of generating image quality and prompt following ability. Code and models will be available on the project page: https://ali-vilab. github.io/ACE_plus_page/.
1 INTRODUCTION
ACE++ addresses the gap between task-specific image editing methods and universal instruction-based generation and editing. It improves LCU with a broader input paradigm and uses staged training plus specialized model variants.
- Universal image editing models lag behind task-specific methods, while versatile image creators still produce unsatisfactory aesthetics.
- LCU++ changes conditional inputs from sequence concatenation to channel concatenation, reducing adaptation costs and extending support to arbitrary editing and reference-generation tasks.
- ACE++ pre-trains on 0-ref tasks before fine-tuning on all ACE tasks to support general instructions.
- The framework offers an all-round model and lightweight models for portrait consistency, subject consistency, local editing, and repainting.
2 RELATED WORKS
Prior work established powerful text-to-image models and emerging unified multimodal generation frameworks, but these systems leave room for broader image editing support. ACE and related approaches motivate a unified task-processing paradigm.
- Multimodal-guided Image Generation and Editing: Large-scale text-to-image models achieve remarkable image fidelity but are less suitable for image editing tasks.
- Unified Image Generation and Editing Framework: OmniGen and ACE explore unified frameworks that jointly process multimodal inputs for multiple visual generation tasks.
3 METHOD
ACE++ introduces LCU++, a universal multimodal input paradigm that combines image, mask, noise, and instruction contexts for diverse generation and editing tasks. Its architecture uses channel-wise condition formation and two-stage training to adapt pretrained diffusion models efficiently.
- 3 METHOD: LCU++ extends a universal multimodal input paradigm across editing, reference-generation, and 0-ref tasks.LCU represents textual instructions and visual condition units, while LCU++ extends the 0-ref format to multiple reference contexts.
- 3.2 IMPROVED LONG-CONTEXT CONDITION UNIT: For spatially aligned editing, LCU++ concatenates conditional and noise contexts along channels instead of sequences, mitigating disruption to pretrained context perception.
- 3.3 MODEL ARCHITECTURE: Each condition unit combines image, mask, and noisy latent features into a feature map, which is embedded into tokens before transformer processing.
- 3.3 MODEL ARCHITECTURE: The model predicts velocity from interpolated noisy latents and minimizes mean squared error against ground-truth velocity.Reference reconstruction loss is zero for 0-ref tasks, while target loss trains the predicted output.
- 3.4 TWO-STAGE TRAINING SCHEME: Training first uses 0-ref tasks to leverage pretrained text-to-image capabilities, then fine-tunes on 0-ref and N-ref tasks for general instructions.The implementation uses FLUX.1-dev as the base model and also trains lightweight models for four vertical domains.
4 EXPERIMENTS
ACE++ is evaluated qualitatively across reference generation, portrait consistency, local and general editing, and local reference editing. The results show broad image-generation and editing capabilities, including zero-shot support for local reference editing.
- 4.1 IMPLEMENTATION DETAILS: ACE++ uses FLUX.1-dev as its base model, pre-trains on 0-ref task data, then fully fine-tunes on the complete dataset.Task-specific models in four vertical domains are lightweight fine-tuned from FLUX.1-Fill-dev.
- 4.1 IMPLEMENTATION DETAILS: The training setup uses AdamW with weight decay 1e-2, learning rate 1e-3, and gradient clipping at an L2-norm threshold of 1.0.The guidance scale is set to ω = 1.0, making the guided prediction equal to the conditional prediction.
- 4.2 QUALITATIVE RESULTS: Reference generation maintains the identity of specific subjects across different scenes and supports portrait-consistent image generation.The qualitative results also report high-quality portrait image generation capability.
- 4.2 QUALITATIVE RESULTS: Local reference editing supports reference-guided generation for specific image regions, including try-on and product-image generation, in a zero-shot manner.These tasks are supported without being trained during the training phase.
5 CONCLUSION
ACE++ combines improved input paradigms and two-stage training to support broad image editing and generation. It offers both an all-in-one model and vertical models for specific application scenarios.
- ACE++ improves ACE through enhanced input paradigms and a two-stage training approach.
- The framework provides an all-in-one editing and generation model alongside vertical models tailored to specific application scenarios.