ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer

• ACE supports a wide range of image generation and editing tasks.Source: Figure 1 · p. 3
• Natural-language instructions let users specify complex and precise editing requests.Source: Figure 1 · p. 3
• Multi-turn interactions are shown as a way to accomplish these editing requests.Source: Figure 1 · p. 3

• Existing visual-generation foundations typically accept pure text, leaving controllable generation and editing to task-specific tuning that is inflexible and inefficient.Source: §1 · p. 3
• ACE uses a Diffusion Transformer and unified Condition Units to accommodate varied multimodal inputs and visual-generation tasks.Condition Units include core elements such as images, masks, and textual instructions.Source: §1 · p. 4
• Long-context Condition Units concatenate the current condition with prior rounds so the model can use historical information in generation.Source: §1 · p. 4
• ACE's data workflow collects image pairs through synthesis or database clustering and uses a fine-tuned multimodal language model to enrich instruction diversity.Source: §1 · p. 4
• The evaluation benchmark draws on real-world and model-generated inputs, covers global and local editing, and includes a user study reporting closer alignment with human perception on most tasks.Source: §1 · p. 4
• ACE shows competitiveness with specialized models and strong generalization across a broader range of open tasks.The evaluation uses a benchmark covering known visual-generation tasks.Source: §1 · p. 4

2 ALL-ROUND CREATOR AND EDITOR

• ACE follows textual instructions for image creation and editing using a Diffusion Transformer and a unified framework for broad task coverage.The model also handles single-task, multi-turn, and long-context tasks with historical information.Source: §2 · pp. 4–5

2.1.1 TASKS

• The taxonomy spans eight visual task types, from text-guided and controllable generation to semantic, element, repainting, layer, and reference editing or generation.Basic generation tasks can be combined into multi-turn scenarios, with historical information supporting long-context tasks.Source: §2.1.1 · pp. 5–6

2.1.2 INPUT PARADIGM

• A Condition Unit combines a textual instruction with image and mask inputs; an optional sequence of historical rounds forms a Long-context Condition Unit.The LCU includes instruction and visual information from up to m prior rounds for the current request.Source: §2.1.2 · p. 6

2.2 ARCHITECTURE

• ACE tokenizes instructions, image latents, and masks into concatenated sequences, adds image-order indicators, and uses long-context self-attention plus per-CU text cross-attention.The indicator embeddings associate textual image references with corresponding images, while per-CU cross-attention aligns text and image embeddings frame by frame.Source: §2.2 · p. 7

3.1 PAIR DATA COLLECTION

• The collection pipeline synthesizes edits with open-source expert models and mines related images from large databases using semantic clustering, task-specific similarity, and pair cleaning.Mining addresses underexplored editing tasks and the overfitting and image-quality risks of relying only on synthetic data.Source: §3.1 · pp. 7–8, Figure 4 · p. 8
• Nearly 0.7 billion image pairs cover 8 basic task types, multi-turn generation, and long-context generation.The pairs are collected through two automatic pipelines.Source: §3.1 · p. 8

3.2 INSTRUCTIONS

• Instruction labeling combines task-specific templates with MLLM-based pair analysis, while repeated LLM reformulation and human correction address limited diversity and imprecise non-natural-image instructions.The paper uses templates for tasks involving non-natural images and MLLM analysis for natural-image pairs.Source: §3.2 · p. 8
• About 800,000 curated instructions fine-tune an Instruction Captioner based on InternVL2-26B to generate source-to-target instructions from image pairs.The captioner can also process image clusters to describe similarities and pairwise differences.Source: §3.2 · pp. 8–9

4.1 BENCHMARKS AND METRICS

• MagicBrush evaluation uses image-to-ground-truth and image-to-prompt comparisons, while the ACE benchmark contains 12,000 manually instructed entries and assesses image quality and prompt following through a user study.The ACE benchmark includes real and generated images, with instructions and masks crafted for each task type.Source: §4.1 · p. 9

4.2 QUALITATIVE EVALUATION

• Qualitative comparisons cover 16 methods across 12 subtasks, with ACE reported to address diverse tasks and outperform task-specific methods.The final three lines also show tasks on which the comparison methods do not perform well.Source: Figure 5 · p. 10, §4.2 · p. 11

4.3 QUANTITATIVE EVALUATION

• 7 of 12 global and 8 of 10 local editing tasks had ACE’s highest prompt-following scores; its best image-quality scores appeared in 5 of 10 global and 7 of 10 local tasks.ACE performed unsatisfactorily on some tasks, including general and style editing; the authors suggest evaluators may prefer larger SDXL-based 1024-resolution outputs to ACE’s 0.6B-parameter, roughly 512-resolution outputs.Source: Table 2 · p. 11, §4.3 · p. 12
• On MagicBrush, ACE significantly outperforms other methods for single-turn instruction-guided editing and has significant advantages in the reported multi-turn settings.Using historical information in a long sequence improves performance compared with not using it.Source: §4.3 · p. 11

5 CONCLUSION

• ACE is proposed as a versatile foundational generative model for creating images and following instructions across a wide range of generation tasks.The conclusion positions it as a productivity tool for the public and professional designers.Source: §5 · p. 12
• The conclusion highlights interactive dialogue and processing long-context historical information as areas of exploration for visual generation.It frames the model as intended to foster innovation and creativity.Source: §5 · p. 12

A RELATED WORK

• Related work notes that task-specific models have difficulty extending to unseen tasks, while instruction-editing methods face a bottleneck in high-quality instruction-paired data.The paper links that data bottleneck with limited generalizability and suboptimal performance in those methods.Source: A RELATED WORK · p. 19

B DATASETS DETAIL

• The internal training dataset contains 0.7 billion pairs across 8 basic task types and 37 subtasks, plus multi-turn and long-context generation.Tasks use textual instructions with zero or more reference images for image generation or editing.Source: B DATASETS DETAIL · p. 19
• The section describes data construction methods for the corpus and points to Fig. 6 for its task distribution and absolute scale.The distribution is shown in Fig. 6a and the absolute scale in Fig. 6b.Source: B DATASETS DETAIL · p. 19

B.1 TEXT-GUIDED GENERATION

• The text-guided generation data uses approximately 117 million images, with MLLM-supplemented captions, to create text-to-image pairs.This portion also serves as an intermediary bridge for combining task instructions into source-to-target image pairs.Source: §B.1 · p. 19

B.2 LOW-LEVEL VISUAL ANALYSIS

• Ten low-level visual features, including segmentation, depth, pose, edges, and scribbles, provide control signals for controllable generation.The features are extracted globally and locally.Source: §B.2 · p. 19, §B.3 · p. 21

B.3 CONTROLLABLE GENERATION

• Controllable generation uses visual conditions as inputs for pixel-precise image generation, with instructions constructed from image captions by the Instruction Captioner.Training pairs use global or regional conditions produced by low-level visual analysis.Source: §B.3 · p. 21

B.4 SEMANTIC EDITING

• Semantic editing covers facial and style changes, plus general edits such as changing backgrounds, posture, or camera view.Facial edits aim to preserve identity while changing selected attributes; style transformations aim to preserve content.Source: §B.4 · p. 21

B.4.1 FACIAL EDITING

• Facial editing separates identity-preserving edits from attribute transformations, with four transformation tasks yielding 1.4 million high-quality pairs.Preservation data include aligned and misaligned faces; transformation tasks cover smiling, beard, makeup, and hair dyeing.Source: §B.4.1 · pp. 22–23

B.4.2 STYLE EDITING

• Style data include over 80 styles and 63,000 pairs, plus around 70,000 input-output pairs spanning about 400 high-quality styles.CLIP filtering removes pairs with too little or too much difference; CSR-based clustering filters style-mismatched synthesized targets.Source: §B.4.2 · p. 23

B.4.3 GENERAL EDITING

• General editing unifies caption-guided generation and instruction-guided adaptation, sourcing content-related image pairs through clustering or IP-Adapter synthesis.The MLLM generates a target-image caption and an instruction for adapting the source image.Source: §B.4.3 · pp. 23–24

B.5 ELEMENT EDITING

• Element editing pairs text removal and rendering in reverse directions and supports object removal and addition while preserving other image content.Text paths use inpainting or text rendering with OCR checks; object data use segmentation and inpainting to create reversible edits.Source: §B.5.1 · pp. 25–26, §B.5.2 · pp. 26–27

B.6 REPAINTING

• Repainting covers unconditional inpainting, text-guided inpainting, and outpainting, using varied masks and local object captions to control completion.Outpainting randomizes the masked areas and directions to support different image-extension scenarios.Source: §B.6 · p. 27, §B.6.1 · p. 27, §B.6.2 · p. 28

B.7 LAYER EDITING

• Layer editing separates a subject from its reconstructed background and combines independent subjects with a background into a composite image.Training uses masked or white-filled regions and multiple references for fusion.Source: §B.7 · pp. 28–29

B.8 REFERENCE GENERATION

• Reference tasks cover segmented multi-condition generation, clustered series generation with 3 to 8 inputs, and style- or face-reference editing.Style references come from the same style category; face references depict the same person, with the edit image's face erased.Source: §B.8.1 · p. 30, §B.8.2 · p. 30

B.9 MULTI-TURN AND LONG-CONTEXT GENERATION

• Multi-turn editing chains independent instruction-based edits, while long-context generation uses prior images and instructions to guide the final image.Training chains are formed from clustered image pairs or sampled single-image tasks.Source: §B.9 · pp. 30–31

C BENCHMARK DETAILS

• Earlier benchmarks are described as having insufficient task coverage and generally poor data quality; MagicBrush has 1,588 samples, including 1,053 single-turn and 535 multi-turn instances.MagicBrush primarily draws from MS-COCO.Source: C BENCHMARK DETAILS · p. 31
• ACE's benchmark contains 12,000 samples across more than 31 tasks, including 5,900 real images and 6,100 generated images.It supports regional and multi-turn editing.Source: C BENCHMARK DETAILS · p. 32

D IMPLEMENTATION DETAILS

• The model uses T5 as its text encoder and DiT-XL/2 as its base network, with capacity near 0.6B parameters.The text-token sequence is limited to 120 tokens; VAE and T5 are frozen during training.Source: D IMPLEMENTATION DETAILS · p. 32
• Training uses AdamW for the DiT module, with weight decay 5e-4 and learning rate 2e-5.The experiments were conducted on A800 hardware.Source: D IMPLEMENTATION DETAILS · p. 32
• A multi-stage strategy first trains single-image instruction following on 0.7 billion data points, then adds multiple-image tasks and high-aesthetic data.The image-token limit increases from 1024 to 4096 for higher-resolution generation.Source: D IMPLEMENTATION DETAILS · p. 32

E MORE EXPERIMENTS

• Image Indicator Embeddings let ACE identify source and reference images when their input order changes.The reported result links textual instructions and images implicitly through the image indicators.Source: E MORE EXPERIMENTS · pp. 32–33
• ACE achieves performance comparable to its baselines on the Emu Edit benchmark, which covers 3,589 examples across eight editing tasks.The evaluation compares image similarity and text-image direction metrics.Source: E MORE EXPERIMENTS · p. 33, Table 5 · p. 33
• 3.56% and 25% improvements in FS and ES show ACE outperforming methods without facial landmarks, though InstantID uses an additional landmark condition.The study defines FS for face identity preservation and ES for instruction-compliant samples above a threshold based on MS1M V3 similarity statistics.Source: E MORE EXPERIMENTS · pp. 33–34, Table 6 · p. 33
• 14% and 16% gains in Normalized Edit Distance and Sentence Accuracy, respectively, show ACE outperforming UDiffText and Anytext on local text rendering.The evaluation crops each generated text line and scores it with an OCR model.Source: Table 7 · p. 33, E MORE EXPERIMENTS · p. 34

F.1 WORKFLOW DISTILLATION

• Workflow distillation trains ACE on workflow inputs and outputs with a fixed trigger instruction, and the model acquires similar abilities.The workflows combine LoRAs, ControlNets, and text-to-image models.Source: Figure 26 · p. 34, Figure 27 · p. 34, §F.1 · p. 35

F.2 CHAT BOT

• The chat bot handles image creation and editing through one diffusion-model backend, while dialogue history informs its response to the current request.The application supports natural-language requests and is described as more efficient than visual-agent pipelines.Source: §F.2 · pp. 35–36, Figure 28 · p. 35, Figure 29 · p. 35

G MORE VISUALIZATION

• Visualizations span low-level analysis, controllable generation, repainting, semantic and element editing, layer editing, reference tasks, and multi-turn generation.The examples include segmentation, depth, pose, text, object, and style editing.Source: G MORE VISUALIZATION · p. 36

H DISCUSSION

• Image generation may support creative, educational, and business work, while raising copyright, misinformation, and bias concerns.The discussion calls for managing these risks responsibly.Source: H DISCUSSION · p. 36
• ACE offers a unified framework for editing tasks, but its text-to-image aesthetic quality trails Midjourney and FLUX, which focus on text-prompted image generation.The authors suggest higher-quality data and a larger model could help bridge the gap.Source: H DISCUSSION · p. 36
• Complex and diverse instructions are harder to interpret with the fixed encoder-decoder language model, and multiple intents require decomposition.The paper identifies accurate capture of user intent as a need for instruction editing.Source: H DISCUSSION · p. 36
• Multiple-image training and inference become more challenging as token counts rise, while excessively long contexts may cause forgotten history to bias outputs.The authors attribute the increasing challenge to limited hardware resources.Source: H DISCUSSION · p. 37
• Future work targets higher-quality data and larger models, LLM/MLLM-based intent and image understanding, and long-sequence multimodal history modeling.The paper also calls for exploring scaling laws in complex generation tasks.Source: H DISCUSSION · p. 37