PaperlayerView exact revision on arXiv

Source-linked AI summary

MRT: Masked Region Transformer for Layered Image Generation and Editing at Scale

Zhicong Tang, Zhao Zhang, Jingye Chen, Mohan Zhou, Yifan Pu, Yuchi Liu, Yalong Bai, Ethan Smith, Yuhui Yuan

arXiv:2605.27235v1cs.CV

TL;DR

Layered image generation remains underdeveloped at scale, and prior methods can produce incomplete elements beyond canvas boundaries. MRT unifies three layered-generation tasks in a masked-region diffusion framework; experiments report improved performance with scaling and joint training, alongside out-of-domain generalization and multi-image fusion.

  • Problem

    Layered image generation remains underdeveloped at scale, while prior methods produce incomplete foreground elements beyond canvas boundaries, limiting layer reusability.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    MRT unifies text-to-layers, image-to-layers, and layers-to-layers with adaptive masking, adds an overflow-aware canvas layer, and distills diffusion for real-time generation.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 5

  • Results

    Experiments report improved performance from scaling and joint training, out-of-domain generalization for image-to-layers, and multi-image fusion for layers-to-layers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MRT provides a framework for efficient synthesis of high-fidelity, semi-transparent, fully editable visual layers.

    Source: §5 · pp. 18–19

    On real-world photographs, MRT can leave shadows on the background because its design-only training data lack physical effects such as shadows, reflections, and refractions.

    Source: §4 · p. 24

Abstract

from arXiv · show

Layered image generation and editing is a fundamental capability that enables layer-wise reuse, editing, and composition of generated visual content, analogous to word-level editing in natural language. Despite its importance, this remains an underexplored area at scale. To address this gap, we present MRT, a 20B-parameter masked region diffusion model tailored for multi-layer transparent image generation and editing, trained on over 10M multilingual design samples spanning diverse aspect ratios and textual prompts. To fully leverage this scale, we make two key technical contributions. First, we unify three complementary tasks including text-to-layers, image-to-layers, and layers-to-layers within a shared masked region diffusion framework, where selective token masking enables flexible layer-wise generation and editing. Second, to enable overflow layer generation, we introduce an overflow-aware canvas layer that handles boundary inconsistencies and supports semi-transparent background synthesis, enabling complete editable layers extending beyond visible canvas boundaries. Additionally, we apply diffusion distillation to achieve 8-step, real-time multi-layer generation with minimal quality degradation. Extensive experiments demonstrate that our framework substantially outperforms prior state-of-the-art approaches, including various commercial systems, across all three tasks, establishing a new benchmark for multi-layer transparent image generation. Notably, our model significantly outperforms the concurrent Qwen-Image-Layered model in image-to-layers quality according to user-study results, while achieving 10-100\times faster inference and reducing activation GPU memory consumption by 50-90\% during image-to-layer inference.

1. Introduction

MRT addresses the underdevelopment of layered image generation with a large-scale dataset and a unified masked-region framework for layer generation and editing. It supports reusable overflow layers and uses distillation for few-step generation.

Motivation

  • Layered image generation remains underdeveloped because comparable large-scale, high-quality datasets are lacking and prior knowledge from state-of-the-art open-source text-to-image models is underused.Source: §1 · p. 2

Dataset and model

  • The dataset contains over ≥10 million samples, 43 million unique layers, and 7 million unique oversized visual elements, and spans diverse resolutions and aspect ratios.GPT-5 mini generates global captions, while visual text layers use ground-truth typography attributes; the dataset is described as an order of magnitude larger than recent work [38].Source: §1 · p. 2

Core framework

  • MRT unifies text-to-layers, image-to-layers, and layers-to-layers generation and editing through adaptive masking that initializes each layer from clean latents or noise according to task requirements.The model is built on Qwen-Image [55], which the paper describes as an approximately 20B-parameter open-source text-to-image diffusion model.Source: §1 · p. 2
  • The full-canvas architecture treats the background as a special transparent foreground layer and encapsulates overflow layers, preserving foreground-layer reusability and arbitrary repositioning.Overflow layers can extend partially beyond the background region.Source: §1 · p. 2

Efficiency

  • Distribution matching distillation enables a few-step multi-layer generator with minimal quality degradation.Source: §1 · p. 2

Experiments

  • Ablations show that scaling the model and dataset improves performance and joint multi-task training further enhances performance and user experience.Image-to-layers generalizes to out-of-domain design and natural images, while layers-to-layers supports multi-image fusion that integrates user images into existing designs.Source: §1 · p. 2

2. Related Work

Prior layered-image methods use simultaneous or sequential generation, while related layout methods generate layouts from visual elements or control generation spatially. Compared with ART [38] and Qwen-Image-Layered, MRT unifies three tasks, supports overflow layers, and enables few-step generation through distillation.

2. Related Work

  • Layered image generation and editing follows simultaneous generation (Text2Layer [63], LayerDiff [17], ART [38], PrismLayer [5], Qwen-Image-Layered) or sequential generation (LayerDiffuse, COLE [22], OpenCOLE [21], LayerD [45]).These are the two paradigms identified for layered image generation and editing.Source: §2 · p. 2
  • Related layout methods either generate layouts from visual elements [2] [3] [4] [6] [7] [10] [11] [15] [18] [19] [20] [24] [25] [27] [46] [53] [54] [56] [58] or control generation via spatial conditioning [1] [4] [10] [26] [29] [51] [52] [64].The passage groups layout generation and control methods into these two categories.Source: §2 · p. 2
  • Compared with ART [38] and Qwen-Image-Layered, MRT unifies text-to-layers, image-to-layers, and layers-to-layers generation, adds native overflow-layer support, and enables few-step generation through distillation.The passage identifies ART and Qwen-Image-Layered as the most closely related work.Source: §2 · p. 2

3. Approach

MRT scales layered design generation with curated data and overflow-aware regional diffusion, unifies text-to-layers, image-to-layers, and layer editing through selective masking, and distills the model for fewer-step inference.

3.1. Scaling-up Layered Data and Diffusion Model

  • Over 10M licensed, professionally created designs support training, while a full-size canvas layer enables complete overflow elements and semi-transparent backgrounds.More than 60% of training samples contain overflow layers; without overflow support, content extending beyond background boundaries is cropped and less reusable. The regional transformer jointly attends to foreground, background, and composed-image tokens.Source: §3.1 · pp. 2–3

3.2. Masked Region Transformer

  • A shared masked region diffusion framework handles text-to-layers, image-to-layers, and layers-to-layers by noising only target tokens and attending jointly to clean conditions and noisy tokens.Text-to-layers generates transparent layers from prompts; image-to-layers conditions layer generation on clean global-image tokens; layers-to-layers conditions new or restylized layers on existing content and references.Source: §3.2 · p. 3, Figure 4 · p. 4
  • Image-to-layers decomposes raster inputs into transparent layers using target layouts, with segmentation and inpainting addressing layer masks and occluded content.Layouts come from human annotation or a detector, and layer-grouping augmentation improves robustness to ambiguous boundaries and generalization to noisy, out-of-domain layouts.Source: §3.2 · p. 4
  • Layers-to-layers supports adding coherent layers from prompts and restylizing selected layers from visual references while preserving the remaining composition.Training selects existing layers as conditions and targets the rest; restylization uses style-transformed variants as training pairs.Source: §3.2 · pp. 4–5

3.3. Accelerated Multi-Layer Generator

  • DMD distills the multi-step teacher into a student that uses fewer denoising steps while maintaining distributional consistency and sample quality.CacheDiT and multi-GPU sequence parallelization are also supported to further accelerate inference.Source: §3.3 · p. 5

4. Experiment … 4.3. Main Results

MRT is evaluated on text-to-layers, image-to-layers, and layered editing, with comparisons showing advantages in layer quality, editability, and inference efficiency. The experiments also describe the benchmarks, evaluation metrics, and training setup used.

4.1. Implementation Details; 4.2. Evaluation Protocol

  • Experiments use DESIGN-MULTI-LAYER-BENCH and an overflow-layer benchmark, with image-quality metrics and user preferences evaluated for text-to-layers and image-to-layers.Training starts from Qwen-Image weights and uses full-parameter fine-tuning; system-level training runs for ∼70,000 iterations at 512 × 512 on 10M samples, then ∼20,000 iterations at 1024 × 1024.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.3.1. Text-to-Layers: Comparison with SoTAs

  • In text-to-layers, participants preferred MRT to ART for instruction following, aesthetics, and layer quality, while MRT alone natively generates editable overflow RGBA layers beyond the background boundary.Prior systems such as ART restrict pixels to the background region, which can crop or omit content.Source: §4.3.1 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7

4.3.2. Image-to-Layers: Comparison with SoTAs

  • For image-to-layers, users preferred MRT over LayerD, RoboNeo, and Lovart for layer quality, content integrity, and decomposition granularity.With a layout supplied by a z-order-aware detector, qualitative results show more complete reusable RGBA layers with sharper boundaries; the paper also demonstrates generalization to natural scenes.Source: §4.3.2 · p. 6

4.3.3. Image-to-Layers: Comparison with con-current Qwen-Image-Layered

  • MRT’s image-to-layers limitations include photorealistic generalization, ambiguous layer granularity, occluded-layer completion, and background inpainting.Occlusion failures can stem from inaccurate amodal bounding regions or difficulty reconstructing complex occluded pixels with limited contextual cues and data diversity.Source: §4.3.3 · p. 7
  • On an out-of-domain image-to-layers test set, MRT beats Qwen-Image-Layered across layer-count groups and wins user-study preferences for quality (79.5%), integrity (68.9%), and granularity (82.6%).The quantitative comparison reports significantly higher SNRmerged and SSIMmerged for MRT; the user study evaluates the two methods on the same test set.Source: §4.3.3 · p. 6, Table 1 · p. 7, Figure 8 · p. 7

4.3.4. Layers-to-Layers: Layered Editing

  • In layered editing, MRT predicts added layers in parallel and harmonizes selected restylized layers in one pass, producing coherent insertions and better geometry preservation than GPT-Image-1.GPT-Image-1 is configured to generate and insert one layer at a time, which may increase latency and propagate inconsistencies across edits.Source: §4.3.4 · pp. 12, 13, 16

4.3.3. Image-to-Layers: Comparison with con-current Qwen-Image-Layered

  • At about 20 layers, MRT reaches up to 108.5× speedup and reduces peak GPU memory consumption by 10.5× →23.6× versus Qwen-Image-Layered.MRT maintains near-constant latency at ∼5s while Qwen-Image-Layered latency scales linearly; the figure reports measurements over 100 samples on a single GPU with identical layer counts.Source: §4.3.3 · pp. 6–7, Figure 19 · p. 17

4.4. Ablation Study and Analaysis

The analyses show that model and dataset scaling improve generation quality, while overflow support, unified task training, caption conditioning, layout augmentation, and distillation reveal distinct quality, robustness, and efficiency trade-offs.

Image-to-layers comparison

  • Blind user study found our method significantly outperformed LayerD and commercial systems Lovart and RoboNeo across quality, integrity, and granularity.Participants compared paired image-to-layers results; the dimensions assessed semantic correctness and transparency, faithful input reconstruction, and appropriate decomposition level.Source: Figure 21 · p. 18

Overflow support

  • With overflow data, the model preserves complete overflow elements at comparable FID, PSNR, and SSIM, addressing a setting found in over 60% of designs that prior works truncate.The comparison concerns text-to-layers generation.Source: Table 2 · p. 18, §4.4 · p. 18

Multi-task training

  • Random task sampling combines T2L, I2L, and L2L without multi-stage fine-tuning at comparable performance, though L2L slightly reduces results, attributed to dataset quality.T2L is text-to-layers, I2L is image-to-layers, and L2L is layers-to-layers.Source: Table 3 · p. 18, §4.4 · p. 18

Textual conditioning

  • Caption conditioning yields modest, consistent metric improvements in I2L but is not essential; text helps disambiguate boundaries and convey semantics in complex overlapping compositions.The analysis concerns global captions for image-to-layers decomposition.Source: Table 4 · p. 18, §4.4 · p. 18

Layer grouping augmentation

  • Randomly merging training layers diversifies layout inputs and improves robustness, including on high-quality DESIGN-MULTI-LAYER-BENCH annotations; larger gains are expected for noisy layouts.The augmentation addresses the gap between precise training layouts and noisy user- or detector-provided test layouts.Source: Table 5 · p. 18, §4.4 · p. 18

Generator distillation

  • DMD2 distillation cuts denoising steps from 50 to 8 (6× speedup) with comparable FID and largely preserved visual quality.The result concerns the multi-layer generator; visual quality is reported as largely preserved.Source: Table 6 · p. 18, §4.4 · p. 18

Model and dataset scaling

  • Scaling from FLUX.1 [dev] (13B) to Qwen-Image (20B) on 0.5M samples lowers FID from 17.79 to 16.15; scaling data to 10M lowers it further to 15.63.The authors note that extended training may yield additional gains.Source: §4.4 · p. 18

5. Conclusion

MRT presents a large-scale diffusion framework for multi-layer transparent image generation, unifying text-to-layers, image-to-layers, and layers-to-layers tasks. Its 20B-parameter model is trained on over 10M multilingual design samples and combines overflow-aware canvas handling with distribution-matching distillation.

5. Conclusion

  • MRT unifies text-to-layers, image-to-layers, and layers-to-layers generation in a shared masked-region diffusion framework.The model is a 20B-parameter diffusion framework trained on over 10M multilingual design samples.Source: §5 · p. 18
  • For layers-to-layers editing, MRT better follows layer-wise addition instructions and improves coherence and style consistency over GPT-Image-1.Figure 22 shows layer addition in its first two rows and layer restylization in its last two rows.Source: Figure 22 · p. 19
  • Despite training only on poster-style design datasets, MRT generalizes to out-of-domain natural scenes in image-to-layers generation.Figure 24 presents qualitative results on natural images.Source: Figure 24 · p. 19
  • An overflow-aware canvas layer supports complete boundary handling, while distribution-matching distillation enables real-time generation of high-fidelity, semi-transparent, fully editable layers.Figure 23 compares the baseline and few-step distilled model.Source: §5 · p. 19, Figure 23 · p. 19

Supplementary Material

The supplementary material examines caption-length effects, multi-layer generator distillation and inference time, and the effect of generated layer counts on image-to-layers quality.

Caption length during training

  • Caption-length training variants are evaluated by FID on VC5K using short and long captions.The compared training setups use short captions, long captions, or a mixture of both.Source: Table 1 · p. 23

Generator distillation

  • A supplementary table presents multi-layer generator distillation alongside inference time.Source: Table 2 · p. 23

Effect of layer numbers

  • Image-to-layers quality is evaluated across different ranges of generated layer numbers.Source: Table 3 · p. 23

1. Additional ablation experiments

The ablations examine caption-length diversity, layer-count scalability, and distilled inference speed. Mixed-length training improves caption generalization, the model handles a broad range of layer counts, and distillation reduces generation time.

1.1. Mixed Training with Variable Caption Length

  • Mixed-length training gives FID 16.13 on short captions and 15.93 on long captions, while short-only training gives 18.56 FID on long captions and long-only training gives 16.15 FID.The passage identifies mixed-length training as achieving the best generalization and describes a domain gap from training on only one caption type.Source: §1.1 · p. 23

1.2. Effect of Layer Numbers on Image-to-layer

  • For image-to-layers, MRT handles compositions ranging from 2 to 50 layers with stable performance and no architectural modifications.The passage says this flexibility covers both simple designs and complex multi-element compositions.Source: §1.2 · p. 23

1.3. Analysis of Distilled Models

  • With DMD2 distillation, inference achieves a 3.2× speed-up in 4.5s at 16 steps, with FID rising slightly from 16.02 to 16.21; at 8 steps, speed-up is 6.26× in 2.3s.Source: §1.3 · p. 23

2. Attention Analysis of Image-to-Layer Model

Attention maps show that the image-to-layer model focuses on semantic boundaries and distinct visual features, supporting clean alpha mattes and coherent textures in generated RGBA layers.

2. Attention Analysis of Image-to-Layer Model

  • Attention localizes on text strokes and fine details while broadly covering background patterns and large shapes, suppressing noise and producing clean alpha mattes and coherent textures.The visualized pixel-wise attention maps correlate with target-element boundaries during decomposition.Source: §2 · p. 23

3. User study details

The section describes blind, randomized pairwise user studies for text-to-layer and image-to-layers evaluation, using task-specific criteria and three-way votes.

3.1. User Study on Text-to-Layer Task

  • MRT was compared with ART in randomized blind pairwise text-to-layer tests, with three-way votes on layout, aesthetics, typography, and layer coherence and quality.Results were displayed side-by-side after participants saw the prompt.Source: §3.1 · pp. 23–24

3.2. User Study on Image-to-Layer Task

  • Three independent blind pairwise image-to-layers comparisons tested MRT against LayerD, Lovart, and Roboneo, with randomized result placement and votes on granularity, layer integrity, and layer quality.The original image served as a central reference alongside the two methods’ results.Source: §3.2 · p. 24

4. Limitations

MRT’s image-to-layer decomposition struggles with shadows in real-world photographs, reflecting a gap between its design-only training data and natural scenes. It nevertheless generalizes reasonably well without real-world multi-layer supervision, and the authors plan to train on more realistic data.

4. Limitations

  • MRT often leaves shadows on the background while excluding them from object layers, causing visual inconsistency; its design-only training data lacks natural effects such as shadows, reflections, and refractions.The authors identify this training-data domain gap as the primary cause of the limitation.Source: §4 · p. 24
  • Despite the domain gap, MRT generalizes reasonably well to real images without real-world multi-layer supervision, successfully separating most objects, which the authors attribute to its Qwen-Image backbone’s visual understanding.The authors describe this as evidence of the approach’s robustness, adaptability, and scalability.Source: §4 · p. 24
  • The authors plan to extend MRT to real-world scenarios by training on datasets with realistic effects such as shadows and reflections to improve coherent, physically plausible layer decompositions.Source: §4 · p. 24

5. Visualizations and Qualitative Analysis

The visual analysis shows MRT producing coherent, editable layered designs across generation, decomposition, and editing, while also exposing limitations in transparency decoding, tiny glyphs, and layer-to-layer fidelity.

5.1. Diverse Text-to-Layer Generation

  • Text-to-layer generation produces coherent designs with more than 25 layers, full-size overflow RGBA layers, and visually rendered multilingual text including Chinese.Overflow layers extend beyond the visible background boundary, preserving editability and reusability; generated compositions maintain spatial consistency and stylistic harmony.Source: §5.1 · pp. 24–25, Figure 8 · p. 33, Figure 9 · p. 34

5.2. Comparative Analysis of Image-to-Layer

  • MRT achieves cleaner boundaries, more precise alpha mattes, and finer semantic decomposition than LayerD, Lovart, and RoboNeo, especially for overlapping elements.The comparison describes RoboNeo artifacts or poor transparency and LayerD’s overly grouped layers as weaknesses.Source: §5.2 · p. 26, Figure 11 · p. 35

5.3. Scalability on Layer Counts in Image-to-Layer

  • Across 6–16-layer decompositions, MRT maintains boundary detection and content preservation without observed quality degradation as layer counts increase.The visualizations cover layer counts from 6 to 16 and report consistent quality across this range.Source: §5.3 · p. 26, Figure 23 · p. 47

5.4. Context-Aware Layer Addition

  • For layer addition, MRT generates prompt-specified elements that match existing lighting, perspective, and style while preserving the composition’s global aesthetic.The task conditions new layers on the full composition and specified bounding boxes.Source: §5.4 · p. 26, Figure 24 · p. 48

5.5. Layer Restylization and Harmonization

  • For restylization, MRT transfers a target design’s palette, texture, and artistic rendering to user-provided assets while preserving their geometry and cross-layer consistency.The transformation is performed in a single pass across target layers.Source: §5.5 · p. 26, Figure 25 · p. 49

5.6. Layout Generalization in Text-to-Layer

  • When text positions conflict with supplied bounding boxes, MRT follows the layout while retaining semantic content, separating spatial arrangement from semantic generation.The examples include moving a title from the top to the bottom without compromising prompt fidelity or content quality.Source: §5.6 · p. 26

5.7. Layout-Guided Image Decomposition

  • In image decomposition, bounding boxes and Z-order from automatic detectors or manual annotation guide MRT to produce semantically meaningful layers that conform to specified boundaries.These spatial cues help resolve ambiguities in raster images and support controllable, predictable decomposition.Source: §5.7 · p. 26

5.8. Generalization to Natural Scenes

  • Despite training only on graphic designs, MRT decomposes real-world photographs into layers, but often leaves cast shadows on the background rather than assigning them to objects.The passage attributes this limitation to the domain gap and complex physical lighting effects in natural scenes.Source: §5.8 · p. 26, Figure 31 · p. 55

Related papers

Loading 2605.27235v1…