PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Layer: Layered Image Generation using Latent Diffusion Model

Xinyang Zhang, Wentian Zhao, Xin Lu, Jeff Chien

arXiv:2307.09781v1cs.CVcs.AIcs.LG

TL;DR

Layered image generation addresses the need for controllable compositing and the limitations of implicit editing regions, incompatible independent generation, and separate segmentation. The paper generates foreground, background, mask, and composed image jointly using a layered-image autoencoder and latent diffusion, and reports better image quality, mask accuracy, and text relevance than baselines. It also creates a large training dataset and establishes a benchmark for layered-image generation.

  • Problem

    Existing editing approaches provide limited explicit control over edited regions, while independently generated layers may be incompatible and separate segmentation can fail across varied objects.

    Source: I.. Introduction · pp. 1–2

  • Method

    The method trains an autoencoder to reconstruct two-layer images and trains latent diffusion models to generate foreground, background, mask, and composed image together.

    Source: I.. Introduction · p. 1

  • Results

    The proposed method generally achieves better FID, mask accuracy, and text relevance than Stable Diffusion-inspired baseline models.

    Source: I.. Introduction · p. 2

  • Takeaways & Limitations

    The paper provides a layered-image generation system, a 57.02M-image dataset, and a benchmark for future research.

    Source: I.. Introduction · p. 2, VI.. Conclusions and Discussions · p. 8

Abstract

from arXiv · show

Layer compositing is one of the most popular image editing workflows among both amateurs and professionals. Motivated by the success of diffusion models, we explore layer compositing from a layered image generation perspective. Instead of generating an image, we propose to generate background, foreground, layer mask, and the composed image simultaneously. To achieve layered image generation, we train an autoencoder that is able to reconstruct layered images and train diffusion models on the latent representation. One benefit of the proposed problem is to enable better compositing workflows in addition to the high-quality image output. Another benefit is producing higher-quality layer masks compared to masks produced by a separate step of image segmentation. Experimental results show that the proposed method is able to generate high-quality layered images and initiates a benchmark for future work.

I.. Introduction

The paper frames layered image generation as a way to support controllable compositing while addressing limitations in implicit editing regions, incompatible generation runs, and unreliable segmentation. It introduces a latent-diffusion system that generates foreground, background, mask, and composition together, with experiments showing improved image quality, mask accuracy, and text relevance.

Motivation

  • Existing editing methods can modify unintended pixels, while implicit editing regions and trial-and-error instructions limit precise text-guided control.These methods are also constrained by generated image quality and the difficulty of describing desired details in plain language.Source: I.. Introduction · p. 1
  • Layered generation explicitly produces a foreground, background, layer mask, and composed image, enabling users to move or replace layers with controlled edits.The mask controls transparency, while alpha blending forms the composed image.Source: I.. Introduction · pp. 1–2

Challenges

  • Generating layers independently does not guarantee compatibility, and segmentation-based pipelines often fail across varied object scales, locations, and counts.The paper therefore treats layered-image generation and training-data synthesis as its two central challenges.Source: I.. Introduction · p. 2, Figure 3 · p. 2, Figure 2 · p. 2

Approach

  • The method learns layered-image latent representations with an autoencoder and trains diffusion models on them, including the composed image during reconstruction.The multi-task objective combines image reconstruction with mask-related losses, and composed-image supervision improves generated image quality.Source: I.. Introduction · p. 2

Training Data

  • The automatic synthesis system filters segmentation and inpainting outputs to produce the 57.02M-image LAION-L2I dataset for diffusion training.The filtering system distinguishes high-quality salient masks and inpainting results.Source: I.. Introduction · p. 2

Results

  • Compared with Stable Diffusion-inspired baselines, the proposed method generally improves FID, mask accuracy, and text relevance for layered images.The paper also introduces an evaluation metric and a benchmark for layered-image generation.Source: I.. Introduction · p. 2

II.. Related Work

The related work spans text-to-image generation, text-guided editing, image matting, and salient-object segmentation. The paper builds on these areas while targeting explicit salient masks and layered compositing through diffusion-based generation.

Text-based image generation

  • Diffusion models have recently improved text-to-image semantic coherence and spatial detail, including through latent-space generation and classifier-free guidance.The paper adopts latent diffusion for layered-image generation.Source: II.. Related Work · pp. 2–3

Text-guided image editing

  • Text-guided editing methods use CLIP, inversion, attention manipulation, finetuning, or diffusion, but many cannot constrain the edited region explicitly.The paper instead generates salient masks alongside images to support localized editing and compositing.Source: II.. Related Work · p. 3

Image matting

  • Image matting estimates an alpha matte using the compositing relation Ii ≈ αiFi + (1 − αi)Bi, which the paper adopts for foreground-background composition.The paper also draws on composition and Laplacian losses for autoencoder training.Source: II.. Related Work · p. 3

Salient object segmentation

  • Salient-object segmentation detects visually salient regions, and mask integrity is important because incomplete masks can split foregrounds and backgrounds and create artifacts.The paper selects ICON to estimate masks for its dataset.Source: II.. Related Work · p. 3

III.. Synthesizing High-Quality Layered Images

The paper constructs LAION-L2I by converting large-scale text-image data into two-layer images with foregrounds, backgrounds, and masks. It uses salient-object segmentation, inpainting, and quality filtering to create paired training examples.

III.. Synthesizing High-Quality Layered Images

  • The section introduces a 57.02M-image LAION-L2I dataset, including 57M training images and 20K test images called LL2I-E.The dataset is designed for large-scale layered-image generation with associated text prompts.Source: III.. Synthesizing High-Quality… · p. 3

A.. Definition of Two-Layer Image

  • A two-layer image is the triplet I = (F, B, m), where foreground F, background B, and mask m share spatial dimensions H × W.The foreground and background are three-channel tensors, while the mask is single-channel.Source: A.. Definition of Two-Layer Image · p. 3

III.. Synthesizing High-Quality Layered Images

  • The construction starts from LAION-Aesthetics, a roughly 600-million-image subset selected for high aesthetic quality and text relevance.The approach creates a two-layer image and text-prompt pair from each source image.Source: A.. Definition of Two-Layer Image · pp. 3–4
  • Salient-object segmentation extracts foregrounds, state-of-the-art inpainting fills missing background regions, and two classifiers remove samples with poor masks or inpainting.Salient segmentation is chosen for scalability and because associated prompts generally describe salient objects, backgrounds, and masks.Source: A.. Definition of Two-Layer Image · p. 4

C.. Extracting foreground and background parts

The dataset is built by estimating foregrounds, backgrounds, and masks from existing images, then filtering defective masks and inpaintings with learned quality classifiers.

C.. Extracting foreground and background parts

  • The initial two-layer dataset is estimated from 512×512 LAION-A images by constructing foreground, background, and mask triplets.The unfiltered version is named LAION-L2I (U).Source: C.. Extracting foreground and… · p. 4
  • Foregrounds are copied from pixels whose ICON-predicted mask values exceed 0.1, while backgrounds come from diffusion inpainting after mask dilation.The inpainting model uses an empty text prompt because adding “background” did not improve quality.Source: C.. Extracting foreground and… · p. 4

D.. Quality filtering

  • The extraction pipeline is vulnerable because mask predictors can omit or include background regions, while inpainting can add artifacts or retrieve foreground objects.The filtering stage addresses these observed failure modes before training.Source: D.. Quality filtering · p. 4, Figure 4 · p. 4
  • Quality labels flag incomplete objects, background contamination, undersized selections, inpainting artifacts, or incoherent patched regions.These criteria target failures that can introduce irrelevant content during diffusion training.Source: D.. Quality filtering · pp. 4–5
  • 57.02M samples remain in LAION-L2I after classifiers discard images with bad masks or poor inpainting results.The classifiers are trained from manually labeled samples and applied to the unfiltered dataset.Source: D.. Quality filtering · p. 5

IV.. Modeling

The modeling approach adapts Stable Diffusion to generate two-layer images by learning a composition-aware latent representation and decoding separate image and mask components.

A.. Text2Layer Formulation

  • The conditional model generates a two-layer image I from text prompt T using the learned distribution pθ.The target includes foreground F, background B, and mask m.Source: A.. Text2Layer Formulation · p. 5
  • CaT2I-AE compresses foreground, background, and mask inputs into latent space and reconstructs them with specialized outputs.The encoder stacks F, B, and m, while the decoder uses separate prediction heads for F, B, and m.Source: A.. Text2Layer Formulation · p. 5
  • An auxiliary decoder branch predicts the original image I, adding supervision beyond the separate foreground and background outputs.The paper reports that this extra supervision slightly improves generation quality.Source: A.. Text2Layer Formulation · p. 5
  • The decoder attaches three small convolutional heads to predict foreground, background, and mask components separately.This design differentiates image-component and mask-component generation in the final decoding stage.Source: A.. Text2Layer Formulation · p. 5

C.. Training Objective

Training combines reconstruction objectives for image components with mask-specific losses, then trains a DDPM on the autoencoder’s latent representation.

C.. Training Objective

  • The full loss combines image reconstruction and mask reconstruction as L = Limg + λLmask, with λ set to 1.Limg covers B, F, and I, while Lmask covers the mask channel.Source: C.. Training Objective · p. 5
  • Mask training uses ℓ1, Laplacian, and composition losses to improve reconstruction of the mask channel.Image components use LPIPS, ℓ1, and adversarial losses inherited from Stable Diffusion’s autoencoder training.Source: C.. Training Objective · p. 5
  • After autoencoder training, a cross-modal attentive UNet trains a DDPM directly on the CaT2I-AE latent space.The noisy latent zt is used with Gaussian noise in the diffusion objective.Source: C.. Training Objective · p. 5

V.. Experiments

Experiments compare CaT2I-AE-SD with Stable Diffusion and latent-UNet baselines at primarily 256×256 resolution using image, mask, and text-relevance metrics.

V.. Experiments

  • The main evaluation focuses on image quality, mask quality, and text-image relevance for two-layer generation on LAION-L2I.All methods except SD-v1-4 target 256×256, and a 512×512 CaT2I-AE-SD variant demonstrates another resolution.Source: V.. Experiments · p. 6, A.. Baseline Methods · p. 6

A.. Baseline Methods

  • CaT2I-AE-SD is compared against SD-v1-4, SD-LAION-L2I, SD-AE-UNet, SD-AE-UNet (ft), and a no-supervision variant.The comparisons test both general Stable Diffusion baselines and the contribution of the composition-aware autoencoder.Source: A.. Baseline Methods · p. 6

B.. Metrics

  • The evaluation measures composed-image fidelity with FID, text relevance with CLIP score, and mask quality with IOU.FID is reported on LL2I-E and LA, while CLIP is computed on LL2I-E and IOU uses two reference-mask sets.Source: B.. Metrics · p. 6
  • Figure 5 presents 256×256 composited images and masks for CaT2I-AE-SD and baselines under two text prompts.Each 2 × 2 block contains the composited images and corresponding masks.Source: Figure 5 · p. 7

C.. Implementation Details

The experiments evaluate CaT2I-AE-SD against baselines using qualitative and quantitative measures of image quality, mask quality, and text-image relevance. Results show stronger layered-image quality and masks, while data filtering and higher resolution further improve performance.

C.. Implementation Details

  • Experiments train SD-AE-UNet and SD-LAION-L2I for 75k steps with DDPM, train CaT2I-AE-SD variants from scratch, and sample two-layer images with DDIM using 50 iterations and guidance scale 7.5.CaT2I-AE-SD (512) resumes from CaT2I-AE-SD and trains for 75k steps with 512 images.Source: C.. Implementation Details · p. 7

D.. Qualitative Results

  • Qualitatively, CaT2I-AE-SD generates sharper details, more accurate masks, and fewer compositing artifacts than SD-AE-UNet baselines.The baseline masks are described as noisy.Source: D.. Qualitative Results · p. 7
  • CaT2I-AE-SD (512) produces components with quality comparable to SD-v1-4 while additionally providing foreground, background, and mask outputs for editing.The generated components are presented as simplifying downstream image editing tasks.Source: D.. Qualitative Results · p. 7, Figure 6 · p. 7

E.. Quantitative Results

  • CaT2I-AE-SD outperforms baseline models in FID on 256×256 two-layer images by about 3.5 on LL2I and 5 on LA.The authors attribute this to separate decoder branches for foreground, background, and mask components.Source: E.. Quantitative Results · p. 7, Table I · p. 8
  • The quantitative evaluation covers image quality, mask quality, and text-image relevance across the LL2I-E and LA test sets.FID is evaluated on both test sets, while CLIP score is evaluated on LL2I-E.Source: Table I · p. 8
  • CaT2I-AE-SD and its no-supervision variant produce more accurate masks than baselines, including on the human-annotated foreground test set.The full model is slightly worse than the no-supervision variant on mask quality, suggesting a trade-off with composed-image quality.Source: E.. Quantitative Results · p. 8, Table II · p. 8
  • CaT2I-AE-SD produces higher text-image relevance than all baselines at 256 resolution, while its 512 variant improves further and is comparable to SD-v1-4.The result is reported on the LL2I-E test set.Source: E.. Quantitative Results · p. 8, Table I · p. 8

F.. Effectiveness of Data Filtering

VI.. Conclusions and Discussions

The paper proposes two-layer layered-image generation with latent diffusion and establishes a benchmark using a 57.02M-image dataset. It also identifies extensions toward more flexible layer generation as future work.

VI.. Conclusions and Discussions

  • The paper proposes two-layer layered-image generation and creates a 57.02M high-quality layered-image dataset for latent-diffusion training.The benchmark evaluates image quality, image-text relevance, and mask quality.Source: VI.. Conclusions and Discussions · p. 8
  • The proposed method is not limited to two layers and can be applied to any fixed number of layers.This extends the formulation beyond the demonstrated two-layer setting.Source: VI.. Conclusions and Discussions · p. 8
  • A conditional model that generates a layer given existing layers is left as future work for arbitrary-layer generation.The paper presents this as a natural extension rather than an implemented capability.Source: VI.. Conclusions and Discussions · p. 9

Appendix I: More Results

The appendix provides additional samples, higher-resolution examples, dataset examples, and documented failure cases for CaT2I-AE-SD and baseline methods. It shows both the breadth of generated outputs and concrete errors in geometry, structure, and masks.

A.. More 256 samples

C.. Failure cases

  • Failure cases include objects with incorrect geometry or structures and masks that are inaccurate.Examples include faces, a dog, and inaccurate masks across the displayed columns.Source: C.. Failure cases · p. 11, Figure 8 · p. 11

D.. Data filtering

  • The appendix visualizes examples before and after filtering low-quality masks and inpaintings in the LAION-L2I dataset.These examples correspond to the data-filtering procedure described in Section 3.4.Source: D.. Data filtering · p. 11

E.. More examples

  • Additional examples depict two-layer images drawn from the 57.02M-image LAION-L2I dataset.The appendix provides ten more examples.Source: E.. More examples · p. 11

F.. Analysis and statistics

The analysis compares LAION-L2I with LAION-Aesthetics across image, mask, prompt, and image-text statistics, finding broadly similar distributions. The appendix details reconstruction losses for image components, composition consistency, and multiscale mask accuracy.

F.. Analysis and statistics

Appendix III: Implementation Details

  • The appendix applies equal-weight Latent Diffusion image losses separately to foreground, background, and composed-image components.Source: Appendix III: Implementation Details · p. 16
  • Training additionally uses composition loss to compare composed images and Laplacian loss to measure predicted-versus-ground-truth mask differences across pyramid levels.The composed image is formed from foreground, background, and mask; epsilon is set to 10^-6 for numerical stability.Source: Appendix III: Implementation Details · p. 16

Related papers

Loading 2307.09781v1…