Transparent Image Layer Diffusion using Latent Transparency

Motivation

• Transparent and layered generation remains underexplored despite strong demand because high-quality data are scarce and latent representations are difficult to modify safely.Commercial transparent assets are costly to access, while latent-distribution changes can degrade pretrained generators.Source: §1 · p. 1

Method

• LayerDiffuse encodes transparency as a regulated latent offset, preserving the original pretrained latent encoder and decoder to maintain image quality.The adjusted latent space supports training and fine-tuning diffusion models.Source: §1 · pp. 1–2
• The framework generates either individual transparent images or multiple transparent layers with consistent, harmonious blending.A shared attention mechanism supports layer consistency, while LoRAs adapt models to different layer conditions.Source: §1 · pp. 1–2

Dataset and applications

• A human-in-the-loop process produced a 1M-image transparent dataset spanning diverse topics and styles.The dataset was extended to multi-layer samples and supports conditional, structure-guided, and style-transfer applications.Source: §1 · p. 2

Results

• 97% of users preferred LayerDiffuse’s natively generated transparent content over generating-then-matting solutions.The study also found generated quality comparable to commercial transparent assets such as Adobe Stock.Source: §1 · p. 2

Contributions

• The paper contributes latent transparency, shared attention for layer blending, pretrained transparent-generation models, and LoRAs for multi-layer generation.It also reports additional ablative architectures for multi-layer generation.Source: §1 · p. 2

2.1 Hiding Images inside Perturbations

• 2.1 Hiding Images inside Perturbations: LayerDiffuse hides transparency in a small latent perturbation while avoiding changes to Stable Diffusion’s overall latent distribution.This builds on neural networks’ ability to encode additional information inside perturbations and reconstruct it later.Source: §2.1 · p. 2

2.2 Diffusion Probabilistic Models and Latent Diffusion

• 2.2 Diffusion Probabilistic Models and Latent Diffusion: Latent diffusion reduces training and computation demands by operating in latent space rather than directly on pixels.Stable Diffusion extends this latent-diffusion framework for large-scale image generation.Source: §2.2 · p. 2

2.3 Customized Diffusion Models and Image Editing

• 2.3 Customized Diffusion Models and Image Editing: Prior work adds text, image, region, or inversion-based controls to diffusion generation and editing.Examples include Textual Inversion, DreamBooth, ControlNet, T2I-Adapter, IP-Adapter, and attention-manipulation methods.Source: §2.3 · p. 2

2.4 Transparent Image Layer Processing

• 2.4 Transparent Image Layer Processing: Existing approaches address decomposition, matting, color separation, segmentation, and layered effects, whereas LayerDiffuse natively generates transparency.The paper contrasts its approach with methods that use matting for post-processing or dataset synthesis.Source: §2.4 · pp. 2–3, Figure 2 · p. 3

2.5 Image Harmonization

• 2.5 Image Harmonization: Harmonization methods align foreground and background color, contrast, and style to produce natural, consistent compositions.LayerDiffuse’s multi-layer generation similarly targets coherent blending between transparent layers.Source: §2.5 · p. 3

3.1 Latent Transparency

• The transparency encoder and decoder reconstruct premultiplied color and alpha channels, and a PatchGAN discriminator loss further improves reconstruction quality.The training objective combines reconstruction, identity, and discriminator terms, with default weights λ_recon = 1, λ_identity = 1, and λ_disc = 0.01.Source: §3 · p. 3, §3.1 · p. 4
• Latent transparency adds alpha support through a regulated latent offset while preserving the original pretrained latent distribution for diffusion fine-tuning.A frozen pretrained decoder measures whether the offset harms reconstruction; lower identity loss indicates that the modified latent remains compatible with Stable Diffusion.Source: §3.1 · pp. 3–4

3.2 Diffusion Model with Latent Transparency

• The diffusion model is fine-tuned on adjusted latents to predict noise from noisy latent images conditioned on timestep and text prompts.Training uses the standard diffusion objective while operating in the transparency-adjusted latent space.Source: §3.2 · p. 4

3.3 Generating Multiple Layers

• Shared attention merges foreground and background denoising through concatenated key, query, and value vectors, while separate LoRAs adapt each layer.Setting the foreground or background noise to zero yields foreground-conditioned or background-conditioned generation, respectively.Source: §3.3 · pp. 4–5

3.4 Dataset Preparation and Training Details

• Human-in-the-loop collection expands the transparent dataset to 45K images before automated filtering produces 1M sample pairs, while multi-layer pairs are synthesized with structured prompts and inpainting.Across collection rounds, human selection removes defects and increases dataset diversity; the final training uses 1M captioned samples and multi-layer pairs.Source: §3.4 · pp. 5–6, Figure 4 · p. 5

4.1 Qualitative Results

• Qualitative results demonstrate diverse single-image transparency, including glass, hair, fur, glowing light, fire, and magic effects across content topics.Source: §4.1 · p. 7
• Multi-layer generation produces blended compositions whose illumination and geometric relationships remain consistent while retaining Stable Diffusion’s aesthetic quality.Source: §4.1 · p. 7

4.2 Conditional Layer Generation

• Conditional generation creates coherent foreground-background geometry and illumination, while iterative generation supports arbitrary layer counts by repeatedly conditioning on blended prior layers.The iterative procedure blends existing layers into one RGB image before generating each new foreground layer.Source: §4.2 · p. 7, §4.3 · pp. 7–8, Figure 8 · p. 8

4.4 Controllable Generation

• ControlNet preserves global structure and enables harmonious compositions with consistent illumination, including foreground-background reflections in reflective-ball examples.Source: §4.4 · p. 8

4.5 Ablative Study

• Adding alpha channels directly to UNet severely degrades pretrained-model quality, whereas modifying the VAE causes unstable training and repeated collapse.The direct-channel approach changes latent feature distributions, while VAE fine-tuning changes the latent distribution too much.Source: §4.5 · pp. 9–10

4.6 Relationship to Image Matting

• Native transparent generation handles semi-transparent fire, white fur on white backgrounds, and shadows that matting methods struggle to separate after blending.The authors suggest native synthesis can produce clean elements and may also provide outputs for training matting models.Source: §4.6 · pp. 10–11

4.7 Perceptual User Study

• In user-preference experiments, users preferred our native transparent image generation over generation-then-matting in more than 97% of cases, while Adobe Stock was marginally preferred, 54.7% versus 45.3%.Source: §4.7 · p. 12

4.8 Raw RGBA Channels

• Raw-channel inspection shows smooth RGB bleeding reduces aliasing in alpha-blended regions, while augmentation-trained decoders tolerate missing or incorrect latent offsets and community models preserve transparency without additional training.The decoder training drops 30% of offsets, and the community-model tests include Minecraft, pixel-art, anime, and prompt styles.Source: §4.8 · pp. 12–13, §4.9 · p. 13, §4.10 · p. 13

• 97% of cases favored the method’s natively generated transparent content over generating-then-matting solutions.Source: §5 · p. 14
• The method encodes alpha transparency into a latent diffusion model to generate individual transparent images or coherent transparent layers.Source: §5 · p. 14
• Training used 1M transparent image-layer pairs collected through a human-in-the-loop scheme.Source: §5 · p. 14
• Generated transparent-image quality was comparable to assets from commercial stock sites.Source: §5 · p. 14

• Pixels with alpha value strictly equal to zero are treated as undefined because their RGB values cannot affect alpha-blended appearance.Source: A PADDED RGB CHANNELS · p. 16
• A local Gaussian filter propagates colors into undefined pixels to avoid high-frequency edge patterns and potential artifacts.The filter uses a 13 × 13 kernel and is applied 64 times to propagate colors throughout undefined regions.Source: A PADDED RGB CHANNELS · p. 16

• The latent transparency encoder matches Stable Diffusion’s latent VAE encoder architecture but accepts four-channel RGBA input and is trained from scratch.Its output convolution is zero-initialized to avoid harmful initial noise.Source: B NEURAL NETWORK ARCHITECTURE · p. 16
• The latent transparency decoder uses a UNet combining Stable Diffusion-style encoder and decoder components with a latent injection and skip connections.The input latent is added to the middle block, while encoder feature maps enter corresponding decoder blocks.Source: B NEURAL NETWORK ARCHITECTURE · p. 16

• The discriminator uses the Latent Diffusion VAE PatchGAN architecture, objective, and training schedule with four-channel input.Source: C PATCHGAN DISCRIMINATOR · p. 16
• Its five convolution-normalization-SiLU layers reduce the input to a 64 × 64 × 1 patch-wise real/fake output.The final layer omits normalization and activation.Source: C PATCHGAN DISCRIMINATOR · p. 16

D SINGLE TRANSPARENT IMAGES

• Figures 18–33 provide additional results for generating single transparent images.Source: D SINGLE TRANSPARENT IMAGES · p. 16
• The results span a dedicated set of single-image examples across Figures 18–33.Source: D SINGLE TRANSPARENT IMAGES · p. 16
• The section documents further single transparent image outputs from the framework.Source: D SINGLE TRANSPARENT IMAGES · p. 16

E MULTIPLE TRANSPARENT LAYERS

• Figures 34–36 provide additional results for generating multiple transparent layers.Source: E MULTIPLE TRANSPARENT LAYERS · p. 16
• The section collects further multi-layer generation examples across three figures.Source: E MULTIPLE TRANSPARENT LAYERS · p. 16
• These results document additional outputs involving multiple transparent layers.Source: E MULTIPLE TRANSPARENT LAYERS · p. 16

F FOREGROUND-CONDITIONED BACKGROUNDS

• Figures 37–38 provide additional results for generating backgrounds conditioned on foreground content.Source: F FOREGROUND-CONDITIONED BACKGROUNDS · p. 16
• The section documents two figures of foreground-conditioned background generation.Source: F FOREGROUND-CONDITIONED BACKGROUNDS · p. 16
• These results show further examples of background generation controlled by foreground inputs.Source: F FOREGROUND-CONDITIONED BACKGROUNDS · p. 16

G BACKGROUND-CONDITIONED FOREGROUNDS

• Figure 39 presents additional results for generating foregrounds conditioned on background content.Source: G BACKGROUND-CONDITIONED FOREGROUNDS · p. 16

D SINGLE TRANSPARENT IMAGES

• The displayed single transparent-image examples cover prompts for an apple, a cat, a man, a man with messy hair, a woman, and a woman with messy hair, each at 896 × 1152 resolution.Source: Figure 18 · p. 17, Figure 19 · p. 18, Figure 20 · p. 19

E MULTIPLE TRANSPARENT LAYERS

• Figures 34–36 present additional examples of multiple transparent layer generation.Source: Figure 34 · p. 33, Figure 35 · p. 34, Figure 36 · p. 35

F FOREGROUND-CONDITIONED BACKGROUNDS

• Figures 37–38 show additional foreground-conditioned background results using foreground inputs and text prompts.Source: Figure 37 · p. 36, Figure 38 · p. 37