PaperlayerView exact revision on arXiv

Source-linked AI summary

AlphaVAE: Unified End-to-End RGBA Image Reconstruction and Generation with Alpha-Aware Representation Learning

Zile Wang, Hao Yu, Jiabo Zhan, Chun Yuan

arXiv:2507.09308v1cs.CVcs.AI

TL;DR

Transparent and layered image generation lacks comprehensive RGBA benchmarks and unified representations. The paper introduces ALPHA and ALPHAVAE, combining alpha-aware evaluation with a unified RGBA VAE, and reports stronger reconstruction and generation using only 8K images. The study is limited to LoRA fine-tuning rather than full-parameter or ControlNet adaptation.

  • Problem

    RGBA reconstruction and generation lack comprehensive benchmarks, while existing methods face limited data and representation challenges.

    Source: §1 · pp. 1–2

  • Method

    ALPHA extends RGB metrics through alpha blending, while ALPHAVAE adds a dedicated alpha channel and composite reconstruction and regularization losses to a pretrained RGB VAE.

    Source: §1 · p. 2, §4.2.3 · p. 6

  • Results

    +4.9 dB PSNR and +3.2% SSIM over LayerDiffuse are achieved with 8K training images, alongside superior transparent image generation after latent-diffusion fine-tuning.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    ALPHA and ALPHAVAE provide a unified evaluation and modeling framework for RGBA reconstruction and transparent image generation with substantially less training data.

    Source: §6 · pp. 9–10

    The study evaluates only parameter-efficient LoRA fine-tuning, not full-parameter fine-tuning or ControlNet adaptation.

    Source: C Limitations · p. 15

Abstract

from arXiv · show

Recent advances in latent diffusion models have achieved remarkable results in high-fidelity RGB image synthesis by leveraging pretrained VAEs to compress and reconstruct pixel data at low computational cost. However, the generation of transparent or layered content (RGBA image) remains largely unexplored, due to the lack of large-scale benchmarks. In this work, we propose ALPHA, the first comprehensive RGBA benchmark that adapts standard RGB metrics to four-channel images via alpha blending over canonical backgrounds. We further introduce ALPHAVAE, a unified end-to-end RGBA VAE that extends a pretrained RGB VAE by incorporating a dedicated alpha channel. The model is trained with a composite objective that combines alpha-blended pixel reconstruction, patch-level fidelity, perceptual consistency, and dual KL divergence constraints to ensure latent fidelity across both RGB and alpha representations. Our RGBA VAE, trained on only 8K images in contrast to 1M used by prior methods, achieves a +4.9 dB improvement in PSNR and a +3.2% increase in SSIM over LayerDiffuse in reconstruction. It also enables superior transparent image generation when fine-tuned within a latent diffusion framework. Our code, data, and models are released on https://github.com/o0o0o00o0/AlphaVAE for reproducibility.

1 Introduction

RGBA generation is underexplored because benchmarks and large-scale data are scarce, while existing methods often separate RGB and alpha processing. ALPHAVAE and ALPHA address these gaps with unified representation learning, RGBA evaluation, and strong results using limited data.

Motivation

  • RGBA generation remains severely underexplored because large-scale datasets are scarce and latent diffusion models are sensitive to latent-space shifts.Existing approaches such as Text2Layer and LayerDiffuse [8] are part of this underdeveloped setting.Source: §1 · pp. 1–2

Benchmark

  • ALPHA extends PSNR, SSIM, and LPIPS to RGBA images by alpha blending over canonical backgrounds, providing a unified transparency-aware benchmark.The benchmark addresses the inability to apply standard RGB metrics directly to four-channel data.Source: §1 · p. 2

Results

  • +4.9 dB PSNR and +3.2% SSIM over LayerDiffuse are achieved with 8K training images versus its 1M samples, while fine-tuned diffusion also improves transparent image generation.The comparison spans multiple quantitative metrics and latent-diffusion generation.Source: §1 · p. 2

Method

  • ALPHAVAE augments a pretrained three-channel VAE with a dedicated alpha channel and specialized losses for unified end-to-end RGBA reconstruction.Its objective combines alpha-blended pixel reconstruction, patch-level fidelity, perceptual consistency, and dual KL constraints.Source: §1 · p. 2

Evaluation

  • The framework is validated across multiple datasets for VAE reconstruction and latent-diffusion transparent image generation using much less training data.The authors report state-of-the-art reconstruction and superior generation in these evaluations.Source: §1 · p. 2

2 Related Work

Prior transparency methods span decomposition, matting, layered generation, and alpha-aware representation learning, but existing RGBA generators still struggle with accurate, coherent alpha channels.

Alpha Processing

  • Image decomposition, layer extraction, matting, and diffusion-based layered generation provide related approaches to modeling transparency and alpha structure.Examples include DreamLayer, Semantic Image Matting [23], Glance and Focus Matting, and Referring Image Matting [20].Source: §2 · p. 3

Transparency-Aware Representation Learning

  • Text2Layer, LayerDiffuse [8], and ART introduce layered, latent-offset, or region-controlled transparency modeling, but do not consistently produce high-quality coherent alpha channels.LayerDiffuse [8] is described as producing representations resembling coarse matting masks.Source: §2 · p. 3

3 ALPHA: Evaluating RGBA Image Reconstruction and Generation

ALPHA evaluates RGBA reconstruction and generation by compositing images over canonical backgrounds and averaging standard RGB metrics, while its dataset repurposes matting data into diverse four-channel examples.

3 ALPHA: Evaluating RGBA Image Reconstruction and Generation

  • ALPHA is the first benchmark designed specifically for four-channel RGBA reconstruction and generation, extending RGB evaluation through background blending.It provides metrics and a dedicated dataset for jointly assessing RGB and alpha quality.Source: §3 · p. 3

3.1 Evaluation Protocol

  • RGBA images are alpha-blended with canonical RGB backgrounds before applying standard metrics, avoiding penalties for visually irrelevant RGB values where α = 0.The resulting composited images remain compatible with RGB metrics while respecting transparency.Source: §3.1 · p. 3
  • The evaluation pipeline records ground-truth and reconstructed sets after compositing, then applies RGB metrics to quantify both visual fidelity and transparency-aware quality.The formulation defines an RGBA counterpart for any RGB metric by averaging across the background set.Source: §3.1 · pp. 3–4
  • The benchmark averages metric values over nine solid backgrounds: black, gray, white, red, green, blue, yellow, cyan, and magenta.These low-frequency, texture-free backgrounds are intended to reduce semantic bias and stabilize metric values.Source: §3.1 · p. 4

3.2 Dataset Collection

  • The dataset converts paired RGB foregrounds and alpha mattes from ten matting datasets into 8,124 aligned RGBA images with diverse transparency patterns.This avoids manual labeling and supports scalable training data construction.Source: §3.2 · p. 4
  • A standardized 5% test split produces 7,722 training images and 402 testing images, covering challenging effects such as hair, glass, and soft shadows.The ALPHA train and test splits are intended to assess transparent image generation across varied scenarios.Source: §3.2 · p. 4

4 ALPHAVAE: Teaching VAE to Reconstruct RGBA Images

ALPHAVAE adapts a pretrained RGB VAE for unified RGBA reconstruction by extending its channel interfaces and training with transparency-aware objectives. Its losses combine blended reconstruction, perceptual consistency, latent regularization, and adversarial fidelity components.

4.1 Model Architecture and Initialization

  • ALPHAVAE extends a pretrained three-channel VAE to four-channel RGBA inputs and outputs by modifying the encoder’s first and decoder’s final convolutions.The RGB parameters are retained for the original channels while new alpha-channel parameters are introduced for fine-tuning.Source: §4.1 · p. 5
  • The model preserves pretrained RGB representations while gradually learning transparency features through its initialization and fine-tuning strategy.The pipeline copies pretrained RGB parameters into the original channel slices and leaves the existing biases unchanged.Source: §4.1 · p. 5

4.2.1 Reconstruction Loss

  • ALPHAVAE replaces direct tensor reconstruction error with Alpha-Blending Mean Square Error, comparing alpha-blended reconstructions and originals across backgrounds.The background distribution is estimated from the ImageNet training split, with its statistics precomputed for the expected loss.Source: §4.2.1 · pp. 5–6

4.2.2 Perceptual Loss

  • Perceptual loss feeds alpha-blended reconstructions and targets into a pretrained three-channel perceptual model, using canonical backgrounds to make RGBA inputs compatible.The perceptual model cannot directly process four-channel images, so blending precedes perceptual comparison.Source: §4.2.2 · p. 6

4.2.3 Regularization Loss

  • Dual KL constraints regularize the encoder toward a standard Gaussian while also aligning its latent distribution with the pretrained reference encoder.The reference constraint supports compatibility between the fine-tuned four-channel encoder and downstream diffusion models.Source: §4.2.3 · p. 6

4.2.4 GAN Loss

  • The training objective additionally incorporates patch-based adversarial fidelity through a discriminator and an adaptive weight used in the final objective.The supplied passages identify the discriminator, adaptive weighting, numerical-stability constant, and final objective construction.Source: §4.2.4 · p. 7

5 Experiments

Experiments evaluate ALPHAVAE’s implementation, reconstruction and generation quality, qualitative transparency preservation, and ablations. Across datasets and tasks, the method improves quantitative metrics, preserves alpha and RGB fidelity, and retains strong generation quality with limited training data.

5.1 Implementation Details

  • The experiments fine-tune SDXL and FLUX VAEs for 30,000 iterations, then adapt the corresponding diffusion modules with rank-64 LoRA for 20,000 iterations.Training uses random solid-color compositing augmentation with probability 0.3 and a global batch size of 8.Source: §5.1 · p. 7

5.2 Quantitative Results

  • +4.86 dB PSNR on AIM-500 with FLUX and +5.75 dB on the Alpha benchmark demonstrate ALPHAVAE’s strongest reconstruction gains over LayerDiffuse.On AIM-500, FLUX PSNR rises from 32.09 to 36.94; rFID also decreases to 11.79, while Alpha-benchmark rFID reaches as low as 1.66.Source: §5.2 · pp. 7–8
  • A 6.15-point FID reduction on the ALPHA split, from 80.27 to 74.12, shows improved transparent-image generation over LayerDiffuse.On AIM-500, the method achieves FID 155.66 versus LayerDiffuse’s 160.27.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 Qualitative Results

  • ALPHAVAE preserves structure, transparency, and RGB appearance better than LayerDiffuse across checkerboard, black, white, and solid-color composites.LayerDiffuse tends to shrink transparency regions, reduce opacity, and introduce reddish color casts after compositing.Source: §5.3 · p. 8, Figure 3 · p. 8, Figure 4 · p. 9
  • The diffusion model generates semantically aligned images from out-of-distribution AIM-500 examples despite training on only around 8K images.The qualitative results indicate a strong generative prior learned from the limited training set.Source: §5.3 · p. 8

5.4 Ablative Study

  • ALPHAVAE achieves the best results across PSNR, SSIM, LAION, rFID, and LPIPS in the reported ablations.Within FLUX-VAE, removing the reference KL term performs particularly well on AIM-500, while the term is intended to preserve generation quality during diffusion training.Source: §5.4 · p. 9, Table 4 · p. 9

6 Conclusion

The paper introduces ALPHA for transparency-aware evaluation and ALPHAVAE for unified RGBA reconstruction. Experiments show strong reconstruction and generation performance with only 8K training images, while motivating extensions to richer editing and video settings.

6 Conclusion

  • ALPHA extends PSNR, SSIM, and FID to RGBA images through alpha blending, while ALPHAVAE adds an alpha channel and specialized losses to a pretrained RGB VAE.Together, they address transparent-image evaluation and reconstruction.Source: §6 · p. 9
  • With only 8K training images, the method outperforms approaches trained on orders of magnitude more data in VAE reconstruction and transparent image generation.The generation result applies when ALPHAVAE is fine-tuned within a latent diffusion framework.Source: §6 · p. 10

A Formula Derivation

The derivation simplifies the background-color expectation by exploiting independent, identically distributed RGB channels. It then expresses the result in an equivalent channel-wise form and visualizes the underlying pixel distribution.

A Formula Derivation

  • The derivation of Equation (8) reduces the analysis to one background pixel because the RGB background channels are independent and identically distributed.Substitution into the preceding expression yields the stated norm involving the second moment of the background.Source: A Formula Derivation · p. 14
  • The implementation uses an equivalent channel-wise expression in which E_ch denotes expectation over RGB channels.This form preserves the same expectation while making channel-wise computation explicit.Source: A Formula Derivation · p. 14
  • Figure 5 visualizes the scaled pixel-value distribution of the ImageNet training split used for the background analysis.Pixel values are shown in the range [0, 255].Source: Figure 5 · p. 14

B Details of Hyperparameters.

The appendix specifies background statistics and weights for reconstruction, perceptual, KL, and adversarial losses. These values make the composite training objective operationally explicit.

B Details of Hyperparameters

  • Background pixels are rescaled to [-1, 1], with E[b] = (-0.0357, -0.0811, -0.1797) and E[b2] = (0.3163, 0.3060, 0.3634).These first and second raw moments are pre-computable terms in the reconstruction loss.Source: B Details of Hyperparameters · p. 15
  • The reconstruction loss has weight 1.0, LPIPS perceptual loss 0.5, latent KL 10^-6, and reference KL 10^-16.The reference KL term receives a much smaller weight than the primary latent KL term.Source: B Details of Hyperparameters · p. 15
  • GAN loss begins after 4000 steps with generator-loss weight 1.0.Source: B Details of Hyperparameters · p. 15

C Limitations

The study evaluates only parameter-efficient LoRA fine-tuning, leaving full-parameter fine-tuning and ControlNet adaptation for future work.

  • LoRA is the only fine-tuning method evaluated; full-parameter fine-tuning and ControlNet are excluded because they require significantly more computation.The authors identify these approaches as a future-work direction.Source: C Limitations · p. 15

D More Qualitative Analysis

Additional qualitative results are presented in Figures 6–25, with all images rendered at 1024 × 1024 resolution.

E Details of Experimental Results

The appendix provides detailed experimental results through Tables 5–30, including metric breakdowns across color categories, subtypes, model variants, and ablations.

Related papers

Loading 2507.09308v1…