PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning

Ciara Rowles, Reshinth Adithyan, Nikhil Pinnaparaju, Vikram Voleti, Mark Boss

arXiv:2605.30257v1cs.CV

TL;DR

Image layer decomposition lacks uniquely correct targets, making paired supervision poorly suited to judging editability. Stable-Layers fine-tunes a pretrained model with VLM-scored reinforcement learning and reports better semantic separation, cleaner layers, and lower per-layer reconstruction error than the baseline.

  • Problem

    Image layer decomposition has many plausible solutions, so single-target supervision poorly captures editability criteria such as semantic separation, clean alpha mattes, and faithful occlusion handling.

  • Method

    Stable-Layers fine-tunes a pretrained decomposition model on unlabeled images using sampled candidates, structured five-criterion VLM scores, and relative grid calibration for GRPO updates.

  • Results

    Stable-Layers yields better semantic separation, cleaner layers, and lower per-layer reconstruction error than the Qwen-Image-Layered baseline on Crello and held-out evaluations.

  • Takeaways & Limitations

    Stable-Layers provides a recipe for improving edit-oriented generators with judge feedback instead of targets, within the demonstrated layer-decomposition setting.

  • Takeaways & Limitations

    The method depends on a proprietary VLM whose API costs and score distributions may vary, while evaluation lacks human studies and covers only up to five layers.

Abstract

from arXiv · show

We present Stable-Layers, a reinforcement learning framework that eliminates the need for paired supervision by fine-tuning a pretrained layer decomposition model using only feedback from a vision-language model (VLM). Starting from Qwen-Image-Layered, we apply Flow-GRPO with LoRA adaptation, sampling multiple candidate decompositions per image, scoring them with a VLM, and optimising the policy from group-relative advantages. The key challenge lies in designing a reliable reward signal: VLMs scoring samples in isolation tend to compress their judgements into a narrow band, leaving GRPO with little within-group variance to learn from. We address this with a two-stage evaluation pipeline that pairs structured per-sample scoring across five edit-centric criteria with a grid-based calibration step in which the VLM re-scores all candidates side-by-side. Stable-Layers produces decompositions with stronger layer separation, fewer blank or artifact-heavy layers, and lower per-layer reconstruction error on the Crello dataset compared to the base model.

1 Introduction

Stable-Layers frames image layer decomposition as an edit-oriented task with ambiguous supervision and multi-dimensional quality criteria. It addresses reward compression and optimization instability by combining a two-phase VLM feedback protocol with a RatioNorm reformulation for packed latent representations.

  • Task and challenges: Image layer decomposition separates images into editable RGBA layers, but supervision is difficult because one image admits many plausible decompositions.Quality depends on downstream usability, including semantic separation, clean alpha mattes, minimal redundancy, and faithful handling of occluded content.
  • Task and challenges: VLM-based rewards face score compression because decomposition quality spans correlated criteria including semantic disentanglement, alpha cleanliness, inpainting plausibility, feature allocation, and content validity.Candidates in a sampled group are often simultaneously good or bad across most criteria, limiting within-group learning signal.
  • Optimization: Qwen-Image-Layered’s packed RGBA latent sequence inflates effective dimensionality D by ∼5×, suppressing per-step log-ratio standard deviation and creating an optimization-stability challenge.The issue arises under GRPO-Guard’s RatioNorm, which uses a spatial mean of per-element log-probabilities.
  • Method and contributions: Stable-Layers introduces a two-phase VLM reward protocol and a RatioNorm reformulation tailored to packed latent representations in flow-matching reinforcement learning.The method uses LoRA adaptation on Qwen-Image-Layered and trains entirely on Fine-T2I images without layer annotations.
  • Method and contributions: Stable-Layers aims to train edit-oriented generators from judge feedback instead of targets by converting VLM judgments over sampled candidates into learning signals that enhance editability.The optimization machinery is presented as a tool for this broader training recipe.

2 Related Work

Prior work applies reinforcement learning, preference optimization, and VLM-based reward modeling to visual generation, while image-layer methods generate or recover layered representations through diverse architectures. Stable-Layers follows Flow-GRPO with stabilization and uses structured multi-criteria VLM judgements for layer-stack credit assignment.

  • RL and reward modelling for visual generation: Diffusion and flow-matching methods optimize visual-generation policies using policy gradients, differentiable backpropagation, preference objectives, or online Flow-GRPO rollouts.DDPO and DPOK formulate diffusion sampling as multi-step MDPs; DRaFT and AlignProp backpropagate through differentiable sampling chains, while Diffusion-DPO uses static comparisons.
  • RL and reward modelling for visual generation: VLM reward research includes numeric-logit extraction, judge-reliability studies, preference training, and self-improving critics.TOPReward avoids text-generated numeric-score brittleness by using token-completion logits, while MJ-Bench evaluates VLM reliability as a judge.
  • RL and reward modelling for visual generation: Stable-Layers follows Flow-GRPO with GRPO-Guard stabilization and replaces scalar or pairwise rewards with structured multi-criteria VLM judgements for richer layer-stack credit assignment.The criteria include alpha cleanliness, semantic separation, and content validity.
  • Image layer decomposition and generation: Image-layer generation methods produce multi-layer raster outputs using inter-layer attention, harmonized decoding, or generate-then-disassemble pipelines.Representative methods include LayerDiff, DreamLayer, LayerFusion, PSDiffusion, and LayeringDiff.

3 Background

The background combines rectified flow and its SDE augmentation with GRPO to provide stochastic, log-probability-based policy optimisation for flow-matching models. Rectified flow defines deterministic trajectories, while Flow-GRPO restores tractable per-step probabilities and GRPO learns from within-group reward advantages.

  • Framework components: The method builds on flow matching, SDE-augmented flow matching, and GRPO for generative modeling and policy optimisation.These three components form the framework’s methodological foundation.
  • Flow Matching and Rectified Flows: Rectified flow linearly interpolates between data and Gaussian noise and learns a velocity field through a regression loss.The interpolation is xt = (1 − t)x0 + tx1, with x0 drawn from πref and x1 from N(0, I).
  • Flow Matching and Rectified Flows: Deterministic ODE sampling lacks tractable per-step log-probabilities, preventing direct policy-gradient use without an SDE formulation.Inference integrates xt−∆t = xt + vθ(xt, t)∆t along a deterministic trajectory.
  • SDE-Augmented Flow Matching (Flow-GRPO): Flow-GRPO adds stochastic dynamics that preserve learned marginals while enabling tractable log-probability computation for reinforcement learning.Its diffusion coefficient is σt = at/(1 − t).
  • Group Relative Policy Optimization: GRPO computes within-group advantages from rewards and optimises a clipped surrogate with a KL penalty to the reference policy.The advantage uses the group reward mean and standard deviation, with ν=10−4 for numerical stability.

1. Generate · 3. Train · 2. Score

Stable-Layers samples candidate decompositions, evaluates them with a two-phase VLM reward, and uses GRPO to update LoRA parameters. The scoring pipeline includes a second-stage grid calibration step.

  • 2. Score: GRPO loss updates the LoRA parameters after candidate scoring.The update is specified as a GRPO loss update applied to LoRA parameters.
  • 2. Score: Phase 2 performs grid calibration as part of the VLM-based scoring process.The passage identifies grid calibration as the second phase.
  • 2. Score: The training pipeline samples G candidates, scores them with a two-phase VLM reward, and replays them through GRPO updates to LoRA parameters.This summarizes the complete pipeline shown in Figure 2.

4 Method

Stable-Layers fine-tunes Qwen-Image-Layered with unlabeled images and post-hoc VLM judgements, using Flow-GRPO to optimize candidate layer decompositions. Its reward pipeline combines structured per-sample scoring with grid-based relative calibration to restore within-group discrimination.

  • Training framework: Stable-Layers requires no layer annotations, paired examples, or synthetic decomposition targets, instead using unlabeled images and post-hoc VLM judgements as supervision.Training follows Flow-GRPO’s three-phase loop: SDE sampling generates G candidates, the VLM scores them, and stored trajectories support GRPO updates.
  • Base model: Qwen-Image-Layered generates N-layer RGBA decompositions conditioned on an input image using a flow-matching transformer architecture.The model includes a 3D VAE, sequence-based transformer, and text encoder for prompt conditioning.
  • VLM reward: Each layer is composited onto a solid white background before VLM evaluation because VLMs cannot meaningfully interpret raw alpha channels.Transparent regions render as white, and Phase 1 presents the RGB composite alongside N white-background layer images at 320 × 320.
  • VLM reward: The VLM evaluates each candidate independently on five criteria using anchored integer scores from 0–5.The rubric includes semantic separation, alpha cleanliness, background inpainting, and feature distribution; the supplied passage truncates before the fifth criterion.
  • VLM reward: Phase 2 tiles all G composites into a labelled 256x256 comparison grid and asks the VLM to re-score candidates relative to one another when Phase 1 scores compress.The calibration step is designed to spread compressed scores and restore within-group variance for GRPO advantage normalisation.
  • Training setup: Output-layer counts are sampled uniformly from [min_layers, max_layers], typically [2], exposing the model to variable decomposition complexity during training.Images are resized to 640 × 640 and normalised to [−1, 1].

5 Experimental Setup

Stable-Layers trains on normalized 640 × 640 images with 2–5 output layers sampled during training, while Crello evaluation uses 2–4 layers for comparison. Experiments compare the calibrated method with the base model, uncalibrated Flow-GRPO, and LayerD, but not supervised fine-tuning requiring unavailable paired decompositions.

  • Data and evaluation setup: Training images are resized to 640 × 640, normalized to [−1, 1], and paired with 2–5 output layers sampled uniformly per step.Crello evaluation restricts generation to L ∈{2, 3, 4} for direct comparison with the base model.
  • Baselines: The experiment compares Stable-Layers with the unfine-tuned Qwen-Image-Layered base checkpoint and Flow-GRPO without grid calibration.The uncalibrated reference uses identical hyperparameters, LoRA configuration, and data, but only Phase 1 individual scoring.
  • Baselines: LayerD provides an additional behavioural comparison because it often returns the input largely intact as a single layer instead of producing multi-layer separation.The paper frames LayerD as a different design choice with different downstream implications.
  • Experimental scope: The study omits supervised fine-tuning with reconstruction loss because natural images lack paired ground-truth layer decompositions.This missing supervision is the gap motivating Stable-Layers.

6 Results

Stable-Layers improves held-out decomposition quality over training, with fewer bad layers, more even feature distributions, better Layer 0 inpainting, and lower Crello per-layer reconstruction error than the base model. Qualitative and ablation results attribute gains to plausible background completion, semantic layer separation, and relative grid calibration, while detailed prompting underperforms.

  • Held-out evaluation: Bad layers fall from ∼1.65 to ∼0.4, while feature distribution evenness rises from ∼0.53 to ∼0.73 and Layer 0 inpainting quality from ∼0.38 to ∼0.62.These held-out metrics evaluate 480 LAION-Aesthetics images across training; lower bad-layer counts and higher evenness and inpainting quality are preferred.
  • Qualitative comparison: Stable-Layers replaces degenerate layer-0 outputs with plausible scene completions and separates semantic foreground elements instead of duplicating the composite across layers.The base model produces black or flat-filled layer 0 and near-complete input copies in foreground slots on both held-out examples.
  • Crello reconstruction quality: Stable-Layers achieves lower mean per-layer RGB L1 error than the base model across all tested output layer counts on the Crello test set.Errors use best-match assignment because fine-tuning may reorder layers without changing decomposition quality.
  • Crello reconstruction quality: Stable-Layers reduces layer-0 dominance, while occasional per-slot regressions reflect content reorganisation under best-match assignment rather than necessarily worse decomposition quality.The fine-tuned model populates previously near-empty slots with real content, which can change slot-level matches.
  • Text conditioning ablation: The detailed fixed prompt underperforms the basic prompt on every axis, with Layer 0 quality degrading from ∼0.44 to ∼0.32 instead of improving from ∼0.40 to ∼0.74.Bad layers fall more slowly and feature distribution evenness plateaus lower in the detailed-prompt run.
  • Calibration ablation: Phase 2 calibration improves image quality: calibrated SSIM averages 0.52 versus 0.45 uncalibrated across steps 80–200, while bad-layer reduction remains similar.Both runs reduce bad layers from 1.008 to 0.4–0.7, but all three Layer 0 quality metrics separate from step 80 onward.

7 Conclusion … E Architecture Details

Stable-Layers uses Flow-GRPO with a two-phase VLM reward protocol to produce a discriminative learning signal for improving layer decomposition. The appendices specify its training procedure, reward prompts, qualitative examples, calibration evidence, and Qwen-Image-Layered architecture with LoRA adaptation.

  • 7 Conclusion: Stable-Layers combines SDE-augmented Flow-GRPO policy optimisation with structured per-sample scoring and relative grid calibration to improve layer separation, content validity, and feature distribution.The conclusion describes this protocol as converting black-box VLM feedback into a sufficiently discriminative learning signal without paired supervision.
  • A Algorithm Pseudocode: The training step encodes a condition image, generates a group of candidate decompositions, scores them, and applies SDE-step updates using group-normalised rewards and stabilisation.Algorithm 1 specifies a LoRA policy, reference policy, reward model, group size, latent dimensionality, clipping parameters, KL weight, and learning rate.
  • B Reward Prompt Template: The reward model is gemini-3-flash-preview, and Phase 1 scores each decomposition from 0-5 on edit-centric criteria before Phase 2 re-scores candidates relatively on a grid.The reward-model snapshot is pinned between October 2025 and submission, and score distributions may drift across versions.
  • C Additional Qualitative Examples: Figure 6 extends qualitative evaluation with Stable-Layers decompositions on held-out LAION-Aesthetics images spanning photographs, product shots, automotive renders, and complex scenes.The gallery covers landscapes, wildlife, portraits, studio products, automotive renders, and compositions with varying foreground complexity.
  • D Additional Calibration Ablation Metrics: Calibrated training maintains a small but consistent lead over uncalibrated training on combined Layer 0 quality and edge-density sharpness from mid-training checkpoints onward.The same qualitative pattern appears in SSIM, and the evidence indicates improved fine-grained background quality without affecting bad-layer reduction.
  • E Architecture Details: Qwen-Image-Layered uses a 3D VAE, a sequence-based transformer, and a text encoder, with RGBA frames compressed 8× spatially into a 16-channel latent space.The transformer uses 2×2 patch-packed latents with token dimension 16 × 4 = 64, while the condition image shares the VAE pipeline and is concatenated along the sequence dimension.
  • E Architecture Details: LoRA adaptation uses rank r=16 and α=16 on all attention projection and feed-forward layers, while all other parameters remain frozen.Per-frame spatial metadata distinguishes generated layer tokens from conditioning tokens in attention.

F Training Implementation Details · G Data Preprocessing · H LayerD Comparison Setup

The implementation uses cost-conscious Flow-GRPO training with replay-based KL computation and stabilised gradient scaling, while preprocessing varies decomposition complexity and LayerD outputs are padded for fixed-slot evaluation.

  • F Training Implementation Details: 8 SDE steps are typically used during training versus 50 at inference, with CFG disabled at training (CFG = 1.0) and enabled at evaluation (CFG = 4.0).This halves forward passes per training step without degrading final sample quality, according to Liu et al. [16].
  • F Training Implementation Details: Trajectory replay recomputes each stored transition mean with the LoRA-adapted transformer and computes the KL reference with adapters disabled at no additional memory cost.Both predictions use the same forward pass structure, with gradients enabled for the current policy computation.
  • F Training Implementation Details: Per-step centring is applied before RatioNorm to preserve O(1) ratio magnitudes, and the policy loss is scaled by δ = 1/∆t to equalise gradient magnitudes across the noise schedule.The modification addresses near-zero log-ratios caused by the high-dimensional latent sequence while retaining centring and variance stabilisation.
  • F Training Implementation Details: K=1 PPO-style epoch per round uses G=16, ϵc=0.2, β=10−3, η=10−5, cadv=5.0, and gradient clip ∥∇∥max=1.0; the 600-step run used 8× NVIDIA H200 GPUs for ∼48 hours.The optimizer is AdamW.
  • G Data Preprocessing: Training images come from Fine-T2I, are resized to 640 × 640 and normalised to [−1, 1], while captions provide text prompts and images are shuffled each epoch.The number of output layers is sampled uniformly from [min_layers, max_layers], typically [2], to expose variable decomposition complexity.
  • H LayerD Comparison Setup: LayerD is compared on the held-out LAION-Aesthetics set used in Figure 4, with outputs padded by empty white layers to reach the four slots scored by the metrics.LayerD often produces fewer than four layers, sometimes leaving the input largely intact or returning one layer containing the full image.

I Crello Evaluation Metric · J Text Conditioning Ablation Prompts · NeurIPS Paper Checklist

The paper evaluates Crello decompositions with an order-aware, best-match reconstruction metric, tests two fixed text-conditioning prompts, and documents claims, reproducibility, limitations, and broader impacts in the NeurIPS checklist.

  • I Crello Evaluation Metric: Crello evaluation compares predicted layers with corresponding ground-truth layers using LayerD’s order-aware Dynamic Time Warping protocol.The protocol measures per-layer reconstruction quality on the Crello dataset.
  • I Crello Evaluation Metric: Best-match assignment prevents semantically correct layer reorderings from being penalized by fixed-index comparison.Each reference layer is matched to the predicted layer with the highest RGB similarity before reconstruction error is computed.
  • I Crello Evaluation Metric: The held-out evaluation set contains 480 fixed LAION-Aesthetics images, fully disjoint from training data.The training data come from a separate dataset, and no LAION-Aesthetics images appear during training.
  • J Text Conditioning Ablation Prompts: The text-conditioning ablation replaces per-image captions with two fixed prompts applied uniformly across training images.The prompts are “a clean, well composed image. Detailed prompt.” and a longer prompt emphasizing distinct objects, clean backgrounds, sharp edges, vivid colors, lighting, composition, and depth layers.
  • J Text Conditioning Ablation Prompts: The detailed prompt mirrors the reward rubric’s evaluation axes but fails to reduce bad layers.Its alignment targets object separation, alpha cleanliness, background coherence, and feature distribution.
  • NeurIPS Paper Checklist: The checklist marks the paper’s claims and theoretical-result documentation as Yes, while describing the method, architecture, hyperparameters, and reward prompts for reproduction.The checklist states that the base model and dataset are publicly available and that trained LoRA adapters are planned for release.
  • NeurIPS Paper Checklist: The checklist acknowledges that code release is under consideration, reports positive and negative societal impacts, and marks human-subjects-related items as not applicable.Potential benefits include accessibility, education, and creative workflows; a negative impact is marginally easing misleading composite imagery production.
Loading 2605.30257v1…