PaperlayerView exact revision on arXiv

Source-linked AI summary

Referring Layer Decomposition

Fangyi Chen, Yaojie Shen, Lu Xu, Ye Yuan, Shu Zhang, Yulei Niu, Longyin Wen

arXiv:2602.19358v1cs.CV

TL;DR

Existing image methods often treat scenes holistically, limiting object-level isolation and manipulation. The paper introduces prompt-conditioned Referring Layer Decomposition, supported by the RefLade dataset, human-preference-aligned evaluation, and the RefLayer baseline. Experiments establish a benchmarkable task and show that performance depends on prompt type, training data, and model design choices, with a documented trade-off between mask preservation and occluded-region generation.

  • Problem

    Holistic image methods lack explicit object and scene-component representations, while prior layer approaches face limited data scale, diversity, scalability, or prompt control.

  • Method

    The paper defines RLD, builds the 1.11M-example RefLade dataset with an automated data engine, introduces human-preference-aligned HPA evaluation, and proposes the RefLayer baseline.

  • Results

    RefLayer experiments show that RefLade-trained models outperform MuLAn-trained baselines at matched layer counts, while spatial prompts outperform pure text prompts on foreground performance.

  • Takeaways & Limitations

    RLD is established as a trainable and benchmarkable framework for prompt-driven decomposition into complete, object-aware RGBA layers.

  • Takeaways & Limitations

    Adding text to mask prompts slightly reduces mask-content preservation while improving generative capability in occluded regions.

Abstract

from arXiv · show

Precise, object-aware control over visual content is essential for advanced image editing and compositional generation. Yet, most existing approaches operate on entire images holistically, limiting the ability to isolate and manipulate individual scene elements. In contrast, layered representations, where scenes are explicitly separated into objects, environmental context, and visual effects, provide a more intuitive and structured framework for interpreting and editing visual content. To bridge this gap and enable both compositional understanding and controllable editing, we introduce the Referring Layer Decomposition (RLD) task, which predicts complete RGBA layers from a single RGB image, conditioned on flexible user prompts, such as spatial inputs (e.g., points, boxes, masks), natural language descriptions, or combinations thereof. At the core is the RefLade, a large-scale dataset comprising 1.11M image-layer-prompt triplets produced by our scalable data engine, along with 100K manually curated, high-fidelity layers. Coupled with a perceptually grounded, human-preference-aligned automatic evaluation protocol, RefLade establishes RLD as a well-defined and benchmarkable research task. Building on this foundation, we present RefLayer, a simple baseline designed for prompt-conditioned layer decomposition, achieving high visual fidelity and semantic alignment. Extensive experiments show our approach enables effective training, reliable evaluation, and high-quality image decomposition, while exhibiting strong zero-shot generalization capabilities.

1 INTRODUCTION

The paper introduces Referring Layer Decomposition (RLD), which extracts complete, targeted RGBA layers from RGB images using spatial, linguistic, or combined prompts. It establishes the task with the RefLade dataset, an automated evaluation protocol, and the RefLayer baseline.

  • 1 INTRODUCTION: The paper frames RLD as a response to holistic image models and limited prior layer-decomposition methods, which constrain object isolation, edit consistency, or data scale.The contribution is positioned as a benchmarkable task combining a dataset, evaluation protocol, and baseline model.
  • 1 INTRODUCTION: Layer representations encode visible and hidden regions of objects as transparent, stackable visual units, supporting fine-grained editing, composition, reuse, and scene understanding.This representation parallels layer-based workflows such as Photoshop.
  • 1 INTRODUCTION: RLD extracts targeted, complete RGBA layers from a single image using spatial, linguistic, or combined user prompts.The task supports points, boxes, masks, and natural-language descriptions for object-specific decomposition.
  • 1 INTRODUCTION: RefLade contains 1.11M image-layer-prompt triplets, including 1M auto-generated examples, 100K manually cleaned layers, and a 10K curated test set.Its scalable data engine integrates prompt interpretation, RGBA synthesis, and automated filtering.
  • 1 INTRODUCTION: RefLayer is a diffusion-based baseline that fuses color-coded spatial prompts into latent representations and uses a parallel alpha decoder for complete RGBA prediction.The model is designed for prompt-conditioned layer decomposition and is evaluated through experiments and ablations on RefLade.

2 RELATED WORK

Prior work spans object-centric representations, RGBA-layer acquisition, and related understanding and editing tasks, but existing approaches have limited coverage, scalability, or output scope. The paper situates RLD as a prompt-driven layer-decomposition task supported by an automated data engine.

  • Image Understanding and Editing: Existing image-understanding and editing tasks address detection, segmentation, generation, editing, inpainting, matting, and amodal completion, but amodal completion does not output decomposed layers.Referring expression and promptable segmentation are related but target segmentation rather than complete RGBA layer output.
  • Compositional Image Representations: Object-centric datasets support operations such as removal, insertion, repositioning, and resizing, but provide limited coverage of real-world objects and spatial relations.Their emphasis on salient objects and specific edits leaves broader compositional coverage unresolved.
  • Compositional Image Representations: The proposed data engine transforms natural images into prompt-aligned RGBA layers through six automatic stages: pre-filtering, scene understanding, layer completion, post-completion, prompt generation, and post-filtering.The stages are designed to produce diverse, realistic, and high-fidelity layers at scale.
  • Compositional Image Representations: Top-down RGBA methods can achieve high fidelity but depend heavily on human guidance, while bottom-up generative methods require high-quality RGBA data, creating a data-availability dependency.These limitations motivate scalable alternatives for acquiring prompt-supervised layers.

3 THE REFLADE

RefLade establishes a scalable, benchmarkable foundation for Referring Layer Decomposition through an automated data engine, diverse prompt-aligned RGBA layers, and human-preference-aligned evaluation. Its large dataset combines broad coverage with quality-controlled subsets and supports evaluation of preservation, completion, and faithfulness.

  • Data Engine: The six-stage data engine filters images, understands scenes, completes occluded regions, refines alpha mattes, generates prompts, and performs quality assurance.Its stages transform natural images into prompt-aligned RGBA layers with complete visual content and semantic grounding.
  • Data Engine: The engine improves its reported success rate from 36% to 70% through substantial enhancements over MuLAn, including pre-filtering and stronger scene understanding.Ensembles combine closed-set, open-vocabulary, and MLLM-based detection with instance and panoptic segmentation.
  • Dataset Composition: 1.11M RGBA layers are generated from 430K images, with bounding boxes and descriptive text, divided into 1M training, 100K quality-tuning, and 10K test subsets.The quality-tuning and test subsets are manually cleaned.
  • Quality Assessment: 74.7% of foreground and 70.2% of background layers meet the quality threshold in an independent audit of the broader training dataset.The audit identifies remaining areas for improvement while validating the data engine’s effectiveness.
  • Evaluation Protocol: The evaluation protocol measures preservation on originally visible content, completion through semantic directional similarity, and faithfulness through distributional similarity.Ground-truth and predicted layers are represented as RGB channels, transparency, and a visibility mask.
  • Evaluation Protocol: HPA aligns strongly with human ELO rankings, whereas LPIPS, FID, and image directional similarity each capture only partial aspects of human judgment.HPA is formed by min-max normalizing and averaging the component metrics after calibrating against pairwise human preferences across nine models.

4 REFLAYER: A BASELINE MODEL

RefLayer formulates referring layer decomposition as conditional image generation, adapting a pretrained diffusion model to produce prompt-specific RGBA layers. It encodes spatial and textual prompts, denoises the target layer, and separately decodes transparency.

  • Model Formulation: RefLayer treats RLD as conditional image generation, enabling direct use of pretrained models with minimal modifications.The model is built upon Stable Diffusion 3 and conditions generation on the original image and prompts.
  • Referring Prompts: Spatial prompts are unified as colored RGB images: blue for background, green for boxes, red for masks, and Gaussian heatmaps for points.The prompt image is encoded into the shared latent space using the same VAE encoder as the original image.
  • Alpha Decoder: A parallel alpha decoder maps the denoised latent representation to a one-channel transparency mask, separating alpha prediction from RGB generation.Its architecture mirrors the VAE decoder while changing the final output to one channel.
  • Training: Training freezes the original VAE encoder-decoder while independently training the diffusion transformer and alpha decoder in a shared latent space.The diffusion model denoises the prompt-corresponding layer, while the alpha decoder learns to decode a mask from a blended-layer embedding.

5 EXPERIMENTS

Experiments show that HPA aligns strongly with human rankings, while RefLade data quality, scale, prompt modality, initialization, and canvas design materially affect RefLayer performance.

  • 5.1 EVALUATING THE HUMAN PERFERENCE ALIGNED SCORE (HPA): HPA agrees strongly with human rankings, whereas LPIPS, FID, and CLIP directional similarity each capture only partial aspects of human judgment.Min-max normalization also outperforms sigmoid and log-based alternatives for combining metrics.
  • 5.2 BENCHMARKING REFERRING LAYER DECOMPOSITION: RefLade-trained models consistently outperform MuLAn-trained models, and foreground performance improves as training data scales from 50K to 1M layers.Background performance is non-monotonic and benefits more from high-quality data than from additional quantity.
  • 5.2 BENCHMARKING REFERRING LAYER DECOMPOSITION: RefLade+Q achieves the best overall results by combining large-scale pretraining with high-quality fine-tuning, improving performance across all metrics.The RefLadeQ tuning set particularly boosts background performance.
  • 5.3 REFLAYER ABLATION STUDY: Mask prompts yield the highest single-modality foreground scores, while text+mask improves occluded-region performance despite slightly reducing HPAfrgd.Point and box prompts provide progressively stronger localization than pure text, with box prompts being more practical for user-facing applications.
  • 5.3 REFLAYER ABLATION STUDY: UltraEdit initialization outperforms SD3 and InstP2P across HPAs, while checkerboard backgrounds with color jittering produce more robust alpha prediction than pure-color canvases.Pure-color backgrounds can cause the alpha decoder to overfit trivial cues.
  • 5.2 BENCHMARKING REFERRING LAYER DECOMPOSITION: The best RefLade+Q model reaches Passrate@K values of 28%, 65%, and 74% for backgrounds and 45%, 74%, and 79% for foregrounds at K=1,5,10.Human annotators count a sample as successful when at least one of K generated outputs is satisfactory.

6 CONCLUSION

The paper establishes Referring Layer Decomposition as a prompt-controlled task and supports it with a scalable data engine, RefLade, evaluation protocol, and RefLayer baseline.

  • 6 CONCLUSION: The data engine converts natural images into prompt-aligned RGBA layers through six stages spanning filtering, scene understanding, completion, refinement, prompt generation, and quality control.The pipeline combines detection, segmentation, inpainting, matting, captioning, and perceptual-semantic checks.
  • 6 CONCLUSION: Layer completion reconstructs occluded content before SAM-V2 segmentation refinement and high-resolution matting produce complete masks and alpha channels.Depth-based masks exclude likely occluders while avoiding background regions during inpainting.
  • 6 CONCLUSION: RefLade supplies large-scale RGBA layers paired with diverse spatial, textual, and multimodal prompts for training and benchmarking RLD.Its post-filter evaluates preservation, visual quality, and semantic correspondence.
  • 6 CONCLUSION: The proposed framework combines automated data construction and human-preference-aligned evaluation to make RLD a trainable and benchmarkable research task.The evaluation checks both perceptual fidelity and referential accuracy.

A.2 SOME EXPERIMENT RESULTS ON DATA ENGINE

Data-engine components improve through model ensembling and multimodal reasoning, while Gemini-2.0 provides especially strong object-completeness judgments for routing layer completion.

  • Layer completion: Gemini-2.0 filters out 60% of samples judged complete, leaving 40% for inpainting-based restoration of occluded regions.The remaining samples are not necessarily incorrectly completed; they proceed to the inpainting stage.
  • GPT as a Tagging Model / GPT for Layer Captioning: GPT-4o generates object tags and rich captions that are grounded spatially or used to describe visual and positional layer attributes.These outputs support prompt generation for referring layer decomposition.
  • Gemini for Judging Object Completeness: Gemini-2.0 achieves 90.3% precision and 56.1% recall for completeness detection, versus GPT-4o’s 66.7% precision and 0.08% recall.Its high precision enables excluding complete objects from generative completion and avoids unnecessary artifacts.
  • Gemini for Quality Control: Gemini-2.0 also evaluates foreground structure and edges and background removal and artifacts for dataset quality control.The paper reports it as more reliable than GPT-4o for fine-grained visual inspection.

A.4 ERROR SOURCE ANALYSIS

Error analysis identifies inpainting and segmentation as the dominant data-engine failure sources, with earlier-stage mistakes potentially cascading through the sequential pipeline.

  • Error analysis: Because the data engine is sequential, errors introduced early can cascade downstream or be partially mitigated by later stages.This makes upstream filtering, detection, and mask construction important for final layer quality.
  • Inpainting Errors: Inpainting errors account for approximately 65% of total errors, mainly because imperfect masks can trigger hallucinations, artifacts, or identity loss during completion.Complex scenes may cause masks to include regions that should remain unchanged.
  • Segmentation Error: Segmentation errors contribute approximately 20% of total errors through incomplete masks, over-segmentation, or inclusion of irrelevant regions.SAM is used in both scene understanding and post-completion refinement.
  • Other Error Source: Additional failures arise from missed detections, inaccurate panoptic segmentation, completeness judgments, and matting artifacts.These sources affect different stages of the six-stage pipeline.

B LIMITATION

The paper identifies omitted visual effects, noisy dataset annotations, mutual occlusion during composition, and limited evaluation of stronger architectures as limitations and future-work directions.

  • The decomposition currently ignores shadows, reflections, rain, dust, and similar effects because transferring them across lighting conditions can create new problems.
  • Mutual occlusion can complicate composing multiple decomposed layers, requiring visibility masks that the decomposition-focused task does not provide.
  • Errors accumulated across the data-engine stages introduce dataset noise, although models can still learn from noisy training data.
  • The baseline has not been evaluated with more advanced architectures, leaving potential performance gains unexplored.
  • The dataset may support finetuning and augmentation for object removal, insertion, repositioning, resizing, and image editing.

C.2 WITH THE DATA ENGINE, DO WE STILL NEED A UNIFIED MODEL?

A unified model remains necessary because the data engine is too expensive for real-time or large-scale deployment, while its noisy outputs can still provide scalable supervision. The paper further positions RLD as a multimodal, end-to-end, benchmarked alternative to prior amodal methods.

  • With the Data Engine, Do We Still Need a Unified Model?: Each data-engine decomposition takes approximately two minutes because the pipeline relies on multiple pretrained models and commercial MLLMs, making deployment impractical at scale.
  • With the Data Engine, Do We Still Need a Unified Model?: A unified model can learn generalizable patterns from noisy, human-free data at scale and potentially outperform the engine that generated its training data.
  • RLD vs. Amodal Segmentation & Completion: RLD accepts spatial and textual prompts, whereas amodal segmentation and completion typically rely on an image plus a binary mask.
  • RLD vs. Amodal Segmentation & Completion: RLD produces complete RGBA layers rather than only amodal masks or de-occluded content, enabling composable outputs.
  • RLD vs. Amodal Segmentation & Completion: RefLayer is an end-to-end unified model, unlike typical two-step pipelines that separately generate completed content and obtain an amodal mask.
  • RLD vs. Amodal Segmentation & Completion: RLD supplies a clear prompt-driven task, benchmark, and human-preference-aligned automatic evaluation where amodal completion lacks standardized resources.

D.2 AMODAL SEGMENTATION EXPERIMENTS.

RefLayer generalizes zero-shot to COCOA amodal segmentation, achieving state-of-the-art occluded-region performance and competitive full-mask results. Small HPA gains on RefLade correspond to substantial downstream mIoU improvements.

  • RefLayer achieves a state-of-the-art mIoUocc result on COCOA without COCOA training, while also attaining a competitive mIoUfull result.The evaluation uses a single prediction per object from text+mask prompts.
  • RefLayer is evaluated on COCOA using visible masks and text labels to generate complete object representations, with the output alpha channel serving as the predicted amodal mask.
  • A 0.0059 higher HPA for RefLayer-γ over RefLayer-α corresponds to gains of 1.76 in mIoUfull and 6.52 in mIoUocc on COCOA.
  • These results indicate that HPA captures preservation, completion, and faithfulness strictly enough that small RefLade improvements can yield large downstream gains.

E QUALITATIVE RESULTS OF THE REFLAYER MODEL

Qualitative results show that spatial prompts improve localization and layer quality over text-only prompts, while RefLayer can preserve visible content and complete occlusions but remains challenged by ambiguity, severe occlusion, and small objects.

  • Text-only Prompting Is More Challenging Than Spatial Prompting: Spatial prompts outperform text-only prompts because explicit localization reduces the reasoning burden required to identify target objects.
  • Text-only Prompting Is More Challenging Than Spatial Prompting: Text-only prompts can extract incorrect regions, whereas a generic box can guide high-quality layer extraction for the same target.
  • Qualitative Results: RefLayer preserves visible content across diverse completions, but serious occlusion, unclear point prompts, and small objects remain difficult cases.
  • General-Purpose LMM Comparison: Gemini 3 struggles to preserve object identity, complete occluded regions, and produce valid RGBA outputs in layer decomposition.
  • Dataset Comparison: RefLayer trained on RefLade produces better RGB appearance and alpha transparency than the same model trained on MuLAn.
  • RefLade Diversity: RefLade contains 95% real and 5% stylized images, with strong coverage of indoor and city scenes at 25.5% and 24.6%, respectively.
Loading 2602.19358v1…