PaperlayerView exact revision on arXiv

Source-linked AI summary

CObL: Toward Zero-Shot Ordinal Layering without User Prompting

Aneel Damaraju, Dean Hazineh, Todd Zickler

arXiv:2508.08498v1cs.CV

TL;DR

Annotated training data for object layers is limited, and existing sources do not fully meet the need. CObL uses concurrent diffusion-based layer generation, trained with synthetic tabletop scenes; its tabletop-photograph performance is described as an imperfect but strong baseline.

  • Problem

    Existing annotation sources do not provide all the object-layer supervision needed, and available synthetic or manual datasets have limits in realism, generality, or annotation coverage.

    Source: §3 · pp. 2–3

  • Method

    CObL uses parallel Stable Diffusion UNets with cross-layer attention and inference-time compositional guidance, trained using synthetic scenes built from rendered geometry and generated appearance.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    CObL’s performance on tabletop photographs is an imperfect but strong baseline, trained with only a few thousand synthetic tabletop scenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Within tabletop scenes, multiple instances of a pretrained text-to-image model can collaboratively reconstruct reasonable object-based decompositions in a zero-shot setting.

    Source: §7 · p. 8

    The findings and model are restricted to tabletop scenes.

    Source: §7 · p. 8

Abstract

from arXiv · show

Vision benefits from grouping pixels into objects and understanding their spatial relationships, both laterally and in depth. We capture this with a scene representation comprising an occlusion-ordered stack of "object layers," each containing an isolated and amodally-completed object. To infer this representation from an image, we introduce a diffusion-based architecture named Concurrent Object Layers (CObL). CObL generates a stack of object layers in parallel, using Stable Diffusion as a prior for natural objects and inference-time guidance to ensure the inferred layers composite back to the input image. We train CObL using a few thousand synthetically-generated images of multi-object tabletop scenes, and we find that it zero-shot generalizes to photographs of real-world tabletops with varying numbers of novel objects. In contrast to recent models for amodal object completion, CObL reconstructs multiple occluded objects without user prompting and without knowing the number of objects beforehand. Unlike previous models for unsupervised object-centric representation learning, CObL is not limited to the world it was trained in.

1. Introduction

The paper frames zero-shot perceptual organization as inferring occlusion-ordered object layers, including occluded portions, from multi-object images. It introduces CObL, which generates layers concurrently using a Stable Diffusion prior and compositional guidance, and contrasts this task with prior completion and object-centric approaches.

1. Introduction

  • The task requires segmenting visible objects, inferring occlusion order, and completing hidden shapes and colors without complete object observations, a known object count, or human prompting.The representation is an occlusion-ordered stack of isolated object layers that composites to the image.Source: §1 · pp. 1–2
  • CObL generates all object layers in parallel with frozen Stable Diffusion UNets, trainable cross-layer attention, and inference-time guidance toward recomposing the input.The guidance encourages the generated stack to composite back to the input image.Source: §1 · p. 2
  • A few thousand synthetic tabletop scenes train CObL’s input-conditioning and cross-layer-attention weights to bridge the sim-to-real gap, forming an imperfect but strong baseline on tabletop photographs.The synthesis pipeline combines physically accurate geometry with generated materials and lighting.Source: §1 · p. 2

2. Related Work

  • Earlier amodal-completion methods are often category-limited, while Stable Diffusion approaches [38] [54] complete one human-indicated object at a time; CObL completes all objects without prompting.Some earlier methods complete boundaries alone [3] [25] [57], while others also complete appearance and depth [12].Source: §2 · p. 2
  • Inpainting models [34] can support amodal completion with an appropriate mask, whereas CObL does not require an input mask.Source: §2 · p. 2
  • Closed-world model [36] produces occlusion-ordered object layers only for objects it has repeatedly seen, whereas open-world slot models [2] [49] lack occlusion ordering and sharp boundaries and omit explicit amodal completion.The model uses Stable Diffusion to help generate object layers from images [48] and introduces two new complementary guidance-loss types [4] [13] [30].Source: §2 · p. 2

3. Methods

To address scarce annotated data for tabletop scenes, the paper introduces a synthetic-data pipeline and CObL, which generates occlusion-ordered object layers from an image. CObL uses Stable Diffusion and inference-time guidance to produce layers that remain natural-looking and composite toward the input scene.

3. Methods

  • The methods address a shortage of annotated data: existing datasets lack sufficient realism or coverage of amodal completions and occlusion ordering.The paper introduces a tabletop-focused synthesis pipeline and evaluates generalization to real images with novel objects.Source: §3 · pp. 2–3

3.1. Pipeline for Synthetic Tabletop Scenes

  • The pipeline combines partial 3D rendering for geometry and shadows with text-to-image generation for natural colors and textures.It uses object names as prompts and can generate varied textures from the same scene geometry.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Model Design and Training

  • CObL concurrently generates object layers with interconnected frozen Stable Diffusion UNets, cross-layer attention, and input-image conditioning through a learned adapter.The trainable parameters are attached to the cross-layer attention and input-conditioning components.Source: §3.2 · pp. 3–4

3.3. Guided Sampling

  • The guided sampling loss combines compositional agreement with prior score matching, which helps avoid unnatural layers that satisfy compositional loss alone.The compositional term compares a composite of decoded layers with the input, while prior score matching encourages latents toward Stable Diffusion’s native distribution.Source: §3.3 · pp. 4–5
  • Periodic layer permutation reduces the impact of bad initializations and improves average sample quality; erase-and-sort operations clean outputs but minimally affect evaluation metrics.The erase-and-sort operations remove spurious hallucinations and sort empty layers.Source: §3.3 · p. 5

4. Datasets

The paper introduces a synthetic tabletop dataset for training and validation and TABLETOP, a real-image dataset for evaluating generalization to objects outside the training distribution. TABLETOP includes 100 scenes with two to six objects, with ground-truth occlusion-ordered layers.

4.1. Synthetic training and validation data

  • The synthetic dataset contains 2,250 scenes generated from 750 tabletop configurations, with three texture sets per configuration; 2,000 scenes train the model and 250 are held out for validation.The validation set includes objects absent from training and unseen ControlNet textures.Source: §4.1 · p. 5

4.2. TABLETOP dataset

  • TABLETOP evaluation shows CObL outperforms inpainting and amodal-completion comparison models, even though those models receive perfect oracle masks and CObL receives none.The comparison models require an additional input mask, whereas CObL does not.Source: §4.2 · p. 5
  • TABLETOP contains 100 real tabletop images with two to six objects, and its ground truth records each unoccluded object and background in an occlusion-ordered layer stack.Performance suffers on images with more than four objects; five- and six-object scenes are included as a challenge for future work.Source: §4.2 · p. 5
  • The real images were captured from a stationary camera, and object layers were made by capturing objects one at a time and refining them with segmentation tools and manual editing.Images and layers are stored in sRGB at 512 × 512 resolution.Source: §4.2 · p. 5

5. Experiments

CObL is evaluated on amodal completion, visible segmentation, and qualitative reconstructions. It outperforms comparison models on all amodal completion metrics and achieves a top-1 ARI of 83.5% for visible segmentation.

5. Experiments

  • Amodal completion is scored with LPIPS and CLIP similarity after Hungarian layer matching, while visible segmentation is scored with ARI.Comparison models are applied sequentially with oracle masks; visible masks are tiled according to occlusion relationships to form a segmentation map.Source: §5.1 · p. 7, §5.2 · pp. 7–8

5.1. Amodal Completion

  • CObL outperforms comparison models on all amodal completion metrics, despite generating every layer without oracle masks.The comparison covers 60 TABLETOP scenes with 2–4 objects; diffusion methods are evaluated over four noise seeds.Source: §5.1 · p. 7, Table 1 · p. 7

5.2. Visible Segmentation

  • CoBL achieves a top-1 ARI of 83.5%, while Mask2Former without any finetuning achieves an ARI of 66.3% on the same task, providing a reference comparison.Source: §5.2 · p. 8

5.3. Qualitative Results

  • CObL better preserves fine-scale details visible in the input, while diffusion-based comparison models tend to hallucinate even in visible regions.Examples include teapot details and text on a book.Source: §5.3 · p. 8, Figure 6 · p. 8

6. Ablations

Ablations examine sensitivity to object count, explicit depth cues, the frozen Stable Diffusion prior, and inference-time guidance.

Number of objects

  • Performance decreases for scenes with more than four objects, seemingly because increasing clutter makes layer separation harder.Figure 7 reports LPIPS and CLIP scores across varying object counts.Source: §6 · p. 8, Figure 7 · p. 8

Depth cues

  • Removing the MiDaS block from image conditioning increases LPIPS by 4%; Stable Diffusion’s strong depth prior may explain reasonable performance without an explicit depth map.The depth-prior explanation is presented as a possibility.Source: §6 · p. 8

Frozen stable diffusion prior

  • Unfreezing the Stable Diffusion UNet increases LPIPS by 9%, and without the natural-image prior, out-of-distribution objects fail to be completed.Source: §6 · p. 8

Guidance

  • Excluding prior score matching and compositional guidance increases LPIPS error by 8%, and the layers no longer composite to the scene.Source: §6 · p. 8

7. Limitations and Discussion

The approach has computational costs and a limited demonstrated domain, alongside biases inherited from its text-to-image model.

Limitations and Discussion

  • Guidance requires storing gradients for N diffusion UNets, limiting the maximum number of output layers; inference is slower than without guidance.Source: §7 · p. 8
  • The model and findings are restricted to tabletop scenes, so the demonstrated scope does not extend beyond that setting.The authors also note recurring failure modes associated with inherited Stable Diffusion biases.Source: §7 · p. 8

S1. Details of synthetic generation pipeline

The synthetic-data pipeline combines rendered 3D geometry and shadows with generated textures and backgrounds to produce object-layer scenes.

S1. Details of synthetic generation pipeline

  • The Blender step places random 3D assets at sampled frame locations and renders shadows, depth, and object masks for each scene.Placements causing total occlusion are rejected and resampled; 750 scenes take approximately three hours to render on an NVIDIA 3080 GPU.Source: S1. Details of synthetic generation… · p. 12
  • ControlNet-Depth generates object textures from rendered depth maps and prompts specifying an object label and visual modifiers.The process uses 20 DDIM steps with a CFG scale of 7.5.Source: S1. Details of synthetic generation… · p. 12
  • Backgrounds are generated from randomly selected prompts conditioned on the input depth map; before compositing, shadow regions darken the background.Source: S1. Details of synthetic generation… · p. 12

S2. Architecture details

CObL uses parallel Stable Diffusion UNets with shared image conditioning and cross-layer attention to generate object layers, then applies inference-time updates to improve the stack. The architecture has 361 million trainable parameters, about 28% of Stable Diffusion 2.1’s size.

S2. Architecture details

  • CObL runs one Stable Diffusion instance per object layer, tying the UNets with lateral attention to generate layers concurrently.Each UNet receives the same empty-string text condition.Source: S2. Architecture details · p. 12
  • Image and depth cues are passed to every UNet through learned-weight adapters, while lateral attention combines Conv3D with inter-layer attention.The learned parameters αin and α can skip their respective blocks when set to 1.Source: S2. Architecture details · p. 12
  • 361 million trainable parameters make up CObL, about 28% of Stable Diffusion 2.1’s size.Source: S2. Architecture details · p. 12

S3. Non-differentiable guidance steps

S4. Detailed Limitations

CObL’s detailed failure modes include merging multiple objects into one layer and splitting one object across layers. The authors associate these issues with the Stable Diffusion prior and the off-the-shelf foreground mask network.

  • Merging assigns multiple objects to one layer and occurs more frequently as the number of objects in an image increases.Source: S4. Detailed Limitations · p. 13
  • Splitting can place visually distinct parts of one object in separate layers; the authors consider some such splits valid perceptual groupings.The mushroom example is split along a color discontinuity.Source: S4. Detailed Limitations · p. 13
  • The authors suggest Stable Diffusion’s large-scene prior, rather than an object-focused prior, as one possible reason for these failures.They note that no model then combined an object-focused prior with Stable Diffusion’s strong prior.Source: S4. Detailed Limitations · p. 13
  • An off-the-shelf foreground mask network can misassign masks for complex, multipart objects, leading to incorrect scene compositions.Source: S4. Detailed Limitations · p. 13

S5. Initialization and Non-convexity

CObL’s non-convex layering problem makes outputs sensitive to latent initialization: poor initializations can trigger failure modes, while different initializations can yield different decompositions. The authors report that a handful of runs can produce a reasonable stack on many real-world scenes.

Loading 2508.08498v1…