ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation

1 Introduction

• Without labels, learning to identify objects and scene composition from images remains difficult, despite its potential to support object-level reasoning without dense annotations.Source: §1 · p. 1
• Current whole-scene methods rely on simple synthetic data, where strong performance has saturated benchmarks and made quantitative model comparison difficult.Source: §1 · p. 2
• Natural images contain diverse, often irregular textures rather than consistent strong colors, which may violate assumptions exploited by methods that segment by color.Greff et al. identify a tendency of their model to segment by color and report that it fails on natural images.Source: §1 · p. 2
• CLEVRTEX introduces textured foreground objects and backgrounds, variants to probe model shortcomings, and test sets with unseen shapes, materials, and camouflaged objects.The authors report that no existing method achieves satisfactory performance on the textured setting.Source: §1 · p. 2

2 Related Work

• Earlier 2D datasets support method development but lack visual cues such as shadows and perspective needed for segmentation that generalizes to real images.Source: §2 · p. 3
• Common 3D benchmarks use simple scenes: CLEVR has 3–10 primitive shapes on a gray photo backdrop, with rubbery or metallic appearances and eight color tints.CLEVR6 filters CLEVR to at most six objects per image.Source: §2 · p. 3
• Unlike datasets with contrasting solid colors, CLEVRTEX uses varied materials with repeating patterns and small details, and objects may blend into the background.Source: §2 · p. 3
• Prior attempts to use unsupervised multi-object segmentation in natural scenes include simulated crockery, foreground/background horse segmentation, and robotic-manipulation datasets.The cited examples include Eslami et al., Monnier et al., and Engelcke et al.; some evaluation data lack segmentation masks or contain only a single foreground object.Source: §2 · p. 3

3 CLEVRTEX

• CLEVRTEX extends multi-object learning scenes with texture, irregular shapes, and diverse materials while retaining control over scene composition.The dataset and generation code are released under a CC-BY license.Source: §3 · pp. 3–4

3.1 Dataset Creation

• Images are generated from photo backdrops with randomized camera and lighting, then populated with 3–10 non-colliding objects sampled from four shapes.Materials are sampled for objects and backgrounds; adaptive subdivision, material maps, and physically based rendering add geometry and visual detail, while rendering provides segmentation maps and checks for fully occluded objects.Source: §3.1 · p. 4

3.2 Statistics

• The dataset contains 50,000 images split into 40,000 training, 5,000 validation, and 5,000 test images, using four shapes and a catalog of 60 materials.Object number, placement, and rotation are sampled; the catalog materials have non-commercial licenses.Source: §3.2 · pp. 4–5

3.3 Variants

• PLAINBG textures objects against a plain background, while VARBG keeps objects simply colored and textures the background, enabling analysis of foreground versus background reconstruction.Each variant contains 20,000 images.Source: §3.3 · p. 5
• GRASSBG fixes the background to mossy grass while randomizing foreground materials, allowing tests of memorization versus reconstruction under background consistency.Source: §3.3 · p. 5
• CAMO uses one randomly sampled material across objects and background, while the 10,000-image OOD test set uses 25 unseen materials and four novel shapes.CAMO is used as a testbed for models trained on CLEVRTEX; OOD is exclusively a test set.Source: §3.3 · p. 5

4 Models

• Pixel-space approaches assign pixels to a fixed set of components through membership maps, generative image distributions, or iterative slot-attention clustering.The benchmark includes MONet [6], IODINE, eMORL, Genesis-v2, and Slot Attention.Source: §4 · p. 5
• Glimpse-based methods extract object-containing patches and compose them over an explicit background, trading smaller regions for the need to decide, extract, and compose patches.The benchmark includes SPAIR, SPAIR* with a VAE background predictor, SPACE, and GNM.Source: §4 · p. 6
• Sprite-based methods decompose images into learned RGBA sprite dictionaries, from whose alpha masks scene segmentation can be recovered.The benchmark compares MarioNette and DTISprites.Source: §4 · p. 6
• GNM and SPACE take a fraction of the time and memory required by even single-GPU pixel-space methods in the reported resource comparison.The comparison measures training time in total GPU hours and inference time per batch on NVIDIA P40 24GB GPUs with 128 × 128 inputs.Source: §4 · p. 6, Table 2 · p. 6

5.1 Benchmark

• On CLEVRTEX, all models struggle with segmentation and reconstruction, often overfitting color regions or producing blurry or flat outputs.SPAIR* fails to recognize objects and instead predicts the background; the authors conjecture its autoregressive handling and spatial transformers may make learning noisy.Source: §5.1 · pp. 7–8
• GNM has the best overall segmentation performance on CLEVRTEX, despite one of the largest reconstruction errors.The authors attribute this to spatial-locality constraints and limited background reconstruction capacity.Source: §5.1 · p. 8
• Pixel-space methods perform best among themselves on overall segmentation, but their CLEVR foreground advantage disappears for background segmentation, where SPAIR* performs best.The authors attribute this to pixel-space methods assigning background regions to nearby objects, while glimpse formation encourages spatially compact objects.Source: §5.1 · pp. 6–8
• Pixel-based approaches and SPACE show higher run-to-run metric variance, while glimpse-based methods are more stable across seeds but more sensitive to hyperparameters.The pixel-based and SPACE variance is linked to occasional failures to use separate components.Source: §5.1 · p. 8
• Some strong CLEVRTEX performers retain much of their segmentation ability on OOD data, but CAMO yields unsatisfactory segmentation for every approach.The authors report that models tend to rely on training-distribution appearances; pixel-space models also tend to reconstruct consistent color regions rather than objects.Source: §5.1 · p. 8

5.2 Variants

• On PLAINBG, textured objects still reduce segmentation performance relative to CLEVR, especially for pixel-space approaches, and increase reconstruction error.The authors connect the reconstruction error to difficulty capturing fine-grained appearance; MN’s sprite dictionary is overwhelmed by foreground diversity.Source: §5.2 · p. 9
• On VARBG, some methods struggle with diverse textured backgrounds, while others improve over PLAINBG and CLEVRTEX with simpler, consistent objects.On GRASSBG, a consistent complex background particularly aids glimpse-based methods; pixel-space methods improve slightly over CLEVRTEX, with overall performance roughly matching it.Source: §5.2 · p. 9

6 Conclusions

• CLEVRTEX raises scene complexity with richer textures, materials, and shapes to encourage progress toward methods applicable to real images.The authors present it as a benchmark because current approaches perform impressively on commonly used datasets such as CLEVR yet are challenged by increased visual complexity.Source: §6 · p. 10
• GNM shows the best overall segmentation performance, but almost all methods struggle with multiple textured scenes and fall short of CLEVR.Pixel-space approaches tend to overfit consistent colors and smooth gradients, while sprite- and glimpse-based approaches tend to memorize repeated patterns and may treat sharp changes as objects.Source: §6 · p. 10
• Current methods perform better when objects or backgrounds have consistent appearance than on full CLEVRTEX, where such consistency is absent.The authors attribute this challenge partly to methods’ reliance on simple visual elements and reconstruction objectives; only on simpler scenes do the best methods segment some objects.Source: §6 · p. 10
• Its three variants and two additional test sets offer diagnostic tools, while the extensive evaluation provides a standardized benchmark for current and future methods.The benchmark is intended to challenge generalization in unsupervised multi-object segmentation.Source: §6 · p. 10
• CLEVRTEX has a limited set of primitive shapes and 60 materials, and remains synthetic rather than closing the gap to real-world data.The authors say generalization to real images is likely out of reach until methods can solve CLEVRTEX.Source: §6 · p. 10
• The simulated 3D primitives contain no people or personal information, and the benchmark’s impact outside research is low because current methods cannot yet properly deal with real images.The authors frame its aims as standardizing evaluation, challenging algorithms, and helping future research compare with prior work.Source: §6 · p. 10

A.1 Motivation

• CLEVRTEX was created as a challenging benchmark for unsupervised multi-object segmentation, trading simpler visuals for textures, irregular shapes, and varied materials.Source: §A.1 · p. 15

A.2 Composition

• The dataset contains 50,000 main instances and 20,000 in each listed variant, plus a 10,000-instance OOD test set; instances include scene maps and metadata, with segmentation targets withheld during training.The dataset is a sample from a near-infinite set of arrangements; recommended splits are 10% test, 10% validation, and 80% train, while OOD is evaluation-only.Source: §A.2 · p. 15

A.3 Collection process

• The data was generated by the authors, rendered with Blender 2.9.3, and produced over several weeks.Source: §A.3 · p. 16

A.4 Preprocessing/cleaning/labeling

• The dataset was generated together with labels, with no preprocessing, cleaning, or labeling step reported.Source: §A.4 · p. 16

A.5 Uses

• The paper benchmarks unsupervised multi-object segmentation; extra maps and metadata may support other modalities for supervision or targets and CLEVR-like question answering, while use is intended for research.Source: §A.5 · p. 17

A.6 Distribution

• The dataset and evaluation code are available on the project website under CC-BY, but source textures cannot be redistributed and some materials require retrieval under specified license terms.The dataset instances themselves have no IP-based restrictions.Source: §A.6 · p. 17

A.7 Maintenance

• The authors and VGG support the dataset; they plan to version it, communicate updates, continue hosting older versions, and provide generation code for extensions.Source: §A.7 · pp. 17–18

A.8 Other questions; A.9 Author statement of responsibility

• The datasheet answers “yes” to being bias-free, cannot guarantee GDPR compliance, and states that the authors accept responsibility for rights and licensing.Source: §A.8 · p. 18, §A.9 · p. 18

B Dataset

• Each instance includes JSON metadata describing object count, background material, object shape, size, rotation, 3D and 2D coordinates, and materials.Metadata also includes object color for VARBG and CLEVR scene directions and object relationships.Source: B Dataset · p. 19
• The dataset provides PNG images, reading code, and evaluation utilities for general metrics and per-shape, material, and size breakdowns.Source: B Dataset · p. 19
• The dataset is provided under the CC-BY license.Source: B Dataset · p. 19

C.1 Data

• Images are center-cropped to 192 × 192 and downsampled to 128 × 128 before model input.This creates partially visible objects, removes empty scene edges, and lowers computational load; many benchmarked models were developed for this resolution.Source: §C.1 · p. 19
• For CLEVR, the benchmark uses a version with segmentation masks for evaluation and standard 70k/15k/15k train/validation/test splits.Source: §C.1 · p. 19
• Helper code is included to load the datasets.Source: §C.1 · p. 19

• mIoU uses Hungarian matching to assign one predicted component to each true mask, maximizing overlap, then averages over objects including the background.Source: §C.2 · p. 19
• Unlike foreground-only ARI, mIoU includes the background and weights objects equally, while ARI’s pair counting gives larger regions more weight.Source: §C.2 · p. 19
• Tables 5 and 6 compare ARI-FG and mIoU on the main datasets and dataset variants; the captions do not specify individual results.Source: Table 5 · p. 20, Table 6 · p. 20

C.3 Hyper-parameters

• Where available, the benchmark uses official implementations; otherwise, reimplementations are checked, and CLEVR-tested methods retain their CLEVR hyperparameters when applicable.Other settings were tuned on a best-effort basis; MONet and IODINE training stopped at 300k iterations after progress appeared to plateau beyond 250k.Source: §C.3 · p. 20
• Slot Attention performed best with three attention iterations, while Efficient MORL used 11 components and dataset-dependent GECO targets.For CLEVRTEX, OOD, and CAMO, Efficient MORL used −73,000; the authors also report stronger evaluation performance for the updated eMORL† models.Source: §C.3 · p. 20
• GNM was sensitive to output standard deviation: 0.2 worked on CLEVR and 0.5 on CLEVRTEX, while 0.4 and 0.6 prevented scene segmentation.Source: §C.3 · pp. 20–21
• GenesisV2 used a 0.5655 GECO goal and 0.7 output standard deviation on CLEVR; lowering the goal to 0.5 on CLEVRTEX outperformed the CLEVR setting.Source: §C.3 · p. 21
• The appendix describes model-specific changes, including background dictionaries for MarioNette, architectural changes for SPAIR*, and protective transforms for DTISprites.Source: §C.3 · p. 21

C.4 Extra Figures

• Extra figures show benchmarked models’ outputs across datasets and variants, plus tiling by glimpse- and sprite-based models when foreground and background reconstruction is insufficient.Source: §C.4 · p. 21

• Lighting positions range from close, warm illumination with pronounced shadows to distant lights producing small, soft shadows.Figure 8 also shows four shapes at three scales; its scenes omit materials.Source: §C.5 · p. 21, Figure 8 · p. 24
• Figure 5 presents model reconstruction and segmentation outputs on CLEVR, CLEVRTEX, and variants; Figure 6 illustrates tiling when foreground and background elements are not reproduced accurately enough.Source: Figure 5 · p. 22, Figure 6 · p. 23
• CLEVRTEX uses four basic objects, with material-specific geometry adjustments, bumping, and transparency mapping to diversify their shapes.Figure 9 isolates these perturbations by omitting other material properties.Source: §C.5 · p. 22, Figure 9 · p. 24
• The camera position is jittered along with the lights, using a perspective camera with a 0.035 m focal length and zero shift.Source: §C.5 · p. 22
• CLEVRTEX and its variants use 80%/10%/10% train/validation/test splits by example index; the OOD variant is test-only.The index-based scheme is motivated by uniform scene-composition sampling and yields roughly proportional material distributions across splits.Source: §C.5 · p. 23, Figure 7 · p. 23
• The material catalog contains 60 materials for CLEVRTEX and its PLAINBG, VARBG, GRASSBG, and CAMO variants; OOD uses 25 materials.Figures 10 and 11 show the respective material sets.Source: §C.5 · p. 23, Figure 10 · p. 25, Figure 11 · p. 26