Source-linked AI summary
Kubric: A scalable dataset generator
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti, Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, Andrea Tagliasacchi
TL;DR
Collecting and annotating real data at scale is costly and raises privacy, fairness, and licensing concerns, while data-generation software remains fragmented. Kubric provides a Python framework linking rendering, simulation, assets, annotations, and scalable execution, demonstrated across 13 generated datasets and multiple vision tasks. The framework is useful for reusable synthetic-data generation, but its current path-tracing design requires substantial computation and lacks several Blender and PyBullet features.
Problem
Collecting, processing, and annotating real data at scale is difficult and expensive, while raising privacy, fairness, and legal concerns; existing generation tools are less mature and fragmented.
Method
Kubric is an open-source Python framework that integrates PyBullet, Blender, reusable assets, rich annotations, and standard data export for scalable synthetic-data generation.
Results
Kubric’s versatility is demonstrated through 13 generated datasets spanning tasks from 3D NeRF models to optical flow estimation.
Takeaways & Limitations
Kubric reduces engineering effort and facilitates reuse and collaboration for generating high-quality synthetic data across varied vision tasks.
Takeaways & Limitations
Kubric lacks several Blender and PyBullet features and currently requires substantial computational resources because it relies on path-tracing rather than rasterization.
Abstract
from arXiv · showhide
Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification.
1. Introduction
Kubric addresses the cost, annotation, control, privacy, fairness, and licensing challenges of collecting real data by providing a scalable framework for realistic synthetic dataset generation.
- High-quality training data at scale can matter as much as or more than architecture and training details.
- Real-data collection remains difficult because detailed annotations are expensive and data diversity, task complexity, privacy, fairness, and licensing are difficult to control.
- Synthetic data supports rich ground-truth annotations, fine-grained control over complexity, and systematic evaluation under violated model assumptions.
- Kubric provides an open-source framework with fine-grained complexity control and rich annotations for photo-realistic synthetic datasets across vision tasks.
- Kubric scales generation across thousands of machines and produces terabytes of data in a standard export format.
- The framework’s versatility is demonstrated through 13 datasets spanning tasks from 3D NeRF models to optical flow estimation.
2. Related Work
Prior synthetic-data resources and pipelines provide useful task-specific annotations but often lack broad annotation coverage, adaptability, and unified scaling support.
- Existing synthetic datasets often target specific tasks and omit annotations such as camera pose, segmentation masks, or optical flow.
- Hand-crafted Blender- and Unity3D-based pipelines can reduce viewing-angle and lighting bias but are usually specialized and difficult to extend.
- Rendering systems differ in ray tracing versus rasterization, global illumination, physics support, scalability, and machine-learning data-loader integration.
- General-purpose pipelines support arbitrary compositions of meshes, textures, scenes, and assets from multiple collections.
3. Infrastructure
Kubric provides a reusable, open-source infrastructure that connects scene generation, simulation, rendering, annotation, and export. Its abstractions, assets, and distributed workflow support realistic, richly annotated data generation from local prototyping to jobs across thousands of machines.
- Architecture: Kubric connects a rendering engine, physics simulator, and data-export infrastructure through a high-level Python library.This design streamlines synthetic-data generation and lowers the barrier to sharing datasets and generation code.
- Ease of use: A simple object-oriented API hides synchronization and interface differences between PyBullet and Blender while providing preprocessed assets from multiple sources.Users manipulate Kubric abstractions rather than directly managing setup, data transfer, or backend conventions.
- Workflow: Kubric combines randomized scene population, optional physics simulation, rendering, and export of images, annotation layers, and metadata.The worker-based workflow creates one random scene per script execution, then repeats independent workers to form the dataset.
- Realism: Blender Cycles provides photo-realistic rendering through effects including global illumination, reflection, refraction, motion blur, and depth of field.The authors accept substantially higher rendering cost than rasterization in exchange for realism and systematic study of visual effects.
- Annotations: Kubric exports image-space annotations such as depth, instance segmentation, optical flow, surface normals, and object coordinates, plus object-centric metadata.Additional metadata includes trajectories, bounding boxes, velocities, physical properties, camera parameters, and collision events.
- Assets: Kubric supplies preprocessed asset collections and collision meshes to reduce the cleanup required to use high-quality 3D assets.Its asset sources use manifests and archive files, while ShapeNet processing includes watertight conversion and approximate convex decomposition.
4. Kubric Datasets and Challenges
Kubric supports diverse synthetic-data challenges across 2D and 3D vision, using varied annotations, scene features, and dataset scales. These challenges evaluate existing methods, test hypotheses, and provide training data under controlled conditions.
- Dataset scope: Kubric-generated datasets span 2D and 3D tasks, range from megabytes to terabytes, and use varied annotations, features, and controllable factors.The challenges use flow, segmentation, depth, camera pose, and object pose, alongside physics and rigged animation.
- Object discovery: MOVi-A through MOVi-E increase visual and dynamical complexity, from CLEVR-inspired objects to scanned objects, cluttered scenes, and moving cameras.MOVi-A–C contain up to 10 moving objects, while MOVi-D–E contain up to 23 objects with few movers.
- Uses of synthetic data: Synthetic challenges serve as benchmarks, sim-to-real training sources, hypothesis tests, and shareable data sources with fewer privacy and legal concerns.Kubric’s common platform makes it straightforward to create, extend, and combine datasets using shared engineering efforts.
- Object discovery: Neither SAVi nor SIMONe reliably decomposes videos across all five MOVi datasets as scene complexity increases.SAVi performs better on videos with moving objects, especially with first-frame bounding-box cues, while SIMONe sometimes captures static objects in MOVi-D and MOVi-E.
- Optical flow: Kubric provides a 3D-rendered optical-flow alternative to FlyingChairs and performs competitively with AutoFlow under the reported comparison protocol.The authors note that AutoFlow’s hyperparameters were learned to optimize Sintel performance, so the comparison is not apples-to-apples.
4.3. Texture-structure in NeRF
Kubric creates controlled datasets for studying how texture frequency relates to NeRF geometry and reconstruction, while also supporting pose-diverse synthetic training examples. The broader motivation is to examine reconstruction behavior beyond standard Lambertian assumptions.
- Texture-structure in NeRF: NeRF surface reconstruction quality can depend on texture, but this relationship remains underexplored for surfaces with different spatial frequencies.Flat-color surfaces may be reconstructed as non-solid volumes rather than faithful surface models.
- Texture-structure in NeRF: The synthetic NeRF dataset assigns each pixel a procedural texture cutoff frequency and measures its association with depth variance and reconstruction error.The experiment uses blue-noise textures with varying spatial frequency on flat surfaces.
- Texture-structure in NeRF: Table 5 reports that increasing texture frequency is associated with lower depth variance, while lower frequency improves color prediction and higher frequency degrades surface geometry.These findings separate color reconstruction accuracy from geometric reconstruction accuracy across texture bands.
- Pose estimation: Synthetic human-pose images supplement real training data with poses underrepresented in datasets such as COCO.The MoveNet experiment mixes real and synthetic data in an 80/20% training-batch ratio.
4.5. Pre-training visual representations
Kubric is used to probe robustness and transfer across several visual representation problems, including NeRF reconstruction, salient-object detection, and synthetic pre-training. The experiments expose where existing methods struggle and where synthetic data can improve coverage.
- Pre-training visual representations: Pre-training ResNet-50 on synthetic Kubric images halves the gap between random pre-training and ImageNet pre-training on standard benchmarks.The pilot combines objects with random background images before transfer using the protocol from prior work.
- Robust NeRF: Standard NeRF-L2 reconstruction is corrupted when scenes are non-static across views, motivating a teleport challenge with randomly inserted transient objects.The associated robust-NeRF comparison evaluates novel-view synthesis without impostors in the test views.
- Robust NeRF: NeRF-L1 can remove transient-object floaters in teleport and jitter scenes, whereas typical NeRF-L2 models render shadows in their place.The test ground truth contains no dynamic objects.
- Salient Object Detection: Kubric’s multi-view salient-object detection datasets contain easy single-object scenes and harder cluttered scenes, each rendered with 10 views and segmentation masks.The training and test sets contain 1000 and 100 scenes respectively for both difficulty levels.
- Salient Object Detection: Existing single-view SOD models perform relatively poorly on cluttered scenes, where nearby clutter is often mistaken for the salient object.The authors identify limited 3D understanding as an important factor in the poor performance and hypothesize that multi-view models will improve.
4.8. Complex BRDFs
Kubric constructs a 3D reconstruction benchmark focused on complex, non-Lambertian surfaces and evaluates models that must account for view-dependent appearance and lighting. Existing approaches perform substantially worse on specular than diffuse shapes.
- Benchmark motivation: Current 3D reconstruction datasets mostly feature Lambertian scenes, leaving highly reflective surfaces underrepresented.For mostly diffuse scenes, geometry and diffuse surface color are the primary relevant scene parameters.
- Evaluation: The benchmark evaluates Light Field Networks and PixelNeRF on scenes containing specular materials.Successful reconstruction requires modeling view-dependent appearance and inferring light-source positions.
- Results: Both evaluated models perform substantially worse on specular shapes than on Lambertian shapes.Figure 13 illustrates the difficulty of representing ShapeNet shapes rendered with specular materials.
- Dataset scale: Kubric also supports broader single-image 3D reconstruction experiments using the full SHAPENET-COREV2 collection of approximately 51,300 models across 55 categories.The expanded dataset enables SOFTRAS to reconstruct a wider range of objects, although complex shapes remain difficult.
4.10. Video Based Reconstruction
Video provides multi-frame consistency for supervising 3D surface reconstruction, but LASR’s reconstruction does not reliably extrapolate beyond observed views, especially under challenging motion and flow conditions.
- Single-shot reconstruction is ill-posed, whereas video supplies multi-frame consistency but introduces challenges from non-static scenes and non-rigid objects.
- LASR jointly recovers object meshes, articulation, and camera parameters from monocular video using silhouettes and optical flow without category-specific shape templates.
- Kubric-generated videos test LASR on rigid and non-rigid objects, including the effects of inaccurate optical-flow estimation.
- LASR fits input views well but fails to extrapolate reliably to unseen views.
- Pillows and bowls achieve the highest SoftRas IoU, while microphones and earphones are the most difficult ShapeNet categories.
4.11. Point Tracking
The point-tracking challenge uses synthetic trajectories with visibility annotations and evaluates a contrastive tracker across localization and occlusion metrics. The method performs well at coarse tracking, but occlusion handling and distribution transfer remain weaknesses.
- Kubric generates trajectories by projecting 3D object points through scenes, providing annotations that are difficult for humans to collect at scale.
- The task predicts point locations and visibility across video frames from one visible query point per trajectory.
- The evaluation combines occlusion accuracy, thresholded tracking accuracy, and a Jaccard-style metric measuring both localization and visibility.
- The contrastive tracker reduces large-threshold error from 25.2% to 5.8% relative to the naïve baseline.
- Occlusion accuracy is poor, with performance below chance at the cycle-consistency threshold, although removing failed points improves average Jaccard.
- Vertical flipping reduces average-Jaccard performance by roughly 4%, indicating limited memorization of scene trajectories rather than collapse.
- Qualitative tracking is strongest for points with little motion or distinctive texture and can sometimes identify occlusions correctly.
- Occlusion can cause large localization errors, motivating global reasoning about object orientation alongside appearance-based features.
4.12. Scene Semantic Segmentation
Kubric creates corresponding 2D image and 3D point-cloud segmentation datasets for comparing models across representations. Results show task-dependent strengths, with 3D supervision becoming especially valuable as scene complexity increases.
- KLEVR, ToyBox5, and ToyBox13 align 2D images with 3D point clouds to compare semantic segmentation methods across representations.
- KLEVR uses a neutral-grey backdrop and five Platonic object shapes, while ToyBox datasets contain ShapeNet objects.
- DeepLab outperforms NeSF by 0.3% to 6.6% in 2D semantic segmentation.
- SparseConvNet achieves 1.9% to 23.1% higher mean intersection-over-union than NeSF in 3D, with larger margins as complexity increases.
- The comparison attributes SparseConvNet’s 3D advantage to access to ground-truth geometry and dense point-cloud supervision.
4.13. Conditional Novel View Synthesis
Kubric generates a large photo-realistic novel-view-synthesis benchmark with varied backgrounds and conditioning and target views. It is designed to test whether models can reconstruct unseen views from limited observations.
- Kubric generates 1M photo-realistic scenes containing groups of ShapeNet objects across 382 randomly chosen background maps.
- Each scene provides ten random 128 × 128 views, with five conditioning views and five target views for reconstruction.
- The benchmark addresses the limitation that scene-specific neural representations require a new model for each scene and cannot learn a dataset-level prior.
5. Conclusions
Kubric is presented as a general framework that reduces the effort of generating and sharing high-quality synthetic data at scale. The paper also identifies current feature and computational limitations.
- Kubric integrates assets from multiple sources, rich annotations, and a common export format for direct use in training pipelines.
- The framework supports generation at scale and is intended to lower barriers, reduce fragmentation, and facilitate sharing of pipelines and datasets.
- Kubric remains incomplete, lacking features such as volumetric effects, soft-body and cloth simulation, and advanced camera effects.
- Its path-tracing renderer requires substantial computational resources, motivating future support for a rasterizing backend that trades speed against render quality.
6. Societal Impact and Ethical Considerations
Kubric may help researchers examine failure modes, privacy, bias, and environmental costs through controllable synthetic data. These benefits do not remove risks from misuse, designer bias, synthetic–real distribution gaps, biased dataset design, or expensive generation.
- Societal Impact: Synthetic dataset construction can help detect dangerous failure modes before deployment and support privacy-preserving system design, but it does not eliminate serious risks to humans.
- Ethical Considerations: Human-driven synthesis can avoid some privacy and legal obstacles, yet designer biases may enter the dataset and require earlier, more controllable mitigation.
- Ethical Considerations: Distributional differences between synthetic and real-world data require explicit evaluation and mitigation to safeguard against real-world bias.
- Ethical Considerations: Kubric may enable datasets tailored to favor particular methods, so experimentation and replication are needed for fairer model assessment.
- Environmental considerations: Controllable synthesis can reduce repeated large-scale training, but generating large datasets remains costly; Kubric’s optical-flow dataset required roughly 3 CPU-years.