Source-linked AI summary
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans
Ainaz Eftekhar, Alexander Sax, Roman Bachmann, Jitendra Malik, Amir Zamir
TL;DR
Omnidata addresses the limited scale, diversity, and task coverage of conventional vision datasets by parametrically generating steerable multi-task data from real-world 3D scans. Its pipeline controls sampling and renders 21 cues per image, and models trained only on its starter dataset reach strong performance across multiple benchmarks. The authors release the pipeline and supporting ecosystem, while noting that scan representations omit scene properties such as reflectance and lighting.
Problem
Existing vision datasets are biased toward recognition, while many non-recognition datasets are small, narrow-domain, or contain few cues.
Method
The pipeline parametrically samples cameras and points of interest from 3D scans and renders steerable multi-task datasets with labels for 21 mid-level cues.
Results
Models trained on the starter dataset achieve state-of-the-art performance on several tasks, including human-level OASIS surface-normal estimation and depth performance comparable to or better than MiDaS.
Takeaways & Limitations
Omnidata provides a bridge between real-world 3D scans, simulators, and static vision datasets while enabling controlled studies of sampling and task relationships.
Takeaways & Limitations
Mesh-based scans omit reflectance models and scene lighting, and reconstruction errors can affect texture and generated-label quality.
Abstract
from arXiv · showhide
This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world. Changing the sampling parameters allows one to "steer" the generated datasets to emphasize specific information. In addition to enabling interesting lines of research, we show the tooling and generated data suffice to train robust vision models. Common architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks, despite having seen no benchmark or non-pipeline data. The depth estimation network outperforms MiDaS and the surface normal estimation network is the first to achieve human-level performance for in-the-wild surface normal estimation -- at least according to one metric on the OASIS benchmark. The Dockerized pipeline with CLI, the (mostly python) code, PyTorch dataloaders for the generated data, the generated starter dataset, download scripts and other utilities are available through our project website, https://omnidata.vision.
1. Introduction
Omnidata addresses limited, recognition-biased vision datasets by generating large, diverse, multi-task datasets from comprehensive 3D scans. Its steerable pipeline supports controlled sampling studies, and its starter dataset trains models that perform strongly across standard vision benchmarks.
- Omnidata accepts textured meshes, meshes with aligned camera images, or 3D pointclouds as inputs for dataset generation.
- The pipeline generates as many cameras and images as desired, with 21 default mid-level cues for every image.Blender produces the labels while exposing control over sampling and generation parameters.
- A UNet reaches human-level surface-normal performance on OASIS, while DPT-Hybrid is comparable to or better than MiDaS for depth estimation without seeing benchmark data during training.The paper also reports strong qualitative detail, especially for fine-grained structures.
- Controlling camera intrinsics, lighting, object-centeredness, photographer’s bias, and domain enables intervention studies without collecting a new dataset.The paper presents this as a possible pathway toward a computer vision dataset design guide.
- Existing non-recognition datasets can be tiny and restricted; NYU, for example, contains 795 training images captured with one camera.The pipeline is intended to generate datasets for non-recognition tasks at more comparable scale and quality.
- 14.5 million multi-domain starter-dataset images with many cues support excellent performance across several tasks and existing benchmarks.The dataset enables matched-pair experiments because every sample receives labels for every task.
- The paper provides Dockerized tooling, a CLI, dataloaders, pretrained models, code, and utilities through its project website.
2. Related Work
Related datasets and simulators often restrict domains, cues, tasks, or realism, while Omnidata combines scan-based generation with controllable multi-task rendering. Its structure also connects to auto-labeling and domain randomization.
- Static 3D Datasets: Existing static 3D datasets commonly focus on restricted scene domains, while high-resolution object meshes often lack corresponding 2D image samples.
- Static 3D Datasets: Hand-annotated links between in-the-wild images and 3D meshes are expensive, time-consuming, and cannot regenerate the original image dataset.The paper contrasts this reverse construction process with its scan-to-dataset pipeline.
- Vision-Focused Simulators: Vision-focused simulators prioritize real-time embodied-agent mechanics and rendering speed, often at the cost of photorealism and cue diversity.
- Multi-Task Datasets: Multi-task datasets often emphasize recognition, use few domains or tasks, or combine separate datasets as nominally different tasks.
- Multi-Task Datasets: Existing multi-task methods may specialize to development datasets and transfer poorly to large realistic datasets or other tasks.This motivates realistic training settings and datasets that generalize to real-world scenarios.
- Data Augmentation + Domain Randomization: Omnidata extends augmentation beyond flips, crops, and texture randomization to dense viewpoints, multi-view consistency, Euclidean transforms, lens flare, and depth of field.
- Auto Labeling: The pipeline uses structure in 3D scans to compute and propagate labels across images, connecting it to auto-labeling procedures.
3. Pipeline Overview
Omnidata converts 3D scene representations into densely sampled, multi-view images with 21 mid-level cues. Its configurable sampling, interconnected rendering stages, and released tooling support large-scale dataset creation.
- Inputs and Outputs: Omnidata accepts an untextured mesh with either a mesh texture or aligned RGB images, plus an optional camera-pose file.A 3D pointcloud can also be meshed and used as input.
- Inputs and Outputs: The pipeline produces 21 mid-level cues for every generated image or video, including labels for all modalities.
- Pipeline Stages: Generation proceeds by creating camera locations and points of interest, rendering fixated views, and producing all mid-level cues for each space-point-view triplet.
- Sampling and Generation: Narrower-baseline dense views can resemble consecutive video frames and are described as crucial for inverse-rendering methods.
- Sampling and Generation: Camera locations use Poisson-disc sampling by default, points of interest are sampled from mesh faces, and visibility filters enforce camera-point coverage.
- Sampling and Generation: Wide-baseline sampling saves unobstructed views for space-camera-point combinations, while smooth-trajectory sampling interpolates fixated camera paths for video generation.The two methods generate images and videos, respectively.
- Cue Rendering: An interconnected pipeline of open-source tools, primarily Blender, renders cues in a dependency order represented by a directed acyclic graph.
- Released Tools: The released ecosystem includes pipeline code, Dockerized software, PyTorch dataloaders, a 14.5-million-image starter dataset, utilities, and pretrained models.
4. Starter Dataset Overview
The Omnidata starter dataset combines roughly 14.5 million scene- and object-centric images from multiple mesh-based sources with 21 mid-level cues. Standard models trained on it achieve strong zero-shot performance across depth, surface-normal, and panoptic-segmentation benchmarks.
- Dataset composition: The starter dataset contains roughly 14.5 million images from scene- and object-centric views across seven mesh-based datasets.It contains 14,601,449 images from 2,414 spaces, with sampled camera field-of-view and roll.
- Benchmark performance: The 5-dataset, 4-million-image subset achieves state-of-the-art depth and surface-normal performance on the zero-shot OASIS benchmark.A smaller 3-dataset, 1-million-image subset produces panoptic-segmentation predictions similar in quality to models trained on COCO.
- Benchmark performance: A starter-dataset DPT-Hybrid outperforms MiDaS DPT-Hybrid on NYU and OASIS depth evaluation after scale-and-shift alignment.Qualitatively, its predictions better recover fine-grained details, curvature, and object shape.
- Benchmark performance: A vanilla UNet trained on the starter dataset matches human-level OASIS AUCp and outperforms related models on most remaining metrics.The model was not trained on OASIS data, and qualitative results may exceed what standard metrics indicate.
- Dataset relatedness: The largest training split improves average cross-dataset transfer, with harmonic means 25.8% and 30.3% above the best single-dataset models for normals and panoptic segmentation.Transfer rankings depend on the task, and individual models typically generalize poorly outside their corresponding test sets.
5. Illustrative Data-Focused Analyses
The paper uses the generated multi-task data to illustrate analyses of augmentation, cue inputs, and multi-task learning. These examples show how controllable 3D data can support robustness studies and systematic comparisons across datasets, tasks, and methods.
- 5.1. New 3D Data Augmentations: 3D image-refocusing augmentation makes use of full scene geometry rather than applying flat 2D blur.Surface-normal models trained only with 3D augmentation were more robust to both 2D blurring and 3D refocusing.
- 5.2. Mid-Level Cues as Inputs: Are They Useful?: Using mid-level cues as sensors improves semantic-segmentation cross-entropy by 23%, 34%, and 30% across Replica, Hypersim, and Taskonomy evaluations.Using cues as intermediate representations improves performance by 13%, 17%, and 19% on the same evaluations.
- 5.2. Mid-Level Cues as Inputs: Are They Useful?: The cue-input experiment evaluates HRNet-18 semantic-segmentation models trained on 10 Replica spaces against Replica, Hypersim, and Taskonomy test data.The models receive predicted or ground-truth mid-level cues in addition to RGB inputs.
- 5.3. Systematic Evaluation of Multi-Task Learning: Multi-task-learning methods show no clear ranking across tasks or datasets, with within-task Spearman’s ρ indistinguishable from 0.Cross-dataset correlation remains weak (ρ < 0.45), while rankings across semantic segmentation and 3D keypoints are anti-correlated (ρ = -0.4).
- 5.3. Systematic Evaluation of Multi-Task Learning: The pipeline enables large, diverse multi-task mid-level benchmarks for more systematic evaluation of multi-task-learning progress.The paper presents these analyses as illustrative rather than comprehensive.
6. Conclusion and Limitations
Omnidata creates steerable, realistic multi-task datasets from comprehensive 3D scans and supports strong real-world vision models. The paper frames the pipeline as a bridge between scans, simulators, and static datasets, while identifying limits in steering analysis and scan fidelity.
- Conclusion: The pipeline creates large, diverse, steerable multi-task datasets from comprehensive environmental scans and supports standard vision methods at state-of-the-art performance on multiple tasks.It is presented as a bridge between real-world 3D scans, simulators, and static vision datasets.
- Conclusion: The pipeline is intended to study how sampling strategies and cue or sensor choices affect representations and model reliability.Its real-world training performance supports investigating these properties beyond purely synthetic settings.
- Limitations: The paper closes by identifying limitations of the pipeline and opportunities for future work.These include studying steerability, improving capture information, and exploring alternative scene representations.
- Limitations: The paper does not systematically analyze how tuning the pipeline’s steering parameters affects datasets or trained models.The starter dataset used fixed generation settings, with only sporadic experiments illustrating the general idea.
- Limitations: The scan representation omits important scene information, including material reflectance models and scene lighting, and reconstruction errors can affect textures and generated labels.The scans commonly have reconstruction errors of up to 2cm in Taskonomy.