Source-linked AI summary
Objaverse-XL: A Universe of 10M+ 3D Objects
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, Ali Farhadi
TL;DR
3D vision has progressed more slowly partly because high-quality training data is difficult to acquire at scale. The paper introduces Objaverse-XL, a deduplicated web-crawled dataset of over 10 million diverse 3D objects, and shows that scaling its use improves zero-shot novel-view synthesis. The dataset remains far smaller than modern billion-scale image-text datasets.
Problem
3D vision has limited high-quality training data because existing datasets are difficult to acquire and scale, constraining progress relative to language and 2D vision.
Method
The paper crawls diverse Internet sources to build Objaverse-XL and trains 3D models, including Zero123 and PixelNeRF, with larger-scale data.
Results
Objaverse-XL substantially improves zero-shot novel-view generalization for Zero123 and PixelNeRF across challenging object modalities, with performance improving as training data scales.
Takeaways & Limitations
Objaverse-XL provides a foundation for large-scale 3D training and supports empirically promising scaling trends for zero-shot novel-view synthesis.
Takeaways & Limitations
Objaverse-XL is still orders of magnitude smaller than modern billion-scale image-text datasets.
Abstract
from arXiv · showhide
Natural language processing and 2D vision models have attained remarkable proficiency on many tasks primarily by escalating the scale of training data. However, 3D vision tasks have not seen the same progress, in part due to the challenges of acquiring high-quality 3D data. In this work, we present Objaverse-XL, a dataset of over 10 million 3D objects. Our dataset comprises deduplicated 3D objects from a diverse set of sources, including manually designed objects, photogrammetry scans of landmarks and everyday items, and professional scans of historic and antique artifacts. Representing the largest scale and diversity in the realm of 3D datasets, Objaverse-XL enables significant new possibilities for 3D vision. Our experiments demonstrate the improvements enabled with the scale provided by Objaverse-XL. We show that by training Zero123 on novel view synthesis, utilizing over 100 million multi-view rendered images, we achieve strong zero-shot generalization abilities. We hope that releasing Objaverse-XL will enable further innovations in the field of 3D vision at scale.
1 Introduction
3D vision has lagged behind language and 2D vision because high-quality 3D data remains difficult to acquire at scale. Objaverse-XL addresses this bottleneck with over 10 million web-crawled objects and improves zero-shot generalization in 3D models.
- 3D vision datasets remain small because creating high-quality assets traditionally depends on expensive professional design, limiting scalable data collection.
- Over 10 million 3D objects from diverse Internet sources make Objaverse-XL an order of magnitude larger than Objaverse 1.0 and two orders larger than ShapeNet.
- Objaverse-XL combines manually designed assets, photogrammetry scans, and professional scans gathered from GitHub, Sketchfab, Thingiverse, Polycam, and the Smithsonian Institution.
- Zero123 and PixelNeRF show significantly better zero-shot novel-view generalization when trained with Objaverse-XL across photorealistic assets, cartoons, drawings, and sketches.
2 Related Work
Prior large-scale datasets have driven advances in language and 2D vision, while 3D research has largely relied on smaller datasets such as ShapeNet. Objaverse-XL extends large-scale dataset construction to 3D with broader scale and diversity.
- Large image and language datasets have powered strong representations and generative systems, but massive dataset efforts have focused mainly on those modalities.
- Figure 2 uses CLIP L/14 t-SNE embeddings to compare rendered-object distributions, with Objaverse-XL more densely capturing the distribution than Objaverse 1.0.
- Objaverse-XL is presented as a public massive 3D dataset intended to accelerate large-scale training for 3D understanding.
- ShapeNet provides textured CAD models for 3D research, but practical use is limited to 51K quality-filtered models from a theoretical collection of 3M.
- Objaverse-XL combines diverse Internet sources and is described as the largest 3D dataset to date, creating opportunities for large-scale 3D applications.
3 Objaverse-XL
Objaverse-XL is a web-scale, deduplicated collection assembled from heterogeneous 3D sources and enriched with source, Blender, and CLIP-derived metadata. Its composition includes scans, designed assets, and untextured shape data, while rendering and filtering address practical data-quality issues.
- 3 Objaverse-XL: Objaverse-XL collects 3D objects from GitHub, Thingiverse, Sketchfab, Polycam, and the Smithsonian Institution.
- 3.1 Composition: GitHub contributes files from over 500K non-forked repositories across common 3D formats, while 5.5 million files were successfully imported and rendered after deduplication.
- 3.1 Composition: Thingiverse contributes around 3.5 million predominantly untextured STL objects, with randomized colors during rendering to broaden image variation.
- 3.1 Composition: Smithsonian 3D Digitization contributes 2.4K CC0 scans of historical and cultural artifacts in standardized compressed GLB format.
- 3.2 Metadata: Metadata includes source information, Blender geometry and scene statistics, CLIP embeddings, and annotations for NSFW content, faces, and photogrammetry holes.
- 3.3 Analysis: Polycam scans can lack back-side information, producing holes in renders from some viewpoints.
4 Experiments
The experiments test whether scaling Objaverse-XL pretraining improves novel view synthesis and zero-shot generalization across Zero123 and PixelNeRF. Results consistently show gains with larger or higher-quality training data, including transfer to downstream datasets.
- Zero123-XL: Novel view synthesis quality consistently improves as Zero123 training data scales, with the 800K and 10M points representing Zero123 and Zero123-XL, respectively.The figure evaluates Zero123 on a zero-shot dataset and scales LPIPS by 10 times for visualization.
- Zero123-XL: Objaverse-XL training improves Zero123 zero-shot generalization on challenging photorealistic, cartoon, drawing, and sketch inputs.Zero123-XL produces views more consistent with input geometry and preserves sketch style.
- Zero123-XL: High-quality alignment finetuning further improves Zero123-XL zero-shot performance on Google Scanned Objects.The model is finetuned on a 1.3 million-object subset selected using preference-related heuristics.
- PixelNeRF: PixelNeRF generalizes better to novel scenes and objects when trained on more than two million objects, with performance improving consistently with scale.The model is trained for single-image-conditioned novel view synthesis.
- PixelNeRF: Objaverse-XL pretraining improves PixelNeRF performance after finetuning on DTU and ShapeNet, measured with PSNR.Both pretraining and finetuning use a single input view.
5 Limitations and Conclusion
The paper concludes that Objaverse-XL provides a 10.2M-asset foundation for 3D research, while identifying remaining scale, data-selection, and task-coverage limitations.
- Limitations: Objaverse-XL remains orders of magnitude smaller than modern billion-scale image-text datasets.The authors identify continued 3D dataset scaling as future work.
- Limitations: The paper focuses on generative tasks, leaving potential benefits for discriminative tasks such as 3D segmentation and detection for future work.The limitation concerns task coverage rather than a reported failure of the dataset.
- Conclusion: Objaverse-XL contains 10.2M 3D assets and shows promising scaling trends for zero-shot novel view synthesis with a fixed model architecture.The authors position the dataset as a foundation for future 3D work.
A.1 Zero123-XL
Additional comparisons show that Zero123-XL generalizes better than Zero123 across camera transformations, following the requested transformation more accurately and producing more plausible outputs.
- Zero123-XL shows better generalization than Zero123 by following camera transformations more accurately and generating more plausible outputs.
C Datasheet
The datasheet describes Objaverse-XL as a dataset created to address the shortage of high-quality, large-scale 3D vision data. It also identifies the research groups involved and the computational support provided.
- Objaverse-XL was created to address the lack of high-quality, large-scale datasets for 3D vision tasks.
- Researchers at the Allen Institute for AI and the University of Washington created the dataset.
- Stability AI supported the main experiments, while the Allen Institute for AI supported dataset collection and rendering.
C.2 Composition
Objaverse-XL contains approximately 10.2 million 3D objects and associated metadata collected from multiple platforms. The dataset is a filtered sample with source-dependent files, no universal labels or splits, and several documented privacy and content risks.
- Contents: The dataset contains 3D objects and associated metadata, while released files and metadata vary by source.Polycam and Sketchfab provide full objects; other sources provide downloadable links and metadata such as licenses and geometric counts.
- Scale and sources: 10.2 million rendered 3D files comprise the dataset, with most coming from GitHub and Thingiverse.Approximately 56% come from GitHub, 35% from Thingiverse, 8% from Sketchfab, and less than 1% from Polycam and the Smithsonian Institute.
- Sampling: Objaverse-XL is a sample from GitHub, Sketchfab, Thingiverse, and Polycam, alongside all Smithsonian objects.
- Task structure: Labels and targets are not provided universally, relationships are treated as independent, and data splits vary by task.
- Deduplication: Exact file duplicates were removed using SHA-256 hashes, but slightly modified near duplicates may remain.
- Risks and access: The dataset relies on external platforms, whose licenses and terms of service users must follow, and may rarely contain confidential, offensive, or sensitive data.
C.3 Collection
The dataset was collected from public-facing platforms using scripts, then filtered and preprocessed for rendering and metadata generation. Collection involved public data without individual notification or consent, and no impact analysis or institutional review was conducted.
- Timing: The new data was collected in Q1 and Q2 of 2023, while Sketchfab data came from Objaverse 1.0.
- Collection process: Python scripts were used to collect the data, at a cost of several thousand dollars including AWS CPU resources.
- Filtering: The dataset was filtered by licensing restrictions, duplicate content, and successful import into Blender.
- Collection sources: Data was collected from public-facing platforms where users had opted to make their data public.
- Research governance: Individuals were not notified about collection, no impact analysis was conducted, and institutional review boards were not involved.
- Preprocessing: Preprocessing included rendering the objects, while the downloaded data preserved the raw individual files and associated cleaning and rendering software was to be released.
C.5 Uses
Objaverse-XL has already been used for multiple tasks and supports many additional 3D, robotic, video, and 2D vision applications. Future uses must follow the dataset and individual-object licenses.
- The dataset has already been used for tasks described in Section 4.
- Potential applications include 3D inpainting, text-to-3D, image-to-3D, robotic simulation, embodied AI, animation-based video models, and segmentation.
- New tasks must comply with the dataset license and the licenses governing individual distributed objects.
C.6 Distribution
Objaverse-XL is publicly distributed through a Python API and hosted on Hugging Face, with the dataset covered by ODC-By 1.0 while individual objects retain their original licenses. The dataset is currently self-contained, and users must assess licensing for downstream applications.
- The dataset is public and distributed through a Python API, with hosting on Hugging Face.
- The dataset was scheduled for public availability toward the end of June 2023.
- The dataset as a whole uses the ODC-By 1.0 license, while individual objects remain subject to their original licenses.
- Users must assess license questions for individual objects according to downstream use, and new tasks must follow applicable licenses.
- The dataset is currently self-contained without immediate update plans; Objaverse 1.0 remains supported, and contributors are encouraged to extend the dataset.