Source-linked AI summary
Objaverse: A Universe of Annotated 3D Objects
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, Ali Farhadi
TL;DR
Existing 3D datasets are much smaller and less diverse than large corpora driving progress in other AI domains. Objaverse addresses this gap with a large, richly annotated 3D asset corpus and demonstrates applications across generative modeling, segmentation, embodied navigation, and robustness evaluation.
Problem
Existing 3D datasets contain far fewer assets and less category diversity than large-scale datasets in other AI areas, motivating an equivalent corpus for 3D assets.
Method
Objaverse 1.0 compiles over 800K diverse, high-quality 3D assets with text descriptions, titles, tags, and detailed metadata from distributable Sketchfab models.
Results
Across four studies, Objaverse supports diverse applications, including more diverse generated objects, improved tail-category segmentation, open-vocabulary navigation across 1.1K categories, and orientation-robustness evaluation.
Takeaways & Limitations
Objaverse provides a broad 3D data resource for advancing computer vision research and enabling new applications.
Abstract
from arXiv · showhide
Massive data corpora like WebText, Wikipedia, Conceptual Captions, WebImageText, and LAION have propelled recent dramatic progress in AI. Large neural models trained on such datasets produce impressive results and top many of today's benchmarks. A notable omission within this family of large-scale datasets is 3D data. Despite considerable interest and potential applications in 3D vision, datasets of high-fidelity 3D models continue to be mid-sized with limited diversity of object categories. Addressing this gap, we present Objaverse 1.0, a large dataset of objects with 800K+ (and growing) 3D models with descriptive captions, tags, and animations. Objaverse improves upon present day 3D repositories in terms of scale, number of categories, and in the visual diversity of instances within a category. We demonstrate the large potential of Objaverse via four diverse applications: training generative 3D models, improving tail category segmentation on the LVIS benchmark, training open-vocabulary object-navigation models for Embodied AI, and creating a new benchmark for robustness analysis of vision models. Objaverse can open new directions for research and enable new applications across the field of AI.
1. Introduction
Large-scale web datasets have driven AI progress, but 3D datasets remain comparatively small and limited in diversity. Objaverse 1.0 addresses this gap with over 800K richly annotated assets and demonstrates applications across 3D generation, segmentation, embodied navigation, and robustness evaluation.
- Existing 3D assets used for generative models number at most in the thousands, while embodied-AI simulators typically contain only dozens to a thousand unique scenes.
- Objaverse 1.0 contains over 800K 3D assets designed by over 100K artists, spanning varied object categories, interiors, exteriors, and detailed metadata.
- 91% of human comparisons found Objaverse-trained generative objects more diverse than objects generated by a model trained on ShapeNet.
- Objaverse assets improve long-tail instance segmentation through Copy+Paste augmentation, increasing training data for challenging LVIS categories.
- Random-view renderings expose dramatic performance degradation in current visual models and support benchmarks for orientation robustness.
- Objaverse-populated ProcTHOR scenes enable open-vocabulary navigation across 1.1K semantic object categories, roughly a 50x increase over the prior setting.
2. Related Work
Large-scale 2D datasets combine scale, diversity, and realism, whereas existing 3D datasets generally fall short on one or more of these dimensions. Language–3D datasets are also small and limited in diversity, leading current text-to-3D models to rely on 2D image–text supervision.
- Large scale datasets: Large image and image–text datasets have accelerated progress in classification, detection, captioning, representation learning, generation, and multitasking.
- 3D datasets: Existing 3D datasets ideally need scale, diversity, and realism, but none match 2D datasets across all three dimensions.
- 3D datasets: KIT, YCB, BigBIRD, IKEA, and Pix3D provide image-calibrated household-object models but contain only a few hundred objects at most.
- Language–3D datasets: Text2Shape provides 15,038 chairs and tables with approximately five captions each, totaling 75,344 text–shape pairs.
- Language–3D datasets: Because language–3D datasets are small and insufficiently diverse, current text-to-3D models rely on 2D image–text supervision instead.
3. Objaverse
Objaverse is a Creative Commons-filtered, richly annotated 3D corpus sourced from Sketchfab and designed for broad computer-vision research. Its scale, metadata, categories, styles, scenes, animations, and articulated assets support diverse downstream uses.
- Objaverse selects freely distributable Sketchfab models through the public API and excludes assets restricted for objectionable or adult content.
- Model metadata: Each model can include a name, fixed categories, unrestricted tags, a natural-language description, a thumbnail, and additional statistics.
- OBJAVERSE-LVIS: Objaverse-LVIS contains 47K objects uniquely assigned to one of 1,156 LVIS categories after CLIP-assisted candidate selection and crowdworker verification.
- Interiors: The corpus includes 16K+ interior scenes, substantially exceeding the roughly 400 existing hand-built interactive embodied-AI scenes.
- Objaverse spans scans, modeled assets, point clouds, physically based rendering, animated objects, articulated parts, and large exterior scenes.
- Statistics: Objaverse 1.0 contains 818K objects designed by 160K artists, over 2.35M tags, over 170K unique tags, and coverage of nearly 21K WordNet entities.
4. Applications
OBJAVERSE supports four applications spanning 3D generation, 2D segmentation, embodied navigation, and robustness evaluation. Its scale and diversity provide varied assets for training models, augmenting data, populating simulations, and testing arbitrary-view recognition.
- OBJAVERSE enables four applications: 3D generative modeling, CP3D instance-segmentation augmentation, open-vocabulary ObjectNav, and visual-robustness analysis.
- 3D generative modeling: 91% of human evaluations rated OBJAVERSE-trained GET3D generations more diverse than generations from a ShapeNet-trained model.The comparison used collections of nine randomly sampled generations from both models.
- Instance segmentation with CP3D: 3DCP renders five views of OBJAVERSE objects and pastes 1–3 randomly chosen instances onto LVIS images with 0.5 probability during training.The augmentation uses LVIS-annotated OBJAVERSE objects and cached multiview renders.
- Open-vocabulary ObjectNav: OBJAVERSE populates 10K procedurally generated ProcTHOR homes, scaling ObjectNav from category targets to open-vocabulary text descriptions.The simulation scales available objects from 2K to 36K and object types from 100 to 1.1K.
- Open-vocabulary ObjectNav: 19.9% success was achieved by the trained ObjectNav agent versus 5.1% for a random policy on unseen houses, layouts, objects, and descriptions.The agent uses frozen ResNet-50 CLIP visual and text branches and DD-PPO training.
- Analyzing robustness: OBJAVERSE renders objects from random orientations to evaluate classification under perspective shifts, where model performance degrades dramatically relative to canonical-view diagnostics.The benchmark reports Top-1 and Top-5 Random Rotation and Any Rotation metrics.
5. Conclusion
OBJAVERSE is an 818K-model 3D asset library with paired textual metadata and diverse applications. The paper demonstrates uses in generation, segmentation, embodied navigation, and rotational-robustness evaluation.
- OBJAVERSE contains 818K high-quality, diverse 3D models with paired text descriptions, titles, and tags.
- Four studies use OBJAVERSE for generative 3D models, instance-segmentation improvements, open-vocabulary ObjectNav, and rotational-robustness measurement.
A. Instance Segmentation with CP3D
The CP3D instance-segmentation study integrates 3D copy-paste augmentation into a Mask-RCNN ResNet-50 pipeline for LVIS. Additional detection results report a gain for rare categories.
- The model is Mask-RCNN with a ResNet-50 backbone and a Gumbel activation replacing softmax for transforming logits into probabilities.No additional model changes are made relative to the referenced implementation.
- 24 epochs of finetuning integrate CP3D augmentation with batch size 64 and learning rate 0.002.
- Two points of AP were gained for rare categories in the additional detection results.The reported metrics distinguish rare, common, and frequent categories using APr, APc, and APf.
- APr, APc, and APf measure average precision for rare, common, and frequent LVIS categories, respectively.Rare categories appear in 1–10 images, common in 11–100, and frequent in more than 100.
B. Open-Vocabulary ObjectNav
Open-Vocabulary ObjectNav trains agents to navigate toward text-described objects in procedurally generated homes populated with OBJAVERSE-LVIS assets. The reported policy reaches 33.0% success after approximately 460 million simulation steps.
- Training: 10,080 houses were procedurally generated for training, populated with OBJAVERSE-LVIS assets except structural components such as doors and windows.Training targeted 9,421 unique assets across 262 categories.
- Model: The model combines visual features, target-description features, previous hidden units, and previous actions to predict the next action.Its high-level architecture employs an RNN and uses CLIP-based target-description representations.
- Testing: 4,500 testing episodes were sampled from 151 previously unseen procedural houses across 30 target categories.Each testing category contributed 150 episodes.
- Results: 33.0% success was achieved after approximately 460 million simulation steps with extended training.The main-paper policy was trained for 18 million steps and already showed reasonable performance given the variety of targets and scenes.
C. Composition
OBJAVERSE includes human-generated data whose category labels were collected through a reviewed and approved Mechanical Turk annotation process. Annotators selected category members from visually and textually retrieved candidate sets.
- Human-subject data: Human-subject data in OBJAVERSE was generated by Amazon Mechanical Turk crowdworkers through a process reviewed and approved for release by an Institutional Review Board.The human-generated portion is documented in the dataset’s composition procedure.
- Candidate selection: For each LVIS category, annotators reviewed sets of 500 candidate objects retrieved using CLIP visual similarity and GloVe metadata similarity.The candidate sets combined thumbnail-based visual resemblance with metadata terms similar to the target category.
- Annotation: Crowdworkers viewed nine candidate objects at a time and marked category members using thumbnails, object names, and visual judgment.They were encouraged to use the object’s name when helpful.
D. Estimating Coverage
The paper estimates OBJAVERSE’s semantic coverage by matching CLIP image embeddings of object thumbnails to text embeddings of WordNet entities. This procedure yields coverage of nearly 21K entities.
- Coverage estimation: Each object’s thumbnail embedding is compared by cosine similarity with text embeddings for WordNet entities.The text prompts use the form “a {entity} is a {definition}”.
- Estimated coverage: 20.8K WordNet entities were estimated from nearest-entity matching, corresponding to nearly 21K entities in the dataset statistics.The estimate is based on the nearest WordNet entity for each object.
- Entity assignment: The WordNet entity with maximum cosine similarity is assigned as the object’s entity.Definitions distinguish senses such as animal and tool meanings for the same word.