Source-linked AI summary
Scalable 3D Captioning with Pretrained Models
Tiange Luo, Chris Rockwell, Honglak Lee, Justin Johnson
TL;DR
High-quality 3D captions are scarce and costly to obtain, limiting large-scale 3D-text resources. Cap3D automates annotation by captioning and filtering multiple rendered views, then consolidating them with GPT4. On Objaverse it creates 660k 3D-text pairs and surpasses crowdsourced captions in quality, cost, and speed, while prompted captioning matches human geometric descriptions and improves text-to-3D finetuning.
Problem
High-quality captions for 3D assets are scarce, while manual 3D annotation is costly and Objaverse metadata is often uninformative.
Method
Cap3D renders multiview images, generates captions with BLIP2, filters them using CLIP, and consolidates them with GPT4.
Results
Cap3D produces 660k Objaverse 3D-text pairs, is preferred 35% more often, costs more than 10× less, runs over 40× faster than crowdsourcing, and improves text-to-3D finetuning over human captions.
Takeaways & Limitations
Automated Cap3D captions provide a scalable resource for 3D-text datasets and text-to-3D model training, while prompting extends performance to geometric description.
Takeaways & Limitations
Unusual camera views can produce inaccurate information that survives filtering and leads to ambiguous, verbose, or imprecise descriptions.
Abstract
from arXiv · showhide
We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM to consolidate captions from multiple views of a 3D asset, completely side-stepping the time-consuming and costly process of manual annotation. We apply Cap3D to the recently introduced large-scale 3D dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted using 41k human annotations from the same dataset, demonstrates that Cap3D surpasses human-authored descriptions in terms of quality, cost, and speed. Through effective prompt engineering, Cap3D rivals human performance in generating geometric descriptions on 17k collected annotations from the ABO dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions, and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E, and DreamFusion.
1 Introduction
Cap3D addresses the scarcity and cost of informative 3D captions by using pretrained image-text and language models to automate annotation. Applied to Objaverse, it produces a large caption dataset and outperforms crowdsourced descriptions in quality, cost, and speed while supporting text-to-3D training.
- 3D asset creation remains costly, while high-quality captions for 3D assets are scarce and Objaverse metadata is often uninformative.
- Cap3D renders 3D assets, captions the views with BLIP2, filters captions with CLIP, and fuses them across views with GPT4.
- 660k 3D-text pairs are collected from Objaverse, with the pipeline designed to scale across larger 3D asset collections and computational resources.
- 35% more human preference, more than 10× lower cost, and over 40× higher speed are reported for automated captions versus crowdsourced captions on Objaverse.The comparison uses over 50k crowdsourced captions on over 40k objects and assumes 8 A40 GPUs.
- Cap3D captions improve text-to-3D finetuning over human captions at the same scale, with further gains at full scale.
2 Related Work
Prior 3D-text datasets and models were limited in scale, object diversity, and supervision, whereas image-text research demonstrated scalable automated data collection. Cap3D follows this direction by adapting pretrained image captioning, alignment, and language models to multiview 3D assets.
- Semi-automated image-text collection showed that trained models can annotate or filter increasingly large datasets with high-quality text.
- Cap3D captions rendered views with BLIP2, refines them with CLIP, and summarizes them with GPT4, using models trained on large datasets.
- Earlier 3D datasets were typically around 50k objects, and labeled 3D-text data relied mainly on human annotation for narrow object categories.
- Objaverse expanded public 3D data to more than ten times the scale of the prior largest public dataset, but its metadata is often short or empty.
- Concurrent work associated text with Objaverse but did not combine multiview captions or address text-to-3D in the same way.
- Recent large-scale text-to-3D systems include 3DGen, Point·E, and Shap·E, although some do not fully disclose their code or data.
3 Method
Cap3D generates a single descriptive caption for a 3D asset by combining multiple rendered views with pretrained captioning, alignment, and language models. The pipeline also supports ethical filtering through image and text processing, but unusual views can produce inaccurate, verbose, or ambiguous captions.
- Captioning Process: Cap3D renders multiple 2D views, generates preliminary captions, filters errors with CLIP, and unifies the surviving captions with an LLM.
- Captioning Process: Eight high-information Blender views are rendered at 512×512, including elevated and lowered viewpoints intended to reveal self-occluded details.
- Captioning Process: BLIP2 generates five captions per rendered image, increasing the chance of correct details before later filtering removes mistakes.
- Captioning Process: CLIP selects the caption with maximum cosine similarity between caption and image encodings for each view.
- Captioning Process: GPT4 consolidates captions across viewpoints to retain significant details while removing unlikely information.
- Ethical Filtering: Rendered captions and images support identifying and mitigating identifiable information and NSFW content in large-scale 3D datasets.
- Ethical Filtering: After license, rendering, face, NSFW, and blocklist filtering, Objaverse contains 661k objects, with roughly 1k faces and fewer than 1k NSFW objects estimated to be missed.
4 Dataset
The paper constructs caption resources for Objaverse and ABO, pairing large-scale 3D assets or geometry-focused product models with automated and human descriptions. Ethical filtering validates the Objaverse collection, while prompted Cap3D matches human-level geometric detail on ABO.
- Objaverse: Manual analysis validates automated face and NSFW filtering in Objaverse; 16 of 17 missed face detections were sports cards.
- Objaverse: Objaverse contains roughly 800k assets from 21k classes and over 100k artists, with metadata that is frequently short or empty.
- ABO Geometry Captions: ABO is a smaller Amazon-products dataset dominated by furniture, where objects vary primarily in geometry and captions therefore emphasize structure over semantic category.
- ABO Geometry Captions: ABO includes 17.2k crowdsourced captions and 6.4k captions from both Cap3D and Cap3D (QA).
- ABO Geometry Captions: Human descriptions contain more geometric detail than standard automated captions, while careful prompting lets Cap3D (QA) rival human descriptions.
5 Experiments
The experiments evaluate Cap3D for caption quality, geometric description, and text-to-3D generation. Cap3D outperforms human and metadata baselines on Objaverse, prompt engineering improves geometric captions on ABO, and Cap3D captions improve text-to-3D finetuning.
- 3D Captioning on Objaverse: BLIP2, GPT4, and CLIP are all important to Cap3D performance.The ablation compares single-view BLIP2 captions with multi-view GPT4 consolidation and CLIP filtering.
- 3D Captioning on Objaverse: Cap3D outperforms human-authored captions and Objaverse metadata across A/B studies and automated metrics on Objaverse.The evaluation uses human-captioned and full-dataset splits, with 95% confidence intervals reported for the 5k-object comparison.
- Geometry 3D Captioning on ABO: Cap3D alone contains little geometric detail, but two-stage prompt engineering makes Cap3D (QA) preferred to human captions on ABO.Cap3D (QA) also produces longer captions with significant fine-grained geometric detail.
- Large-Scale Text-to-3D Generation: Finetuning text-to-3D models on Cap3D captions improves performance beyond human-caption finetuning on the same dataset and improves further at full scale.Point·E improves with Cap3D captions, while Shap·E shows no CLIP-metric improvement but achieves its best relative behavior and largest FID improvement on the full Cap3D dataset.
- Large-Scale Text-to-3D Generation: DreamField performs strongly on CLIP metrics, even outperforming ground-truth images, indicating overfitting to the CLIP evaluation metric.The experiments therefore also report ViLT precision to mitigate CLIP bias.
6 Conclusion
The paper concludes that Cap3D provides large-scale 3D-text data and supports text-to-3D training, while evaluation should account for metric bias. Its data-collection costs are quantified for automated and human annotation variants.
- Evaluation: DreamField can exceed ground-truth images on CLIP metrics because optimization overfits the CLIP metric.ViLT precision is proposed as a more fair complementary evaluation measure.
- Conclusion: Cap3D creates the largest publicly available high-quality 3D-text dataset by an order of magnitude.The paper also collects geometric captions for fine-grained 3D objects and frames both datasets as benchmarks for text-to-3D tasks.
- Conclusion: Cap3D’s automated pipeline leverages pretrained models and produces captions at scale for 3D assets.The conclusion identifies automated annotation and geometric-caption collection as the paper’s two dataset contributions.
- Computation: Human annotation cost and speed are computed from crowdsourcing expenditures and annotation progress across the full annotation process.These quantities provide the basis for comparison with automated annotation.
- Computation: Processing 1k objects with the standard Cap3D pipeline costs $8.35, while Cap3D (QA) costs $12.14 per 1k objects.The Cap3D (QA) cost reflects doubling BLIP2 runs for two-stage question answering.
Appendix B Additional 3D Captioning Results
The appendix presents additional random Cap3D captioning examples and comparisons with human captions and Objaverse metadata. The examples use eight views for Cap3D and human annotation comparisons.
- Caption Comparisons: Comparative examples place Cap3D-generated captions alongside human-annotated captions and Objaverse metadata.Figures 14 and 15 use two displayed views while Cap3D and human annotations use eight.
Appendix C Additional Text-to-3D Results
The appendix provides additional text-to-3D results for pretrained and finetuned methods. The figures compare generated outputs with prompts and reference objects across several model families.
- Compared Methods: The appendix includes pretrained and finetuned Shap·E and Point·E models alongside DreamFusion, DreamField, and 3D Fuse optimization baselines.These comparisons cover the methods evaluated in the text-to-3D experiments.
- Qualitative Results: Additional figures show text-to-3D outputs for prompts and references, with the left five method columns finetuned on Cap3D captions.The detailed experimental setting is described in Section 5.3.
- Example Prompts: The examples include an axe with a dragon head, sword, and colorful handle, and a Five Nights at Freddy’s fox character holding a gun.These captions identify two of the illustrated text-to-3D prompts.
Appendix D Limitations and Failure Cases
Cap3D’s multiview pipeline can fail when unusual views produce inaccurate captions that filtering cannot remove, and complex indoor scans remain difficult to process.
- Unusual camera views can cause BLIP2 to generate inaccurate information that CLIP fails to filter.GPT4 then struggles to reconcile the resulting disparate view descriptions, producing ambiguous, verbose, or imprecise captions.
- Complex indoor 3D scans can remain difficult to distinguish, sometimes even for humans.
- A single rendered view cannot capture all details of a 3D object.
- In one failed case, GPT4 plus CLIP cannot correct BLIP2 errors and produces a fuzzy description.
- Another failed case involves inaccurate information across views, yielding roughly described details that do not accurately depict an indoor scene.
Appendix E ABO Captioning: Automated Metrics
ABO evaluates geometric captioning where similar objects make retrieval difficult and automated captions describe geometry poorly, limiting the usefulness of automated metrics.
- Automated metrics are poor measures on ABO because many similar objects make captions difficult to distinguish through retrieval.The full ABO A/B evaluation covers 6.4k objects, while retrieval metrics use a representative random 5k-object subset.
- Automated image-caption alignment is not expected to measure geometric correctness well because automated captioning performs poorly on geometry.
- Cap3D (QA) uses two BLIP2 prompts to identify the object and then ask about its structure and geometry.
- ABO excludes simple-geometry categories and retains 6.4k objects for rendering and captioning.
Appendix H Crowdsourced A/B Testing Details
Crowdsourced A/B testing compares captions on a five-point preference scale, with randomized presentation and manual removal of fraudulent judgments.
- Workers compare an image and two captions on a 1–5 scale, where 3 denotes a tie.Instructions prioritize accuracy, then informative detail, then brevity, and method order is randomized for each instance.
- Manual screening removes workers who always select one score or systematically prefer shorter or longer captions.Discarding these judgments creates uneven observation counts across experiments.
- Objaverse Split (1) contains 36k A/B observations across 22k objects for Cap3D versus Human.
- Objaverse Split (2) contains 10k Cap3D-versus-Human observations and 7k Cap3D-versus-Metadata observations across 4.7k objects.The Metadata comparison has fewer than the target 10k observations, but its results are reported as conclusive because Metadata ratings are extremely poor.
- A/B testing cost $1.8k across Objaverse and ABO, using an estimated rate of 7.5 selected tests per minute.
Appendix I Additional Experimental Details
Additional experiments report evaluation and training costs for captioning, text-to-3D finetuning, and optimization across several model implementations.
- Crowdsourced A/B testing takes about 12 hours for 10k responses across 5k objects.CLIP Score takes about 5 minutes, while ViLT R-Precision takes about 8 hours on an A40 for 5k object-caption pairs.
- Full-scale text-to-3D finetuning takes about 3 days, compared with 1 day on the smaller human-captioned set.Point·E and Shap·E use AdamW, CosineAnnealingLR, and an initial learning rate of 1e-5.
- ShapE (stf) takes 16sec for text-to-3D and rendering, whereas ShapE (NeRF) takes 193sec.
- PointE text-to-3D takes 37sec, including 28sec for synthesis and 9sec for rendering.
- The publicly available PointE image-to-3D model has 1B parameters, compared with 40M for the largest publicly available PointE text-to-3D model.
- Optimization plus final rendering takes 40 minutes for 3DFuse, 95 minutes for Stable DreamFusion, and 35 minutes for DreamField on one A40 GPU.