PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable 3D Captioning with Pretrained Models

Tiange Luo, Chris Rockwell, Honglak Lee, Justin Johnson

arXiv:2306.07279v2cs.CV

TL;DR

High-quality captions for 3D assets remain scarce, and Objaverse metadata is not informative. Cap3D combines pretrained image-captioning, image-text alignment, and language models to produce captions from multiple views; on Objaverse, its captions were preferred 35% more often, cost more than 10 times less, and were over 40 times faster than crowdsourced captions.

  • Problem

    High-quality paired text for 3D assets remains scarce, while existing Objaverse metadata does not provide informative captions.

    Source: §1 · p. 2

  • Method

    Cap3D uses pretrained image-captioning and image-text alignment models to generate and filter captions from 3D renders, then an LLM fuses them across views.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    On Objaverse, Cap3D captions were preferred 35% more often by humans, cost more than 10 times less, and were over 40 times faster than crowdsourced captions, assuming only 8A40 GPUs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The resulting caption and geometric-caption datasets provide benchmarks for text-to-3D tasks at scale and in geometric detail.

    Source: §6 · p. 9

    Cap3D can produce ambiguous, verbose, or imprecise captions when unusual views yield inaccuracies that CLIP misses, and it struggles with complex indoor 3D scans.

    Source: Appendix D · p. 22

Abstract

from arXiv · show

We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM to consolidate captions from multiple views of a 3D asset, completely side-stepping the time-consuming and costly process of manual annotation. We apply Cap3D to the recently introduced large-scale 3D dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted using 41k human annotations from the same dataset, demonstrates that Cap3D surpasses human-authored descriptions in terms of quality, cost, and speed. Through effective prompt engineering, Cap3D rivals human performance in generating geometric descriptions on 17k collected annotations from the ABO dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions, and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E, and DreamFusion.

1 Introduction

Cap3D automates 3D asset annotation by combining pretrained image-captioning, image-text alignment, and language models to produce captions across multiple views. Applied to Objaverse [9], it yields 660k 3D-text pairs and is evaluated against crowdsourced captions and on downstream text-to-3D model finetuning.

1 Introduction

  • Cap3D addresses the high cost of 3D asset creation and scarcity of high-quality 3D captions, while Objaverse [9] metadata alone is not informative caption text.The introduction contrasts limited 3D-text data with abundant high-quality text-image pairs that have enabled progress in image-text learning, text-conditioned image synthesis, and image captioning.Source: §1 · p. 2
  • Cap3D applies BLIP2 to asset renders, filters captions with CLIP [16], and uses GPT4 to fuse filtered captions across views.The approach leverages pretrained image-text and language models trained on varied, large-scale data to address the scarcity of informative 3D captions.Source: §1 · p. 2
  • Cap3D is agnostic to 3D asset sources and scales with more assets and computational resources; its Objaverse [9] collection contains 660k 3D-text pairs.The authors state that the released dataset includes automated and human-annotated captions, point clouds, and rendered images, and supports ethical filtering through both image and text.Source: §1 · p. 2
  • Cap3D captions are preferred 35% more often by humans, cost more than 10 times less, and are over 40 times faster than crowdsourced captions, assuming only 8A40 GPUs.For ABO geometry captioning [35], automated captions initially underperform humans, but question-answering formulation yields stronger performance than crowdsourced workers.Source: §1 · p. 2
  • At the same scale, Cap3D captions yield better text-to-3D finetuning performance than human captions, with further gains at full scale.The paper also reports meaningful improvements when finetuning state-of-the-art methods on Cap3D data.Source: §1 · p. 2

2 Related Work

Cap3D adapts scalable image-text modeling to address scarce 3D-text supervision, captioning Objaverse [9] and using ABO [35] to evaluate fine-grained geometry.

Image-Text Data and Modeling

  • Transformer architectures and diffusion models scaled best with large image-text datasets, and Cap3D uses transformer-based captioning while adopting diffusion for text-to-3D experiments.Earlier systems used CNNs and LSTMs for captioning and representation learning, and GANs and VQVAEs for text-to-image generation.Source: §2 · p. 3
  • Cap3D follows model-assisted dataset scaling by captioning rendered views with BLIP2, refining them with CLIP [16] [61], and summarizing with GPT4 [62].Concurrent work captioned 2D images with an older system [66] or used metadata [65].Source: §2 · p. 3

3D-Text Data and Modeling

  • Earlier 3D datasets contained about 50k objects [68] [69] [71], while labeled 3D-text data was scarce and concentrated on ShapeNet chairs, tables and chairs [73] [74], and ScanNet [75] [76].Prior work explored 3D captioning [77] [78] [79] and text-to-3D [73] [77] [83] at small scale; some text-to-3D methods avoided 3D supervision [3], resulting in slow optimization.Source: §2 · p. 3
  • Objaverse [9] is over 10 times larger than the prior largest public 3D dataset [68], and Cap3D assigns one caption per object after filtering.Concurrent Objaverse text efforts [65] do not fuse captions across views or rely on metadata [65], and do not address text-to-3D.Source: §2 · p. 3
  • Cap3D reports that finetuning Point·E on its captions improves performance, although Point·E and Shap·E had already trained text-to-3D models on large-scale 3D data.Related work includes 3DGen [2] for text- and image-to-3D, and scaled image-3D generation; these studies had not fully disclosed code or data.Source: §2 · p. 3

3 Method

Cap3D builds a single descriptive caption by rendering multiple views, selecting and consolidating image captions, and filtering assets and text for ethical and licensing concerns. Its method combines view-level error correction with safeguards for large-scale 3D datasets.

3.1 Captioning Process

  • Cap3D renders eight views per object to expose details hidden or distorted in a single forward-facing view.Views are rendered at 512×512 from high-information camera angles, including two slightly below and the rest slightly above the object.Source: §3.1 · p. 3
  • BLIP2 generates five candidate captions per rendered image, while ABO geometry experiments use a two-stage question-answering process.The two stages identify the pictured object, then generate five answers about its structure and geometry.Source: §3.1 · pp. 3–4
  • CLIP selects the candidate caption with the highest image-text cosine similarity, and GPT-4 consolidates selected captions across views into a comprehensive summary.GPT-4 can preserve significant details while removing unlikely ones; the authors explain that consolidating after CLIP avoids making GPT-4 parse more incorrect details and lets CLIP correct errors at the view level.Source: §3.1 · pp. 3–4

3.2 Ethical Filtering

  • Cap3D’s rendered views and captions support identifying and mitigating identifiable information and NSFW content, while the smaller, mostly furniture ABO dataset is manually checked.Objaverse filtering removed CC BY-NC-SA and CC BY assets and objects lacking sufficient camera information before sensitive-content screening.Source: §3.2 · p. 4
  • 661k Objaverse objects remained after filtering, which detected over 19k objects and conservatively estimated roughly 1k missed faces and fewer than 1k missed NSFW items.Filtering included license and camera-information checks, face and NSFW detection, and a blocklist; the face detector removed 18.6k objects, the NSFW classifier 217, and the blocklist 226.Source: §3.2 · pp. 4–5

4 Dataset

The section collects captions for Objaverse, a large, varied dataset of artist-created assets, and ABO, a smaller real-product dataset focused on geometry. It builds automated and crowdsourced caption sets for Objaverse and compares human and prompted Cap3D geometric descriptions on ABO.

4 Dataset

  • Manual validation of automated filtering found that 16 of 17 missed face detections were sports cards.The filtering analysis manually checks face and NSFW-content detection.Source: Table 2 · p. 5

4.2 ABO Geometry Captions

  • ABO [35], primarily furniture, contrasts with Objaverse by having few classes that vary mainly in geometry; the section collects 17.2k crowdsourced captions and 6.4k each of standard Cap3D and Cap3D (QA) captions.ABO metadata is typically short and offers limited detail; its captioning task emphasizes structure rather than semantic category.Source: §4.2 · pp. 5–6
  • With two-stage QA prompting, Cap3D can rival human descriptions of ABO object geometry, although unprompted automated captions are less detailed.ABO captions focus on geometric structure without color or texture; the comparison covers crowdsourced descriptions, standard Cap3D captions, and Cap3D (QA).Source: §4.2 · p. 6, Figure 4 · p. 6

5 Experiments

Experiments evaluate Cap3D captions on Objaverse and ABO, then test their value for text-to-3D finetuning and state-of-the-art model performance. Results favor Cap3D on Objaverse, show that geometry-focused prompting improves ABO captions, and indicate benefits from finetuning, with metric-specific qualifications.

5.1 3D Captioning on Objaverse

  • On Objaverse’s human-captioned split, Cap3D outperforms human captions and metadata across A/B studies and automated metrics; multiple views and GPT4 help, while CLIP filtering removes errors.Filtering reduces GPT input from 40 captions to 8, lowering reported cost from $15.33 to $4.18; large-scale A/B testing also favors Cap3D over Human.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Geometry 3D Captioning on ABO

  • On texture-free, colorless ABO inputs, Cap3D (QA) is preferred to Human, while unprompted Cap3D lacks geometric detail and metadata is weakest.The two-stage prompt produces longer captions with finer-grained geometric detail.Source: §5.2 · p. 7, Table 4 · p. 8

5.3 Large-Scale Text-to-3D Generation

  • On the 30k Objaverse set, finetuning Point·E on Cap3D captions outperforms finetuning on human captions, with the full Cap3D set performing best.Shap·E does not improve on CLIP metrics after finetuning, but performs least poorly on the full Cap3D set and improves most on FID.Source: §5.3 · p. 8, Table 5 · p. 8
  • Finetuning generally improves FID; Point·E and Stable Diffusion + Point·E also improve on CLIP-related metrics over pretrained counterparts.Shap·E’s downturns may reflect unstable finetuning and training only its diffusion stage.Source: §5.3 · p. 8, Table 6 · p. 9
  • DreamField’s strong CLIP-metric performance, including outperforming ground-truth images, indicates overfitting to CLIP; the authors propose ViLT precision as an additional metric.DreamField remains strong on ViLT precision.Source: §5.3 · p. 9

6 Conclusion · Appendix A Price Breakdown Details

Cap3D produces large-scale 3D-text and fine-grained geometric-caption datasets that support text-to-3D benchmarks. The appendix details the compute and GPT-4 costs of producing captions, including the two-stage QA variant.

6 Conclusion

  • Table 7 notes that CLIP-based overfitting yields higher CLIP scores than ground truth, while ViLT provides a fairer score.The table covers text-to-3D optimization baselines.Source: Table 7 · p. 9
  • Cap3D creates the largest publicly available high-quality 3D-text dataset by an order of magnitude and a geometric-caption dataset for fine-grained 3D objects.The geometric captions pair real-object 3D assets with real images; together, the datasets support text-to-3D benchmarks at scale and in geometric detail.Source: §6 · p. 9

Appendix A Price Breakdown Details

  • BLIP2 processes about 337.5 objects per hour on one A40 GPU, while CLIP processes about 27,000 iterations per hour on the same GPU.BLIP2 runs at about 2,700 iterations per hour and requires eight runs for eight rendered views; processing 1,000 objects takes about 2.96 hours.Source: Appendix A · p. 15
  • Processing 1,000 objects with BLIP2 costs $3.79, and CLIP filtering costs $0.38 on one A40 GPU; using eight A40s costs the same as one.The stated cloud rate is $1.28 per hour, and multiple GPUs process different rendered views in parallel.Source: Appendix A · p. 15
  • GPT-4 input costs $15.33 per 1,000 objects without CLIP filtering and $4.18 with filtering.Without filtering, prompts average 511.1 tokens across 40 captions; with filtering, they average 139.3 tokens across eight captions.Source: Appendix A · p. 15
  • For Cap3D (QA), total cost rises from $8.35 to $12.14 per 1,000 objects because BLIP2 runs twice, doubling its cost from $3.79 to $7.58.Cap3D (QA) is used only on ABO and was not costed in the main paper.Source: Appendix A · p. 15

Appendix B Additional 3D Captioning Results

Appendix B presents random Cap3D captioning examples and comparisons of generated captions with human annotations and Objaverse metadata, showing two views per object although Cap3D uses eight.

Appendix B Additional 3D Captioning Results

  • Figures 7–12 show random Cap3D-generated captions for Objaverse objects [9], with two views displayed per object although Cap3D uses eight.Source: Figure 7 · p. 15, Figure 8 · p. 16, Figure 9 · p. 16
  • Figure 13 adds another random Cap3D example from Objaverse [9], while Figures 14–15 compare Cap3D-generated captions with human annotations and Objaverse metadata [9].The comparison figures show two views per object; Cap3D and human annotations use eight views.Source: Figure 13 · p. 18, Figure 14 · p. 19, Figure 15 · p. 19

Appendix C Additional Text-to-3D Results

Appendix C presents additional text-to-3D results across pretrained and Cap3D-finetuned models and optimization baselines. The figures use test-set prompts and references, with the left five-column methods fine-tuned on Cap3D-generated captions.

Appendix C Additional Text-to-3D Results

Appendix D Limitations and Failure Cases

Cap3D can produce ambiguous, verbose, and imprecise descriptions when unusual rendered views yield inaccurate captions that filtering does not remove. It also struggles with complex indoor scans, while no single view captures an object’s complete details.

Appendix D Limitations and Failure Cases

  • Unusual above- or below-object views can lead BLIP2 to produce inaccurate captions that CLIP fails to filter, leaving GPT4 to consolidate disparate information into ambiguous, verbose, and imprecise descriptions.Figure 22 illustrates a failed case in which GPT4 and CLIP do not correct BLIP2’s errors.Source: Appendix D · p. 22, Figure 22 · p. 22
  • Cap3D struggles to process some inherently complex indoor 3D scans, which can be difficult to distinguish even for humans.In the Figure 23 example, inaccurate information across views leads to rough details that fail to depict the indoor scene accurately.Source: Appendix D · p. 22, Figure 23 · p. 22
  • No caption from a single view can describe all the details of a given 3D object.Source: Appendix D · p. 22

Appendix E ABO Captioning: Automated Metrics … Appendix G Crowdsourced Captioning Details

The appendices caution that automated metrics are poor measures of ABO caption quality, and document Cap3D’s prompts, rendering and dataset choices, human-captioning split, and crowdsourcing process.

Appendix E ABO Captioning: Automated Metrics

  • Automated captioning scored best on ABO’s automated metrics, but retrieval scores were very low and the authors caution that human and qualitative evaluations suggest the opposite.The metrics are limited because many ABO objects are similar, making retrieval difficult, and automated captions describe geometry poorly; the table uses a random 5k-object subset rather than the full 6.4k.Source: Table 8 · p. 23, Appendix E · p. 23

F.1 Prompt used in Cap3D

  • Cap3D’s BLIP2 prompts first identify the object, then ask about its structure and geometry using that identified object.The second prompt substitutes the response to the first prompt for <object>.Source: §F.1 · p. 23
  • The GPT4 prompt distills multiple descriptions into one concise caption while avoiding background, surface, and posture details.Longer prompts sometimes elicited GPT4 reasoning in confusing output captions, so the authors sought a shorter prompt partly to reduce cost.Source: §F.1 · p. 23

F.2 Rendering Details

  • For rendering, objects are normalized to a unit cube, recentered, and viewed by eight surrounding cameras, including two slightly below the object.Three area lights serve as key, fill, and rim lights; the authors select 660k Objaverse objects and 6.4k ABO objects, excluding six simple-geometry ABO categories.Source: §F.2 · pp. 23–24

F.3 Human Captioning Split

  • Human captions use a manually selected Objaverse subset with good renders of nontrivial but decipherable objects, excluding unsuitable samples and furniture; ABO follows the split of [77].The excluded cases include trivial or unclear objects such as simple rocks, abstract art, and difficult-to-distinguish scans.Source: §F.3 · p. 24

Appendix G Crowdsourced Captioning Details

  • Hive workers received task instructions and gold-standard examples, were monitored and banned for bad captions, and were paid approximately $50 per 1k tasks.Assuming three objects per minute, the authors estimate $9 per hour; total captioning expenditure across Objaverse and ABO was $7k.Source: Appendix G · p. 24

Appendix H Crowdsourced A/B Testing Details

Crowdsourced A/B tests compared caption pairs using randomized left/right order and a 1–5 preference scale, with workers instructed to prioritize accuracy, informative detail, then brevity. Suspected scammers were manually removed, leaving uneven but sufficient observations for conclusive findings.

Appendix H Crowdsourced A/B Testing Details

  • Workers rated image-caption pairs on a 1–5 scale, with 3 as a tie, and method order randomized for each instance.Instructions included gold-standard examples and prioritized accuracy, informative detail, then brevity.Source: Appendix H · p. 27
  • Manual removal of suspected scammers produced uneven observation counts, but the authors report enough judgments for conclusive findings.Common scam patterns included always choosing the same rating or always choosing the shorter or longer caption.Source: Appendix H · p. 27
  • Experiments covered Objaverse splits and the full ABO set, with observations ranging from 7k to 36k across individual comparisons.Objaverse: Split (1), Cap3D vs. Human, 36k observations across 22k objects; Split (2), Cap3D vs. Human, 10k across 4.7k objects, and Cap3D vs. Metadata, 7k across 4.7k objects (below the 10k target, but results were conclusive); Split (3), Cap3D vs. BLIP2, 20k across 5.0k objects, and Cap3D vs. +GPT4, 29k across 5.0k objects. ABO: Human vs. Cap3D, 21k across 6.4k objects; Cap3D (QA) vs. Human, 17k across 6.4k; Cap3D (QA) vs. Cap3D, 13k across 6.4k; and Cap3D (QA) vs. Meta, 12k across 6.4k.Source: Appendix H · p. 27
  • A/B testing across Objaverse and ABO cost $1.8k in total.Workers were paid approximately $20 per 1k tasks; assuming 7.5 tests per minute, this corresponds to $9 per hour.Source: Appendix H · p. 27

Appendix I Additional Experimental Details

The appendix documents caption-evaluation workloads, text-to-3D finetuning configurations, and evaluation and optimization runtimes.

Appendix I Additional Experimental Details

  • Captioning experiments used one full-scale evaluation run with 95% confidence intervals; CLIP Score took about 5 minutes, ViLT R-Precision 8 hours, and A/B testing 12 hours.The timing covers a 5k object-caption-pair test set for CLIP Score and ViLT R-Precision, and 10k responses across 5k objects for crowdsourced A/B testing.Source: Appendix I · p. 30
  • Finetuning used one train and evaluation run, and Shap·E often output NaNs requiring checkpoint restarts, possibly explaining its lack of improvement.Training took about 3 days on the full set and 1 day on the human set; Point·E and Shap·E used AdamW, CosineAnnealingLR, and an initial learning rate of 1e-5, with batch sizes 256 and 64, respectively.Source: Appendix I · p. 30
  • LoRA training failed under multi-GPU training because of a HuggingFace internal DDP error, but single-GPU training still yielded improvement.LoRA used AdamW, CosineAnnealingLR, an initial learning rate of 1e-4, and batch size 3; ControlNet used AdamW, a constant learning rate of 1e-5, and batch size 8. Training used 4 A40s except for LoRA.Source: Appendix I · p. 30
  • 37 seconds per iteration for PointE text-to-3D, 114 seconds for LoRA + PointE image-to-3D, and 124 seconds for ControlNet + PointE image-to-3D, including rendering.The image-to-3D methods include 100 seconds for image-to-3D and 9 seconds for rendering; PointE text-to-3D includes 28 seconds generation and 9 seconds rendering.Source: Appendix I · p. 30
  • 193 seconds per iteration for ShapE NeRF and 16 seconds for ShapE stf, including text-to-3D and rendering; publicly available PointE image-to-3D uses 1B parameters versus 40M for text-to-3D.The appendix notes that the 1B-parameter image-to-3D model is slower than the largest publicly available text-to-3D model.Source: Appendix I · p. 30
  • 40 minutes for 3DFuse, 95 minutes for Stable DreamFusion, and 35 minutes for DreamField per object, using one A40 GPU.Optimization plus final rendering used default parameters, and each method was run once.Source: Appendix I · p. 30

Related papers

Loading 2306.07279v2…