T$^3$Bench: Benchmarking Current Progress in Text-to-3D Generation

1 Introduction

• T3Bench fills two evaluation gaps with diverse challenging prompts and automatic metrics that assess generated-scene quality and text alignment.Prior work mostly used simple prompts, subjective studies, or single-view CLIP-based measures.Source: §1 · p. 2
• Its three prompt suites progress from single objects to surroundings and multiple objects, targeting scalability, robustness, view consistency, and text alignment.Source: §1 · p. 2
• The quality metric combines multi-view CLIP and ImageReward scores with regional convolution, while alignment uses multi-view captioning and GPT-4 evaluation.The metrics correlate with human scores above 0.75 Spearman correlation.Source: §1 · p. 3
• Benchmarking 10 prevalent methods exposes common insufficiency on complicated scenes, especially those involving multiple objects.Source: Figure 1 · p. 2, §1 · p. 3
• The analysis identifies the transition from 2D guidance to a consistent 3D scene as a primary bottleneck for text-to-3D.Source: §1 · p. 3

2 Related Works

• Predominant text-to-3D methods use large pretrained text-to-image diffusion models to avoid requiring 3D training data.Source: §2 · p. 3
• Existing methods suffer from view inconsistency because score distillation does not explicitly account for the stochasticity of diffusion guidance.Source: §2 · p. 3
• Prior evaluation relies on labor-intensive user studies, single-view CLIP R-precision, or partial consistency measures that omit comprehensive quality and prompt alignment.Source: §2 · p. 4
• Unlike 2D images, 3D results contain semantic information across multiple viewpoints, requiring evaluation methods that capture more than one view.Source: §2 · p. 4

3.1 Diverse Prompts with Increasing Complexity

• T3Bench comprises prompt suites for single objects, objects with surroundings, and multiple objects, increasing difficulty to expose weaknesses beyond simple object generation.GPT-4 generates candidate prompts, which are manually filtered and deduplicated with ROUGE-L.Source: §3.1 · p. 5

3.2 Unified 3D Representation

• The benchmark converts NeRF outputs to meshes using DMTet or Marching Cubes, providing explicit geometry for localization, normalization, and evaluation.Meshes also better support editable 3D assets and compact, clear geometry.Source: §3.2 · p. 5

3.3 Evaluation Metrics on Quality and Alignment

• The metric design is intended to detect Janus inconsistency: incorrect views reduce regional multi-view quality scores rather than being hidden by a single favorable view.Source: Figure 3 · p. 6
• Quality assessment uses multi-focal and multi-view capture because single views cannot adequately measure geometry, occlusion, area coverage, or view consistency.A level-2 icosahedron supplies 161 capture locations after mesh normalization.Source: Quality Assessment · p. 6
• Regional convolution smooths text-image scores over neighboring icosahedron viewpoints, balancing local coverage against the weaknesses of simple averaging or maximization.The implementation uses three iterations with convolution weight w = 1, then selects the highest viewpoint score as the final quality score.Source: Quality Assessment · p. 7
• Alignment is measured by generating a 3D caption from BLIP captions at 12 icosahedron viewpoints and comparing it with the prompt using ROUGE-L and GPT-4.The GPT-4 evaluation emphasizes recall of prompt information without penalizing correct additional features.Source: Alignment Assessment · pp. 7–8

4.1 Metric Evaluation

• Multi-view capturing + ImageReward and 3D captioning + GPT-4 align most closely with human quality and alignment annotations, respectively.The comparison evaluates alternative single-view, retrieval-based, captioning, and GPT-4 combinations against expert judgments.Source: §4.1 · pp. 8–9, Table 2 · p. 9
• Regional convolution makes the quality metric penalize view-inconsistent regions, and Janus-problem scenes generally receive lower scores.The study compares paired and synthetically altered scenes, finding a clearer score discrepancy after regional convolution; distributions also show lower scores for Janus scenes.Source: §4.1 · pp. 9–10, Figure 4 · p. 10, Table 3 · p. 10

4.2 Benchmarking Results

• T3Bench evaluates 10 prevalent methods on 300 GPT-4-generated prompts spanning single objects, surroundings, and multiple objects.The setup uses 100 prompts per category, normalized quality and alignment scores, and standardized representation and rendering procedures.Source: §4.2 · pp. 10–11, Table 4 · p. 12
• A simple SDS and Stable Diffusion combination can cause density collapse, while SJC reduces collapse but produces sparse floating density that harms mesh extraction.Magic3D’s refinement improves quality despite first-stage failures, and LatentNeRF reduces failure rates by optimizing in the latent domain.Source: §4.2 · p. 11
• Current optimization approaches typically require thirty minutes or more, while Gaussian-splatting acceleration can produce poor meshes, excessive smoothness, and disordered textures.The authors identify feed-forward frameworks and improved Gaussian splatting as possible directions for efficiency gains.Source: §4.2 · p. 11
• ProlificDreamer’s VSD produces rich detail but may add extraneous geometry and retain Janus faces, reducing alignment as object count increases.The method optimizes a distribution of 3D scenes rather than directly optimizing rendered results, but it lacks explicit 3D or multi-view priors.Source: §4.2 · pp. 12–13
• Quality is relatively strong for single objects but degrades with surroundings or multiple objects, while alignment declines especially for multiple-object prompts.SDS becomes less stable as surroundings and relative object positions admit more denoising possibilities; ProlificDreamer and MVDream show advantages in complex settings, with different trade-offs.Source: §4.2 · p. 13
• High-quality Stable Diffusion images correlate weakly with resulting 3D quality, highlighting 2D-to-3D structure learning and view consistency as central challenges.The benchmark also finds that quality and alignment are distinct: realistic textures can raise quality even when prompt information is incompletely or imprecisely represented.Source: §4.2 · p. 14

• T3Bench provides diverse prompt suites and automatic multi-view quality and alignment metrics that closely correlate with human judgments.Source: §5 · p. 14
• The benchmark evaluates 10 prevalent text-to-3D methods and highlights common and method-specific problems.Source: §5 · p. 14

Size of Data

• A single text-to-3D prompt requires at least half an hour and potentially several hours, making large prompt sets difficult to test.Source: §6 · pp. 14–15

Indirect Evaluation

• Indirect evaluation inevitably loses information when 3D scenes are rendered into 2D views, despite multi-view capturing and processing.Source: §6 · p. 15
• No 3D captioning framework matches BLIP's 2D image-captioning performance, and multi-view caption merging is not always flawless.Source: §6 · p. 15

7.1 Question Generation Single Object

• The single-object prompt set asks for 20 diverse object descriptions without backgrounds, using “A pig wearing a backpack” as a format example.Source: §7.1 · p. 19

Single Object with Surroundings

• The single-object-with-surroundings prompts describe 20 diverse objects together with contextual settings, exemplified by a bicycle leaning against a brick wall.Source: §7.1 · p. 19

7 Example Prompts

• The multiple-object prompt set requests 20 diverse scenes containing multiple objects, with examples involving a child and dog or two coffee cups.Source: §7.1 · p. 19

7.2 Multiple Caption Merging

• The caption-merging prompt instructs GPT-4 to distill descriptions from multiple views of one 3D object into a concise caption while avoiding background, surface, and posture.Source: §7.2 · p. 19

7.3 LLM Likert Scale Scoring

• The Likert-scoring prompt asks an assessment expert to rate prediction recall and correctly captured colors or features on a 1–5 scale.Source: §7.3 · p. 19
• Figure 6 schematizes icosahedrons with different levels.Source: Figure 6 · p. 20

8.1 Metric Evaluation

• Human annotators score 630 mesh samples from 1 to 5 on subjective 3D quality and prompt alignment after examining each mesh interactively.Source: §8.1 · p. 20

8.2 Capture Viewpoint Selection

• Capture viewpoints are selected from recursively subdivided unit-icosahedron vertices to provide uniform coverage around each 3D mesh.The construction begins with an icosahedron and recursively adds edge vertices before normalizing them to unit length.Source: §8.2 · pp. 20–21

8.3 Capturing Poses Derivation

• The camera pose derives look-at, horizontal, and up vectors from the shot location and vertical axis, then assembles them into camera matrix P.Source: §8.3 · pp. 21–22

9 Design Choices for Regional Convolution

• Correlation experiments vary convolution weights from 0 to 2 and apply different numbers of convolutions against human annotations.Source: §9 · p. 22
• Spearman and Kendall correlations rise with stronger and repeated neighbor aggregation, whereas excessive smoothing reduces Pearson correlation by losing fine details.The authors attribute the trade-off to larger receptive fields preserving evaluation order while compressing texture differences.Source: §9 · p. 23
• The selected configuration uses convolution weight w = 1 for three iterations, balancing adequate smoothing against over-smoothing.Source: §9 · p. 23

10 2D Guidance Analysis

• All correlations between diffusion-model 2D image scores and final text-to-3D quality are relatively low, with stronger correlations for simpler prompt sets than for multi-object prompts.The reported ordering is Single Object greater than Single Object with Surroundings, which is greater than Multiple Objects.Source: §10 · p. 23, Table 5 · p. 24
• Acceptable 2D images can still yield poor 3D scenes because text-to-3D methods may use the guidance ineffectively or receive view-inconsistent guidance.Source: §10 · p. 23
• ProlificDreamer shows higher 2D-to-3D correlation, suggesting better use of 2D images to form higher-quality 3D scenes.Source: §10 · p. 24
• MVDream’s retrained multi-view diffusion provides the highest correlation guidance, but its 2D generation averages 32.9 versus Stable Diffusion’s 44.0 on the multi-object set.The result indicates useful multi-view guidance alongside degraded diffusion generation for surroundings and multiple objects.Source: §10 · p. 24
• The bottleneck lies in learning consistent 3D structure from 2D guidance and in the view consistency of that guidance, rather than Stable Diffusion’s basic generative capability.Source: §10 · p. 24

11.1 Single-view vs. Multi-view Capturing

• Multi-view capturing identifies poor geometry and viewpoint artifacts hidden by apparently good front views, then adjusts the evaluation scores accordingly.Source: §11.1 · pp. 24–25, Figure 8 · p. 25
• When the front view is partially obscured, multi-view capturing reveals that it does not fully represent the generated object’s subjective quality.Source: §11.1 · pp. 24–25

11.2 More results

• Additional case studies present generations and evaluations from different text-to-3D methods across Figs. 9–12.Source: §11.2 · p. 25, Figure 9 · p. 26, Figure 10 · p. 27