PaperlayerView exact revision on arXiv

Source-linked AI summary

3DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models

Yuhan Zhang, Mengchen Zhang, Tong Wu, Tengfei Wang, Gordon Wetzstein, Dahua Lin, Ziwei Liu

arXiv:2503.21745v3cs.CV

TL;DR

3D generation lacks comprehensive human-preference data and evaluation methods that consistently align with human perception. This paper introduces 3DGen-Bench, collected through 3DGen-Arena and used to train 3DGen-Score and 3DGen-Eval for unified text-to-3D and image-to-3D assessment. The resulting evaluators show strong human-preference alignment and complementary strengths, while the authors identify a need for more advanced evaluator architectures.

  • Problem

    3D evaluation lacks a comprehensive human-preference dataset and transparent metric covering text-to-3D and image-to-3D generation.

    Source: §1 · pp. 2–3

  • Method

    The paper builds 3DGen-Arena, collects multidimensional preferences from public users and experts, and trains CLIP-based 3DGen-Score plus MLLM-based 3DGen-Eval.

    Source: §1 · p. 3

  • Results

    The proposed evaluators show stronger alignment with human judgments than existing metrics, with 3DGen-Score emphasizing efficiency and scalability and 3DGen-Eval interpretability and robustness.

    Source: §1 · p. 3, §5.3 · p. 13, §5.6 · p. 16

  • Takeaways & Limitations

    3DGen-Score is suitable for efficient method-level comparison, whereas 3DGen-Eval is recommended for detailed quality analysis of individual samples.

    Source: §5.6 · p. 16

    The authors note that a more advanced 3DGen-Evaluator is still needed, including improvements to model architectures or embedding techniques.

    Source: §7 · p. 19

Abstract

from arXiv · show

3D generation is experiencing rapid advancements, while the development of 3D evaluation has not kept pace. How to keep automatic evaluation equitably aligned with human perception has become a well-recognized challenge. Recent advances in the field of language and image generation have explored human preferences and showcased respectable fitting ability. However, the 3D domain still lacks such a comprehensive preference dataset over generative models. To mitigate this absence, we develop 3DGen-Arena, an integrated platform in a battle manner. Then, we carefully design diverse text and image prompts and leverage the arena platform to gather human preferences from both public users and expert annotators, resulting in a large-scale multi-dimension human preference dataset 3DGen-Bench. Using this dataset, we further train a CLIP-based scoring model, 3DGen-Score, and a MLLM-based automatic evaluator, 3DGen-Eval. These two models innovatively unify the quality evaluation of text-to-3D and image-to-3D generation, and jointly form our automated evaluation system with their respective strengths. Extensive experiments demonstrate the efficacy of our scoring model in predicting human preferences, exhibiting a superior correlation with human ranks compared to existing metrics. We believe that our 3DGen-Bench dataset and automated evaluation system will foster a more equitable evaluation in the field of 3D generation, further promoting the development of 3D generative models and their downstream applications. Project page is available at https://zyh482.github.io/3DGen-Bench/.

1 Introduction

3D evaluation has lagged behind rapid model development because existing metrics and benchmarks inadequately capture human preferences across diverse 3D assets. 3DGen-Bench addresses this gap with a large-scale preference dataset, unified evaluation of text-to-3D and image-to-3D generation, and two human-aligned automated evaluators.

Motivation

  • Existing 3D metrics inadequately capture human judgments because CLIP lacks 3D-specific knowledge, user studies are difficult to standardize, and prior benchmarks retain metric or bias limitations.CLIP-based quality assessment and BLIP-based captioning remain limited, while GPT-4V-based evaluation is difficult to interpret because of closed-source biases.Source: §1 · p. 2

Contribution

  • 3DGen-Bench builds the first comprehensive human-preference benchmark spanning text-to-3D and image-to-3D generation.The benchmark is designed to provide large-scale preference data for aligning automated assessment with human judgment.Source: §1 · p. 3
  • Over 1,000 prompts, 19 representative models, more than 10,000 assets, and over 58,000 battle pairs support broad benchmark coverage.Prompts cover single-object, multi-object, and micro-scene cases across text and image conditions.Source: §1 · p. 3
  • 3DGen-Arena collects anonymous pairwise preferences from public users and experts across five independent quality dimensions.Expert annotations additionally include absolute scores for individual 3D models.Source: §1 · p. 3
  • 3DGen-Score and 3DGen-Eval jointly provide automated evaluators that predict human preferences for 3D generation.The former is CLIP-based, while the latter uses a multimodal language model and produces explanations.Source: §1 · p. 3

Results

  • Comparative experiments show that the proposed evaluators outperform existing metrics in pairwise alignment and ranking correlation with human references.The models also offer complementary strengths in efficiency, scalability, interpretability, and robustness.Source: §1 · p. 3

2 Related Work

Prior work has rapidly expanded conditional 3D generation, but evaluation remains constrained by incomplete references, costly user studies, and potentially biased automated metrics. Human-preference methods motivate a unified, scalable framework for 3D assessment and model optimization.

2.1 3D Generative Models.

  • Conditional 3D generation includes optimization-based, feed-forward, transformer-based, diffusion-based, and Gaussian-splatting approaches.These methods target text-conditioned generation, image reconstruction, novel-view synthesis, and 3D scene modeling.Source: §2.1 · pp. 3–4

2.2 3D Evaluation Metrics.

  • Existing 3D evaluation relies on CLIP alignment, reference datasets, user studies, or GPT-4V, but faces limited coverage, high cost, or concerns about credibility and interpretability.The paper positions human-preference data as a way to capture subtleties missed by prior metrics while improving scalability.Source: §2.2 · p. 4

2.1 3D Generative Models.

  • The benchmark’s model zoo is intended to cover varied generation paradigms and representations used in contemporary 3D generation.Figure 2 identifies the specific generative models selected for evaluation.Source: §2.1 · p. 4, Figure 2 · p. 5

2.3 Human Preference.

  • Human-preference learning supports both model ranking and model tuning, motivating the paper’s combination of a human-evaluation leaderboard with an automatic reward model.The paper extends arena-style preference collection to 3D evaluation.Source: §2.3 · pp. 4–5

3 3DGen-Bench Dataset

3DGen-Bench combines a diverse model zoo and prompt suite with five-dimensional human evaluation collected through an anonymous arena and expert annotation process. The resulting comparison and absolute-score data provide complementary preference labels for benchmarking 3D generators.

3.1.1 3D Generative Models Involved

  • The benchmark covers 9 text-to-3D and 13 image-to-3D models spanning optimization-based, feed-forward, point-cloud, triplane, NeRF, and Gaussian representations.Models are rendered as RGB, normal, and texture-free geometry videos for consistent evaluation.Source: §3.1.1 · pp. 5–6

3.1.2 Prompt Generation

  • The prompt suite contains 1,020 diverse prompts across more than 270 categories, six subjects, multiple-object cases, and micro-scenes.It includes 510 text prompts and 510 curated image prompts, split into training, validation, and test sets at a 9:1:2 ratio.Source: §3.1.2 · p. 6

3.2.1 Evaluation Criteria

  • Evaluation uses five independent dimensions: Geometry Plausibility, Geometry Details, Texture Quality, Geometry-Texture Coherence, and Prompt-Asset Alignment.The criteria assess shape validity, surface detail, material quality, geometry-texture consistency, and prompt fidelity including identity consistency for image-to-3D.Source: §3.2.1 · p. 6, Figure 4 · p. 7

3.2.2 3DGen-Arena

  • 3DGen-Arena presents anonymous pairwise battles with choices for each dimension and three 360° views showing normal maps, textureless geometry, and textured renderings.The platform also supports named battles and direct chat, and had collected 8,045 public votes.Source: §3.2.2 · pp. 6–7

3.3 Large-scale Human Annotation

  • Expert annotation combines battle-format comparisons with absolute numerical scores, using detailed guidelines, performance monitoring, and independent labels.The study involved 47 professional annotators and approximately $4,700 in annotation expenditure.Source: §3.3 · p. 7

3.3.1 Comparison Votes

  • Comparison annotation retained 68,400 unique votes after validating 13,680 battle pairs with conflict, cross-annotation, and tie/both-bad checks.The raw collection contained 114,576 votes across five dimensions.Source: §3.3.1 · p. 8

3.3.2 Absolute Scores

  • Absolute-score annotation produced 56,100 final dimension entries from 11,220 models after ranking-consistency, error-rate, and annotator-conflict validation.Two independent annotators scored each model using dimension-specific ranges, with an absolute deviation threshold of 1.Source: §3.3.2 · p. 8

4 3DGen-Evaluator

The 3DGen-Evaluator uses complementary CLIP-based and MLLM-based models to evaluate text-to-3D and image-to-3D generation from multi-view renderings. 3DGen-Score emphasizes efficient, scalable human-aligned scoring, while 3DGen-Eval adds open-world reasoning and interpretable explanations.

4 3DGen-Evaluator

  • Multi-view image encoding replaces direct 3D encoding, enabling a complementary evaluation system that combines 3DGen-Score's efficiency with 3DGen-Eval's interpretability.The CLIP-based model provides numerical scores, whereas the MLLM-based model offers visual-text reasoning and explanations; MLLM outputs can be irreproducible because prompts or input timing may alter results.Source: §4 · p. 8

4.1 CLIP-based 3DGen-Score Model

  • 3DGen-Score processes prompts with multi-view normal and RGB renderings through separate encoders, producing five-dimensional win-rate tuples for pairwise model comparisons.Its prompt encoder supports text and image prompts, while separate normal and RGB visual encoders simplify multiview processing and model expansion.Source: §4.1 · p. 8, Figure 5 · p. 9
  • 3DGen-Score estimates alignment, geometry plausibility, geometry-texture coherence, geometry details, and texture quality from prompt-rendering similarities and learned linear predictors.Normal-prompt similarity estimates geometry plausibility, normal-RGB similarity estimates geometry-texture coherence, and two linear predictors estimate geometry details and texture quality.Source: §4.1 · p. 9
  • 3DGen-Score uses two-stage training: contrastive vision-encoder adaptation first, followed by human-preference supervision for its scoring functions.The first stage supplements generated assets with aligned OmniObject3D and Cap3D examples; the second minimizes divergence between normalized scores and human comparison labels while selectively unfreezing the last four vision-encoder layers.Source: §4.1 · pp. 9–10

4.2 MLLM-based 3DGen-Eval Model

  • 3DGen-Eval adapts MV-LLaVA to concatenated RGB and normal renderings, improving geometry-related evaluation beyond RGB-only inputs.The model uses a 2×2 image-grid format and adds normal renderings, while its first training stage teaches the visual encoder both concatenated-image and normal-image understanding.Source: §4.2 · pp. 10–11
  • 3DGen-Eval's instruction tuning separately learns absolute scoring and pairwise comparison across five dimensions, while GPT-4V-guided explanations enhance interpretability.The absolute-score module establishes overall understanding of each dimension, and the pairwise module identifies subtle differences between instances using human comparison data.Source: Figure 7 · p. 11, §4.2 · p. 11

5 Evaluation

The evaluation compares standard metrics and the proposed models against human preferences across text-to-3D and image-to-3D generation. The experiments assess baseline alignment, generalization, training strategy, and recommended use cases.

5.1 Baseline Metrics

  • The baseline comparison averages four-view scores across seven metrics, including CLIP-based, preference-based, point-cloud, and GPT-4V-based evaluators.The test data contain 40 uniformly sampled objects, while a fair comparison with T3Bench is unavailable in this section.Source: §5.1 · p. 12

5.2 Human Preference Evaluations

  • MVDream, LucidDreamer, and Magic3D lead text-to-3D human-preference rankings, while Wonder3D, OpenLRM, and Stable Zero123 lead image-to-3D rankings.Their average Elo scores are 1177.66, 1112.21, and 1088.93 for text-to-3D, and 1304.05, 1279.67, and 1200.69 for image-to-3D, respectively.Source: §5.2 · pp. 12–13, Figure 8 · p. 12

5.3 Comparisons on Human Preference Correlation

  • 3DGen-Score consistently outperforms other metrics in pairwise and ranking alignment with human judgments across both generation tracks and evaluation dimensions.3DGen-Eval also improves over MV-LLaVa and can outperform GPT-4V in some dimensions.Source: §5.3 · pp. 13–14, Table 2 · p. 13, Table 3 · p. 13

5.4 Generalization Performance

  • 3DGen-Score generalizes to unseen Instant-Mesh models and external 3DRewardDB data, despite estimated normal maps and differences in evaluated dimensions.The external-data experiments still report strong performance across diverse data sources and estimation noise.Source: §5.4 · p. 15, Table 6 · p. 15, Table 7 · p. 15

5.5 Ablation Experiments

  • Two-stage training generally outperforms single-stage training: stage two improves human alignment, while stage one further strengthens robustness to new models.The ablations evaluate both pairwise human alignment and robustness on Instant-Mesh comparisons.Source: §5.5 · pp. 15–16, Table 8 · p. 16, Table 9 · p. 16

5.6 Discussion

  • 3DGen-Score is recommended for efficient method-level comparisons, whereas 3DGen-Eval is recommended for detailed, interpretable analysis of individual samples.For geometry-related text-to-3D dimensions, the authors recommend consulting 3DGen-Eval results.Source: §5.6 · p. 16

6 Application

The paper extends 3DGen-Score beyond evaluation by using it as a reward model to optimize MVDream. The resulting optimization improves output quality and coherence, but reward strength must remain within an effective range.

6 Application

  • 3DGen-Score serves as a reward model that simulates human feedback to refine generative-model behavior.The reward averages scores across five evaluation dimensions and is applied to MVDream.Source: §6 · pp. 16–17
  • Reward optimization improves MVDream outputs through noise reduction, stronger 3D-prior simulation, and texture refinement.The reported visual examples show improvements in quality and coherence across critical dimensions.Source: §6 · p. 17, Figure 9 · p. 17
  • Optimizing a single evaluation dimension can affect other dimensions, indicating that reward effects are not fully isolated.The dimension study reports interactions across dimensions rather than independent improvements.Source: §6 · p. 17, Figure 10 · p. 18
  • The reward coefficient λr has a non-linear effect: benefits appear only within an optimal range, while excessive values can trigger generation collapse.Low values underuse improvement potential, whereas high values diminish benefits before collapse signs emerge.Source: §6 · p. 18, Figure 11 · p. 19

7 Conclusions

3DGen-Bench introduces a large-scale human-preference dataset and automated evaluation system for 3D models, while identifying limitations in model coverage and 3D representation. The dataset, metadata, licensing information, and supporting documentation are publicly released.

7 Conclusions

  • 3DGen-Bench is presented as the first large-scale human preference dataset for 3D models, with automated evaluators aligned more closely with human judgment than existing metrics.The dataset contains 11,200 models, over 68,000 preference votes, and more than 56,000 score labels.Source: §7 · p. 19
  • The collection covers 19 open-source generative models, leaving closed-source and newer state-of-the-art systems as targets for future leaderboard updates.The authors explicitly identify incomplete model coverage as a limitation.Source: §7 · p. 19

Data Availability.

  • The dataset, metadata, licenses, prompts, models, renderings, annotations, and construction details are publicly documented through the repository and appendix materials.Use of the 3D assets remains subject to the original methods’ licensing and copyright policies.Source: Data Availability · p. 19, Appendix · p. 19

A.1 Dataset Link and Documentation

The project releases its curated prompts, generated assets, renderings, and human annotations with documentation and an online arena for voting. Licensing varies by data source, and 3D assets retain the licenses of their originating methods.

A.1 Dataset Link and Documentation

  • The repository releases curated prompts, generated 3D models with renderings, annotated human preferences, dataset documentation, and Croissant metadata.The 3DGen-Arena application is also hosted online for voting across the released assets.Source: §A.1 · pp. 19–20
  • The online 3DGen-Arena app enables users to vote among diverse 3D assets through hosted battle-based interactions.The app is available through a Hugging Face Space.Source: §A.1 · p. 20

A.2 Licensing Terms

  • Prompts and annotations use the MIT License where applicable, while 3D assets follow the original licenses of their respective methods.All uses must comply with the corresponding licensing terms and copyright policies.Source: §A.2 · p. 20

B Additional Details on Dataset Construction

The dataset construction combines diverse prompt generation, broad coverage of text-to-3D and image-to-3D systems, expert and public annotation platforms, and Elo-based leaderboards. Additional absolute-score collection addresses balanced assessment across generated assets.

B.1 Prompt Generation

  • 510 text prompts and 510 image prompts are constructed across six object domains, with manual screening and targeted regeneration balancing categories and difficulty.The process begins with 600 raw text prompts and uses descriptive templates targeting geometric or appearance features.Source: §B.1 · p. 20

B.2 3D Generative Models

  • The benchmark includes 9 text-to-3D and 13 image-to-3D generative models spanning established and recent systems.The listed systems define the model coverage used for benchmark construction.Source: §B.2 · p. 20

B.3 Annotation Cost

  • 47 expert annotators completed the annotation process at $6 per hour, with an approximate total cost of $4,700.The cleaned annotation results are published on Hugging Face.Source: §B.3 · p. 20

D Human Leaderboard

  • The human leaderboards rank text-to-3D and image-to-3D systems by average Elo scores computed from comparison annotations.Additional anonymous-vote rankings are visualized for text-to-3D and the top nine image-to-3D models.Source: Table R1 · p. 21, Table R2 · p. 21, Figure S1 · p. 22

C.1 3DGen-Arena

  • 3DGen-Arena supports anonymous battles, named battles, and single-model chat, enabling both comparative voting and individual model interaction.The anonymous and named modes differ in whether voters know model identities.Source: §C.1 · p. 21

C.2 More Platforms

  • Absolute-score annotation first ranks models generated from the same prompt, then labels them systematically to improve consistency and reduce bias.Comparison votes use fixed battle pairs to maintain balanced data distribution, while real-name participation adds accountability.Source: §C.2 · pp. 21–22
  • The absolute-score distribution is imbalanced because high-quality generated 3D models are relatively scarce in the collected dataset.This distribution is shown for generated models receiving absolute-score annotations.Source: Figure S2 · p. 22

D Human Leaderboard

  • Expert annotations produce 13.8k comparison records, while anonymous arena voting contributes 8,045 additional votes across text-to-3D and image-to-3D tracks.Elo ratings are used to construct human leaderboards from these comparison annotations.Source: D Human Leaderboard · p. 22

E.1 3DGen-Score Predictor

3DGen-SP extends 3DGen-Score to predict absolute scores across five evaluation dimensions using additional high-quality data and a two-stage training strategy. Ranking-alignment tables evaluate correspondence with human judgment, including an additional Instant-Mesh setting.

E.1 3DGen-Score Predictor

  • 3DGen-SP adds five predictor heads to estimate absolute scores for each evaluation dimension, extending the pairwise 3DGen-Score model.The model is introduced because absolute-score annotations remain underused by the original win-rate predictor.Source: §E.1 · p. 22
  • The training data combine 19k high-quality models with 11.2k human-annotated models after augmenting the imbalanced absolute-score dataset with OmniObject3D and Cap3D.Scanned OmniObject3D models receive full scores, while Cap3D models rated good or higher by GPT-4 are filtered in.Source: §E.1 · pp. 22–23
  • Ranking alignment is evaluated with Kendall’s tau, where higher values indicate stronger agreement with human judgment, including a separate comparison involving Instant-Mesh.The corresponding evaluations are reported in Tables R3 and R4.Source: Table R3 · p. 23, Table R4 · p. 23
  • Training uses contrastive encoder adaptation followed by MSE supervision over 30k samples, with selected final CLIP-ViT-H/14 layers unfrozen.The implementation uses a batch size of 32, learning rate 3e-6, and Adam optimization.Source: §E.1 · p. 23

E.2 Evaluation

The evaluation measures how consistently scoring models reproduce human model rankings, with 3DGen-SP outperforming CLIP across tested scenarios and dimensions, including novel models.

Alignment with Human

  • 3DGen-SP consistently outperforms CLIP in ranking correlation with human annotations across all tested scenarios.Models are ranked by average predicted test-set scores, then evaluated using Kendall’s tau.Source: §E.2 · p. 23

Robustness to New Model

  • 3DGen-SP remains stronger than CLIP across all dimensions when evaluated on Instant-Mesh pairs, indicating robustness to novel models.The Instant-Mesh experiment tests generalization beyond the models used in the primary evaluation.Source: §E.2 · p. 23

F.1 3D Prior Exploration

The supplementary experiments explore point-based MLLM evaluation, compare the scoring model with T3Bench, and document prompt and annotation designs for 3DGen-Eval.

F.1 3D Prior Exploration

  • Point-based MLLMs convert generated 3D models into colored point clouds, generate captions, and submit prompt-caption pairs to ChatGPT for assessment.The pipeline samples 100 battle pairs and reports rankings in Figure S3.Source: §F.1 · pp. 23–24, Figure S3 · p. 24

F.2 Comparison with T3Bench

  • 3DGen-Score consistently outperforms T3Bench in pairwise human-alignment probability across all dimensions.The comparison uses five T3Bench-compatible models because format conversion was challenging for some 3DGS models.Source: §F.2 · p. 24, Table R5 · p. 24

F.3 Prompt Design for 3DGen-Eval

  • 3DGen-Eval uses GPT-4V to generate model descriptions and dimension-specific explanations for absolute scores and pairwise comparison votes.These responses are structured into LLaVA-compatible templates for instruct tuning.Source: §F.3 · p. 24, Figure S4 · p. 25, Figure S5 · p. 25
  • The comparison prompt presents two models generated from the same prompt using four-view normal and RGB renderings.Normal renderings expose geometry, while RGB renderings convey color, material, style, and texture.Source: §F.3 · p. 25
  • Comparison votes encode whether model_0 wins, model_1 wins, or the models tie using the values 0, 1, and 0.5.The resulting annotations are collected in list format for each comparison dimension.Source: §F.3 · p. 25
  • The instruct-tuning prompts train 3DGen-Eval with human-supervised absolute scores and GPT-4V-supervised explanations, alongside comparison votes.The supplementary figures show templates for both training modes.Source: Figure S6 · p. 25, Figure S7 · p. 25
  • 3DGen-Arena provides anonymous and named model battles, single-model chat, and separate interfaces for comparison votes and absolute scores.These interfaces support the annotation workflows documented in the supplementary screenshots.Source: Figure S8 · p. 26, Figure S9 · p. 26, Figure S10 · p. 26

Related papers

Loading 2503.21745v3…