PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation

Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, Gordon Wetzstein

arXiv:2401.04092v2cs.CV

TL;DR

Text-to-3D evaluation lacks flexible metrics that align with human judgment, while user studies are difficult to scale. The paper uses GPT-4V to generate tailored prompts, compare 3D assets under user-defined criteria, and aggregate pairwise results into Elo rankings. Experiments report that the approach outperforms existing metrics across various criteria and aligns with human preferences.

  • Problem

    Existing text-to-3D metrics often focus on single criteria and may not align with human judgment, while user studies are costly and difficult to scale.

  • Method

    GPT-4V generates customizable evaluation prompts, compares pairs of 3D assets using user-defined criteria, and supplies results for Elo-based model ranking.

  • Results

    The approach outperforms existing metrics across various criteria and achieves consistent alignment with human judgment.

  • Takeaways & Limitations

    The metric provides a customizable and scalable alternative to user preference studies for evaluating text-to-3D generative models.

  • Takeaways & Limitations

    Experiments and user studies were conducted on a relatively small scale, and GPT-4V can produce hallucinations and systematic positional biases.

Abstract

from arXiv · show

Despite recent advances in text-to-3D generative methods, there is a notable absence of reliable evaluation metrics. Existing metrics usually focus on a single criterion each, such as how well the asset aligned with the input text. These metrics lack the flexibility to generalize to different evaluation criteria and might not align well with human preferences. Conducting user preference studies is an alternative that offers both adaptability and human-aligned results. User studies, however, can be very expensive to scale. This paper presents an automatic, versatile, and human-aligned evaluation metric for text-to-3D generative models. To this end, we first develop a prompt generator using GPT-4V to generate evaluating prompts, which serve as input to compare text-to-3D models. We further design a method instructing GPT-4V to compare two 3D assets according to user-defined criteria. Finally, we use these pairwise comparison results to assign these models Elo ratings. Experimental results suggest our metric strongly align with human preference across different evaluation criteria.

1. Introduction

Text-to-3D research is advancing rapidly, but evaluation metrics remain inadequate, inflexible across criteria, and costly to replace with scalable human studies. The paper proposes GPT-4V as a customizable, scalable, human-aligned evaluator.

  • Text-to-3D evaluation metrics have not kept pace with advances in generative methods.
  • Existing metrics often target a single criterion, such as text–asset alignment, and may miss geometric or texture detail.
  • User studies provide adaptable, human-aligned assessment but are costly, time-consuming, and difficult to scale.
  • The approach relies on GPT-4V’s demonstrated promise for generating prompts, understanding human intention, and reasoning about 3D worlds from visual information and language.
  • The proposed proof of concept uses GPT-4V to generate evaluator-tailored prompts, compare 3D shapes by user-defined criteria, and assign Elo ratings.

2. Related Work

Related work spans text-to-3D generation, traditional 3D evaluation metrics, user studies, and large multimodal models. The paper positions automatic GPT-4V evaluation as a customizable alternative to costly preference studies.

  • Text-to-3D generation: Text-to-3D generation has largely relied on NeRF optimization because available 3D data are limited.
  • 3D Evaluation Metrics: Distribution-based metrics for unconditional or class-conditioned 3D generation are difficult to apply to text-conditioned tasks without comprehensive references.
  • 3D Evaluation Metrics: User studies offer versatile and accurate evaluation but are costly, time-consuming, and difficult to scale.
  • 3D Evaluation Metrics: The proposed automatic metrics are intended to align with human preferences while offering high customizability.
  • Large multimodality models: Large multimodal models combine visual and language capabilities, motivating their use for multimodal evaluation tasks.

3. Method Overview

The method ranks text-to-3D models under user-defined criteria through controllable prompt generation, pairwise 3D asset comparison, and Elo scoring.

  • The prompt generator produces evaluation prompts with customizable complexity and creativity levels.
  • The 3D asset comparator evaluates pairs of shapes generated from a prompt according to specified evaluation criteria.
  • Pairwise comparison outcomes are aggregated through the Elo rating system to rank the evaluated models.

4. Prompt Generation

Prompt Generation constructs evaluator-controlled text prompts from reusable components and GPT-4V instructions, enabling focused evaluation across varying complexity and creativity. The generated prompts can make evaluation settings more challenging and expose model weaknesses.

  • 4. Prompt Generation: Because evaluating all possible user prompts is computationally infeasible, the method generates prompts intended to mimic user-input distributions.
  • 4.2. Meta-prompt: The generator controls prompt difficulty by varying complexity and creativity, with more complex or creative prompts often producing more challenging evaluations.
  • 4.1. Prompt components: A typical text-to-3D prompt comprises subjects, properties, and compositions that combine into a sentence or clause.
  • 4.1. Prompt components: GPT-4V creates comprehensive subject and property lists that serve as building blocks for constructing the meta-prompt.
  • 4.2. Meta-prompt: The meta-prompt instructs GPT-4V to compose input prompts according to evaluator-specified requirements.
  • 4.2. Meta-prompt: Evaluator requirements can target complex prompts containing multiple subject interactions and properties.
  • 4.2. Meta-prompt: High-complexity prompts may contain more objects and multifaceted descriptions, while creative prompts combine subjects, verbs, or adjectives unconventionally.
  • 4.2. Meta-prompt: Controlling prompt distributions enables focused examination of model performance, since models tend to struggle to follow highly creative descriptions exactly.

5. 3D Assets Evaluator

The evaluator converts GPT-4V pairwise judgments of rendered 3D assets into model-level Elo scores, while using perturbed inputs to account for response variability.

  • 3D asset comparison: Given a text prompt and two model-generated 3D shapes, GPT-4V selects the better asset according to user-defined evaluation criteria.Each asset is represented through multi-view renderings before comparison.
  • 3D asset comparison: The comparator provides GPT-4V with multiple viewpoints and surface-normal renders to convey each 3D asset’s appearance and geometry.The renderings are arranged into images passed with textual instructions.
  • Robust comparison: Because GPT-4V responses vary probabilistically, the method ensembles outputs from perturbed prompts, layouts, view counts, criteria, and random seeds.This estimates comparison preferences more robustly than relying on one response.
  • Elo scoring: The method aggregates comparisons across sampled prompts and models, treating outcomes as probability-distributed observations.The comparisons span prompts sampled according to user-defined criteria.
  • Elo scoring: The resulting rating procedure adapts Elo, a system widely used to estimate relative performance from probabilistic pairwise competition outcomes.The paper initializes scores at 1000 and optimizes them with Adam.
  • Elo scoring: Elo scores are estimated by maximizing the likelihood of observed wins, using a matrix whose entries count how often one model beats another.The final objective is optimized from the comparison counts.

6. Results

Experiments evaluate the metric against human preferences across five criteria and show strong cross-criterion alignment, while holistic Elo profiles expose model strengths and weaknesses and support diversity evaluation.

  • Experimental setup: The study benchmarks 13 text-to-3D generative models using official implementations or Threestudio alternatives with default hyperparameters.The benchmark includes ten optimization-based and three feed-forward methods.
  • Experimental setup: The evaluation focuses on five criteria, including text–asset alignment and 3D plausibility, while comparing against CLIP similarity and aesthetic score baselines.The criteria are intended to reflect important text-to-3D evaluation needs.
  • Experimental setup: Human reference preferences come from 20 computer-vision and graphics graduate students, with three annotators assigned to each comparison question and criterion.These annotations are used to compute reference Elo ratings.
  • Alignment with human annotators: The proposed metric achieves the best Kendall’s tau correlation in 4 out of 5 criteria and the best average correlation with expert rankings.Prior metrics generally perform well on only one or two criteria.
  • Alignment with human annotators: The metric achieves top-two pairwise agreement across all but one criterion when matching individual GPT-4V decisions against human annotations.The agreement is computed from the probability that metric and human choices coincide.
  • Holistic evaluation: MVDream ranks first across all five criteria and scores about 100 Elo points above its nearest runners-up.Latent-NeRF, ProlificDreamer, and Instant3D occupy the next three positions with differences under 30 Elo points.
  • Holistic evaluation: Radar charts reveal criterion-specific strengths and weaknesses among similarly ranked models, such as ProlificDreamer’s weaker 3D plausibility.The authors present this holistic view as guidance for future algorithm development, while noting reliability can remain limited.
  • Extending evaluation criteria: With modified text and image prompts, the method extends beyond the five studied criteria to compare the diversity of outputs across random seeds.Diversity is described as an aspect often overlooked by existing text-to-3D metrics.

7. Discussion

The framework combines customizable prompt generation, GPT-4V pairwise asset comparison, and Elo ranking to evaluate text-to-3D models across criteria. It outperforms existing metrics, but remains limited by scale, reliability, adversarial vulnerability, computation, and quadratic comparisons.

  • Framework: The framework uses a prompt generator and customizable 3D-aware GPT-4V prompts to compare assets and rank models with Elo.The two components support evaluator-defined prompts and criteria before aggregating pairwise comparisons into model rankings.
  • Results: Experimental results show the approach can outperform existing metrics across various evaluation criteria.
  • Limitations: The experiments and user studies were conducted at relatively small scale because of limited resources.The authors identify larger-scale verification as future work.
  • Limitations: GPT-4V may hallucinate or exhibit systematic image-position biases that can introduce errors into the evaluation metric.Ensembling can mitigate these issues, but the authors leave efficient fundamental solutions open.
  • Limitations: The metric may be vulnerable to adversarial patterns that obtain high scores without producing high-quality 3D assets.
  • Limitations: API access limits and a quadratically growing number of comparisons may constrain scalability.

B. Method Details

The method uses customizable GPT-4V conversations to generate evaluation prompts and compare paired 3D assets from multiple views. Pairwise outcomes are converted into Elo ratings that model stochastic preferences.

  • Prompt Generator: The prompt-generation pipeline is designed as a customizable GPT-4V conversation that adapts to evaluator needs.
  • Prompt Generator: The generator combines task instructions, subject categories, property descriptions, and user-curated example prompts.Properties can describe states such as a sleeping cat, while examples guide the generator’s output style.
  • Prompt Generator: Users can generate prompts with different creativity and complexity levels, producing varied difficulty and focus.
  • 3D Asset Evaluator: Multiple perturbed instruction-and-image inputs are ensembled to obtain a more accurate estimate.
  • 3D Asset Evaluator: GPT-4V compares two assets generated from the same text description according to evaluator-defined criteria.The evaluator first analyzes the paired assets and then produces a final comparison in a specified format.
  • 3D Asset Evaluator: The evaluator can use RGB renders, surface normal maps, or both, with one, four, or nine evenly distributed viewpoints.Criteria descriptions and evaluation instructions can also be adjusted for the task.
  • Elo Rating: Elo assigns each model a strength score whose pairwise win probability increases with the rating difference.Equal ratings imply a 1/2 win probability, while c controls the scale of the difference.
  • Elo Rating: Observed pairwise wins and losses are used to estimate ratings by maximizing the likelihood of the comparison outcomes.

C. Experimental Details

The benchmark covers 13 generative models using official or Threestudio implementations, while reference labels come primarily from expert annotators. General-user studies were noisier and therefore were not used as the reference.

  • Benchmark: The benchmark includes 13 generative models: ten optimization-based methods and three feed-forward methods.
  • Benchmark: Experiments use official implementations when available, otherwise relying on Threestudio implementations and default hyperparameters.Image-to-3D methods receive Stable Diffusion XL images conditioned on text.
  • User Studies: The expert study uses 20 computer-graphics graduate students to compare 13 methods across 78 method pairs and sampled captions.

D. Ablation Studies

The ablation studies test input perturbations and ensembling, using an L1-distance between GPT-4V and expert selection probabilities because API limits prevent full alignment-score computation. Lower L1-distance indicates better human alignment.

  • Ablation Design: The ablations examine different input perturbations and methods for combining the resulting ensemble outputs.
  • Evaluation: Limited GPT-4V API access prevents computing the method-level and pair-level human-alignment scores used in the main experiments.
  • Evaluation: The alternative ablation metric compares GPT-4V’s selection probability p_i with the expert probability q_i across sampled asset comparisons.
  • Evaluation: Lower L1-distance indicates better alignment with human judgment.

D.1. Ablation for GPT-4V Input Perturbation

The ablations show that GPT-4V’s human alignment depends on balancing visual inputs, prompt focus, view count, and perturbation diversity. Ensembling several complementary strategies reduces sensitivity to individual choices and improves alignment.

  • Visual Information: RGB renders are most important for human-aligned decisions, while surface normals are required for many geometry-related evaluations.RGB-only inputs skip Color-Geo and Geometry alignment evaluation, whereas normal maps alone provide little improvement even for Geometry Details.
  • Text Instruction: Focused single-criterion instructions improve most evaluation dimensions, especially Text-Asset Alignment, but require more computation.Jointly evaluating all criteria reduces API calls, creating a compute–accuracy trade-off.
  • View Number: One view improves low-level criteria, whereas four or nine views better support global criteria such as Text-Asset Alignment and 3D Plausibility.The trade-off follows from a fixed visual context size that limits the simultaneous availability of global and local information.
  • Augmentation: Watermarks slightly improve alignment by reducing ambiguity when instructions refer to image positions.The tested augmentations also include horizontal flipping and rearranging RGB images with their corresponding normal maps.
  • Other Findings: Emphasizing normal maps produces slightly worse results than the baseline, both in Geo-first and Pure Normal settings.The findings indicate that geometry-focused ordering or normal-only inputs do not improve the baseline evaluation configuration.
  • Output Ensemble: Using 4–5 strategies across views, augmentations, visual information, and criterion-specific queries reduces L1-distance variance and outperforms prior metrics.With enough ensembles, alignment becomes less sensitive to the particular perturbation choices; Figure S4 reports consistent gains as ensemble size increases.

E. Additional Results

Additional experiments provide preliminary GPT-4V rankings of text-to-3D models across six criteria and examples of paired-asset comparisons. MVDream ranks first across criteria, while other methods show criterion-dependent strengths.

  • Ranking: MVDream ranks first across all six criteria, with Latent-NeRF generally ranking second or third.Prolific Dreamer shows comparable performance to Latent-NeRF except for Plausibility.
  • Ranking: Instant3D ranks fourth on every criterion except Plausibility, where it ranks third above Prolific Dreamer.Dreamfusion achieves good Plausibility, which the passage associates with its geometry regularization.
  • Additional Results: The reported rankings are preliminary because the method remains largely limited by API request limits.The examples nevertheless demonstrate paired visual comparisons accompanied by close observation and detailed analysis.

F. Failure cases

GPT-4V can make specific errors when evaluating text-to-3D assets, including missed object-count mismatches and confusion caused by low-level geometric noise. More views may help address some of these failures.

  • GPT-4V sometimes fails to detect mismatches in the number of balloons between two objects.
  • GPT-4V may overlook severe 3D plausibility problems when both objects contain low-level geometric noise.The analysis reports that sensitivity to noise can obscure larger structural defects.
  • Presenting a larger number of views may potentially rectify failures caused by noise and limited geometric understanding.
Loading 2401.04092v2…