PaperlayerView exact revision on arXiv

Source-linked AI summary

Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

Anson Y. Lam, Shuqing Li, Michael R. Lyu

arXiv:2610.00447v1cs.AIcs.CLcs.CVcs.GRcs.MM

TL;DR

Can text-to-3D comparisons change when scenes are fixed? This audit varies render and caption protocols and finds configuration-sensitive scores and winners, while full-search simultaneous analysis does not confirm reversed generator superiority.

  • Problem

    The paper asks which scene-generator comparisons remain credible when render and caption choices vary while generated scenes stay fixed.

    Source: §1 · p. 1

  • Method

    The audit freezes 300 scenes from six generators while varying eight render and caption factors across 19 alignment evaluators and one quality-control evaluator.

    Source: §1 · p. 2

  • Results

    Peak configuration variance exceeded between-generator variance for 17/19 alignment evaluators, and 18/19 changed point-estimate winners under some configuration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Report (generator, score, card ID) with configuration, uncertainty, stable comparison scope, and independent sensitivity evidence; stronger superiority claims need independent prompts and blinded human comparisons.

    Source: §5 · p. 9, §6 · p. 10

    Protocol-margin envelopes describe only tested configurations, not confidence intervals or claims about untested combinations, new prompt samples, or human scene quality.

    Source: §3.4 · p. 5, §5 · p. 9

Abstract

from arXiv · show

Can a text-to-3D leaderboard change when every generated scene stays fixed? We audit this question for rendered-image evaluation, where camera settings and caption wording become part of the measurement protocol. Across 300 frozen scenes from six generators, we vary eight render and caption factors for 19 alignment evaluators plus one perceptual-quality control, then test four targeted scene degradations. Peak configuration variance exceeds between-generator variance for 17/19 alignment evaluators, with prompt-bootstrap lower bounds above 1 for 11/19. Rankings are more stable than scores, yet 18/19 evaluators change their point-estimate winner under some configuration. Pairwise protocol margin envelopes show which comparisons keep their direction across the tested settings. Selected pairs have opposite pointwise intervals, but no reversal survives simultaneous inference over the full search. Thus the observed winner changes are descriptive, not confirmed changes in generator superiority. Sensitivity remains separate: no evaluator, even the prompt-free control, exceeds 67% tie-adjusted directional discrimination on layout scrambling, which is diagnostic rather than human-validated ground truth. The audit separates score stability, decision uncertainty, and targeted sensitivity, and recommends reporting (generator, score, card ID) with protocol-dependent comparisons and selection-aware uncertainty.

1 INTRODUCTION

The paper asks whether text-to-3D comparisons remain credible when camera and caption choices are explicit, and audits protocol changes while holding generated scenes fixed. It finds substantial score and winner sensitivity, then frames reporting around protocol-specific comparisons and uncertainty.

1 INTRODUCTION

  • A text-to-3D leaderboard can change without changing generated scenes because camera, lighting, and caption wording are part of the evaluation recipe.The motivating decision question is which generator comparisons remain credible when those choices are made explicit.Source: §1 · p. 1
  • The fixed-artifact audit varies one measurement-protocol factor at a time across 19 alignment evaluators and one perceptual-quality control.It separates reliability under configuration choices from sensitivity to targeted scene changes.Source: §1 · p. 1
  • Configuration variance exceeds between-generator variance for 17/19 alignment evaluators, while 18/19 change their point-estimate winner under some configuration.Bootstrap lower bounds exceed 1 for 11 evaluators; selected pairs have opposite pointwise intervals, but no reversal survives full-search simultaneous inference.Source: §1 · p. 2
  • The audit contributes a 300-scene, six-generator design, decision analyses separating score drift from uncertainty, and a versioned Evaluator Configuration Card.The card ties each reported score to the rendering and caption choices behind it.Source: §1 · p. 2
  • The proposed practical sequence is to declare the protocol, inspect method differences across a stated configuration set, quantify paired uncertainty, and test scene sensitivity independently.The paper calls this an audit before a superiority claim.Source: §1 · p. 2

2 RELATED WORK

Prior text-to-3D evaluation commonly reduces scenes to rendered images and text queries, while metric-audit research examines stability and sensitivity. This paper distinguishes its scope as auditing render and caption choices within alignment evaluators on fixed 3D scenes.

2 RELATED WORK

  • Rendered-image text-to-3D alignment evaluation uses varied views and metrics, but under-specified rendering and caption wrappers mean a metric name alone does not define the measurement.Examples include top-down CLIPScore, perspective renders, and metrics such as BLIP-2, Long-CLIP, and VQAScore.Source: §2 · p. 2
  • Metric-audit work identifies score artifacts, prompt sensitivity, and controlled-degradation blind spots, while AutoMetrics separates stability from sensitivity to worse-quality perturbations.RLiable provides uncertainty-aware comparison tools, and Evaluation Cards record evaluation-run provenance.Source: §2 · pp. 2–3
  • The paper’s scope is a reporting contract for score comparability, not a new scene-quality metric.Its configuration card records render and caption choices for published vision evaluators on frozen 3D scene artifacts.Source: §2 · p. 3
  • SceneCritic studies same-artifact instability for one VLM judge, whereas 3D-DefectBench audits a VLM defect-detection pipeline against human labels.The present work instead varies render and caption settings within 19 alignment evaluators and reports pairwise protocol envelopes with selection-aware uncertainty.Source: §2 · p. 3

3 EXPERIMENTAL SETUP

The setup treats rendered-image scoring as a protocol applied to frozen scenes, separating evaluator-side configuration sensitivity from response to targeted scene edits. It uses distinct statistics for score movement, rank changes, pairwise margins, and uncertainty.

3.1 AUDIT DESIGN

  • The audit varies evaluator-side choices one at a time with scenes fixed, then separately edits scenes to measure responses to targeted degradations.Neither part alone establishes broad validity: stable scores can track the wrong signal, and response to one degradation is not general validation.Source: §3.1 · p. 3

3.2 SCENES, PROTOCOLS, AND EVALUATORS

  • Fifty stratified prompts yield 300 frozen scenes from six generators; one Blender pipeline sweeps eight render and caption factors across 19 alignment evaluators and one quality control.The prompt-bootstrap intervals are diagnostic summaries, not population estimates.Source: §3.2 · pp. 3–4

3.3 CONTENT PROBES

  • Four paired edits probe content response; scrambling preserves object identity while changing layout, making it the cleanest spatial probe rather than pairwise human ground truth.The expected response is a dataset-level diagnostic because some original scenes may already be wrong and some prompts are generic.Source: §3.3 · p. 4

3.4 STATISTICS AND UNCERTAINTY

  • Configuration-to-signal ratio compares within-scene level variance with between-generator mean-score variance; rank concordance and winner changes separately track ordering and decision robustness.A ratio above 1 means configuration levels contribute more variance than generator means on that evaluator's scale, not that configuration matters more to people.Source: §3.4 · pp. 4–5
  • Pairwise protocol envelopes preserve ordering when strictly one-signed and flag configuration dependence when crossing zero, but cover only tested protocols.They are descriptive ranges, not confidence intervals, and do not generalize to untested combinations, new prompt samples, or human scene quality.Source: §3.4 · p. 5
  • Prompt-bootstrap intervals are pointwise descriptive, while winner-reversal inference uses an approximate simultaneous band across 110,295 searched contrasts; point-estimate changes alone imply no significance.The search spans 19 alignment evaluators, 15 generator pairs, and 387 shared configurations.Source: §3.4 · p. 5

4 RESULTS

With scenes fixed, rendered-image scores and point-estimate winners still vary across protocol settings; score movement, pairwise uncertainty, and targeted scene sensitivity yield distinct conclusions. Caption templates produce the largest shifts for most alignment evaluators, while layout-scramble discrimination remains weak and selecting the best tested views raises scores.

4.1 RENDER-SIDE SENSITIVITY: THE CAMERA MOVES THE SCORE

  • 179% is the BLIP-2 ITM score shift from top-down to eye-level pitch; CLIP encoders shift 18–33%.Across eight lighting maps, brightness correlates r = 0.83 with mean score shift, but the association does not isolate brightness from other map properties.Source: §4.1 · p. 5

4.2 PROMPT-SIDE SENSITIVITY: THE TEMPLATE IS THE LARGEST KNOB

  • Caption-template choice moves scores more than any render factor for 18/19 alignment evaluators; BLIP-VQA is the exception, with pitch dispersion of 42% versus template dispersion of 36%.For CLIP ViT-L/14, mean per-scene normalized range is 51% across the grammar and 28% across common caption forms; the best grammar beats the default by a median 18%.Source: §4.2 · pp. 5–7, Figure 2 · p. 7

4.3 SCORE MOVEMENT DOES NOT BY ITSELF ESTABLISH A WINNER REVERSAL

  • Peak configuration variance exceeds between-generator variance for 17/19 alignment evaluators, and 18/19 change their point-estimate winner across settings.Four evaluators have opposite pointwise intervals for selected pairs, but none survives full-search simultaneous inference; 4–11 of 15 pairs per evaluator retain strict ordering across tested levels.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.5 BEST-VIEW SELECTION CAN INFLATE A REPORTED SCORE

  • Selecting the highest-scoring tested pitch/yaw raises median scores by 1.03–1.17× for CLIP encoders, 1.37× for VQAScore, and 1.27× for the GPT-4o judge.Telephoto, bright, and shallow-pitch levels score highest for nearly every evaluator; the increases are lower bounds on the maximum from a larger pose search.Source: §4.5 · p. 8

4.4 CONTENT PROBES: WEAK TARGETED ARRANGEMENT SENSITIVITY

  • 74–90% tie-adjusted directional discrimination detects keeping only the largest object, while no evaluator exceeds 67% on layout scrambling.Most evaluators score 54–62% on scrambling; the ceiling belongs to prompt-free CLIP-IQA, and scrambling is a dataset-level diagnostic rather than human-validated layout ground truth.Source: §4.4 · p. 8, Figure 3 · p. 9

5 DISCUSSION

Score stability, ranking agreement, and targeted sensitivity are distinct forms of evidence, and none alone establishes scene quality or generator superiority. The discussion recommends protocol-specific comparisons, uncertainty accounting, and independent sensitivity checks.

5 DISCUSSION

  • A one-signed protocol margin envelope supports only a consistent sampled ordering across declared configurations, not a correct ordering.An interval containing zero leaves the corresponding superiority claim unresolved under that analysis.Source: §5 · p. 9
  • Identical scores across views are not the target because camera changes can reveal, hide, or clarify visual evidence.Human judgments of camera-pair legibility and intact–scrambled alignment remain separate validation targets.Source: §5 · p. 9
  • Report the evaluator protocol, configuration set, paired uncertainty, and property-matched or human-grounded sensitivity checks before making superiority claims.The recommended report identifies results as (generator, score, card ID), with camera, caption, uncertainty, companion checks, and evidence status auditable.Source: §5 · p. 9

6 CONCLUSION

The conclusion treats text-to-3D evaluation as protocol-dependent: scores and winners can shift while some pairwise directions remain stable. It calls for auditable reporting with uncertainty and independent sensitivity evidence.

6 CONCLUSION

  • Configuration variance often exceeds between-generator variance, and descriptive winners can change without those changes surviving full-search simultaneous analysis.Pairwise margin envelopes identify comparisons that retain their direction across declared settings and those requiring configuration-dependent claims.Source: §6 · p. 10

AI USE STATEMENT

  • Generative AI tools were used for paraphrase probes, optional language backbones, hosted evaluation, code, and manuscript editing; the authors take responsibility for the final work.The statement includes text, claims, code, and artifacts produced with AI assistance.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The audit uses existing generated scenes as fixed evaluator inputs and makes claims about evaluation protocols, not which generator is best.The work introduces no human participants or personally identifiable data; added blinded human comparisons require an applicable ethics determination.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Independent regeneration is not yet possible from the checkout alone because complete raw records, a verified intake manifest, an observed evaluator card, and the anonymous release archive are absent.The repository provides validators, and the hash-bound manifest contract is designed to bind source revision and raw inputs to regenerated numbers.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX MAP

  • Appendices B–K cover related-work boundaries, card scope, render examples, caption grammar, lighting, protocol envelopes, inference, hosted-score provenance, paraphrase audits, and per-evaluator tables.Appendix A maps these supporting materials while keeping the main text focused on audit logic and decision-level evidence.Source: A APPENDIX MAP · p. 15

B CLOSEST-WORK BOUNDARY

  • Table 5 distinguishes adjacent components of cited work from the paper’s combined intervention, and “Related” marks an adjacent design rather than the same intervention.“Not studied” is limited to the cited paper and does not claim absence across the full literature.Source: B CLOSEST-WORK BOUNDARY · p. 15

B.1 BROADER RELATED WORK

  • The related work spans procedural, diffusion, instruction-driven, constraint-program, VLM-optimization, and reasoning-driven scene generators with varied evaluation choices.Broader audits motivate validation, while the main-text comparison is restricted to protocol effects on fixed structured room scenes.Source: §B.1 · p. 15

C EVALUATOR CONFIGURATION CARD AND SCOPE DETAILS

The Evaluator Configuration Card is a domain-specific run-side contract for auditing single-image text-to-3D evaluator reports. The audit’s intrinsic measurements do not estimate human-perceived meaningful score differences.

C EVALUATOR CONFIGURATION CARD AND SCOPE DETAILS

  • Table 5 distinguishes adjacent components in cited work from the paper’s combined intervention; “Related” does not mean the same intervention.The table’s “Not studied” label is limited to the cited paper.Source: Table 5 · p. 16
  • The audit is intrinsic and does not estimate which score difference a person would consider meaningful.Its OFAT sweeps do not estimate interactions, and external validity requires blinded human comparisons of representative scenes.Source: C EVALUATOR CONFIGURATION CARD AND… · p. 16

D QUALITATIVE RENDERS

  • Figure 4 shows the same generated scene under two render-side configuration factors, illustrating that such changes are ordinary configuration choices rather than exotic manipulations.Source: D QUALITATIVE RENDERS · p. 16

C EVALUATOR CONFIGURATION CARD AND SCOPE DETAILS

E TEMPLATE GRAMMAR

The appendix details the caption-template grammar and reports how wording, lighting, and protocol choices affect evaluator scores or comparisons. It also specifies the scope of the tested protocol margin envelopes.

E TEMPLATE GRAMMAR

  • The 337 caption templates are source-to-sink grammar paths; Table 7 lists ordered slots, while Figure 5 depicts the same grammar as a token graph.Realized captions capitalize their first letter; the identity path emits the prompt verbatim.Source: E TEMPLATE GRAMMAR · p. 17, Table 7 · p. 17, Figure 5 · p. 18

E.1 WHICH WORDING THE EVALUATORS REWARD

  • Rendering scores highest and photo lowest among head nouns, a range of 11% of the slot mean, with rendering leading for all 19 evaluators.Choices were averaged over matched subsets to reduce confounding from other caption slots.Source: §E.1 · p. 17
  • Figure 4 contrasts focal lengths of 16, 50, and 200 mm at fixed top-down framing and studio, city, and sunset environment-map lighting.These are two render-side configuration factors applied to the same scene.Source: Figure 4 · p. 18
  • Naming a scene frame spans 14% around the slot average, while omitting the viewpoint costs about 19% relative to top-down.The scene-frame comparison places “virtual scene” about 9% above the slot average and omission about 5% below.Source: §E.1 · p. 18
  • The best template beats the default by a median 18% (up to 52%), while the default ranks 202 of 337.The best template is “a top-down rendering of a virtual scene”; article and connective choices are close to inert.Source: §E.1 · p. 19

F ENVIRONMENT-MAP SCALARS

  • Across lighting maps, brightness (EV) correlates r = 0.83 with aggregate score shift, but this association does not isolate brightness from other map properties.EV is log-average exposure and CCT is correlated color temperature; the default city map is the brightest.Source: F ENVIRONMENT-MAP SCALARS · p. 19, Table 8 · p. 19

G PAIRWISE COMPARISON OVER THE TESTED PROTOCOLS

  • Table 9’s alignment envelopes cover the baseline and 386 shared levels of seven rankable factors, while the quality control’s 50 baseline and render protocols are separate.Scene-specific paraphrases are excluded because they are not shared configuration levels; categories describe saved point estimates.Source: G PAIRWISE COMPARISON OVER THE TESTED… · p. 19

H SELECTION-AWARE SAME-PAIR INFERENCE

The paper tests whether apparent winner reversals survive selection-aware inference across a shared family of evaluator, generator-pair, and configuration contrasts. Although descriptive and pointwise reversals occur, none has opposite simultaneous intervals.

H SELECTION-AWARE SAME-PAIR INFERENCE

  • The simultaneous family covers 19 alignment evaluators, 15 generator pairs, and 387 configurations, totaling J = 110,295 paired contrasts.It includes the baseline and 386 shared levels of seven rankable factors, but excludes the image-only control and prompt-specific paraphrase strings.Source: H SELECTION-AWARE SAME-PAIR INFERENCE · p. 19
  • Fifty prompt blocks are resampled jointly across evaluators, methods, and configurations, using 2,000 draws with seed 42 to set the maximum-deviation critical value.Standard errors remain fixed at their original-sample values, and exact-constant contrasts are ineligible for inferential declarations.Source: H SELECTION-AWARE SAME-PAIR INFERENCE · p. 20
  • The resulting q0.95 = 4.397 gives approximate exploratory coverage for the fixed six-generator field and empirical prompt distribution, not a finite-sample family-wise guarantee or prevalence estimate.Table 9 distinguishes strict ordering, sign crossing, tie boundary, and a stable winner across the finite tested protocol set; these descriptive ranges are not confidence intervals or human-grounded superiority claims.Source: H SELECTION-AWARE SAME-PAIR INFERENCE · p. 20, Table 9 · p. 20
  • 18 evaluators show descriptive reversals, 4 show opposite pointwise intervals, and 0 show opposite simultaneous intervals in the primary analysis.The pointwise count is three at seed 1234; none of these comparisons establishes either selected winner as superior to all five alternatives.Source: H SELECTION-AWARE SAME-PAIR INFERENCE · p. 20

I HOSTED-SCORE PROVENANCE AND ZERO-SCORE SENSITIVITY

The appendix checks how GPT-4o’s stored zero scores affect scramble discrimination and separately audits the prompt rewrites. Zero-involved pairs yield deterministic bounds, while the rewrite auditors accept fewer than half of rewrites.

I HOSTED-SCORE PROVENANCE AND ZERO-SCORE SENSITIVITY

  • The full-sample scramble discrimination bound is 49.7–57.3% when arbitrary credit is allowed for the 23 pairs involving at least one zero score.The other 277 pairs contribute 149 units of directional credit, or 53.8% on that selected subset; the bounds concern zero-score interpretation, not sampling uncertainty.Source: I HOSTED-SCORE PROVENANCE AND… · p. 21, Table 10 · p. 21
  • Retrying 30 stored zero-valued cells resolved 13 to nonzero scores with a mean of 20.0, while 17 remained zero and carried explicit low-alignment justifications.The retry policy used GPT-4o at temperature 0 for up to six attempts; 13 persistent zeros were in HSM scenes.Source: I HOSTED-SCORE PROVENANCE AND… · p. 21

J PARAPHRASE AUDIT

  • The two hosted auditors accepted 245/500 and 238/500 rewrites, agreed on 376/500, and similarity scores did not separate accepted from rejected rewrites.The passage cautions that a machine verdict is not a human judgment.Source: J PARAPHRASE AUDIT · p. 21

K PER-EVALUATOR DETAIL

Per-evaluator details show that template sensitivity and weak scramble discrimination also appear outside the dominant SceneEval prompts for GPT-4o. Within OpenCLIP, greater scramble discrimination accompanies higher template sensitivity.

K PER-EVALUATOR DETAIL

  • Tables 13, 14, 16, and 17 report per-factor full dispersion, configuration-to-signal ratio ρ, rank stability, and tie-adjusted directional discrimination; Table 15 reports leave-one-generator-out sensitivity.The audit groups 20 evaluators by family and reports CLIP-IQA apart from the alignment metrics because it does not condition on the prompt.Source: K PER-EVALUATOR DETAIL · p. 21, Table 12 · p. 22
  • OpenCLIP ViT-bigG/14 raises tie-adjusted scramble discrimination from 54% to 64% versus ViT-L/14, while template ρ rises from 1.08 to 1.98.Using sample rather than population variance rescales every ρ by 5/6 and still leaves 13 alignment evaluators and the prompt-free control above 1.Source: K PER-EVALUATOR DETAIL · p. 21
  • For GPT-4o, template dispersion is 342% on SceneEval versus 501% on the other three benchmarks, while scramble discrimination is 55% versus 51%.Both effects are stronger off SceneEval for this judge; the source breakout does not establish the same pattern for all 20 evaluators.Source: K PER-EVALUATOR DETAIL · p. 22

L BEST AND WORST LEVEL PER RENDER FACTOR

Across render factors, evaluators consistently score telephoto, bright, and shallow-pitch levels highest and wide-angle, dark, and eye-level levels lowest, without establishing a human-grounded correct setting. The reported tables and figure summarize evaluator-specific sensitivity, stability, and uncertainty.

L BEST AND WORST LEVEL PER RENDER FACTOR

  • No render level is established as human-grounded ground truth, yet telephoto focal lengths, bright maps, and shallow pitches score highest for nearly every evaluator, while wide-angle, dark, and eye-level score lowest.The paper says standardizing one configuration improves comparability only when declared and shared; it cannot establish construct validity or repair conclusions drawn under hidden configurations.Source: L BEST AND WORST LEVEL PER RENDER FACTOR · p. 22, Table 18 · p. 26
  • Figure 6 shows that prompt-bootstrap lower bounds for peak configuration-to-signal ratio exceed one for 11 of 19 alignment evaluators; CLIP-IQA is shown separately.A high ratio alone does not establish a changed pairwise ordering.Source: Figure 6 · p. 23
  • Tables 13 and 14 report evaluator-by-factor score dispersion and configuration-to-signal ratio ρ, where ρ ≥ 1 means configuration variance equals or exceeds between-generator variance.The tables cover resolution, background, template, and paraphrase among the listed factors.Source: Table 13 · p. 23, Table 14 · p. 24
  • Table 15 reports prompt-bootstrap uncertainty and leave-one-generator-out sensitivity for peak ρ, alongside the baseline Top-1/2 gap; that gap does not identify the winner pair at every configuration.CLIP-IQA has no alignment Top-1/2 gap because it does not condition on the prompt.Source: Table 15 · p. 24
  • Table 16 reports Kendall’s W and distinct top-1 winners across factor levels, excluding paraphrases because their per-scene strings cannot be aligned across scenes.W = 1 with a single winner denotes perfect stability.Source: Table 16 · p. 25
  • Table 17 defines tie-adjusted directional discrimination as 1 for a strict drop, 0.5 for a tie, and 0 for a rise, with 50% as chance.Its intended order is a dataset-level diagnostic expectation, not per-pair human ground truth; “Mean (4)” is the plain average of four probe columns and is reported as Content acc. in Table 2.Source: Table 17 · p. 25
Loading 2610.00447v1…