PaperlayerView exact revision on arXiv

Source-linked AI summary

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, Jindong Chen

arXiv:2606.01057v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Procedural 3D modeling via code lacks a standardized benchmark for assessing VLM agents, despite the expertise required to author complex assets. 3DCodeBench evaluates 12 VLMs with multimodal prompts, procedural code, automated metrics, iterative agentic settings, and human-preference rankings, finding that models execute code but struggle with geometric reasoning and physical plausibility while refinement improves performance.

  • Problem

    The field lacks a standardized, reliable benchmark for evaluating VLM agents’ ability to author complex procedural 3D assets, a labor-intensive task requiring specialized coding and geometric expertise.

  • Method

    3DCodeBench curates 26K multimodal prompt–code–object triplets across 212 categories, evaluates 12 VLMs with iterative feedback, and complements automated metrics with pairwise human-preference Elo rankings.

  • Results

    Models generally produce executable code but struggle with complex geometric reasoning and physical plausibility, while multi-turn refinement using execution feedback improves performance.

  • Takeaways & Limitations

    Effective procedural 3D modeling depends on agentic harnesses that process deterministic environmental feedback to support iterative refinement.

  • Takeaways & Limitations

    The benchmark’s planned extensions to multi-asset scenes and other platforms indicate that its current scope does not yet cover those settings.

Abstract

from arXiv · show

Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack. Authoring such procedural content, however, demands deep expertise in 3D software APIs, parametric design, and code-level geometric reasoning. In this paper, we propose 3DCodeBench, a systematic benchmark for evaluating vision-language model (VLM) agents for procedural 3D generation in 3D modeling software. Specifically, 3DCodeBench evaluates how effectively 12 advanced VLMs can serve as procedural 3D modelers by translating text and image references into procedural code for 3D modeling software. Recognizing that automated metrics may not fully capture the perceptual quality of 3D shapes, we build 3DCodeArena, a ranking platform based on pairwise human preferences over generated 3D outputs. From extensive evaluations and results, we observe that: (1) Failures mostly arise from API mismatches, while successful renders still suffer from disconnected or floating 3D geometric components. (2) Test-time scaling, such as higher thinking budgets and multi-turn refinement, improves performance overall. Our findings highlight a critical need for high-quality procedural coding data to advance commercial VLMs. Furthermore, effective procedural 3D modeling requires a robust execution environment that provides high-fidelity feedback for iterative refinement. We release 3DCodeBench, including the curated large-scale dataset of multimodal (text/image) prompts, procedural code, 3D object triplets, evaluation protocol, and the public 3DCodeArena platform as a foundational toolkit for exploring VLM-based procedural 3D modelers.

1. Introduction

3DCodeBench addresses the lack of a standardized, reliable benchmark for evaluating VLM agents as procedural 3D modelers. It combines a curated multimodal code dataset, agentic evaluation, automated metrics, and human-preference Elo rankings to assess procedural 3D generation.

  • Motivation: Procedural 3D modeling is commercially important but remains labor-intensive and difficult to author.Its applications include gaming, industrial design, and high-fidelity simulation environments for robotics training.
  • Limitations: Existing methodologies lack aligned procedural data and standardized, reliable evaluation of model capabilities.Static-mesh repositories provide extensive assets but lack the underlying procedural code needed to measure generative agency.
  • 3DCodeBench: 26K text/image prompts ↔ standalone code ↔ 3D object pairs across 212 object classes form the benchmark’s curated dataset.The dataset is produced from procedural factories with agentic curation and human verification; classes such as flying bird, crab, and dragonfly average over 400 lines of code.
  • Evaluation framework: The framework evaluates VLMs as active agents through multi-turn execution-error feedback, retries, visual self-critique, and API documentation augmentation.This design reflects the iterative nature of 3D design rather than restricting evaluation to single-shot prompting.
  • 3DCodeArena: 3DCodeArena uses pairwise human preferences to produce Elo rankings for the perceptual quality and aesthetic appeal of generated 3D shapes.The arena complements automated metrics with systematic human evaluation.
  • Findings: Physical plausibility remains the primary bottleneck beyond executability, while multi-turn refinement improves performance through deterministic execution feedback.Models frequently produce disconnected parts and incorrect structural alignments, indicating shortcomings in physical-world understanding.

2. Related Work

Prior work has evolved from static 3D repositories toward dynamic procedural pipelines, while existing evaluations largely use unit tests or target narrower procedural-generation capabilities. These limitations motivate benchmarks for executable asset generation from scratch.

  • Procedural and Agentic 3D Generation: Automated 3D content creation has progressed from static repositories to dynamic procedural pipelines such as Infinigen, BlenderProc, and Kubric.Static resources include ShapeNet, ABO, Thingi10K, Objaverse, OmniObject3D, ScanNet, and Cap3D.
  • Evaluating Procedural 3D Generation: General-purpose code benchmarks such as HumanEval, MBPP, and CodeContests rely on unit tests for evaluation.These benchmarks provide precedents for code evaluation but are not presented as comprehensive evaluations of procedural 3D generation.
  • Evaluating Procedural 3D Generation: Existing procedural 3D benchmarks target specific capabilities, including scene editing, low-complexity voxel generation, and layout prediction rather than executable asset generation from scratch.BlenderGym focuses on scene editing, VoxelCodeBench on single-shot inference for low-complexity voxel structures, and SceneScript on layout prediction.

3. 3DCodeBench: Towards 3D Generation via Writing Procedural Code

3DCodeBench evaluates VLM agents that translate text or image conditions into executable Blender Python scripts producing reference-matched 3D meshes. It combines an automated curation pipeline, a diverse procedural-code dataset, quantitative mesh evaluation, and human pairwise preference ranking through 3DCodeArena.

  • Benchmark formulation: 3DCodeBench formalizes procedural generation as a policy producing code that a deterministic 3D-software operator executes into a target mesh.The benchmark instantiates this software-agnostic formulation with Blender 5.0 and Blender Python scripts.
  • Dataset curation: The curation pipeline converts deeply nested Infinigen procedural factories into standalone Python scripts using execution feedback, solution retrieval, and human quality control.Its tools include code simplification, sandboxed Blender simulation, visual critique, and manual checks for execution, captions, and reference-image alignment.
  • Dataset scale and complexity: The benchmark spans 212 asset categories, with scripts averaging 531 lines and a median length of 387 lines, requiring reasoning about 3D structure and new API functions.The taxonomy covers organic entities, manufactured objects, and architectural fragments, while some geometry-node factory scripts exceed 1,000 lines.
  • Dataset scale and complexity: 12,963 curated instances form a larger corpus of text prompts, standalone 3D code, and objects for supervised fine-tuning and procedural-code research.The broader data includes canonical multi-view references, textured and geometry-only Blender scripts, baked GLB meshes, and three caption styles.
  • Evaluation protocol: Evaluation combines mesh-based quantitative metrics with 3DCodeArena, where human voters rank model pairs separately for text-to-3D and image-to-3D.Quantitative evaluation checks end-to-end executability and, for successful scripts, computes mesh-grounded similarities and four-view perceptual fidelity.

4. Experiments

Experiments evaluate 12 frontier VLMs under successful-rendering conventions and show that automated multi-view metrics align strongly with human preferences. Test-time reasoning and error-feedback retries improve reliability, while extra input views and autonomous harnesses provide limited or no consistent shape-quality gains.

  • Automated evaluation: SigLIP-2 view similarity is the strongest linear predictor of human preference, with Pearson r=0.964, while DINOv3 achieves the highest rank correlation at Spearman ρ=0.972.These correlations across all 12 evaluated models support using scalable automated metrics as proxies for perceptual evaluation.
  • Test-time scaling: Gemini 3.1 Flash Lite gains approximately 19 executability points from minimal to high thinking budgets, whereas Pro-class backbones change by fewer than five points.Thinking-budget gains are therefore concentrated among lightweight reasoners and saturate early for frontier models.
  • Test-time scaling: Conditional SigLIP-2 varies by at most 0.012 across input-view budgets within each backbone, with extra views yielding no consistent gain over N=1.Conditional Uni3D gains are likewise described as modest and stable across N∈{1, 2, 3, 4}.
  • Agentic workflows: Aggregate executability rises from 0.702 to 0.974 with multi-turn error-feedback, a +27.2 pp improvement, and 8 of 22 cells reach the 1.000 ceiling.The loop grants up to two stateless retries using prior code and truncated Blender tracebacks, while improving penalized quality across all 22 cells.
  • Agentic workflows: Coding-agent harnesses lift Executability but produce shape fidelity indistinguishable from a single prompt when scored on the same instance subset.The comparison wraps in-budget backbones in their native coding-agent harnesses under a fixed wall-clock budget.
  • Qualitative analysis: Qualitative comparisons show frontier models capturing basic silhouettes but often producing disconnected geometric fragments or simplistic, floating primitives.The comparison covers six held-out prompts and evaluates Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.5 against references.

5. Conclusion and Future Work … A.2. Per-Model Main-Results Table

The paper introduces 3DCodeBench and 3DCodeArena as a framework for evaluating procedural 3D modeling agents, while proposing extensions to multi-asset composition, cross-platform evaluation, and larger-scale data curation. Its appendix analyzes cost–quality tradeoffs across paid models and reports per-model metrics at each model’s best thinking level.

  • 5. Conclusion and Future Work: 3DCodeBench evaluates VLM agents for procedural 3D modeling using a diverse dataset of 212 object categories paired with executable code.The benchmark combines automated metrics with 3DCodeArena human preferences.
  • 5. Conclusion and Future Work: Future work will extend the benchmark to multi-asset scene composition and cross-platform evaluation in SideFX Houdini or Unreal Engine.These extensions aim to disentangle API memorization from generalized procedural modeling capabilities.
  • 5. Conclusion and Future Work: Scaling the curation pipeline is proposed as a way to create larger datasets for pre-training next-generation 3D-aware VLMs.The paper presents 3DCodeBench and 3DCodeArena as a foundational framework for advancing autonomous agents.
  • A.1. Cost–Quality Pareto Frontier: Figure A.1 compares each paid model’s 3DCodeArena Bradley–Terry Elo with its per-query API cost and identifies the cheapest model reaching each Elo level.The two free Gemma backbones are omitted, leaving 10 paid frontier VLMs; the dashed line marks the Pareto frontier.
  • A.1. Cost–Quality Pareto Frontier: Four of the five Pareto-frontier points are Gemini models, while GPT-5.5 is the only non-Gemini point and reaches the highest Elo.The frontier includes Gemini 3.1 Flash Lite ($0.01, 877), Gemini 3 Flash ($0.02, 1039), Gemini 3.5 Flash ($0.04, 1119), Gemini 3.1 Pro ($0.12, 1147), and GPT-5.5 ($0.32, 1163).
  • A.1. Cost–Quality Pareto Frontier: Elo rises quickly at low cost and then flattens, with Gemini 3.5 Flash coming within about 44 Elo of the best model at roughly one-eighth of its cost.The best model referenced in this comparison is GPT-5.5 at 1163 Elo.
  • A.2. Per-Model Main-Results Table: For each of the 12 evaluated VLMs, the appendix selects the single thinking-effort level that maximizes SigLIP-2 and reports metrics averaged across text-to-3D and image-to-3D tracks.Claude and GPT use their base run, Gemma uses the API-permitted high level, and only Gemini models have multiple thinking levels to select from.

A.3. Elo vs. Metrics under Single-Shot Run · A.4. Aggregation Comparison: Single-Shot vs. Thinking-Average · A.5. Models Evaluated

Single-shot analyses preserve the main metric–human-preference relationships and produce rankings similar to best-level and thinking-average aggregation. The benchmark evaluates twelve VLMs across compute tiers under standardized zero-shot Blender execution, excluding two models that fail the text-to-3D executability threshold.

  • A.3. Elo vs. Metrics under Single-Shot Run: Single-shot rankings are essentially unchanged, with image-grounded similarity and 3D-shape metrics remaining strong monotone predictors of human preference.The cross-modal and 3D–3D Uni3D panels are the strongest; executability is the weakest relationship.
  • A.4. Aggregation Comparison: Single-Shot vs. Thinking-Average: The three aggregation methods produce very similar model rankings, with Gemini differences reaching 0.08 between best-level and all-level mean.The alternatives are single-shot base runs and thinking-level averages; Claude and GPT use their base runs when thinking ablations are unavailable.
  • A.4. Aggregation Comparison: Single-Shot vs. Thinking-Average: Single-shot values average one model call per instance across both text-to-3D and image-to-3D tracks.Open-weight Gemma is evaluated at its API-default thinking level because it has no separate base run.
  • A.4. Aggregation Comparison: Single-Shot vs. Thinking-Average: Thinking-level averages combine all available effort levels, with Gemini and Gemma averaged across four levels and three seeds per level.Claude and GPT fall back to their single-shot base runs, and values remain conditional means averaged across both tracks.
  • A.5. Models Evaluated: The benchmark evaluates twelve frontier VLMs from four providers across lightweight, mid-tier, and frontier compute tiers.The lightweight tier includes Claude Haiku 4.5, Gemini 3.1 Flash Lite, GPT-5.4-mini, Gemma 4 26B, and Gemma 4 31B; mid-tier and frontier models are listed separately.
  • A.5. Models Evaluated: All models are queried zero-shot with task-specific prompts, using text descriptions or four canonical reference views depending on the track.Decoding uses temperature 0.7, a 65,536-token output cap, Markdown-fence stripping, and Blender 5.0 execution.
  • A.5. Models Evaluated: Gemini 2.5 Pro and GPT-5.4 Nano are excluded after falling below the 10% single-shot text-to-3D executability floor.Gemini 2.5 Pro records 0.071 text-to-3D and 0.217 image-to-3D executability; GPT-5.4 Nano records 0.061 and 0.165, respectively.
  • A.5. Models Evaluated: The multi-turn protocol permits two additional stateless retries per failed instance, while the agentic harness gives provider-native CLI agents 600–900 s to write, run, debug, and iterate on Blender scripts.Gemma is omitted from the agentic-harness regime because it has no first-party CLI agent.

A.6. Thinking-Level Ablation: 3D-Shape and DINOv3 Metrics

The thinking-budget sweep evaluates four additional 3D-shape and DINOv3 metrics on the same cells as the SigLIP-2 analysis. Lightweight reasoners improve substantially with more thinking, whereas frontier models plateau early and Gemini 3.1 Pro behaves differently across tracks.

  • Metrics: The ablation adds Uni3D 3D–3D paired cosine, Uni3D cross-modal cosine, and DINOv3 view-paired cosine alongside the SigLIP-2 sweep.DINOv3 view-paired cosine is used on image-to-3D, while text-to-3D uses the Uni3D text–3D variant because it lacks rendered reference views.
  • Thinking-budget effects: Gemini 3.1 Flash Lite, Claude Haiku 4.5, and GPT-5.4-mini gain substantially from minimal to high thinking budgets across every metric and panel.These results match the qualitative pattern reported for SigLIP-2.
  • Thinking-budget effects: Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.5 plateau early, with Opus already at the per-track ceiling at minimal thinking.Frontier reasoners therefore show less benefit from increasing the thinking budget than lightweight reasoners.
  • Thinking-budget effects: Gemini 3.1 Pro decreases mildly on text-to-3D but increases sharply on image-to-3D as thinking rises.This track-dependent reversal is consistent with the headline single-turn ablation finding.

A.7. Multi-View Image Budget Ablation

The ablation tests whether increasing canonical reference views from one to four improves image-to-3D generation across six backbones. Performance peaks at different view budgets by model capacity, with only Gemini 3.1 Pro avoiding regression at four views.

  • Setup: The study sweeps N∈{1, 2, 3, 4} canonical views, adding azimuths 45°, 135°, 225°, and 315° in order.All six open-budget image-to-3D backbones receive progressively larger reference-view budgets.
  • Findings: 0.758±0.031 Exec. is Gemini 3.1 Pro’s peak at N=4, while N=2 is statistically tied and N=2, 3, 4 substantially beat N=1.Gemini 3.1 Pro is the only backbone that does not regress at four views.
  • Findings: As capacity decreases, the optimum shifts from N=3 for Gemini 3 Flash to N=2 for Gemma 4 31B and N=1 for Gemma 4 26B.Gemma 4 26B is flat-to-degrading after one view, whereas Gemini 3.1 Flash Lite is view-budget-insensitive within seed noise.
  • Findings: Smaller backbones gain less marginal information from additional views but incur the same context cost, causing earlier saturation or distraction.The interpretation contrasts their behavior with the largest backbone’s ability to absorb four reference views without trading off response quality elsewhere.

B. Evaluation metric implementation … D. Iterative inference and multi-stage pipelines

The appendix specifies a deterministic Blender 5.0 evaluation and inference protocol spanning executability, multimodal similarity, 3D-shape metrics, strict code-generation prompts, iterative repair, visual critique, image-mediated generation, and agent harnesses.

  • B. Evaluation metric implementation; B.1. Executability: The render driver executes each script in a fresh Blender 5.0 subprocess for 240 s and evaluates four canonical views across N=212 test instances.Executability failures are categorized as ERR_EXEC, ERR_NO_MESH, ERR_RENDER, or ERR_TIMEOUT, with exception fingerprints diagnosing recurring API mismatches.
  • B.2. Image-Grounded Similarity (Image-to-3D Track); B.3. Text-Render Similarity (Text-to-3D Track): Image-to-3D evaluation compares matched generated and reference views using separate SigLIP-2 semantic and DINOv3 shape-structural similarities.The text-to-3D track instead compares rendered views with SigLIP-2 text embeddings and reports reference-render performance as a soft GT ceiling.
  • B.4. 3D-Shape Similarity: 3D-shape evaluation samples K=8192 normalized surface points and reports yaw-aligned symmetric squared Chamfer plus Uni3D 3D–3D and cross-modal cosine similarities.Uni3D uses an EVA-Giant point encoder and a 1024-dim CLIP-aligned latent; cross-modal comparisons target either the prompt or canonical 45° reference view.
  • B.5. Conditional vs. Penalized Aggregation: Headline Table A.1 metrics use penalized aggregation, assigning failed instances a finite run-relative Chamfer penalty rather than rewarding suppressed outputs.Conditional aggregation isolates geometric quality from code reliability, while penalized aggregation incorporates failures into model scores.
  • C. Inference setup; C.1. Text-to-3D System Prompt; C.2. Image-to-3D system prompt: Inference fixes raw Blender 5.0 Python output, a closed library allow-list, deterministic execution, one object, no scenery, no rendering, and no file I/O.Text prompts provide descriptions, while image prompts provide reference renders and require reconstructing the depicted object as a self-contained script.
  • C.3. Multi-Turn Error-Feedback User Template: Multi-turn error feedback is stateless: each retry receives the original task, previous code, attempt status, rendered-view count, and truncated Blender diagnostics.The template directs models to fix API, mesh, rendering, or timeout failures and output a complete corrected script rather than a patch or explanation.
  • C.4. Visual Self-Critique Prompts: Visual self-critique runs on baseline-OK instances, comparing rendered views with text or image references and prioritizing missing parts, floating pieces, proportions, alignment, detail, and overall shape.The protocol allows acceptance or a complete rewrite under a limited iteration budget, with a stated conservatism bias favoring good-enough renders over risky fixes.

D.1. Multi-Turn Error-Feedback Retry

Multi-turn retry gives failed renders up to two stateless error-feedback attempts, using the original task, prior script, and truncated stderr. Across the fixed 212-instance evaluation, it substantially improves executability and penalized benchmark quality, though at measurable computational cost.

  • Setup: Each failed single-turn render receives up to two stateless retries, and the loop stops after the first success or third attempt.Retries include the original task, previous full script, and stderr truncated to a 3 K-character cap.
  • Findings: Executability lifts substantially across all 22 cells, with approximately 90% of baseline failures caused by localized Blender 5.0 API mismatches.Visible tracebacks make these mismatches suitable for copy-pasteable fixes.
  • Findings: The penalized-mean SigLIP-2 delta is positive in every cell, ranging from +0.048 to +0.365 for image and +0.018 to +0.083 for text.The change is computed over all 212 instances, with failures contributing 0 and Chamfer using a 1.5×max penalty.
  • Cost: Total retry cost on baseline-failed instances is $55.54, while wall-clock time ranges from approximately 10 minutes to approximately 2 hours across four parallel workers.Gemma is free, GPT-5.4-mini and GPT-5.5 cost approximately $15 each, and Claude rows cost $1.1–$2.3.

D.2. Visual Self-Critique Loop … F. LLM/VLM-as-a-judge against the human arena

The paper evaluates iterative visual self-critique, an intermediate text-to-image-to-3D pipeline, human pairwise preferences in 3DCodeArena, and LLM/VLM judges against arena verdicts. Results show task-dependent critique effects, regressions from image-only intermediates, a structured blind-voting platform, and evaluation of render- and code-based judging.

  • D.2. Visual Self-Critique Loop: The loop shows previous code and renders, optionally reference images, and permits up to two corrected-script iterations with revert-on-break protection.The model can accept a result or emit a corrected full Python script; failed fixes restore the prior executable state.
  • D.2. Visual Self-Critique Loop: The visual self-critique loop shows task-asymmetric effects: text-to-3D improves across all four backbones, whereas image-to-3D uniformly flips direction.Text-to-3D gains are +0.003 to +0.009 mean Δ SigLIP-2, with win/lose ratios of 1.24–2.63.
  • D.2. Visual Self-Critique Loop: On image-to-3D rows, Gemma 4 31B accepts 49/119 baselines (41%), versus 3–15 (3–13%) for smaller backbones, while reverts show the opposite trend.Reverts are 11% on 31B versus 26–28% on smaller backbones.
  • D.3. Text-to-Image-to-3D Pipeline: The pipeline compares direct text-to-3D, image-only Nano Banana Pro mediation, and combined text-plus-photo conditioning, scoring all configurations against the original description.Nano Banana Pro generates one studio three-quarter-view photo per instance at $0.134 each, or $28.27 for all 212 instances.
  • D.3. Text-to-Image-to-3D Pipeline: Image-only text-to-image-to-3D regresses against direct text-to-3D on four of five backbones, with the largest losses on the smaller Flash models.Reported drops include −0.059 and −0.067 for the smaller Flash class and −0.018 for 3.1 Pro.
  • E. 3DCodeArena: human-preference voting interface: 3DCodeArena presents blind pairwise comparisons of executable GLB meshes, randomizes presentation order, and records one of four voter verdicts.Voters choose A is better, B is better, Tie, or Both bad after independently viewing neutral-gray, auto-fitted meshes.
  • E. 3DCodeArena: human-preference voting interface: The arena adapts its interface by modality: text-to-3D shows the natural-language prompt, while image-to-3D adds a four-image reference strip and maintains separate Elo scales.The same vote pool supports a public leaderboard and later judge evaluation.
  • F. LLM/VLM-as-a-judge against the human arena: The judge study tests whether frontier LLM/VLMs recover human arena verdicts from identical pairwise comparisons using render-based and Blender-free code-only modes.A snapshot of n=2,508 vote rows supplies the human ground truth for exact-match agreement and decisive-row accuracy.

F.1. Data and Protocol · F.2. Judge Prompt

The evaluation uses a 2,508-pair judge set drawn from 3DCodeArena votes, with four Google models assessing anonymized image- or code-mode inputs under controlled sampling and position-bias procedures. Judges return standardized winner labels and reasoning, following prompts that prioritize object identity, structure, detail, reference faithfulness for images, and Blender executability for code.

  • F.1. Data and Protocol: 2,508 pairs remained after removing 52 votes with missing artifacts from 2,560 arena votes across 212 instances.The set contains 1,455 image-track and 1,053 text-track pairs, distributed as 1,012 a, 1,105 b, 192 tie, and 199 both_bad.
  • F.1. Data and Protocol: Four Google models judged every pair in image and code modes using anonymized System A/System B inputs.Image mode provided four renders per side plus 1−4 reference images for image-track prompts; code mode provided full Blender Python source capped at 60 K characters.
  • F.1. Data and Protocol: Sampling was greedy at T=0 with low thinking, and each verdict used strict single-line JSON with winner ∈{a, b, tie, both_bad} plus one-to-three-sentence reasoning.Gemma had no tunable thinking setting, and fewer than 1% of pairs reached the 60 K-character code cap.
  • F.1. Data and Protocol: A deterministic approximately balanced swap randomized pair-side presentation so positional preferences averaged out over A/B identity.The swap bit was derived from Random(vote_id).random() < 0.5, and verdicts were not unswept after the call.
  • F.1. Data and Protocol: The full sweep required 4 × 2 × 2,508 ≈20 K API calls over approximately four wall-clock hours.Thirteen pairs failed JSON parsing across all eight cells and were dropped.
  • F.2. Judge Prompt: Image-mode judges assessed four canonical renders from the same prompt and selected which anonymous system better matched the prompt.The verdict options were a, b, tie, and both_bad.
  • F.2. Judge Prompt: Image judging prioritized object identity, structural correctness, geometric detail, and image-to-3D reference faithfulness, while excluding color and material.The criteria were weighted in the listed order.
  • F.2. Judge Prompt: Code-mode judging evaluated scripts without execution for object intent, structural decomposition, geometric detail, and executability under Blender 5.0.The code and image prompts shared the strict JSON contract requiring winner and reasoning fields.

F.3. Headline Results: The Four-Verdict View · F.4. A/B-Only View: Accuracy and Correlation on Decisive–Decisive Rows

The four-verdict view finds image judging substantially more reliable than code judging, while the A/B-only view shows Pro leading image accuracy and Pro–Flash parity on code accuracy. Image references improve image-track judgments, but that advantage disappears in code mode.

  • F.3. Headline Results: The Four-Verdict View: Image judges achieve 62.5%−64.7% overall agreement and approximately 73% on decisive rows, exceeding the 25% four-class chance line and 44.1% majority baseline.These rates summarize agreement with the human arena under the four-verdict formulation.
  • F.3. Headline Results: The Four-Verdict View: Code judges reach 51.7%−56.9% overall agreement and 59.6%−67.4% on decisive rows, making them usable for trend-level comparisons but not interchangeable with renders.Code judging is approximately 7−13 percentage points below image judging overall.
  • F.3. Headline Results: The Four-Verdict View: Pro alone uses tie and both_bad, whereas Flash, Flash Lite, and Gemma use binary a/b votes with non-decisive calls at 0%−3.1% versus the human rate of 15.6%.Pro abstains on 8.7% of image calls and 12.6% of code calls, so four-verdict exact match penalizes its decisive-row ties.
  • F.4. A/B-Only View: Accuracy and Correlation on Decisive–Decisive Rows: The A/B-only analysis drops tie and both_bad from both sides, evaluating agreement on 1,864−2,116 doubly-decisive rows per cell.It additionally reports Cohen’s κ and Pearson ϕ, with ϕ computed on a=0, b=1.
  • F.4. A/B-Only View: Accuracy and Correlation on Decisive–Decisive Rows: Pro leads image judging at 77.1% accuracy and κ=+0.542, ahead of Flash at 75.6%/+0.513, Flash Lite at 74.7%/+0.494, and Gemma at 74.0%/+0.479.Pro covers 94.2% of human-decisive rows versus 98%−98.3% for the others, trading approximately 4 percentage points of throughput for a 1.5-point accuracy gain.
  • F.4. A/B-Only View: Accuracy and Correlation on Decisive–Decisive Rows: Code A/B accuracy is 62.6%−67.7% with κ=+0.249 to +0.348, while Pro and Flash are within 0.3 percentage points at 67.7% versus 67.4%.Flash Lite trails meaningfully at κ=+0.249, with reasoning traces emphasizing surface-code style over geometric intent.
  • F.4. A/B-Only View: Accuracy and Correlation on Decisive–Decisive Rows: Image-track accuracy is 2−3 percentage points higher than text-track accuracy for every image judge, with Pro at 78.0% versus 76.0%.The image-track supplies four reference views as an additional anchor.
  • F.4. A/B-Only View: Accuracy and Correlation on Decisive–Decisive Rows: The image-track advantage vanishes in code mode, where references are unavailable, consistent with the gain arising from the visual anchor rather than prompt-track difficulty.This contrast isolates the effect of additional visual references on image judgments.

G. Sampling temperature ablation

The ablation finds that T=2.0 is uniformly worst, while T∈{0, 0.7, 1.0} performs similarly on strong models and Flash Lite peaks at T=0.7. T=0.7 is recommended as a Pareto-frontier setting, whereas T≥1.5 is discouraged because greedy decoding can sharply increase Flash-class costs.

  • G. Sampling temperature ablation: T=2.0 is uniformly worst across every evaluated quality cell.The sweep covers text-to-3D and image-to-3D metrics for Gemini 3 Flash, Flash Lite, and Pro, with 3 seeds per cell.
  • G. Sampling temperature ablation: Flash Lite peaks at T=0.7 with image-to-3D quality of 0.627±0.016, then degrades above T=1.0.The spread between T=0.7 and T=2.0 is approximately 17 percentage points.
  • G. Sampling temperature ablation: On Flash and Pro, T∈{0, 0.7, 1.0} overlaps within 1σ on every metric.For Pro text-to-3D Exec, the corresponding values are 0.692/0.684/0.693.
  • G. Sampling temperature ablation: At T=0, Flash Lite uses approximately 63 K thinking tokens per instance versus approximately 4 K at T=0.7.The reported costs are $20.34 per 212-instance pass at T=0 and $2.47 at T=0.7.
  • G. Sampling temperature ablation: The study recommends T=0.7 as a Pareto-frontier setting and recommends against T≥1.5.The headline numbers in Table A.1 use temperature=0.7.
Loading 2606.01057v1…