PaperlayerView exact revision on arXiv

Source-linked AI summary

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

Xinyu Fang, Yuchi Wang, Weihao Yu, Rongyao Fang, Zhaohai Li, Zhixun Li, Tiancheng Gu, Xilin Wei, Yuhang Zang, Hongsheng Li, Dahua Lin, Guofeng Zhang, Shuai Bai, Zhibo Yang, Dayiheng Liu

arXiv:2610.23986v1cs.CV

TL;DR

Practical 3D creation benchmarks cover few realistic open-ended requests, despite the diverse references and deliverables these workflows involve. BLENDERWORLD evaluates such work with varied Blender tasks, multi-view assessment, and structural and visual scoring. Across 13 agents, GPT-6-Astra has the highest overall Score, yet perfect rubric completion remains uncommon.

  • Problem

    Existing benchmarks provide limited coverage of realistic, open-ended 3D requests, which may rely on diverse references and lack a single ground-truth asset.

    Source: §1 · pp. 1–2

  • Method

    BLENDERWORLD evaluates 223 Blender tasks using task-conditioned multi-view rendering, 2,651 binary rubrics, programmatic scene inspection, and visual-quality assessment.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    Across 13 agents, GPT-6-Astra achieves the highest overall Score of 79.9, but passes every rubric criterion on only 42.2% of tasks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Current agents still struggle with view-consistent 3D structure, fine-grained geometry, physical contact, and turning visual inspection into effective refinements.

    Source: §5.3.3 · p. 9

    Aggregate visual-query counts do not establish how effectively agents use visual feedback to diagnose defects or make valid corrections.

    Source: §5.3.2 · p. 8

Abstract

from arXiv · show

Recent advances in multimodal large language models (MLLMs) have enabled them to act as autonomous agents, creating and editing 3D models via code or direct software manipulation. In practical workflows, these models must infer object shapes, materials, spatial layouts, and lighting from diverse references while strictly following instructions to reconstruct, modify, or design 3D content. However, existing benchmarks provide limited coverage of such open-ended requests across real-world scenarios. To bridge this gap, we introduce BlenderWorld, the first comprehensive benchmark dedicated to evaluating frontier models on realistic 3D creation tasks. BlenderWorld comprises 223 tasks spanning architecture, products, natural environments, and scientific visualization, driven by text, image, video, and structured specification inputs. To rigorously assess these complex deliverables, we establish a unified evaluation protocol: we render the submitted Blender projects from multiple viewpoints and evaluate them against 2,651 task-specific binary rubrics alongside holistic visual quality scores. Benchmarking thirteen frontier models reveals that GPT-6-Astra achieves the highest overall fused score of 79.9, followed by Opus-5 (73.0) and GPT-5.6-Sol (67.7). Ultimately, our findings reveal a critical bottleneck: while current models can parse broad instructions, they still profoundly struggle with fine-grained 3D spatial coherence and accurate structural reproduction in Blender, leaving substantial headroom for future 3D-native agentics.

1 Introduction

Visual agents can interpret references and operate design tools, but realistic 3D creation remains difficult to evaluate because requests vary widely and often lack a single ground-truth asset. BLENDERWORLD addresses this gap with diverse Blender tasks and an evaluation protocol combining multi-view renders, binary rubrics, and visual-quality scores.

1 Introduction

  • 3D modeling tests long-horizon tool use, spatial reasoning, visual verification, and sustained interaction across applications ranging from household objects to city blocks.The paper frames these capabilities as important for extending AI agents’ productivity frontier.Source: §1 · p. 1
  • Existing Blender benchmarks focus on narrower task classes, while realistic open-ended requests may lack ground-truth assets and permit multiple valid solutions.Reliable assessment therefore needs both visual and structural evidence beyond a single rendered view.Source: §1 · pp. 1–2
  • Figure 1 pairs references on the left/top with GPT-6-Astra results on the right/bottom to illustrate the benchmark taxonomy and examples.The taxonomy spans diverse 3D creation tasks and categories.Source: Figure 1 · p. 2, §1 · p. 2
  • BLENDERWORLD covers realistic reconstruction, modification, and design through 223 tasks across 16 application scenarios, with text, image, and video references and structured specifications in 59 tasks.Agents must deliver editable Blender projects; evaluation combines task-conditioned multi-view rendering, 2,651 binary rubrics, and continuous visual-quality dimensions.Source: §1 · p. 2
  • Table 1 compares task coverage and artifact evaluation across programmatic 3D benchmarks, including BLENDERWORLD’s combination of multimodal references, task-specific rubrics, selected views, and editable-project inspection.These evaluation features address the need to inspect structure as well as rendered appearance.Source: §1 · p. 2, Table 1 · p. 3

2 Related Work

Related work spans productivity benchmarks for software workflows and 3D benchmarks for modeling, editing, reconstruction, or CAD. BLENDERWORLD focuses on practical 3D production briefs and combines editable-project inspection with task-specific compliance and visual-quality assessment.

Productivity-Oriented and Real-World Agent Benchmarks

  • Productivity benchmarks cover professional software workflows and browser or desktop tasks; BLENDERWORLD extends practical-deliverable evaluation to everyday 3D production briefs.Briefs can combine photographs, illustrations, videos, specification files, and user instructions, with editable Blender projects as deliverables.Source: §2 · p. 3

Procedural 3D and CAD Benchmarks

  • Existing 3D benchmarks target object modeling, scene editing, reconstruction, dynamics, or CAD, whereas BLENDERWORLD combines reconstruction, modification, and reference-based design across nine task categories and 16 application scenarios.Generation agents build editable projects while evaluators select task-relevant views and inspect scene structure.Source: §2 · p. 3
  • BLENDERWORLD assesses brief compliance with task-specific binary rubrics and rendered results with visual-quality scores, alongside evaluation of editable project structure.This combines task-relevant views with scene inspection.Source: §2 · p. 3

3 BLENDERWORLD

BLENDERWORLD defines interactive 3D creation tasks around production briefs and editable Blender deliverables. It constructs application-grounded, validated briefs and task-specific rubrics, then covers a broad set of scenarios, references, and production constraints.

3.1 Task Definition

  • Each task combines a natural-language request with optional image or video references and specification files; agents use observations to act on a Blender workspace and deliver an editable project with requested media.The evaluation checks deliverables against briefs and references while allowing multiple valid 3D implementations.Source: §3.1 · p. 3

3.2 Benchmark Construction

  • Benchmark construction varies production requirements and reference roles, grounds briefs in application scenarios, validates structural and semantic consistency, and iteratively reviews binary rubrics before freezing them.Rubrics are linked to brief sources and specify evidence type and importance weight.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Dataset Statistics

  • The benchmark contains 223 briefs across nine main categories, 22 subcategories, and 16 application scenarios; visual references appear in 145 tasks and specification files in 59.Among 2,651 criteria, function and geometry account for 41.7%, 83.3% have major importance, and 58.2% are assigned to scene inspection.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4 Evaluation and Metrics

BLENDERWORLD evaluates submitted artifacts using adaptive multi-view rendering and complementary visual and programmatic judges. It combines requirement satisfaction with holistic artifact quality into task scores and reports averages across the benchmark.

4.1 Evaluation Protocol

  • Submitted artifacts are archived and rendered automatically; the rendering agent selects eight complementary task-conditioned views while preserving the submitted scene and allowing limited rerenders of invalid views.Original exported media remain available alongside evaluator-generated views.Source: §4.1 · p. 5, Figure 4 · p. 6
  • Binary criteria are routed to a programmatic Agent-as-Judge for scene structure or a VLM-as-Judge for visual properties, using rendered views, exported media, and sampled video frames as appropriate.When visual evidence conflicts, original exported media take precedence.Source: §4.1 · p. 5, Figure 4 · p. 6
  • Artifact quality is scored on eight dimensions, including scene fidelity, physical plausibility, style and color, spatial layout, lighting and materials, and completeness.Each dimension receives an integer score from 1 to 10 grounded in task requirements and available references.Source: §4.1 · p. 6

4.2 Metrics

  • Requirement-satisfaction and artifact-quality component scores are combined with equal emphasis; the reported Score averages fused task scores over all 223 tasks, with failed evaluations remaining in the denominator.The component scores use binary rubric decisions weighted by importance and eight dimension ratings.Source: §4.2 · p. 6

5 Experiments

The experiments evaluate thirteen agents on 223 tasks using rubric-based requirement scores and visual-quality scores. GPT-6-Astra leads overall, but results also reveal category-specific weaknesses, divergent tool-use patterns, and recurring 3D coherence failures.

5.1 Evaluation Setup

  • The evaluation compares thirteen agents on 223 tasks, reporting mean Text, multimodal, and overall scores plus Rubric, Dimension, and Perfect metrics.Agents run once per task in isolated Linux containers, with evaluation rubrics hidden during generation.Source: §5.1 · p. 6

5.2 Main Results

  • 79.9 overall Score puts GPT-6-Astra ahead of Opus-5 at 73.0, though it passes every rubric criterion on only 42.2% of tasks.Its lead over Opus-5 is larger on multimodal tasks (80.7 vs. 72.1) than text-only tasks (78.6 vs. 74.8).Source: §5.2 · p. 7, Table 2 · p. 7

5.3.1 Performance across Task Categories

  • GPT-6-Astra leads eight of nine categories, but Opus-5 leads Rendering & Simulation and natural scenes and characters remain more challenging.Astra’s strongest categories are Procedural & Game Assets (86.8) and Products & Manufacturing (83.9); its Art & Animation Rubric score is 89.6 versus Dimension 64, while Rendering & Simulation scores are 70.6 and 71.6, respectively.Source: §5.3.1 · pp. 7–8, Figure 5 · p. 8

5.3.2 Tool Use

  • Opus-5 averages 45.5 Blender executions and 27.1 image views per task, roughly twice GPT-6-Astra’s 23.3 executions and 15.0 views, despite similar tool-budget fractions.Frequent visual inspection alone does not ensure effective correction: GPT-6-Astra scores 80.7 multimodally with 15.0 inspections, while some agents inspect more often but score moderately or poorly.Source: §5.3.2 · p. 8, Figure 6 · p. 8

5.3.3 Failure Analysis

  • Recurring failures include missing physical interactions, viewpoint-dependent omissions, distorted anatomy, and characters floating above surfaces.The audit interprets these as evidence that agents can rely on 2D visual shortcuts while struggling to construct coherent, physically sound 3D scenes.Source: §5.3.3 · pp. 8–9, Figure 7 · p. 9

6 Conclusion

BlenderWorld evaluates practical 3D reconstruction, modification, and design tasks through multi-view rendering, task-specific criteria, and visual-quality assessment. Across thirteen agents, GPT-6-Astra scores highest, but full rubric satisfaction and coherent 3D structure remain challenging.

6 Conclusion

  • Across 13 agents, GPT-6-Astra achieves the highest overall score of 79.9 but passes every rubric criterion on only 42.2% of tasks.Qualitative failures include incomplete interactions, inconsistencies across views, geometric distortions, and contact violations.Source: §6 · p. 9

7 Future Work

The authors propose expanding evaluation methods, task complexity, rubric granularity, application coverage, and experimental scope.

7 Future Work

  • Interactive judging could test rigs, animations, and simulations, while parameter extraction, deterministic scripts, reference solutions, and geometric metrics could support reproducible or precision-critical checks.Planned checks include dimensions, tolerances, geometric constraints, and voxel intersection-over-union.Source: §7 · p. 9
  • Planned tasks add localized editing, mesh repair, topology optimization, retopology, engineering drawings, tolerances, and revisions to existing projects.These tasks would test preservation of existing content, interacting constraints, and editability during revision.Source: §7 · p. 9
  • Professional 3D artists would help refine rubrics for modeling reliability, geometric stability, structural consistency, and production readiness.The authors also plan graded part-level assessments and studies of rubric agreement across viewpoints, animation frames, and domains.Source: §7 · pp. 9–10
  • The benchmark may extend to Maya, 3ds Max, additional Blender versions, MCP tools, and workflows using external asset repositories.Comparisons with and without repository access would assess how external assets affect performance.Source: §7 · pp. 9–10
  • Further experiments would broaden model, reasoning-effort, harness, platform, and interface coverage while documenting resource budgets and measuring quality, cost, latency, and variability.Planned comparisons include Linux, Windows, and macOS, plus graphical-interface operation versus API/MCP workflows.Source: §7 · pp. 9–10

A Task Coverage

The benchmark combines varied real-world 3D task inputs with source-traceable, weighted acceptance criteria and complementary visual and scene-based evaluation. Its runtime supports Blender editing and inspection while recording resource use across tasks.

A Task Coverage

  • The task set includes briefs with text, specifications, images, or video, and spans 16 application scenarios across 223 tasks.The five input combinations include 56 briefs without references or attachments, 22 with specifications only, 93 with images only, 37 with images and specifications, and 15 with videos.Source: A Task Coverage · p. 13, Table 3 · p. 13

B Rubric Construction

  • Requirements are traced to their sources, grouped into binary acceptance criteria, and weighted by importance.Criteria combine related requirements, use one of 13 categories, and receive weights of 3, 2, or 1 for major, secondary, or detail importance.Source: B Rubric Construction · p. 13, Table 4 · p. 13
  • Criteria are assigned to rendered views or scene inspection according to the evidence needed to verify them.The frozen set assigns 1,107 criteria to render and 1,544 to scene_summary.Source: B Rubric Construction · pp. 13–14

C Scoring Details

  • Eight visual-quality dimensions use integer ratings from 1 to 10, anchored from absent or broken to essentially faithful.Judges cite decisive files or views and record observed issues for each rating.Source: C Scoring Details · p. 14, Table 5 · p. 14

D.1 Runtime Configuration

  • The reference runtime uses isolated Linux workspaces with one NVIDIA GPU per task and archives each final workspace.The listed software includes Blender 4.2.10 LTS and the specified agent runtimes and interfaces.Source: §D.1 · p. 14

D.2 Available Tools; D.3 Resource Accounting

  • Agents can execute Blender scripts, inspect scenes and renders, retrieve assets, and process media through the provided harness and tools.Resource counters cover all 223 tasks and include unsuccessful calls; tool-call categories describe requested operations, not guaranteed successful executions or matched compute costs.Source: §D.2 · pp. 14–15, §D.3 · p. 15

E.1 Detailed Analysis of Judge Model

The judge comparison finds that overall scores align most closely with human preferences, although agreement falls on close-score pairs. Results also distinguish high weighted rubric coverage from complete acceptance and show that fully passing counts conceal variation in unmet criteria.

E.1 Detailed Analysis of Judge Model

  • The judge evaluation compares machine scores with majority preferences from three human reviewers on artifact pairs, excluding three split-vote cases from 150 pairs.It examines Rubric, Dimension, and equally weighted Overall scores and reports pairwise agreement and preference correlations.Source: §E.1 · p. 15
  • 82.99%, 77.55%, and 76.87% are the full-set pairwise agreement rates for Overall scores from GPT-5.6-terra, Gemini-3.8-Flash, and DeepSeek-V4.1-Flash, respectively.On 70 small-gap cases, their agreement falls to 68.57%, 58.57%, and 55.71%; Dimension scores outperform Rubric scores in agreement across all three judges.Source: §E.1 · p. 15, Table 6 · p. 16

E.2 Completion across Rubric Thresholds

  • 143 GPT-6-Astra tasks and 85 Opus-5 tasks reach a Rubric threshold of 90, while 94 and 57, respectively, reach 100.Because every criterion has positive weight, threshold 100 corresponds to Perfect; the threshold changes while the fixed task set remains the same.Source: §E.2 · p. 15, Figure 8 · p. 16

E.3 Component Scores and Unmet Criteria

  • GPT-6-Astra has 94 fully passing outputs and 64 with one unmet criterion, while Opus-5 has 57 fully passing outputs and 51 with one unmet criterion.A single unmet criterion may combine several conditions, so the count does not establish that repair is easy.Source: §E.3 · p. 16

E.4 Overlap in Fully Completed Tasks

Fully passing task sets overlap only partly, and quality profiles differ across dimensions and rubric categories. Overall rankings are stable under tested fusion weights, even as rubric and visual-quality scores diverge for individual artifacts.

E.4 Overlap in Fully Completed Tasks

  • GPT-6-Astra and Opus-5 each pass 40 tasks, but 54 pass only under GPT-6-Astra and 17 only under Opus-5.Another 112 tasks pass under neither system, showing that their successful task sets are not fully nested.Source: §E.4 · p. 16, Figure 10 · p. 17

E.5 Dimension and Rubric Profiles

  • Realism is the lowest-rated dimension for every system; GPT-6-Astra averages 6.61/10 on realism versus 7.89/10 on completeness.Opus-5 passes 87.6% of function criteria, compared with 73.5% for geometry, 71.8% for material, and 66.1% for physics; GPT-6-Astra exceeds 80% in every category.Source: Figure 11 · p. 18, §E.5 · p. 18

E.6 Sensitivity to the Fusion Weight

  • The rubric–dimension Spearman correlation is 0.630 across 2,899 artifacts, while model rankings remain unchanged across the five tested rubric weights except for Grok-4.6 and Opus-4.8.The scores can differ substantially for individual artifacts because rubric compliance and visual quality measure distinct aspects.Source: §E.6 · pp. 18–19, Figure 12 · p. 19

E.7 Task-Category Profiles for All Systems

Category profiles show GPT-6-Astra leading most categories, while selected failure traces reveal how narrow checks can miss 3D structure or the original specification.

E.7 Task-Category Profiles for All Systems

  • GPT-6-Astra leads eight categories, but Opus-5 leads Rendering & Simulation with 74.5 versus GPT-6-Astra’s 71.1.GPT-6-Astra’s category means range from 71.1 in Rendering & Simulation to 86.8 in Procedural & Game Assets; all systems score higher on Procedural & Game Assets than on Art & Animation or Characters & Animals.Source: §E.7 · p. 19

E.8 Connecting Defects to Actions

  • GPT-6-Astra’s flat photographic quads explain why the people appear from the submitted camera but disappear in side views.The frozen judgment identifies one-sided photographic cards; texture refinement cannot supply the missing volume needed for alternate views.Source: §E.8 · p. 19, Figure 15 · p. 21
  • GPT-5.6-Sol’s projection check achieved precise shoe-to-tower alignment but missed that the person’s body was unsupported above the grass.The validation required alignment error below 1 × 10−6, yet the figure remained suspended; the example illustrates a limitation in capturing 3D contact and physical structure.Source: §E.8 · p. 21, Figure 15 · p. 21
  • Opus-5’s verifier passed a 150.0 mm floor-to-frame height even though the required timber legs were 150 mm and the built timber measured 102 mm.The task specification had drifted to include the casters in the 150 mm target, leaving the original component-length requirement unsatisfied.Source: §E.8 · p. 21, Figure 15 · p. 21

E.9 Paired Case Studies across Six Tasks

Six paired case studies contrast completed deliverables and construction workflows with differences in evaluation scores and verification choices. Across the examples, passing binary criteria does not always coincide with stronger dimension scores.

E.9 Paired Case Studies across Six Tasks

  • Both booth outputs pass all binary criteria, while the case materials show Opus-5’s and GPT-6-Astra’s different booth deliverables and scores.The booth brief calls for a recreated exhibition layout with branded panels and a visible lunar rover; GPT-6-Astra’s listed result is Rubric 100.00, Dimension 85.00, Total 92.50.Source: Figure 16 · p. 22, §E.9 · p. 22
  • The lion cub workflows construct fur and rigging, yet their dimension scores are 37.50 for Opus-5 and 53.75 for GPT-6-Astra.The task requests realistic anatomy, distinct fur, whiskers, and a nose across three 1080p views.Source: Figure 16 · p. 22, §E.9 · p. 22
  • For the sofa, Opus-5 scores higher on Dimension and has better lighting, while GPT-6-Astra scores higher on Rubric.Figure 15c details the timber-length error in this case.Source: Figure 16 · p. 22
  • Opus-5’s chest workflow uses high/low meshes and baked details; its listed result is Rubric 100.00, Dimension 82.50, Total 91.25.The task requires a chest with a lid, hinges, lock, and riveted trim, baked onto a mesh of at most 3,000 faces.Source: §E.9 · p. 23
  • Both membrane workflows construct two leaflets and pass all binary criteria, while their construction traces describe curved membrane structures and cutaway views.Opus-5’s listed result is Rubric 100.00, Dimension 66.25, Total 83.12.Source: Figure 17 · p. 23, §E.9 · p. 23
  • For the train task, Opus-5 checks axle/path parameterization while GPT-6-Astra fits the camera numerically; both deliver 80-frame clips at 30 fps.GPT-6-Astra has higher frozen Rubric and Dimension scores in this case.Source: Figure 17 · p. 23

Related papers

Loading 2610.23986v1…