PaperlayerView exact revision on arXiv

Source-linked AI summary

BlenderAlchemy: Editing 3D Graphics with Vision-Language Models

Ian Huang, Guandao Yang, Leonidas Guibas

arXiv:2404.17672v3cs.CVcs.GR

TL;DR

Blender design can require extensive repeated editing, and language alone does not convey the visual consequences of actions in the software. BlenderAlchemy iteratively searches visual-program edits using a vision-language edit generator, a state evaluator, and generated reference images to ground text intentions. The paper reports improved text-based procedural material editing over prior work and demonstrates applicability to geometry and lighting tasks.

  • Problem

    Blender design involves repeated editing, while language alone cannot capture the visual consequences of actions and task-specific methods are non-trivial to combine.

    Source: §1 · pp. 1–2

  • Method

    BlenderAlchemy iteratively proposes and visually evaluates program edits, using generated reference images to ground abstract textual intentions.

    Source: §3.2 · p. 7, Visual Imagination · p. 8

  • Results

    BlenderAlchemy reports outperforming prior work in text-based procedural material editing and demonstrates applicability to geometry and lighting tasks.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The system demonstrates vision-language-model editing of Blender programs across material, geometry, and lighting design tasks.

    Source: §6 · p. 16

    GPT-4V inference is expensive and high-latency; the reported 4 × 8 system costs just under $3 per material, with costs likely to rise when many candidates are needed.

    Source: F Limitations · p. 33

Abstract

from arXiv · show

Graphics design is important for various applications, including movie production and game design. To create a high-quality scene, designers usually need to spend hours in software like Blender, in which they might need to interleave and repeat operations, such as connecting material nodes, hundreds of times. Moreover, slightly different design goals may require completely different sequences, making automation difficult. In this paper, we propose a system that leverages Vision-Language Models (VLMs), like GPT-4V, to intelligently search the design action space to arrive at an answer that can satisfy a user's intent. Specifically, we design a vision-based edit generator and state evaluator to work together to find the correct sequence of actions to achieve the goal. Inspired by the role of visual imagination in the human design process, we supplement the visual reasoning capabilities of VLMs with "imagined" reference images from image-generation models, providing visual grounding of abstract language descriptions. In this paper, we provide empirical evidence suggesting our system can produce simple but tedious Blender editing sequences for tasks such as editing procedural materials and geometry from text and/or reference images, as well as adjusting lighting configurations for product renderings in complex scenes.

1 Introduction

Blender design requires repeated visual experimentation, while language alone does not capture the visual consequences of software actions. BlenderAlchemy uses vision-language models to search program edits guided by visual evaluation and imagined references, and reports gains on procedural material editing alongside applicability to geometry and lighting tasks.

1 Introduction

  • Creating high-quality graphics in Blender can require hours of repeated work on materials, object placement, and lighting.Artists mentally picture a target, experiment with parameters, and visually inspect whether edits move toward it.Source: §1 · p. 1
  • Language-based graphic-design automation remains challenging because language does not capture the visual consequences of Blender actions, and combining task-specific tools is non-trivial.Earlier approaches often focus on specific editing tasks such as material synthesis.Source: §1 · p. 2
  • Its iterative search combines a vision-aware edit generator with a visual state evaluator that selects among rendered program variants against the design goal.The system can revert to a prior best candidate when a new iteration lacks a viable edit, and uses generated reference images to guide text-only requests.Source: §1 · p. 3
  • BlenderAlchemy reports outperforming prior work in text-based procedural material editing and demonstrates applicability to geometry editing and lighting configurations.The system is presented as a visual-program editor driven by text or image user input.Source: §1 · pp. 3–4

2 Related works

Prior research addresses material generation and language-guided design through specialized representations or language-model workflows, but the cited approaches leave gaps in editability, visual feedback, or generality. BlenderAlchemy aims to use visual perception for a range of Blender design tasks.

2 Related works

  • Prior material-generation approaches produce image- or latent-based representations that can be difficult to edit in existing 3D graphics pipelines.Some methods instead combine learning with symbolic material representations or use rule-based procedural generation.Source: §2 · p. 4
  • The cited LLM-based Blender systems do not use visual information to inform or refine their outputs, which the paper associates with unsatisfactory results.Textual or symbolic condensation of 3D visual states is described as working poorly for graphic design.Source: §2 · pp. 4–5
  • BlenderAlchemy applies VLM-based visual perception to 3D graphic-design tasks, a challenging application that the paper describes as relatively unexplored.VLMs have also been proposed as flexible evaluators for guiding planning and search.Source: §2 · p. 5

3 Method

BlenderAlchemy represents a Blender scene as a base state plus executable task-specific programs, then iteratively proposes and visually evaluates edits. Its search combines constrained parameter tweaks with broader structural edits, can revert to a prior best state, and uses generated images to ground abstract textual intentions.

3 Method

  • The method edits programs that define Blender actions to move an initial scene toward a user intention.The paper frames the task as decomposing an initial Blender input into programs and a base state, then editing those programs to produce a desired visual state.Source: §3 · p. 5

3.1 Representation of the Blender Visual State

  • An initial Blender environment is represented as a base state and task-specific programs executed by Blender's Python executor.Programs can separately handle parts of the workflow, such as a mesh's material or the scene lighting; splitting tasks avoids the output-length and search-space limits of a single program.Source: §3.1 · pp. 5–6

3.2 Iterative Refinement of Individual Visual Programs

  • The edit generator proposes multiple program variants conditioned on the current state and intention, while a visual evaluator selects the better rendered state through pairwise comparisons.The selected candidate can include the best state from a prior iteration, enabling reversion when a search step is unsuccessful.Source: §3.2 · p. 7, Hypothesis Reversion · p. 7
  • The search alternates between numerical parameter tweaks and broader program-space edits, while generated images ground abstract text intentions in visual references.The generated references guide low-level visual comparisons and provide users an interpretable intermediary for confirming vague goals.Source: Hypothesis Reversion · p. 8, Visual Imagination · p. 8

4 Experiments

The experiments test procedural material editing from text and images, with comparisons against BlenderGPT and diffusion-based methods. They also demonstrate the system on geometry and lighting tasks.

4 Experiments

  • The experiments also demonstrate BlenderAlchemy on procedural materials, geometry, and lighting setups.These are described as three tedious parts of 3D design.Source: §4 · p. 8

4.1 Procedural Material Editing

  • Text-based tests edit one Infinigen wooden material into materials described by diverse prompts, while image-based tests adapt different starters toward one target image.The text-based system uses GPT-4V for generation and evaluation and DallE-3 for visual imagination, with 8 hypotheses per iteration over 4 iterations.Source: Text-based material-editing · p. 9, §4.1 · p. 9
  • BlenderAlchemy produces deeper edits that better align with prompts than BlenderGPT, whose edits are shallower and sometimes miss requested visual qualities.For example, BlenderAlchemy creates the sharper angles requested for digital camouflage, while BlenderGPT gets the colors right but not the sharp digital look.Source: Text-based material-editing · pp. 9–10, Figure 5 · p. 12
  • In user comparisons, BlenderAlchemy is preferred to Paint3D 73% of the time and TEXTure 56% of the time; users prefer evaluator-selected outputs in 71% of comparisons.The study collected 592 comparisons from 24 Turkers across 32 text prompts.Source: Text-based material-editing · pp. 10–11
  • The no-vision variant outperforms BlenderGPT on CLIP similarity, and adding vision to the state evaluator alone improves results further.Table 1 reports average ViT-B/32 and ViT-L/14 CLIP similarity scores for the text description.Source: Text-based material-editing · p. 10, Table 1 · p. 11
  • For image-based editing, the generator describes visual differences, maps them to relevant code lines, then proposes edits; outputs can transfer target attributes without perfectly matching the image.The image-based system does not use visual imagination.Source: Image-based material-editing · p. 11, Figure 6 · p. 12

5 Ablation Experiments on Material Editing

The ablations examine how search dimensions and system components affect material editing. The section also demonstrates geometry editing and iterative lighting-and-material editing in product scenes.

5 Ablation Experiments on Material Editing

  • Balanced search dimensions outperform sequential refinement and unrefined querying at the same 32 edit-generator calls, with 8 × 4 or 4 × 8 identified as a sweet spot.The tested dimensions trade off overly broad exploration against overly deep exploitation of suboptimal candidates.Source: Value of the multi-hypothesis,… · p. 13, Table 2 · p. 13
  • Removing hypothesis reversion causes alignment to diverge over edits, with larger CLIP-similarity drops on leap-edit steps.The table reports ViT-B/32 CLIP scores across edits 1–4 for the text-based material task.Source: Value of the multi-hypothesis,… · p. 13, Table 3 · p. 13
  • Leap-only edits strongly diverge from user intent, while tweak-only edits improve alignment slowly but remain more conservative than the combined variant.The comparison is among -Tweak, -Leap, and Ours in Table 3.Source: Table 3 · p. 13, Tweak/Leap Edits · p. 14

5.1 Geometry Editing

  • BlenderAlchemy edits geometry by interpolating blend shapes, modifying geometry-node graphs, and placing scene objects; one facial-expression example coordinates 19 blend shapes.The rose graph contains 60 geometry nodes, and its edits address petal number and angle and leaf density.Source: §5.1 · pp. 15–16

5.2 Lighting Setup Editing

  • The system iteratively adjusts lighting and material using separate edit-generator and state-evaluator pairs to align a BlenderKit product scene with a text intention.The lighting example dims environment lighting for a nighttime scene and produces a glowing-hot material.Source: Figure 10 · p. 15, §5.2 · p. 16

6 Conclusion & Discussion

BlenderAlchemy uses vision-language models and visual information to refine Blender programs toward user intentions. The paper demonstrates edits to materials, geometry, and lighting, and points to extending the approach to other workflows.

6 Conclusion & Discussion

  • BlenderAlchemy iteratively refines Blender programs using visual information to explore and prune edits toward user intention.It also uses text-to-image models to guide edits with visual imagination.Source: §6 · p. 16
  • The demonstrated editing tasks include materials, geometry, and lighting.Source: §6 · p. 16
  • The authors hope future work will extend BlenderAlchemy to other workflows.Source: §6 · p. 16

A Explaining System Design Decisions Through Qualitative Examples

Qualitative ablations illustrate how edit types, visual imagination, visual perception, and hypothesis reversion shape material-editing outcomes.

A Explaining System Design Decisions Through Qualitative Examples

  • Without leap edits, the system changes continuous parameters but fails to alter the wood material’s swirl structure; without tweaks, edits change the swirls drastically and then plateau after reversion.The leap-only variant produces darker stripes, while the tweak-disabled variant stays at the pale material from edit 1.Source: A Explaining System Design Decisions… · pp. 22–23
  • Without visual imagination, material verisimilitude plateaus quickly because the edit generator lacks a visual target for adjusting shader-node parameters.Source: A Explaining System Design Decisions… · p. 23
  • Without visual perception for the generator and evaluator, or for the generator alone, edits mainly lighten the wood to gray rather than producing marbled granite.Source: A Explaining System Design Decisions… · p. 23
  • Disabling hypothesis reversion lets a less granite-like candidate replace edit 1, costing two edit cycles before the result returns only to edit 1’s prompt fit.Source: A Explaining System Design Decisions… · p. 23

B Prompting the State Evaluator and the Edit Generator

The appendix describes how the material-editing prompts are adapted for ablations and how lighting prompts are adjusted. Material renders use a fixed 512 × 512 view of a sphere.

B Prompting the State Evaluator and the Edit Generator

  • For visual-imagination ablations, the prompts remove the target image and use the target description instead; perception ablations make the evaluator judge code-only candidates and the generator produce edits from text and source code.Source: B Prompting the State Evaluator and… · pp. 23–24
  • The marbled-granite graph maps “white and gray” to color-ramp colors and “glossy finish” to the principled BSDF specularity input.Source: Figure 13 · p. 24
  • For digital camouflage, the system uses a Voronoi texture node for sharp angles and selects color-ramp colors matching the requested greens, browns, and tans.Source: Figure 14 · p. 24
  • Material images are rendered at 512 × 512 from the same camera, showing one side of a sphere with the material applied.Source: B Prompting the State Evaluator and… · p. 24
  • Lighting-edit prompts follow the material prompts with minor changes that replace material references with lighting-setup references.Source: B Prompting the State Evaluator and… · p. 24

C Analysis of Program Edits

BlenderAlchemy makes substantial procedural-material changes in both shader graphs and scripts, while alternating small tweaks with larger leaps produces distinct edit-size patterns. The observed leap edits were smaller on average in the alternating method than in leap-only edits.

C Analysis of Program Edits

  • Edits to the wood shader graph change node types, connectivity, continuous values, and categorical node settings.Examples include adding a Voronoi node, changing Principled BSDF inputs, changing color-ramp values, and switching noise textures from 4D to 3D.Source: C Analysis of Program Edits · pp. 24–25
  • For digital camouflage, BlenderAlchemy replaces much of the wooden-material script and adds a parameterized Voronoi Texture node.The randomness setting is set to eliminate randomness and create sharper edges.Source: C Analysis of Program Edits · p. 25, Figure 21 · p. 29
  • Alternating tweak and leap edits yields alternating edit-size distributions, with leap edits averaging smaller sizes than leap-only edits.The analysis measures added and deleted characters and lines across four iterations; edits are prompted but not strictly constrained to either type.Source: C Analysis of Program Edits · pp. 25, 30

D BlenderAlchemy Materials In Scenes

The in-scene examples apply synthesized materials to downloaded meshes in product and characterful scenes. The process expands short material descriptions and starts from manually selected procedural materials.

D BlenderAlchemy Materials In Scenes

  • The scene demonstrations begin with concise material descriptions expanded into detailed prompts using GPT-4, following the semantic-upsampling idea in [25].The expanded descriptions specify visual characteristics such as color, pattern, and finish.Source: D BlenderAlchemy Materials In Scenes · pp. 30–31, Figure 23 · p. 31
  • Starter materials are selected manually from previously synthesized BlenderAlchemy materials or Infinigen procedural materials.The selected inputs vary by target, including metallic swirl, acid trip, celestial nebula, chunky rock, and stone materials.Source: D BlenderAlchemy Materials In Scenes · pp. 31–32
  • BlenderAlchemy applies seven synthesized materials to car, shoe, and katana scenes using a 4 × 8 system with visual imagination enabled.The car examples use brushed aluminum, paint splash, and Tron Legacy materials; shoe slabs use ice and surface-of-the-sun materials; katana blades use Damascus steel and rusted metal.Source: D BlenderAlchemy Materials In Scenes · p. 32

E Societal Impact

BlenderAlchemy is intended to help creators make more, not replace their creative intent. Human oversight remains part of the process, with Blender's interface and multiple edit hypotheses offering ways to manage problematic outputs.

E Societal Impact

  • The project aims to enable creators to create more, not replace their intent, by starting from an existing project file and a specification of intent.This contrasts with approaches that attempt to generate everything from scratch [15] [46] [57].Source: E Societal Impact · p. 32
  • Users must carefully review outputs because LLM and VLM biases can carry into BlenderAlchemy's creations.Edits are made directly in Blender, whose existing interface allows problematic edits to be corrected, removed, or rejected; multiple hypotheses also increase the chance of a usable and unproblematic option.Source: E Societal Impact · pp. 32–33

F Limitations

BlenderAlchemy's demonstrated scope is iterative editing, but its use of current VLMs brings cost, speed, and code-reliability constraints. The authors also identify the lack of a skill library and do not demonstrate generation from an empty scene.

F Limitations

  • Each 4 × 8 material example costs just under $3, and GPT-4V is described as expensive and high-latency.Synthesizing many candidates to obtain one usable material may further increase average cost per usable result.Source: F Limitations · p. 33
  • BlenderAlchemy does not develop a skill library, whose tools are expected to be domain-specific.The authors leave such libraries for future work.Source: F Limitations · p. 33
  • The paper demonstrates editing 3D graphics, not iteratively generating a full scene from an empty one.The authors suggest stronger VLMs may be needed for end-to-end generation without humans in the loop, except in simple settings such as [57] and [46].Source: F Limitations · p. 33
  • GPT-4V still hallucinates Blender code, including nonexistent library imports and incorrectly dimensioned field values.The system uses rejection sampling when Blender returns errors, which can increase runtime and cost.Source: F Limitations · p. 33

Related papers

Loading 2404.17672v3…