Source-linked AI summary
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Vivian Liu, Jo Vermeulen, George Fitzmaurice, Justin Matejka
TL;DR
The paper addresses the limited understanding of how text-to-image AI can support conceptual CAD and 3D design workflows. It introduces 3DALL-E, which integrates DALL-E, GPT-3, and CLIP into CAD to generate image inspiration from text and workspace images, and evaluates it with 13 designers. Participants reported uses including reference images, combating design fixation, and design inspiration, while their prompts revealed workflow patterns and typically two-to-six concepts; the authors also identify scope and privacy limitations.
Problem
How text-to-image AI can contribute to CAD and product-design workflows and benefit 3D designers’ creative processes remains underexplored.
Method
The paper integrates DALL-E, GPT-3, and CLIP into Fusion 360 through 3DALL-E, a plugin that helps designers construct text and image prompts from goals and work in progress.
Results
An exploratory study with 13 designers found high enthusiasm, uses for reference images and inspiration, prompting patterns across workflows, and prompts commonly ranging from two to six concepts.
Takeaways & Limitations
3DALL-E can augment conceptual CAD by helping designers combat design fixation, explore references, and connect their work with text-to-image generation.
Takeaways & Limitations
3DALL-E is object-oriented for CAD product designs, experienced technical difficulties, could not be directly compared with generative design, and raises data-privacy concerns.
Abstract
from arXiv · showhide
Text-to-image AI are capable of generating novel images for inspiration, but their applications for 3D design workflows and how designers can build 3D models using AI-provided inspiration have not yet been explored. To investigate this, we integrated DALL-E, GPT-3, and CLIP within a CAD software in 3DALL-E, a plugin that generates 2D image inspiration for 3D design. 3DALL-E allows users to construct text and image prompts based on what they are modeling. In a study with 13 designers, we found that designers saw great potential in 3DALL-E within their workflows and could use text-to-image AI to produce reference images, prevent design fixation, and inspire design considerations. We elaborate on prompting patterns observed across 3D modeling tasks and provide measures of prompt complexity observed across participants. From our findings, we discuss how 3DALL-E can merge with existing generative design workflows and propose prompt bibliographies as a form of human-AI design history.
1 INTRODUCTION
3DALL-E addresses the underexplored use of text-to-image AI for conceptual CAD by connecting designers’ goals and work-in-progress models to image inspiration. The paper presents the system and an exploratory study of its use in 3D design workflows.
- Motivation: Conceptual CAD requires balancing functional, aesthetic, and feasibility objectives while exploring external resources for inspiration.Designers commonly consult 3D repositories, tutorials, and image searches, but few computational methods support conceptualization.
- Research questions: The paper investigates how text-to-image AI can assist 3D designers with conceptual CAD and where it may benefit creative workflows.
- System: 3DALL-E integrates DALL-E, GPT-3, and CLIP within Fusion 360 to translate design goals and viewport context into multimodal prompts.GPT-3 supplies design language and 3D keywords, while CLIP highlights suggestions that match the current viewport.
- Contributions: The paper contributes a CAD plugin, an exploratory user study with 13 designers, analyses of prompting patterns and complexity, and prompt bibliographies for human-AI design history.
2 RELATED WORK
Related work frames prompting as natural-language interaction with generative models and situates 3DALL-E among creativity-support, CAD, and text-to-image or text-to-3D systems. The paper distinguishes its systematic connection of CAD progress to image generation.
- Prompting: Prompting lets users provide natural-language context to LLMs and query world knowledge, completions, summaries, or translations.
- Prompting: Online generative-AI communities disseminate prompting guidance using style and medium keywords such as “isometric” and “3D render”.
- Prompt-based workflows: Prior systems explored prompt decomposition, visual prompt chains, accessible ML prototyping, and pipelines combining language models with text-to-image models.
- Novelty: 3DALL-E is presented as the first system to systematically generate image prompts from CAD software and incorporate 3D design progress into text-to-image generation.
- Generative models: Text-to-3D methods remain constrained by inference time, computing requirements, diversity, fidelity, stylistic range, and variable binding.
- Creativity support: Creativity-support research has used chat, affective part matching, sentence-based scene creation, design galleries, timelines, natural-language exploration, and collaboration support.
- CAD workflows: Conceptual CAD commonly uses 2D representations and external inspiration because early-stage designs are ambiguous, incomplete, expressive, and uncertain.
- CAD workflows: Existing CAD workflow research characterizes multiple paths to the same 3D model, while related generative-design systems provide cognitive support for novel solutions.
3 DESIGNING WITH 3DALL-E
3DALL-E helps CAD users turn design intentions and workspace context into editable prompts, then explore generated references and variations. Its interface organizes suggestions around geometry-relevant design concepts.
- Constructing prompts: Users begin by entering a design goal, after which GPT-3 supplies suggestions for 3D keywords, designs or styles, and parts.Suggestions provide relevant design language, such as egg, Eames, or Muskoka chair designs.
- Constructing prompts: Suggestions are color-coded by category and vary in opacity according to their CLIP-based alignment with the image prompt.Designs, styles, and parts use blue, green, and orange respectively; darker highlights indicate stronger matching.
- Design scope: The plugin focuses on geometry-relevant suggestions—parts, designs, and styles—rather than material or abstract functional goals.
- Image prompts: Users can include an automatically extracted viewport image with the text prompt or choose a text-only prompt.The image prompt becomes active when users select the image preview, and viewport rendering provides a clean prompt without manual erasing or masking.
- Exploring results: After prompt editing, users send the prompt to DALL-E, retrieve results, star favorites, request variations, download images, and review generation history.
4 SYSTEM IMPLEMENTATION
The Fusion 360 plugin combines DALL-E, GPT-3, and CLIP in one interface, using logged interactions and automated viewport images to support prompt construction and later analysis.
- Implementation: 3DALL-E was implemented as a Fusion 360 plugin using the Fusion 360 API, Python, Javascript, Selenium, and Flask.The authors state that its basic required functionality could be implemented in most 3D modeling tools.
- Prompt generation: GPT-3 generates design, style, and part suggestions through structured queries and rephrases selected suggestions into a combined prompt.
- Prompt generation: Ten 3D keywords are sampled from 121 high-frequency CAD-specific terms extracted from Fusion 360 Screencast transcripts.
- Prompt alignment: CLIP converts text-image similarity logits into normalized probabilities rendered as suggestion opacity highlights.
- Viewport capture: The system automatically saves a transparent Fusion 360 viewport PNG every 0.3 seconds for image-prompt construction.
5 EVALUATION
In an exploratory study of 13 designers, 3DALL-E supported positive experiences across editing and from-scratch modeling, while revealing distinct prompting behaviors and some control challenges.
- Evaluation: 13 designers evaluated generation patterns, prompt construction, and prompt complexity across editing and from-scratch modeling tasks.The study examined how designers used text-to-image generations within different workflow stages.
- Evaluation: 12/13 participants rated editing enjoyment positively, compared with 11/13 for from-scratch creation; both tasks had a median enjoyment rating of 6.Participants also generally found at least one design satisfying their goal.
- Evaluation: 10/13 participants in each task felt 3DALL-E would help their usual workflow, with median usefulness ratings of 5 for editing and 7 for creation.
- Prompt construction: Participants rated GPT-3 suggestions positively, while CLIP highlighting received positive ratings from 8/13 editing participants and 7/13 creation participants.Participants tended to click suggestions with stronger text-image alignment.
- Image prompting: Image prompts were incorporated positively by 10/13 editing participants and 8/13 creation participants, helping tailor generations toward ongoing designs.Participants also used generations as references and shifted between AI-first and AI-last workflows.
- Prompt construction: 3DALL-E supplied 63.61% of all prompt keywords, providing at least half of the keywords for 9/13 participants in each task.Participants therefore heavily used the GPT-3 suggestion function.
6 PROMPT COMPLEXITY
The study measured prompt complexity by counting concepts in text-only and image-plus-text prompts. Participants commonly explored two to six concepts but could readily assemble prompts exceeding six concepts.
- Measurement: Prompt complexity was measured as the number of concepts forming a prompt, using GPT-3-rephrased suggestions or linguistic rules for participant-written prompts.Noun phrases and verbs were counted while prepositions, function words, stop words, and count words were excluded.
- Observed complexity: Most prompt activity concentrated between two and six concepts, although participants explored a wide range of concepts.The analysis covered text-only and image+text prompts, excluding variation images.
7.1 3DALL-E Use Cases for CAD Design
Participants used 3DALL-E primarily for inspiration, including reference images, appearance exploration, presentation concepts, collaboration, and design considerations. The generated images supported exploration even when they were not technically accurate or guaranteed feasible.
- Inspiration: Inspiration was the most commonly acknowledged use case, particularly during early design stages, where participants used 3DALL-E to generate more design options.Participants connected this use to overcoming design fixation and time constraints.
- Reference images: Participants imported generations as reference images and specified top, side, or front views to support three-dimensional modeling.Most generations were angled or perspective-based unless viewpoints such as “top view” or “flat” were explicitly prompted.
- Reference images: Even when generations were not technically accurate or clean enough for technical drawings, participants still found them useful for modeling references.Participants also used images more loosely to adjust proportions, remove material, or guide sustainable design decisions.
- Appearance exploration: Participants adapted model appearances toward generated images by applying software materials, textures, or generations directly onto simple geometries.One participant used generated images as textures in an outdoor movie theater scene.
- Collaboration and presentation: 3DALL-E supported product presentations and collaboration by visually situating designs and providing shared materials between mechanical engineers and industrial designers.Participants also described diverse prompting trajectories as useful for team brainstorming and exploration.
- Design considerations: Generations inspired participants to consider functionality, manufacturability, uniqueness, design conventions, and plausibility, although feasibility, educational value, and patentability were not guaranteed.Examples included reasoning about robot motor power, manufacturing details, and whether an unusual product might be novel.
- Limitations: Participants identified weaknesses for machining, simulation, internal components, and function-focused parts that are difficult to describe or enclose visually.One participant specifically noted difficulty generating housing around an internal part.
7.2 Comparing with Traditional Workflows
3DALL-E introduced prompting into workflows that participants usually described as visual and geometry-centered. Participants used it for problem solving and dynamic feedback, but also raised concerns about design agency, ownership, and creativity.
- Workflow differences: Participants were impressed that 3DALL-E generated images from prompts containing technical jargon, but prompting remained distinct from their usual multi-phase, low-fidelity-to-high-fidelity workflows.The exploratory study tested the plugin across disciplines and design settings.
- Workflow differences: 3D design workflows were primarily visual and based on direct geometry manipulation, leading some participants to prefer image-based prompting for generating variants.Other participants described text in CAD as parameters, properties, equations, and configurations.
- Problem solving: Participants used 3DALL-E as a problem solver and experimented with image prompts that connected separate geometries or filled gaps while modeling.One participant compared this use to an automated modeling command.
- Driving the design: Some participants felt that generated suggestions could drive their design process, raising concerns about who controls the design trajectory.One participant described eventually depending on the system rather than continuing to imagine independently.
- Ownership: Participants generally still considered designs their own, relating AI-assisted modeling to borrowing and assembling premade online files while retaining control over placement.The ownership discussion remained grounded in participant perceptions rather than a legal determination.
- Human expertise and creativity: Participants emphasized that expert understanding of manufacturing, markets, and style remained necessary, while widespread AI inspiration could risk convergence on similar designs and reduced creativity.One participant characterized AI as a source of inspiration rather than the solution.
7.3 Comparison with Existing Generative CAD Tools
Participants contrasted 3DALL-E’s creative, text-based image generation with generative design’s constraint-driven geometry completion. They viewed the tools as potentially complementary rather than directly comparable.
- Study context: Five of 13 participants had experience with Fusion 360’s generative design mode, which completes designs from physical constraints and geometric filters.The study did not directly compare the systems because hardware constraints made 3DALL-E incompatible with generative design.
- Interaction and goals: Generative design has a higher entry barrier because users must calculate loads and resolve constraints before generating results.This burden distinguishes it from the more exploratory interaction participants associated with 3DALL-E.
- Speed and integration: During a 30-minute task, participants could browse hundreds of 3DALL-E results with initial results arriving within seconds, contrasting with generative-design waits of hours or days.Participants suggested that 3DALL-E could generate shapes before generative design optimized them.
8 DISCUSSION
The discussion finds strong promise for text-to-image assistance in 3D workflows, especially for early conceptual design, while identifying provenance, privacy, scope, and integration challenges.
- Participants used 3DALL-E to obtain varied reference images, combat design fixation, and inspire design considerations in conceptual CAD.
- 8.1 Prompt Bibliographies: Prompt bibliographies could record prompts, inspirations, and designer intentions within existing software histories, helping clarify human and AI contributions.
- Language-model scaffolding can help designers discover wording that text-to-image systems are likely to understand.
- Introducing 3DALL-E earlier in the workflow produced a slightly more positive experience, although relevance filtering remains important.
- 8.5 Future Work and Limitations: The workflow generalizes to other design software through application-specific vocabulary and image prompts, but current implementation targets CAD product design rather than 3D art broadly.
- 8.4 Benefits of Text-to-Image for CAD: 3DALL-E provides visual assets and design knowledge throughout conceptual CAD, with potential to help generative design align shapes with designers’ visual goals.
- 8.5 Future Work and Limitations: Future work should connect text-to-image assistance with sketch-based modeling, whose clean primitives could provide control over generated-image composition.
- 8.5 Future Work and Limitations: Data privacy is a key future concern because company design know-how and intellectual property may be passed to AI systems.
9 CONCLUSION
The conclusion reports an exploratory evaluation of 3DALL-E in 3D workflows and summarizes its use cases, prompting analyses, and proposed prompt bibliographies. It frames the system as a way to augment conceptual CAD processes with text-to-image assistance.
- 3DALL-E was evaluated in an exploratory study with 13 designers and supported reference images, collaboration, and design considerations.
- Prompting patterns varied according to whether participants used 3DALL-E first, last, or throughout their design process.
- The paper measures prompt complexity across participants and proposes prompt bibliographies for tracking human-AI design history.