Source-linked AI summary
Does AI Save Time on Product Design? A Randomized Controlled Experiment of AI Prompt-to-Design Workflows
Remy Stewart, Olabode Anise, Andrew Hogan, Augustus Griffin
TL;DR
Evidence quantifying AI time savings on everyday design tasks remains limited compared with software engineering. This study randomized designers and product managers to Figma Make or manual editing for three standardized tasks, finding 20% improvement in cumulative time savings among task completers.
Problem
Experimental evidence quantifying AI time savings on everyday design tasks is limited, despite designers citing time savings as a reason to adopt AI tools.
Source: §1 · p. 2Method
A randomized between-subjects experiment compared Figma Make with manual Figma editing across three standardized design tasks.
Source: §3.1 · p. 5Results
20% improvement in cumulative time savings was observed among task completers, with larger gains among product managers than product designers.
Source: §1 · p. 2Takeaways & Limitations
Prompt-to-design tools may reduce time and effort for specific UI tasks while lowering participation barriers for non-designers.
Source: §5.1 · p. 13Three standardized tasks were attempted in a fixed order, so the observed time differences may not extend to other task types or orders.
Source: §5.2 · p. 14
Abstract
from arXiv · showhide
AI tools for digital product design now offer prompt-to-design capabilities, allowing designers and their non-designer colleagues to create prototypes through conversational workflows with large language models (LLMs). While these tools promise time savings, experimental evidence in product design remains limited compared with evidence from software engineering. We conducted a randomized controlled trial with 50 product designers and 50 product managers to evaluate prospective time savings from leveraging Figma Make in design work. Participants attempted three standardized design tasks with or without access to Figma Make. Among participants who completed the study tasks, access to Figma Make was associated with approximately 20% shorter completion times, with larger gains among product managers. Our findings suggest that prompt-to-design tools may enable product managers to further contribute to design work, while the benefits for professional designers may be task dependent.
1 Introduction
Although designers cite time savings as a reason to adopt AI, experimental evidence on everyday design-task completion remains limited. This study tests Figma Make in a randomized trial and reports shorter completion times among task completers, with gains varying by role and task complexity.
1 Introduction
- Experimental research has quantified AI time savings mainly in software engineering, leaving everyday design-task savings comparatively underexamined.The study uses an RCT, a method valued in this research for mitigating confounders.Source: §1 · p. 2
- Prompt-to-design tools use iterative conversations with LLMs to create designs and supporting code, potentially lowering expertise barriers for non-designers.Product managers, UX researchers, and software engineers are among the roles participating alongside product designers.Source: §1 · p. 2
- The study tests whether Figma Make reduces time on everyday design tasks and enables non-designers to contribute to design work.Its research questions also ask about role differences and effects on design quality, task ease, and platform usability.Source: §1 · p. 2
- 20% improvement in cumulative time savings was observed among task completers, with larger gains for product managers and no statistically significant aggregate gain for designers.Product-manager savings occurred on less complex tasks, while designer savings emerged only on the most complex tested task.Source: §1 · p. 2
2 Related Work
Prior work finds mixed productivity effects from AI and emphasizes trade-offs and contextual variation. Prompt-to-design tools extend conversational AI workflows into design creation, while also broadening who can participate in design work.
2.1 Measuring AI Tooling’s Impact on Productivity
- AI productivity findings vary by study and experience: reported outcomes include time reductions, productivity regressions, and larger gains for less experienced workers.Other coding studies report faster task completion alongside quality or verification costs.Source: §2.1 · p. 3
- Design-AI research has examined creativity and related outcomes, while scholarship questions net productivity gains and highlights broader dimensions such as craftsmanship and user empowerment.Interviews and surveys describe faster ideation alongside poor output, missing human context, and cognitive overload that can reduce productivity benefits.Source: §2.1 · p. 3
2.2 The Rise of Prompt-to-Design AI Tools
- Prompt-to-design tools use conversational prompts and iterative workflows to create UI/UX designs and supporting code, extending AI from content generation and chatbots toward deployable design outputs.The paper uses a broader conceptualization than “vibe design,” which prior work framed around design and user-testing agentic support.Source: §2.2 · p. 4
- Prompt-to-design tools may reduce the expertise needed for design and enable non-designers to move from observers to collaborators.Prior work describes benefits including collapsed design stages, reduced cognitive burden, and faster movement from ideas to prototypes [27].Source: §2.2 · p. 4
- This study evaluates Figma Make’s effects on task-completion time for product designers and product managers in a laboratory randomized experiment.Figma Make generates interactive prototypes and underlying code, and is targeted beyond designers, including product managers.Source: §2.2 · p. 4
3 Methodology
The study randomized product designers and product managers to Figma Make or manual Figma editing for three standardized, reference-based UI tasks. Moderators verified task completion and recorded times, while analyses modeled repeated task observations and adjusted for participant characteristics.
3.1 Experiment Design
- A between-subjects randomized experiment compared Figma Make with manual Figma editing, using the same three tasks and post-task survey measures in both conditions.Participants were randomly assigned after eligibility screening; treatment and control received identical starting designs and reference images.Source: §3.1 · p. 5
3.1.1 Recruitment.
- The sample comprised 50 product designers and 50 product managers, with 25 from each role assigned to each condition.Eligibility required at least two years of relevant experience, monthly Figma use, and work involving a digital product.Source: §3.1.1 · p. 5
3.1.2 Experiment Flow.
- Participants completed three fixed-order social-media UI tasks: dark-mode styling, adding a menu option, and creating an interactive comment flyout.Both conditions used the same starting frames and static target images; treatment participants could prompt Figma Make and make manual edits afterward.Source: §3.1.2 · pp. 5–6
3.1.3 Moderation & Data Collection.
- Moderators verified designs against reference images, while a Figma widget timed each task and timestamped logs enabled telemetry cross-checks.Participants also rated task ease and completed an exit survey about usability, perceived design quality, and AI experience.Source: §3.1.3 · pp. 6–7
3.2 Data Analysis
- GEE models estimated task-level and cumulative completion outcomes while accounting for repeated observations within participants.Models adjusted for demographics, design background, and AI experience; completion times were log-transformed and results reported with cluster-robust standard errors and 95% confidence intervals.Source: §3.2 · p. 7
4 Results
Across the tested tasks, Figma Make was associated with shorter completion times, with larger time savings among product managers. Treatment participants also reported greater task ease and usability, while their design-quality ratings differed from controls.
4.1 Descriptive Statistics
- Task 2 averaged 10 minutes and Task 3 averaged 26; designers used Figma and its AI features more often than product managers.Task completion exceeded 90% for Tasks 1 and 2 but was 73% for Task 3.Source: §4.2 · p. 8, §4.1 · p. 8
4.2 RQ1 - Time Savings from Figma Make
- Task 3 completion probability was 19.9 percentage points higher with Figma Make (SE=6.8%, p=0.004), while Tasks 1 and 2 and the three-task average showed no significant differences.Source: Table 2 · p. 8, §4.2 · p. 9
- An equally weighted task-time comparison found a 20% reduction with Figma Make (AME=10:53, SE=3:25, p=0.0014).Cumulatively, the reported reduction was 27% (AME=15:08, SE=3.18, p<0.0001).Source: §4.2 · p. 9, Table 3 · p. 10
4.3 RQ2 - Designers versus Non-Designers Time Savings
- Product managers saved 22 minutes 46 seconds cumulatively (35%), while designers showed a significant 7-minute-41-second reduction only on Task 3 (26%).Source: §4.3 · p. 10, Table 3 · p. 10
- With treatment-by-role interaction, product managers’ completion time fell 33% versus product managers without Make; the cross-role designer comparison was not significant.The estimated reduction was 21 minutes 58 seconds for product managers (SE=5:47, p=0.0001); the 8% reduction for treatment designers versus control managers was not statistically significant.Source: §4.3 · p. 11
4.4 RQ3 - Design Quality, Task Ease, and System Usability
- Across participants, Make increased task-ease scores by 0.8 points on the 1–7 scale and SUS scores by 10 points on the 0–100 scale.Product managers drove the gains: task ease improved 37% and SUS 32%; neither measure changed significantly for designers.Source: §4.4 · p. 12, Table 5 · p. 12
- Treatment versus control differed significantly in perceived instruction match (χ2(3)=8.05, p=0.045, V=0.28) and design quality (χ2(4)=15.46, p=0.004, V=0.39); treatment more often reported “somewhat matched” (66% vs 46%) and “much better” quality (20% vs 2%).With ordinal Mann–Whitney testing, instruction-match results were no longer significant overall but remained significant for product managers; quality-result trends remained the same.Source: §4.4 · pp. 12–13
5 Discussion
Figma Make was associated with shorter completion times overall, but gains differed by role and were clearest for product managers; the discussion cautions that the findings are bounded by the study’s tasks and participant sample. The authors identify future work on broader tasks, longitudinal use, expert quality assessment, and other tools and populations.
5.1 Synthesis of Results
- Estimated completion times for completed tasks were approximately 20% lower with Figma Make than control when averaging across three tasks.The authors note that whether the benefits extend to broader design work remains uncertain.Source: §5.1 · p. 13
- Product managers using Figma Make had shorter completion times and higher task-ease and perceived-usability scores, and rated their designs more favorably against their expected outside-study work.The authors suggest the tool may help product managers translate design intentions into interface changes.Source: §5.1 · p. 13
- Product designers showed no statistically significant aggregate time reduction, although treatment participants completed the more complex, time-intensive Task 3 faster.The authors suggest tools may benefit involved, time-intensive tasks in established workflows even without overall time savings.Source: §5.1 · p. 13
- Prompt-to-design tools may reduce time and effort for specific UI tasks while supporting established workflows and lowering participation barriers for non-designers.Source: §5.1 · p. 13
5.2 Limitations
- Timing analyses included only completed tasks, and completers could differ between conditions, particularly for product managers on Task 3.The authors advise interpreting time reductions alongside completion results.Source: §5.2 · pp. 13–14
- Three fixed-order tasks with specified target designs may not represent professional projects’ ambiguity, stakeholder feedback, or changing requirements.Observed completion differences may not extend to other tasks or task orders.Source: §5.2 · p. 14
- Moderator variability may affect verification; rendering failures and extra prompts may lengthen treatment times, while the experienced Figma-user sample limits generalizability.The study’s results may not extend to newer Figma users, other design roles, or other prompt-to-design systems; product managers were the only non-design role included.Source: §5.2 · p. 14
5.3 Future Directions
- Future work should test broader and open-ended tasks, track whether gains persist or are offset by later review and iteration, and use expert quality rubrics across tools and populations.The authors also call for longitudinal field studies and assessments of visual quality, accessibility, and design-system or brand alignment.Source: §5.3 · p. 14
A.2 Task 2 - "Add a “Help & Support” option to the settings drop down menu."
The supplied passage identifies a Task 3 comment-flyout instruction.
A.2 Task 2 - "Add a “Help & Support” option to the settings drop down menu."
- Task 3 begins with an instruction to create a comment flyout from the homepage.Source: §A.2 · p. 19
B.1 System Usability Questions
The appendix lists survey questions about Figma usability, self-assessed design quality, treatment-only perceptions, and AI-use frequency.
B.1 System Usability Questions
- The System Usability questions cover frequency of use, complexity, ease, support needs, integration, consistency, learnability, cumbersomeness, confidence, and required learning.Source: §B.1 · p. 19
B.2 Design Quality Questions
- Design-quality questions ask how closely designs matched instructions, how their quality compared with outside-study work, and whether participants had used Figma Prototyping.Source: §B.2 · p. 19
B.3 Treatment-Only Questions
- Treatment-only questions ask whether participants trusted the AI to interpret their intentions and believed Figma AI would increase their design-process contribution.Source: §B.3 · pp. 19–20
B.4 AI Use Questions
- The AI-use questions ask how often participants use AI tools for work and for design.Source: §B.4 · p. 20
C Participant Attribute Descriptive Statistics
Table 7 presents participant demographics and background characteristics.
C Participant Attribute Descriptive Statistics
- Table 7 presents participant demographic and background-characteristic distributions across the table and its continuation pages.Source: Table 7 · pp. 20–22
D Regression model covariate terms
Table 8 presents GEE covariate coefficients for completion and cumulative time. It includes participant characteristics and prior use of Figma and other AI tools, with reference categories identified for several covariates.
D Regression model covariate terms
- Table 8 reports GEE covariate coefficients for completion and cumulative time.Source: Table 8 · p. 22
- Listed participant-characteristic reference categories include undergraduate education, freelance/contractor status, 2–5 years of experience, banking and financial services, and organizations with 1–1,000 employees.Figma design-use frequency has “Few times a year or less” as its reference category.Source: D Regression model covariate terms · p. 22
- AI-use covariates include prior Figma AI use and the frequency of using AI tools for work and design.The stated reference category for each frequency measure is “Few times a week.”Source: D Regression model covariate terms · p. 23