Source-linked AI summary
FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows
Darshan Deshpande, Yoshinari Fujinuma, Martyna Markiewicz, Devanshu Bansal, Shivani Jain, Nicholas Saban, Chirag Maheshwari, Anand Kannappan
TL;DR
VLMs underperform on subjective design tasks partly because high-quality human workflow data is scarce. FIGMATRACE builds an expert-curated skill taxonomy and phase-segmented dataset of long-horizon Figma work, and training on it improves out-of-domain agentic performance while phase-based conversion beats length-based sharding.
Problem
VLMs lack sufficient high-quality human workflow data capturing the diverse preferences and decisions involved in expert creative design.
Method
FIGMATRACE defines expert-curated design skills and tasks, records expert Figma workflows, and converts them into validated trajectories using design-phase segmentation.
Results
Training on FIGMATRACE improves out-of-domain agent performance, with phase-based conversion exceeding maximum context-length truncation by 7.3 absolute points.
Takeaways & Limitations
Human-captured, intent-segmented design trajectories can improve VLM performance beyond Figma tasks and support learning of broader agentic navigation and design skills.
Takeaways & Limitations
SFT sometimes repeats actions or abandons browser-chrome targets for central content, behaviors attributed to noisy preprocessing and canvas-centric bias.
Abstract
from arXiv · showhide
Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as \textsc{Claude-Opus-5} and \textsc{GPT-5.6-Sol} on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing \textsc{Qwen3.8-27B} outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.
1 INTRODUCTION
VLMs perform well on verifiable tasks but struggle with subjective design, motivating human workflow data that captures expert preferences and decisions. FIGMATRACE addresses this gap with curated skills, long-horizon tasks, and phase-based trajectories, improving performance across agentic benchmarks.
- VLMs handle verifiable domains but struggle to capture human nuances in subjective tasks such as design taste.
- Existing approaches face difficult validation because automated annotations, synthetic grounding, and multimodal judging do not fully capture human design taste.
- FIGMATRACE contains 126 long-horizon tasks across 8 designer workflows, 10 expert-curated skills, and 3469 trajectories.
- The dataset records expert actions and screens, then converts them into ultra-long-horizon trajectories with expert-reviewed design-phase labels.The trajectories span up to 5 million tokens, and labels are extracted and verified at scale.
- Training on FIGMATRACE improves performance across goal-oriented and narrative-oriented benchmarks, with up to 46% absolute improvement on AndroidControl.The reported benchmarks include GUIOdyssey, Mind2Web, and VideoGUI.
2 RELATED WORK
Prior design datasets emphasize finished artifacts or synthetic and unlabeled recordings rather than complete expert trajectories. FIGMATRACE contributes preprocessed recordings paired with gold actions and intent segmentation for long-horizon Figma work.
- Earlier design datasets focus on final artifacts, teaching agents what designs look like rather than how experts create them.
- Screen-recording datasets commonly lack aligned gold actions or rely on synthetic action generation and decomposition.
- Existing preprocessing methods annotate planning or detect atomic actions, but short-trajectory methods provide little structure above individual actions.
- FIGMATRACE pairs preprocessed screen recordings of full expert trajectories with actions for long-horizon sessions requiring creative design skills.
3 FIGMATRACE
FIGMATRACE grounds task construction in an expert-curated design-skill taxonomy and realistic workflows, then cleans recordings and segments them by meaningful design phases. This produces validated trajectories that preserve visible effects, skills, and intent.
- Skill taxonomy: Three experts created a taxonomy of 10 creative skills covering core Figma practices and abilities applicable across design applications.The skills include visual perception, translation, asset creation, debugging, flow design, accessibility, and content reasoning.
- Task coverage: Eight task categories require one or more taxonomy skills and span deterministic verifiable tasks alongside subjective non-verifiable workflows.Examples include pixel-perfect replication, flaw repair, accessibility remediation, platform adaptation, theming, sketch-to-Figma, and prototype wiring.
- Action processing: The pipeline filters irrelevant mouse movements, maps OS actions to Playwright MCP actions, and inserts observe probes when environment state changes without input.Observe probes are inserted every 2 seconds inside gaps longer than 4 seconds.
- Frame extraction: Frame extraction identifies pre-action and settled post-action frames through a low-resolution scan followed by targeted full-resolution decoding.The initial pass decodes videos up to 4.5 hours at 8 frames per second.
- Effect filtering: Effect filtering drops actions with negligible visual change while applying a stricter scene-change threshold to observe probes.Actions below 5 × 10^-4 changed fraction are dropped; observe probes require 2 × 10^-2.
- Phase segmentation: Gemini-3.6-Flash segments recordings into contiguous spans from 11 phase labels, retaining boundaries supported across multiple shardings.Boundaries are kept when at least two shardings place them within ±30 seconds.
4 EXPERIMENTAL SETUP
The experiments fine-tune four VLMs on sampled human-work data using long-context training and evaluate generalization beyond Figma. The setup also compares phase-based trajectories with maximum-length truncation.
- Training setup: Four models are trained with ms-swift on 92,472 randomly sampled actions from 35 sessions totaling an average of 47.6 hours of human work.
- Training setup: The training framework is selected for long-context VLM support through context parallelization.
- Evaluation: Closed-model evaluations use high reasoning effort, while FIGMATRACE models are trained without reasoning because trajectories lack pre-annotated reasoning chains.
5 RESULTS AND DISCUSSION
FIGMATRACE-trained models improve out-of-domain GUI performance, with phase-aware trajectory curation outperforming maximum-length truncation. Qualitative analysis links gains to element selection, coordinate understanding, and decisiveness, while also identifying repetition and screen-center biases.
- RQ1: QWEN3.8-27B outperforms CLAUDE-OPUS-5 by up to 6.4% on GUI-Odyssey and 11.8% on AndroidControl.On ScreenSpot-Pro Creative, performance rises from 29.3% to 36.7%, an absolute increase of 7.4 points.
- RQ2: 7.3 absolute points separate phase-based curation from maximum context length-based truncation across the evaluated GUI benchmarks.The advantage is strongest for AndroidControl cases beginning mid-action; Mind2Web and VideoGUI show smaller effects.
- RQ3: Two-thirds of GUI-Odyssey gains involve selecting the correct UI element, reducing median error from ≈457 px to ≈15 px.Category improvements include Media (+22 pp) and Social (+17 pp).
- RQ3: 10/150 analyzed GUI-Odyssey base-model items emit raw-coordinate errors, compared with none after SFT.The analysis attributes these failures to confusion between pixel coordinates and norm-1000 coordinates, especially in tall frames.
- RQ3: Every AndroidControl gain occurs when the base model provides no coordinates or selects the wrong element, whereas SFT always answers.When correcting element choice, SFT lands at an average of ≈11 px.
- RQ3: SFT sometimes repeats nearly identical coordinates and abandons browser-chrome targets for screen-center content.The paper plausibly links repetition to noisy leaked actions and screen-center focus to a canvas-centric bias; these failures cluster in utility and browser flows.
6 CONCLUSION
The paper introduces a skill taxonomy and FigmaTrace dataset for general-purpose design workflows, then evaluates its effects on in-domain and out-of-domain agent performance. It also compares phase-based trajectory conversion with maximum-length extraction and analyzes model success and failure modes.
- The paper proposes a creative-skill taxonomy and a dataset containing 2883 training and 586 evaluation trajectories.
- Training on FIGMATRACE improves performance on Figma design tasks and out-of-domain agentic tasks.
- Phase-based video-to-trajectory conversion outperforms standard maximum context length extraction.
- Qualitative analysis categorizes model behaviors associated with downstream successes and failures.
A.1 PHASE TAXONOMY
A phase is defined as a contiguous recording segment during which an expert maintains one intent, with phases covering the recording exactly.
- A phase is a contiguous stretch of recording throughout which the expert holds a single intent.Phases tile the recording exactly, so every frame belongs to exactly one phase.
- The phase label vocabulary is closed, and the complete vocabulary is listed in Table 4.Free-form labels can drift between calls, motivating a fixed label set.
A.2 SELECTION OF EXPERTS
Subject matter experts were selected using minimum Figma experience and age requirements, followed by a starter task to vet work quality.
- All contracted experts needed at least two years of Figma experience and had to be at least 18 years old.
- Every expert completed a starter task because the SMEs were hired through Upwork.The starter task was used to vet the quality of their work.
A.3 SKILL DISTRIBUTION OF THE DATASET
FIGMATRACE’s skill-labeled trajectories are dominated by structural craft, with visual perception the second most frequent category. The phase vocabulary is presented as a closed set ordered by typical workflow position.
- Structural craft dominates the skill distribution, followed by visual perception.The paper attributes this pattern to auto-layout and component hygiene being primary requirements in Figma design workflows.
- Figure 4 reports skill shares across all skill-labeled FIGMATRACE trajectories.
- Table 4 presents the closed phase vocabulary ordered by typical workflow position rather than frequency.
A.4 REVIEWERS AND ANNOTATORS
The design-file readiness checklist emphasizes clear navigation, reusable and semantically named components, responsive structure, and realistic, accessible content. It also records when practices are optional, agent-dependent, or intended to support cleanup and orientation.
- File Structure & Agent Navigation: Files should use clear page organization, descriptive frame names, and no loose or orphaned elements.These practices reduce navigation noise and help agents orient through metadata and screen structure.
- Components & Design System: Components should be reusable, semantically named, locally available, and distinguished from one-off layout frames.Repeated UI elements are expected to remain components, while accidental detachments should be resolved.
- Components & Design System: Component properties and descriptions should expose editable content and usage guidance for agent discovery.The checklist notes that descriptions can specify what a component does, when to use it, when not to use it, and searchable keywords.
- Layout & Composition: Relational components should use Auto Layout, deliberate resizing constraints, and higher-order compositions rather than only atomic parts.The checklist connects these requirements to responsive behavior and coherent layout composition.
- Color-Reference to Design System: Design systems should use named color and spacing references, while token variables are most useful when agents operate through MCP.The checklist marks variable-based references as less critical for browser CUA, where values can be read visually.