Teaching an Agent to Sketch One Part at a Time

Motivation

• Existing methods largely generate full sketches at once, overlooking progressive sketching and limiting localized editing, interpretability, and fine-grained creative control.Meaningful parts can be independently removed, replaced, modified, or explored before proceeding to later parts.Source: §1 · p. 2
• Unlike prior work, including SketchAgent, the proposed method targets free-text guided part-by-part generation of detailed vector sketches within a unified, human-friendly workflow.SketchAgent uses a closed-source VLM backend and produces simplistic icon-style outputs.Source: §1 · p. 2

Data

• The generic annotation pipeline uses proposal, critique, and revision stages to produce scalable part decompositions and path-to-part assignments for vector sketches.It yields an overall caption and semantic part descriptions, and can be applied to any vector sketch data.Source: §1 · pp. 2–3
• Applying the pipeline to ControlSketch produces ControlSketch-Part, a high-quality part-annotated dataset released with a benchmark for multi-turn text-to-vector sketch generation.The dataset addresses the cost and scalability challenges of collecting professional part annotations.Source: §1 · pp. 2–3

Training

• The training framework combines two-stage supervised fine-tuning with multi-turn process-reward GRPO reinforcement learning using intermediate-state rewards.SFT teaches output format and initializes a single-turn sketching policy, while RL aligns multi-turn rollouts.Source: §1 · pp. 2–3

Results

• Automated metrics, user studies, and qualitative evaluations show significant improvements over prior work while enabling multi-turn vector sketch generation and progressive text-guided editing.These capabilities support interactive sketching with branching possibilities and creative exploration.Source: §1 · pp. 2–3

2.1 Text-to-Vector Sketch Synthesis

• Learning-based methods generate strokes autoregressively or with diffusion, while test-time optimization improves flexibility and visual quality but jointly optimizes strokes without meaningful ordering or semantic part structure.Learning-based methods include Sketch-RNN, BézierSketch, SketchODE, ChiroDiff, and StrokeFusion [42]; optimization-based methods include CLIPDraw, DiffSketcher, AutoSketch, and SketchDreamer [25].Source: §2.1 · p. 3
• SketchAgent performs text-guided sequential sketching with a closed-source Claude Sonnet model, but its zero-shot design produces doodle-style outputs with limited fidelity, domain adaptability, and spatial grounding.Its zero-shot nature constrains output quality and adaptation to specific domains.Source: §2.1 · pp. 3–4

2.2 Reinforcement Learning for Large Language Models

• Reinforcement learning frames autoregressive and agentic systems as sequential decision processes, with GRPO demonstrating efficient performance across reasoning tasks [30].This perspective addresses settings where errors can accumulate over multiple steps.Source: §2.2 · p. 4
• Reason-SVG and Rendering-Aware Reinforcement Learning [27] use GRPO with visual feedback, but neither uses intermediate generation states to provide dense credit assignment.Reason-SVG combines programmatic correctness and visual similarity after SFT, while Rendering-Aware Reinforcement Learning [27] uses rendered-SVG similarity as a visual reward.Source: §2.2 · p. 4

3 Automated Part Annotation

• The annotation pipeline enriches each vector sketch with an overall caption, semantic descriptions of its parts, and an assignment of every path to a part.These annotations progressively derive semantic structure from the raw SVG input.Source: §3 · p. 5, §3.1 · p. 5

3.1 Data Collection Pipeline

• The seven-step pipeline decomposes sketches into parts, critiques and refines them, assigns paths under schema constraints, critiques assignments using diagnostic visualizations, refines them, and generates a consistent caption.The assignment requires each path to belong to exactly one labeled part, while every part contains at least one path.Source: §3.1 · pp. 5–6

3.2 Our Dataset: ControlSketch-Part

• The procedure is designed for SVG or vector-convertible sketch datasets and is applied to the complex, realistic-looking ControlSketch dataset.ControlSketch contains 35,000 image-sketch pairs across 15 object categories, whose labels are not used during training.Source: §3.2 · p. 6
• Using Gemini 3.0 Pro and a two-to-five-part schema, the procedure yields ControlSketch-Part with captions, part descriptions, and path-to-part assignments.Examples of the resulting annotations include color-coded part descriptions and stroke groups.Source: §3.2 · p. 6, Figure 3 · p. 6

4 Method

• The agent iteratively draws one sketch part at a time, conditioning on the current canvas, captions, prior part descriptions and paths, and remaining-part count.Outputs contain cubic Bézier paths represented by eight coordinates and SVG commands M and C, separated by newlines.Source: §4 · pp. 6–7

4.1 Stage 1: Supervised Fine-Tuning

• SFT trains valid paths and single-part extensions using up to 20 random part permutations per sketch, exposing the agent to arbitrary turn orderings and oracle partial sketches.The goal is to prepare the model for autoregressive multi-turn generation.Source: §4.1 · pp. 7–8

4.2 Stage 2: RL with Multi-turn Process-Reward GRPO

• GRPO [30] bridges the gap between oracle intermediate states seen during SFT and the agent-generated states encountered during inference, improving the deteriorating later-step regime.Its group-mean baseline avoids an additional value model and is more efficient than predecessors such as [22] [29].Source: §4.2 · p. 8
• The multi-turn GRPO objective applies stepwise rewards across equal-length trajectories and optimizes token-level clipped policy updates with KL regularization to a reference model.Responses are passed through a validity verifier before reward computation, and the learning objective builds on DeepSeekMath [30].Source: §4.2 · pp. 8–10
• Process rewards combine DreamSim visual similarity with a path-count reward, while regularizing only the final path count because per-part signals are empirically too noisy.DreamSim compares each rendered intermediate canvas with the corresponding ground-truth rendering using CairoSVG and a pretrained DreamSim model; the path-count reward addresses GRPO’s bias toward longer trajectories identified by Liu et al. [21].Source: §4.2 · p. 9

5.1 Experimental Setup

• Training separates SFT and RL data, reserving the high-quality ControlSketch-Part dataset for RL while using a cheaper automatically annotated dataset for SFT.This follows established two-stage training practice intended to reduce imitation bias and preserve exploration during RL.Source: §5.1 · pp. 10–11
• The method is benchmarked against SketchAgent, Gemini 3.1 Pro, and SDXL + SwiftSketch using Long-CLIP faithfulness scores and double-blind pairwise preference studies.Long-CLIP compares sketch embeddings with embeddings of concatenated part descriptions, while users evaluate whole-sketch quality and, against SketchAgent, part-by-part generation quality.Source: §5.1 · p. 11

5.2 Experiment Results and Analysis

• The full SFT + RL model achieves the best performance across all methods, while user studies consistently favor it across step-quality and final-output comparisons.The study collected 3,092 pairwise comparisons, and participants preferred the proposed method across both evaluation settings and all baselines.Source: §5.2 · p. 12
• Qualitatively, the proposed sketches contain smooth paths, natural styles, and identifiable parts, whereas baselines exhibit simpler geometries, missing or distorted components, or errors inherited from image generation.SketchAgent and Gemini 3.1 Pro favor geometric and symmetric layouts, while SDXL + SwiftSketch can lose details or compositional relationships because of text-to-image limitations.Source: §5.2 · pp. 12–13

5.3 Ablation Study

• The ablation reaches 0.298 Long-CLIP score with multi-turn process-reward GRPO, versus 0.286 for multi-turn outcome-reward and 0.281 for single-turn RL.Intermediate-state rewards provide dense credit assignment, and the ablation indicates complementary benefits from multi-turn training and process-level rewards.Source: §5.2 · p. 14, §5.3 · p. 14, Table 1

6 Conclusion

• The resulting agent generates structured sketches one part at a time, outperforms prior methods across automatic metrics and user studies, and supports high-quality localized editing.The framework uses SFT for output-format grounding and policy initialization, followed by multi-turn process-reward GRPO with intermediate visual rewards.Source: §6 · pp. 14–15
• The final training framework closes the distribution gap between oracle training states and free-form inference by optimizing visual quality through intermediate visual rewards.This process-reward stage follows supervised fine-tuning and enables one-part-at-a-time generation.Source: §6 · pp. 14–15

Appendix

• The appendix provides additional training pseudocode, annotation prompts, sketching results, dataset examples, and discussion of failure cases, limitations, and future work.These materials appear in Sections A–E.Source: Appendix · p. 19

A Pseudo Code

• Algorithm 1 initializes the policy and repeatedly samples trajectory batches before updating the policy with group-relative advantage estimation and the GRPO objective.The pseudocode includes reference-policy and old-policy updates within nested iterations.Source: A Pseudo Code · p. 19

B Prompt Templates for the Data Collection Pipeline

• The automatic annotation pipeline uses seven prompt templates with placeholders for renderings, diagnostic visualizations, SVG text, prior decompositions, assignments, critiques, and step instructions.These templates support structured multimodal annotation inputs.Source: B Prompt Templates for the Data… · pp. 19–20

Step 1: Initial Part Decomposition; Step 2: Part Critique; Step 3: Part Refinement

• The decomposition stages produce collectively exhaustive, non-overlapping semantic parts, audit requirement compliance, and revise errors while constraining the number and format of returned parts.The prompts require concrete object-part descriptions, prohibit vague or drawing-level labels, and allow unchanged outputs when current parts are satisfactory.Source: Step 1: Initial Part Decomposition · pp. 20–21

Step 4: Initial Path Assignment; Step 5: Path Assignment Critique with Diagnostic Visualization; Step 6: Path Assignment Refinement

• The path-assignment stages assign every SVG path to exactly one provided part, require every part to be used, and revise assignments using critiques and diagnostic visualizations.Audits check prompt compliance, semantic correspondence, missing paths, and incorrect assignments before minimal sufficient edits.Source: Step 4: Initial Path Assignment · p. 23, Step 5: Path Assignment Critique with… · pp. 23–24

Step 7: Caption Generation

• Caption generation produces a strictly objective, literal description of visible objects using candidate parts while excluding artistic attributes, speculation, and unintended marks.Captions include only essential iconic information and may describe clearly visible orientation, posture, or motion.Source: Step 7: Caption Generation · pp. 25–26

Additional part-by-part results

• Additional figures A1–A6 show the model’s part-by-part results through cumulative frames, with part descriptions and captions above sketches and newly added parts color-coded to corresponding labels.Source: Figure A1 · p. 27, Figure A2 · p. 28, Figure A3 · p. 29

D Additional ControlSketch-Part Dataset Examples

• Additional ControlSketch-Part dataset examples are provided in Tables A1–A3.Table A1 introduces the additional examples, while Tables A2 and A3 continue the examples.Source: Table A1 · p. 42, Table A2 · p. 43, Table A3 · p. 44

E.1 Failure cases

• Fixed path-count rewards can cause premature stopping once the dataset’s ground-truth count is reached, even when a part remains incomplete.This failure arises because all ControlSketch sketches contain a fixed number of paths.Source: §E.1 · p. 45
• The premature-stopping behavior can omit an entire component, exemplified by the missing right wheel in Fig. A16a.The figure presents this as a concrete failure case.Source: §E.1 · p. 45, Figure A16 · p. 45
• The agent can produce erroneous topologies for unfamiliar semantic concepts that are rare in the training data.This failure concerns structural depiction rather than merely spatial placement.Source: §E.1 · p. 45
• A vertically oriented oval rear wheel is incorrectly depicted, illustrating the difficulty of representing rare semantic structures.This example appears in Fig. A16b.Source: §E.1 · p. 45, Figure A16 · p. 45
• Although RL training substantially reduces part-misalignment errors, occasional misplacements remain.Residual spatial errors persist despite the improvement from reinforcement learning.Source: §E.1 · p. 45
• In one example, the jacket is placed too far right, creating an unnatural gap between the jacket and upper legs.This residual misalignment is shown in Fig. A16c.Source: §E.1 · p. 45, Figure A16 · p. 45

Object-level generalization

• Generalization is limited by the ControlSketch dataset’s object-category coverage, although relatively general OOD part descriptions occasionally produce reasonable approximations.The authors suggest substantially larger and more diverse training data could improve generalization.Source: §E.2 · pp. 45–46
• OOD object inputs expose two failures: meaningless sketches for most unseen categories and nearest-neighbor sketches resembling seen classes instead of intended objects.Examples include outputs resembling bear, horse, fish, car, and crab for intended tiger, camel, shark, bus, and spider classes.Source: §E.2 · p. 46, Figure A17 · p. 46

Out-of-domain parts

• The agent usually fails to accurately add out-of-domain parts, including pincers on an angel, walking legs on a fish, and feathered wings on a chair.Fig. A18 compares each original incomplete sketch with the attempted result after adding the specified unrelated part.Source: §E.2 · p. 46, Figure A18 · p. 47

Coordinate precision

• Imprecise numerical-coordinate prediction, compounded by temperature-based sampling, can cause drastically degraded stroke placement, though such low-quality outputs are very infrequent edge cases.Fig. A19 shows the lowest-quality validation outputs observed across in-distribution categories.Source: §E.2 · pp. 46–47

• A planning agent could coordinate multiple agents to generate different sketch parts in parallel, while refining unsatisfactory intermediate outputs could improve overall sketch quality.These directions target both generation efficiency and correction of poor intermediate results.Source: §E.3 · p. 47
• Richer natural-language reasoning, such as chain-of-thought before generating each part, could strengthen the generation process.Source: §E.3 · p. 47
• The agent’s sketching capabilities could extend to visual reasoning tasks by generating auxiliary figures for applications such as geometry problems.Source: §E.3 · p. 47
• Self-evaluation could assess intermediate outputs and selectively regenerate unsatisfactory parts, mitigating error accumulation and improving robustness over longer generation sequences.Source: §E.3 · p. 47