Source-linked AI summary
Teaching an Agent to Sketch One Part at a Time
Xiaodan Du, Ruize Xu, David Yunis, Yael Vinker, Greg Shakhnarovich
TL;DR
Text-to-vector sketch generation largely lacks detailed, free-text-guided part-by-part workflows. The paper combines a generic VLM annotation pipeline and ControlSketch-Part with SFT followed by multi-turn process-reward GRPO, producing structured one-part-at-a-time generation that outperforms prior methods and supports localized editing.
Problem
Existing methods mostly generate complete sketches at once and lack unified, free-text-guided part-by-part generation of highly detailed vector sketches.
Method
The paper builds ControlSketch-Part through multi-stage VLM part annotation and trains a sketching agent with supervised fine-tuning followed by multi-turn process-reward GRPO.
Results
The resulting agent generates structured sketches one part at a time, outperforms prior methods across automatic metrics and user studies, and supports localized editing operations.
Takeaways & Limitations
Structured part-level data and intermediate visual rewards support interpretable, controllable, and locally editable text-to-vector sketch generation.
Takeaways & Limitations
After SFT, visual quality deteriorates as autoregressive generation progresses because training uses oracle intermediate states while inference uses the agent’s own generations.
Abstract
from arXiv · showhide
We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal language model-based agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation.
1 Introduction
Existing text-to-vector methods mostly generate complete sketches at once, limiting progressive, part-by-part workflows. The paper addresses this gap with part-annotated data and multi-turn training for interactive generation and editing.
- Most existing methods generate the full sketch at once, overlooking the progressive, step-by-step nature of sketching.
- Meaningful part groupings make sketches more interpretable and enable efficient localized removal, replacement, or modification.
- Prior work generally lacks free-text guided part-by-part generation of detailed vector sketches in a unified model.
- The proposed annotation pipeline uses proposal, critique, and revision stages and is generic across vector sketch data.
- ControlSketch-Part trains a VLM through supervised fine-tuning followed by multi-turn process-reward GRPO using intermediate-state rewards.
- The resulting agent supports multi-turn sketch generation and progressive text-guided editing, with improvements shown through automated metrics, user studies, and qualitative results.
2 Related Works
Prior text-to-vector sketch methods include learning-based generation, test-time optimization, and sequential prompting approaches. The paper positions its method against limitations in adaptability, fidelity, and visual-feedback-based training.
- Learning-based approaches generate sketches autoregressively using polylines, Bézier curves, or continuous-time functions.
- SketchAgent performs text-guided sequential sketching with a closed-source Claude Sonnet model in a zero-shot prompting framework.
- SketchAgent produces doodle-style outputs with limited adaptability to higher visual fidelity or specific domains and low spatial grounding accuracy.
- Multimodal reinforcement-learning work includes two-stage SFT-plus-GRPO training with programmatic correctness and visual-similarity rewards.
3 Automated Part Annotation
The annotation pipeline progressively converts vector sketches into captions, semantic parts, and path assignments through VLM proposal, critique, and refinement stages. Applied to ControlSketch, it produces ControlSketch-Part with structured part-level annotations.
- Each annotated sketch receives an overall caption, semantic part descriptions, and a path-to-part assignment for every path.
- The pipeline begins by rendering the sketch and asking a VLM to propose non-overlapping, collectively exhaustive semantic parts.
- A VLM critic audits the proposed decomposition against the instructions and rendering, reporting violation types, severity, reasons, fixes, and whether revision is needed.
- Path assignments use unique part labels, assign each path to exactly one part, require every part to contain a path, and undergo diagnostic-visualization critique and refinement.
- The procedure is designed for SVG or vector-convertible sketch datasets and is applied to the complex, realistic-looking ControlSketch dataset.
- ControlSketch-Part is produced with two-to-five parts per sketch using Gemini 3.0 Pro, adding captions, part descriptions, and path-to-part assignments.
4 Method
The method trains a vision-language agent to generate vector sketches iteratively, one part per turn, using structured canvas feedback and a two-stage SFT-to-RL process. Multi-turn process-reward GRPO combines intermediate visual similarity with final path-count regularization to address the mismatch between oracle partial sketches during SFT and the agent’s own prior generations.
- Iterative generation: At each turn, the VLM receives the current canvas, global caption, next-part description, prior parts with paths, and remaining-part count, then outputs cubic Bézier paths.Paths are represented using SVG M and C commands with eight coordinates per cubic Bézier curve.
- Training framework: Training has two stages: SFT teaches valid single-turn path generation, while multi-turn process-reward GRPO improves visual output.SFT uses cross-entropy loss; RL uses process rewards during multi-turn rollouts.
- Supervised fine-tuning: SFT augments sketches with up to 20 random part permutations, exposing the agent to arbitrary part orderings and corresponding partial sketches.Each permutation supplies ordered part descriptions, incomplete strokes, and incomplete renderings while retaining the same global caption.
- Reinforcement learning: RL bridges SFT’s oracle-state training and inference-time self-generated states, where autoregressive generation otherwise suffers deteriorating visual quality as steps progress.The RL stage uses GRPO, which estimates a group baseline from multiple sampled trajectories without a separate value model.
- Process-reward GRPO: Because intermediate ground-truth states are available and trajectories share the same number of parts, rewards and advantages are normalized within each generation step.This provides step-specific process supervision rather than a single unified baseline across all rewards.
- Reward design: The process reward combines DreamSim similarity between current generated and ground-truth renderings with a path-count reward that regularizes only the final output.DreamSim is computed at each step; path-count regularization avoids noisy per-part signals and discourages overly long trajectories.
5 Experiments
The experiments compare the proposed sketching agent with three baselines using automatic metrics, user studies, qualitative examples, and ablations. Results favor the full SFT + RL model and indicate benefits from multi-turn process-level rewards.
- Experimental setup: The method is benchmarked against SketchAgent, Gemini 3.1 Pro, and SDXL + SwiftSketch.SketchAgent generates sketches sequentially, Gemini 3.1 Pro generates whole sketches directly, and SDXL + SwiftSketch combines text-to-image and image-to-sketch generation.
- Experimental setup: Both automatic metrics and double-blind user studies evaluate final visual quality and, for SketchAgent, part-by-part generation quality.Long-CLIP measures similarity between the rendered sketch and concatenated part descriptions; user studies separately assess whole-sketch quality and alignment of the generation procedure with part descriptions.
- Experiment results: The full SFT + RL model achieves the best performance across all methods and surpasses the SFT-only variant.The authors report this pattern for Long-CLIP, pairwise preference studies, and the contribution of both training stages.
- Experiment results: Progressive, part-by-part generation is associated with stronger performance than holistic approaches among prior methods.SketchAgent performs best among the prior methods, while Gemini 3.1 Pro falls short of specialist agents.
- Qualitative results: Changing an early part description while keeping later descriptions fixed produces differences localized to the affected part in progressive editing examples.Different initial canvases with identical part descriptions also lead to different outputs.
- Ablation study: The ablation scores are 0.281 for single-turn RL, 0.286 for multi-turn outcome-reward GRPO, and 0.298 for multi-turn process-reward GRPO.The controlled ablation uses Qwen2.5-VL-3B and supports benefits from multi-turn training and intermediate-state rewards.
6 Conclusion
The paper combines ControlSketch-Part with a two-stage SFT+RL framework to train a VLM agent for structured, part-by-part vector-sketch generation and editing. Its annotation pipeline and multi-turn process-reward training support semantic decomposition, path assignments, and visual feedback during generation.
- ControlSketch-Part enriches vector sketches with semantic part decompositions, per-part text descriptions, and path-to-part assignments.
- The training framework first uses SFT to establish output format and a single-turn sketching policy, then applies multi-turn process-reward GRPO with intermediate visual rewards.The GRPO procedure samples trajectories and updates the policy using group-relative advantage estimation and a GRPO objective.
- The resulting agent generates structured sketches one part at a time and supports localized editing operations such as stroke removal and replacement.
- The supplementary material includes pseudocode, annotation prompt templates, additional generation results, dataset examples, and failure cases with limitations and future work.
- The automatic annotation pipeline uses staged prompts for decomposition, auditing, revision, path assignment, and path-assignment auditing.The prompts require exhaustive, non-overlapping parts, strict JSON outputs, complete path assignments, and semantic verification.
C Additional Part-by-Part Results
Figures A1–A15 show additional part-by-part results of the model. Each sketch presents cumulative frames with newly added parts color-coded to their corresponding labels.
- Figures A1–A15 provide additional part-by-part results of the model.
- The cumulative frames show newly added parts color-coded to match their corresponding part labels.
D Additional ControlSketch-Part Dataset Examples
Tables A1–A3 provide additional examples from the ControlSketch-Part dataset.
- Tables A1–A3 present additional examples of the ControlSketch-Part dataset.
E.1 Failure cases
The failure cases expose limitations caused by fixed path-count supervision, unfamiliar semantic topologies, and occasional part misplacement.
- Fixed path counts can encourage premature stopping once the reference count is reached, leaving a corresponding part incomplete.One example omits the right wheel.
- The agent can produce erroneous topologies for unfamiliar semantic concepts, such as a vertically oriented oval rear wheel.
- Despite substantial mitigation of part-misalignment errors through RL training, occasional misplacements remain.One example places the jacket too far right, creating an unnatural gap between the jacket and upper legs.
E.2 Limitations
The agent’s main limitation is weak generalization beyond the training data, especially for unseen object categories and unrelated parts. Precise coordinate prediction and sampling stochasticity can also produce rare, severely degraded in-distribution outputs.
- OOD object categories expose two failure modes: meaningless sketches or outputs resembling seen nearest-neighbor classes instead of intended objects.Examples include tiger rendered like a bear and bus rendered like a car.
- General part descriptions occasionally yield reasonable OOD approximations, but broader and more diverse training data is expected to improve generalization.
- The agent usually fails to accurately add parts from unrelated categories to familiar objects.Examples include pincers on an angel, legs on a fish, and feathered wings on a chair.
- Imprecise numerical-coordinate prediction, compounded by temperature-based sampling, can drastically degrade stroke placement.The authors describe these lowest-quality outputs as infrequent edge cases across in-distribution categories.
E.3 Future Work
Future work could extend the one-part-at-a-time pipeline with parallel planning, intermediate-output refinement, richer reasoning, visual self-evaluation, and applications beyond sketch generation.
- A planning agent could coordinate multiple agents to generate different parts in parallel, extending the current one-part-at-a-time pipeline.
- Refining unsatisfactory intermediate outputs could improve overall sketch quality during generation.
- Chain-of-thought reasoning before each part could add richer natural-language reasoning to the generation process.
- The agent could be extended to generate auxiliary figures for visual reasoning tasks such as geometry problems.
- Self-evaluation and selective regeneration of poor intermediate parts could mitigate error accumulation over longer generation sequences.