Source-linked AI summary
SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement
Yiqi Liang, Xiaodong Cun
TL;DR
Text-guided complex scene sketching remains underexplored, despite the need to organize multiple concepts and their spatial relationships. SketchDirector turns prompts into editable vector assets, iteratively composes layouts, and refines detail while preserving structure. It achieves the highest performance across five VLM-assessed metrics among five baselines.
Problem
Complex scene sketching must organize multiple concepts, spatial relationships, and coherent details, while text-guided generation remains underexplored.
Source: §1 · p. 2Method
Results
SketchDirector achieves the highest performance across all five VLM-assessed metrics among five baseline methods and produces clearer object boundaries than the compared methods.
Source: §4.2 · pp. 7–8Takeaways & Limitations
MCSketch8K's compositional layouts and object-level manipulation annotations provide structured supervision for object decomposition, spatial reasoning, and controllable editing.
Source: §1 · p. 2The method struggles with complex physical and semantic relations, refinement can duplicate objects, and iterative multimodal verification incurs computational costs.
Source: §4.4 · p. 10
Abstract
from arXiv · showhide
Existing sketch generation methods often rely on image guidance and are largely limited to individual objects or simplified scenes, leaving text-guided complex scene sketching underexplored. We present SketchDirector, an agentic framework that mimics the human sketching workflow for automatic multi-concept scene sketching. Given a scene-level text prompt, SketchDirector first employs a large language model~(LLM) to decompose the scene into multiple draft assets, where each is represented by a set of Bézier curves. Then, an agentic reasoning loop iteratively plans and executes their spatial arrangement, guided by self-evolving memory. Finally, a detail refinement module leverages diffusion priors to enhance sketch quality while preserving geometric structure. To support future research and applications in multi-concept scene sketch generation, we further introduce the \textit{MCSketch8K} dataset, comprising editable vector assets, compositional layouts, and object-level manipulation annotations. Extensive experiments show that SketchDirector outperforms state-of-the-art baselines in semantic and compositional quality while effectively preserving overall contour organization during refinement. The project page is: \href{https://yiki77.github.io/SketchDirector/}{https://yiki77.github.io/SketchDirector/}.
1 Introduction
Complex scene sketching must organize multiple concepts, spatial relationships, and coherent details in one scene. SketchDirector addresses this challenge with editable vector assets, iterative layout reasoning, and structure-preserving refinement, alongside the MCSketch8K dataset.
1 Introduction
- Complex scene sketching requires semantically aligned concepts, reasonable layouts, and coherent details within a unified scene.The paper identifies organizing concepts and reasoning about their spatial relationships as central challenges.Source: §1 · p. 2
- Artists iteratively revise, group, and rearrange sketch elements, while digital Bézier-curve vectors enable efficient programmatic manipulation.Artists also adjust refinement from rough drafts to detailed renderings to suit creative goals.Source: §1 · p. 2, Figure 2 · p. 2
- SketchDirector decomposes text prompts into object-level Bézier draft assets and uses executable operations in an iterative Plan–Execute–Verify–Correct layout loop.The loop updates the scene and evaluates composition through geometric checks and multimodal verification.Source: §1 · p. 2
- Detail refinement optimizes object curve segments alongside the global canvas to vary object detail while maintaining textual and structural consistency.MCSketch8K contains 1,000 scene sketches, over 4,000 vector object assets, and 800 object categories, with compositional layouts and manipulation annotations.Source: §1 · p. 2
- The framework combines closed-loop agentic reasoning and Bézier-based local–global refinement, while MCSketch8K provides vector assets, scene layouts, and manipulation annotations.The paper presents these as its key techniques and dataset contribution.Source: §1 · p. 2
2 Related Work
Prior work moves from editable vector scene sketches toward concept-aware and agentic approaches. The reviewed methods still leave gaps in object-level planning, multi-object coordination, or high-level scene reasoning.
Scene Sketch Generation
- Scene sketching progressed from photograph-to-raster contour translation toward scalable vector representations and text-guided generation.Existing vector methods often optimize scenes holistically, leaving objects implicit and limiting editing to strokes rather than objects.Source: §2 · pp. 2–3
Multi-concept Sketch Generation
- SketchTriplet, SketchAny, and CoProSketch add concept structure but retain limits in layout planning, instance control, or independent concept generation (Wu, Wang, and Yang 2025; Zhang et al. 2026; Zhan et al. 2025).The reviewed methods treat objects as fixed composition units, provide limited control over instances, or still generate at scene level.Source: §2 · p. 3
Agents for Sketch Generation
- Sketch agents support sequential generation, conversational refinement, or style-aware completion, but lack high-level scene reasoning and multi-object coordination (Du et al. 2026).SketchDirector addresses these gaps with hierarchical planning, object-level control, and iterative refinement.Source: §2 · p. 3
3 SketchDirector
SketchDirector turns a text description into editable vector assets, composes them through iterative agentic layout reasoning, and refines details while preserving the scene structure and layout. Its pipeline combines executable object-level edits, geometric and semantic verification, and memory-guided correction.
3 SketchDirector
- SketchDirector decomposes textual scenes into object assets, composes them through a closed-loop layout process, and refines details while preserving asset structure and canvas layout.The framework represents sketches as SVG assets parameterized by black Bézier curves.Source: §3 · p. 3
3.1 Assets Generation
- Each scene description is parsed into drawable objects with identifiers, descriptions, and depth order, then each object receives a separate Bézier-curve draft asset.Draft generation fits sampled outline points to cubic Bézier curves and uses a contour-derived white mask to support later collage-like assembly.Source: §3.1 · p. 3
3.2 Operating Action Library
- The action library gives the agent composable object-level operations for moving, scaling, rotating, layering, deleting, or modifying assets.A single revision can combine primitives, such as scaling, moving, and layering an object to place it behind another.Source: §3.2 · pp. 3–4
3.3 Agentic Layout Reasoning
- Layout reasoning iterates through planning, execution, and verification, with executable programs updating both object attributes and scene state to produce a traceable composition trajectory.The agent plans with functional tools and spatial clues, then uses the trajectory to identify problematic operations during correction.Source: §3.3 · p. 4, Planning and Execution · p. 4
- Layouts must pass canvas-boundary, scale, and visual-semantic checks; detected violations feed correction, while short-term failure records and retrieved successful experiences guide later planning.Successful experiences are retrieved by Jaccard similarity between current and historical object sets.Source: Multimodal Verification · pp. 4–5, Self-Evolving Correction · p. 5
3.4 Detail Refinement
- Detail refinement resamples existing contours into a fixed number of cubic Bézier primitives and progressively optimizes local objects and the global canvas while retaining curve order and scene structure.Resampling subdivides existing curves with de Casteljau, preserving contour geometry; optimization updates control-point geometry while keeping the inherited curve ordering fixed.Source: §3.4 · p. 6, Curve Contour Resampling · p. 6, Progressive Joint Optimization · p. 6
4 Experiment
VLM assessments and user preferences favor SketchDirector, while ablations examine its reasoning and refinement components. The experiments also identify limits in relational modeling, instance consistency, and computational cost.
4.1 Experimental Setup
- Evaluation uses 30 MCSketch8K descriptions; ablations use another 30 prompts, and detail refinement takes about five minutes per object.The implementation uses cubic Bézier strokes and renders at 780×780 pixels.Source: Implementation Details · p. 6, §4.2 · p. 7, §4.3 · p. 8
4.2 Experiment Results
- SketchDirector achieves the highest performance across all five VLM-assessment metrics against five baselines.The comparison includes LLM-based and optimization-based methods.Source: Table 1 · p. 7, §4.2 · p. 7
- Across both comparison settings and all baselines, 68 participants favored SketchDirector on object completeness, layout alignment, and visual aesthetics.The study collected 5,100 responses per criterion.Source: §4.2 · p. 8
- Qualitatively, SketchDirector produces clearer object boundaries and presentation, with diverse layouts from the same prompt.Figure 7 also illustrates varied object configurations under similar semantic descriptions.Source: §4.2 · p. 8, Figure 7 · p. 8
4.3 Ablation Studies.
- GPT-5.5 scores highest across verification metrics and requires the fewest reasoning iterations among the tested backbones.The study also tests Gemini-3.1-Pro, Claude-Opus-4-7, and Qwen3.7-Plus.Source: Impact of VLM Backbones · pp. 8–9, Table 2 · p. 9
- Removing planning degrades layouts; geometric and semantic verification address spatial validity and scene-description alignment, while memory modules provide complementary capabilities.Short-term memory chiefly reduces repeated exploration, while long-term memory supports reuse of prior layout experience.Source: Table 3 · p. 9, Impact of Agentic Layout Reasoning · p. 9
- Curve-based resampling preserves contour topology; local and global supervision and progressive optimization help avoid distorted objects, weak scene coherence, and unstable updates.Point samples can drift independently, distorting characteristic object shapes.Source: Impact of Detail Refinement · pp. 9–10, Figure 11 · p. 10
4.4 Limitations.
- SketchDirector struggles with complex physical or semantic relations, can duplicate instances during refinement, and incurs costs from verification and iterative optimization.The authors suggest stronger relation modeling, instance-aware constraints, and more efficient verification and optimization as future directions.Source: §4.4 · p. 10, Figure 13 · p. 10
5 Conclusion
SketchDirector turns text descriptions into editable vector assets and executable layout operations through an agentic, multi-stage workflow. The paper also introduces MCSketch8K and reports improved semantic and compositional quality while preserving structure during refinement.
5 Conclusion
- SketchDirector converts text descriptions into editable vector assets and executable layout operations through concept representation, iterative agentic reasoning, and detail refinement.The reasoning stage uses Plan–Execute–Verify–Correct and self-evolving memory; refinement progressively optimizes Bézier-based vectors.Source: §5 · p. 10
- MCSketch8K is a benchmark for multi-concept scene sketch generation containing editable vector assets and composition annotations.Source: §5 · p. 10
- Experiments show better semantic and compositional quality than existing methods while preserving structural integrity during refinement.Source: §5 · p. 10