Sketch-n-Sketch: Output-Directed Programming for SVG

INTRODUCTION

• ODP addresses a trade-off: direct-manipulation GUIs offer intuitive point-and-click creation, while text-based programming offers flexibility not afforded by GUIs.The introduction also notes efforts to combine the strengths of these paradigms, including scripting APIs and visual programming approaches.Source: INTRODUCTION · p. 2

Output-Directed Programming (ODP)

• ODP constructs a plain-text program through mouse-based operations on its output, while preserving ordinary text editing for unsupported changes.Each operation transforms the program text, and text edits do not disable later direct manipulation.Source: INTRODUCTION · p. 2

INTRODUCTION

• The paper’s motivating possibility is that ODP could support program construction, potentially accelerating common tasks for experts and offering novices an approachable programming pathway.These are described as possibilities for general-purpose programming.Source: INTRODUCTION · p. 2
• Earlier ODP systems mixed text edits with direct manipulation, and many changed output constants; this work asks what programs output manipulation alone can construct.The question is how thoroughly direct manipulation can subsume text-based editing.Source: INTRODUCTION · p. 2

New ODP Techniques in SKETCH-N-SKETCH

• The new techniques enable construction of 16 example programs without text edits, extending ODP beyond manipulating only final outputs.The techniques expose intermediate execution values, support expression focusing for contextual drawing, and provide GUI-accessible refactorings.Source: INTRODUCTION · p. 2
• The techniques use generic run-time tracing to associate output values with source locations, and the authors see the principles as relevant beyond vector graphics.SKETCH-N-SKETCH itself is specialized for programs that output vector graphics.Source: INTRODUCTION · p. 2

OVERVIEW

• SKETCH-N-SKETCH pairs a source editor with an SVG output pane that also provides graphical transformations; text edits remain possible during construction.The paper’s walkthrough does not use text edits.Source: OVERVIEW · p. 2
• The lambda-logo walkthrough proceeds through drawing shapes, relating features, abstracting the design into a reusable function, and refactoring the generated code.The final code for this example is shown in Figure 11.Source: OVERVIEW · p. 3

Drawing Shapes

• Dragging with the square tool inserts a square definition and its use in the shape list, with position and size derived from the drag.This imitates a drawing application and avoids manually consulting drawing-library documentation for the initial shape.Source: OVERVIEW · p. 3
• Snap-drawing two lines to square features encodes persistent alignment constraints through shared variables for the square’s top-left coordinates and width.The figure’s freeze annotations, written !, prevent specified constants from changing when shapes move.Source: OVERVIEW · p. 3, Figure 4 · p. 3

Relating Properties

• Selecting two lines exposes their color and stroke-width sliders, which can introduce shared variables so both lines use identical properties.The programmer applies the color change and repeats the workflow for stroke width.Source: Relating Properties · pp. 3–4
• GROUP gathers selected shapes into a list definition that replaces the individual shapes in the final shape list.The resulting list can itself be selected through a list widget.Source: Relating Properties · p. 4
• Clicking a canvas label renames its variable, while ABSTRACT extracts the selected list into a function parameterized by its remaining free variables.The example extracts squareLineLine as logoFunc with parameters y, x, w, color, and strokeWidth.Source: Relating Properties · p. 4
• Type inference recognizes a reusable function's coordinate and width inputs, making it available as a custom drawing tool that inserts calls when used on the canvas.The example's function accepts an (x,y)-coordinate and a width.Source: Relating Properties · p. 4

Refactoring

• The lambda logo program was completed entirely through canvas operations using drawing, relating, abstracting, and refactoring tools.Call widgets expose function arguments and focus its return expression, where the programmer edits the polygon's fill and stroke.Source: Refactoring · p. 5

DESIGN AND IMPLEMENTATION

• The workflow evaluates a text program to SVG, overlays widgets for selection, then re-evaluates the transformed program to render updated output.Programmers can draw shapes, drag items to change numbers, or choose transformations from a floating menu.Source: DESIGN AND IMPLEMENTATION · p. 5
• Canvas transformations apply to selected output sub-values, including whole shapes and positional, size, color, or stroke features.The selected sub-values serve as arguments interpreted according to each transformation.Source: DESIGN AND IMPLEMENTATION · p. 5
• The programmer can select intermediate execution values or focused sub-expressions as transformation arguments, modifying computations without direct representations in the final output.These selections extend manipulation beyond the final output alone.Source: DESIGN AND IMPLEMENTATION · p. 5
• The implementation also discusses transformations as ordinary refactorings and uses generic provenance tracing to connect canvas selections with program locations.The passage describes these techniques as not specific to SVG.Source: DESIGN AND IMPLEMENTATION · p. 5

Intermediate Value Widgets

• Point widgets expose intermediate coordinate pairs, and distance features between selected points allow Cartesian distances to be selected.The system also draws an offset arrow when arithmetic changes an x or y coordinate; dragging it changes the offset amount.Source: Intermediate Value Widgets · p. 6
• The Point and Offset tool adds point definitions or offset definitions through canvas clicks or drags, and snap-drawing can share offset amounts to create symmetry.Offset endpoints can also serve as snap targets for later drawing.Source: Intermediate Value Widgets · p. 6
• A list widget marks a list of graphical items, allowing lists of shapes, points, or nested lists to be selected without a graphics-specific grouping construct.A group is represented as a list of shapes.Source: Intermediate Value Widgets · p. 6

Expression Focusing

• Focusing a shape group or function changes the scope of drawing operations, so a recursive design can be created by drawing a function inside itself.A call widget focuses the function call that produced graphical elements; a literal definition can also be focused through its right-hand side.Source: Expression Focusing · p. 6

Generic Refactoring

• ABSTRACT performs Extract Method by building a function parameterized over an expression’s free variables, while focused function calls can have arguments added, removed, or reordered.MERGE performs clone elimination.Source: Generic Refactoring · p. 7

Provenance Tracing

• The evaluator tags each computed value with its source expression and pointers to prior values, enabling dependency tracing; list-element pointers additionally support containment queries.The tracing discards some control-flow information, notably pattern-match destructuring; the authors suggest Transparent ML tracing as a possible future alternative.Source: Provenance Tracing · p. 7
• Sketch-n-Sketch can become sluggish on larger examples, usually because of comparisons between large traces or widget processing rather than tracing itself.Tracing theoretically adds memory and time overhead to evaluation.Source: Provenance Tracing · p. 7

CASE STUDY OF ODP EXAMPLES

• 16 parametric designs spanning 427 lines of code were built entirely through output-directed manipulations, without code-box edits.The examples include six designs that end as parameterized drawing tools, six with repetition by position, one with repetition by other features, and one recursive design.Source: CASE STUDY OF ODP EXAMPLES · p. 7

Authoring

• Design construction did not always follow the four stages in fixed order; the authors often specified relationships with points and offsets before attaching shapes through snap-drawing.Offsets, snaps, and MAKE EQUAL sufficed for reasonable parameterizations but were not always convenient, and laying out offsets beforehand required forethought.Source: Authoring · p. 7
• The #Ex column records how many examples used each tool, and snap-drawing was the most widely used functionality after shape drawing and variable renaming.The examples used neither MERGE nor most tools for specifying relationships; three tools used only once were all involved in the Koch snowflake recursion workflow.Source: Authoring · p. 7
• Snap-drawing was preferred over MAKE EQUAL for spatial constraints, while RELATE was used only for constraints involving thirds.MAKE EQUAL is more flexible but requires extra clicks and can offer many hard-to-distinguish constraint encodings; its ranking heuristic favors nearby, later program terms.Source: Authoring · p. 7

Recursive Koch Snowflake

• The Koch snowflake requires manipulating intermediate values because the program has no output during most of development, and focused editing helps build its recursive function.The recursive motif uses helper functions to compute one-third points and a third point completing the motif.Source: Recursive Koch Snowflake · p. 7
• Drawing a function inside itself inserts a recursive skeleton and call, with a temporary guard preventing infinite recursion until a termination condition is chosen.The guard supports fixed-depth termination by tracking whether the function has appeared earlier in the call stack; the available condition decrements a depth argument on recursive calls.Source: Recursive Koch Snowflake · p. 8
• The base-case and recursive-case outputs are specified as lists, and the recursive case concatenates the lists returned from base-case calls.The author can refocus a call to the base case to edit its output while working on the recursive case.Source: Recursive Koch Snowflake · p. 8

Repetition

• Seven examples use repetition tools that repeat a selected shape over an existing point list or over a call to a function returning points.Figure 19 illustrates code produced by these repetition tools.Source: Repetition · p. 9
• For non-spatial variation, REPEAT BY INDEXED MERGED combines manually laid-out shapes into one function called with successive index values, while FILL HOLE synthesizes expressions that resolve their syntactic differences.The workflow was used for Figure 15x and xiii.Source: Repetition · p. 9

Limitations: Remaining Tasks

• 4 of the 15 WWID: PBD tasks interpretable as parametric drawings are fully completed by the system.Two other tasks can be partially completed: “Box Volume” needs an interaction to compute and display numeric volume, while “Xs” needs more precise control over which definitions are abstracted.Source: Limitations: Remaining Tasks · p. 9
• The nine remaining tasks are diverse, with no single feature helping more than two or three; needs include arbitrary text boxes, list operations, geometric constraints, and non-recursive conditional branches.The geometric needs include reasoning about line intersections with shape edges, specifying overlapping and containment constraints, and solving different constraint types simultaneously.Source: Limitations: Remaining Tasks · p. 9

RELATED WORK

• The prior system’s bounding-box syntax, special shape-composition functions, and required top-level program structure could make tools unavailable after text edits left that subset.The current work relaxes those restrictions: drawing tools operate by type, groups are ordinary lists, and provenance tracing lets tools transform shapes not defined at top level.Source: RELATED WORK · p. 10
• Four examples could also be built without text edits in the prior system, but two had undesirable parameterizations; the new version improves code generation and simplifies math expressions.The authors attribute simpler math to discarding bounding boxes, reusing variables, simplifying MAKE EQUAL expressions, and connecting to the REDUCE solver; fewer source lines are described as incidental.Source: RELATED WORK · p. 10
• The current system lacks a path tool, and tools for re-parameterizing shapes en masse could be useful when a bounding-box parameterization is desired.The path tool was unneeded for the examples in this work.Source: RELATED WORK · p. 10

• The work expands Sketch-n-Sketch’s expressive power with nontrivial programs built entirely through direct manipulation, while using graphics as a laboratory for broader output-directed programming.The authors say their long-term goal is not a practical drawing tool, but advancing ODP for future programming settings.Source: CONCLUSION AND FUTURE WORK · p. 10

ODP for Novices

• ODP may help novices such as domain experts or students produce rudimentary programs, but design considerations for novices remain open.Source: CONCLUSION AND FUTURE WORK · p. 10

Widget Visibility

• Many intermediate execution steps can clutter the canvas, so the implementation hides most widgets by default and uses heuristics to show them.Widgets from intermediate expressions in standard-library code are generally not shown; future controls could include source annotations or UI options.Source: CONCLUSION AND FUTURE WORK · p. 10

Customizing Widgets

• A proposed toSvg API could render intermediate data structures recursively, much as toString renders text.Source: CONCLUSION AND FUTURE WORK · p. 10

Synthesizing Program Transformations

• Transformations are hand-coded one by one, making their implementation time-consuming and error-prone while limiting composability.The authors suggest synthesis techniques as a possible way to streamline transform specifications.Source: CONCLUSION AND FUTURE WORK · p. 10

User Interactions for Deciphering Human Intent

• Because one user action can admit multiple valid program changes, the system shows code and output differences and asks users to choose; richer intent-resolution interactions remain a challenge.The authors point to change-impact analysis and additional questions as possible improvements.Source: CONCLUSION AND FUTURE WORK · p. 10

APPENDIX

• The appendix supplies design and implementation details, revisits examples from prior work, and expands two Figure 15 examples.Source: APPENDIX · p. 13

Type Inference

• Functions accepting two points, or a point and horizontal or vertical distances, can become custom drawing tools; type inference propagates roles to identify distances and suggest defaults.Standard-library annotations seed role inference, which can also use built-in usage rules.Source: Type Inference · p. 13

Value Holes

• A value hole temporarily inserts a desired value into the syntax tree, then uses provenance to select an expression that evaluates to it.The authors use this workflow to express and resolve equality constraints in template code.Source: Value Holes · p. 13

Specifying the Focused Expression

• Special comments mark the focused function and call, and persist across arbitrary text edits unlike internal expression identifiers.Source: Specifying the Focused Expression · p. 13

Completing the Examples from the Prior Sketch-n-Sketch

• Completing prior examples requires path, offset, symmetry, bounds-relative polygon, and min/max tools; path drawing was disabled pending updates for snap support.The prior examples discussed here still required text edits in both the prior system and this one.Source: Completing the Examples from the Prior… · p. 13

Example (xii): Tree Branch

• For the tree branch, shared snapped offsets define a rhombus abstracted over its center and dimensions; fixed-separation points along the branch provide leaf attachments for repeating it.REPEAT OVER LIST maps a function abstracted over one point across the attachment points.Source: Example (xii): Tree Branch · p. 14

Example (xiii): Target

• Indexed repetition handles attributes beyond position: circle radius and color differences become programming-by-example holes, resolved using the index, while a slider controls circle count.The demonstrated target reverses the index order so i = 0 corresponds to the smallest circle; synthesis selects alternating colors and computes radii.Source: Example (xiii): Target · p. 14