SketchLattice: Latticed Representation for Sketch Manipulation

1. Introduction

• Pixelative representations lack intuitive structural cues, while sequential RNN methods require vector stroke data that is often unavailable for sketches drawn on paper.The paper identifies structural information as important for abstraction modeling and structural manipulation tasks.Source: §1 · pp. 1–2
• SketchLattice samples points from a sketch image with a lattice graph, retaining structural cues without requiring vector data.The authors describe the representation as amenable to structural deformations and abstraction modeling.Source: §1 · p. 2
• A graph model encodes the lattice representation with 13.5 times fewer parameters than recent state-of-the-art techniques.The paper links this smaller model to easier deployment and reduced optimization and training difficulty.Source: §1 · p. 2
• The approach supports sketch healing, density-controlled abstraction in generated sketches, and image-to-sketch synthesis by dropping points along an object's edge.The network can also recreate vector sketches from lattice points sampled from input sketches.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

• Pixelative methods apply CNNs to sketch images [33] [4], but the sparse line drawings are redundant inputs and these methods do not effectively capture structural cues.The passage notes that these approaches do not encode object position and orientation, leading to sub-par generation results.Source: §2 · p. 2
• Sequential sketch-specific representations are constructed stroke-by-stroke and enable RNN-based sketch modeling, including LSTM generation and synthesis [7], but require vectorized input, limiting use when vector sketches are absent.Source: §2 · p. 2
• GCNs have been applied to sketch embedding, while SketchLattice uses geometric proximity in a lattice representation to retain spatial evidence without stroke order.The paper also proposes using this graphical embedding for sketch generation and image-to-sketch translation.Source: §2 · p. 2

3. Methodology

• A sketch image is transformed into lattice points, graph-encoded as a latent vector, and decoded by an LSTM into a vector sketch.The network also takes an image edge map as input.Source: §3 · p. 3, Figure 2 · p. 3

3.1. Latticed Sketch

• SketchLattice samples stroke-pixel intersections with a uniform grid, preserves sketch topology without vector data, and varies abstraction through sampling density.Increasing lattice density produces more detailed sketches; decreasing it produces more abstract sketches.Source: §3.1 · p. 3

3.2. Graph Construction

• Sampled points become graph nodes through learned coordinate embeddings, while geometric proximity defines weighted edges using a distance threshold of 0.2.The adjacency link strength is 1 minus normalized distance.Source: §3.2 · pp. 3–4

3.3. Graphical Sketch Encoder

• Graph layers propagate neighbor-aware features that are robust to missing lattice parts, then FC, batch normalization, and Tanh produce sketch embedding Ψ.Each encoding layer uses two MLP units followed by ReLU, with dropout and residual connections.Source: §3.3 · p. 4

3.4. Sketch Generation by LSTM decoder

• The decoder projects Ψ into μ and σ, samples z with IID Gaussian noise, and conditions an LSTM on z to predict vector sketch strokes.The predicted stroke representation is converted to final stroke coordinates using Gaussian mixture models.Source: §3.4 · p. 4

3.5. Model Training and Deployment

• The encoder and decoder train end-to-end with vector sketches, but inference operates on image sketches and generates vector sketches from their lattice representation.The latent vector z conditions the decoder during reconstruction.Source: §3.5 · p. 4

4.1.1 Experimental Settings

• The experiments evaluate healing on 10 QuickDraw categories, with 70k training and 1k testing sketches per category, using sketch recognition accuracy and sketch-to-sketch retrieval accuracy as evaluation measures.Compared baselines include SketchRNN, SketchPix2seq, and SketchHealer.Source: §4.1.1 · p. 5

4.1.2 Results

• Generated sketches remain satisfactory through Pmask ≤30%, become more abstract as corruption rises, and become difficult to recover faithfully at Pmask = 50%.Increasing the number and position of lattice points also controls abstraction: adding points for a butterfly’s body and antennae yields more detail than sampling its wings alone.Source: §4.1.2 · pp. 5–6, Figure 3 · p. 6, Figure 4 · p. 6
• SketchLattice outperforms baseline methods on recognition accuracy, maintains stable recognition accuracy through Pmask = 30%, ranks second on Top-1 retrieval, and uses 13.5 times fewer parameters than SketchHealer.SketchHealer’s retrieval performance drops sharply without stroke order; the table reports recognition accuracy and Top-1 retrieval across corruption levels.Source: Table 1 · p. 6, §4.1.2 · p. 6
• On 10 selected QuickDraw categories, sketches’ latent vectors cluster by category in t-SNE visualization, suggesting category-level discriminative ability.The visualization uses 100 test sketches per class.Source: §4.1.2 · p. 7, Figure 6 · p. 7
• Ablations show recognition accuracy improves as lattice density increases, saturates beyond n = 32, benefits from more nearby graph neighbors, and drops significantly without residual connections.The study compares Top-1 and Top-3 recognition accuracy at Pmask = 10% and 30%.Source: §4.1.2 · p. 7, Table 2 · p. 7
• For the most complex sketches—the top 25% by stroke count—SketchLattice outperforms competitors; in a 10-participant study, its healed sketches were preferred at both 10% and 30% corruption.The study used 50 samples across 10 classes, each with two corrupted instances.Source: §4.1.2 · pp. 7–8, Table 3 · p. 7

4.2. Image-to-Sketch Synthesis

• For image-to-sketch synthesis on QMUL-Shoe-v2, human judgments found SketchLattice outputs more human-like than LS-SCC and equally effective at depicting shoes, though both methods fell short of human-drawn sketches.The comparison used 6,000 training pairs and 648 test pairs; generated examples include photo edges, model outputs, and human sketches.Source: §4.2 · p. 8, Figure 7 · p. 8, Table 5 · p. 8

• SketchLattice removes the bottleneck of requiring vector data while preserving the essential structural cues vector data provides.Source: §5 · p. 8
• Its amenability to structural changes allows better abstraction modeling.Source: §5 · p. 8
• Across sketch healing and image-to-sketch synthesis, the representation outperforms state-of-the-art alternatives while using significantly fewer parameters.Source: §5 · p. 8