Source-linked AI summary
From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches
Ahmet Neçirvan Doğan, Eren Acar Başaran, Metin Sezgin, Zahra Montazeri
TL;DR
Sketch-to-3D pipelines do not provide a direct way to encode where a designer wants light and shadow. This paper uses star and moon marks to predict an editable directional light and displace a garment surface; on held-out renders, 79% of moon marks darken after displacement.
Problem
Existing sketch-to-3D pipelines lack an input for shading intent, so drawn shading is treated as line art or appearance rather than editable scene lighting.
Source: §1 · p. 1Method
From a marked garment sketch and texture prompt, the system predicts a directional light and a displacement field that bends the surface so moon-marked regions fall into shadow.
Source: §1 · p. 2, §4.6 · p. 6Results
79% of moon marks darkened after displacement on held-out renders, compared with 44% of equally sized control discs away from marks.
Source: §5.3 · p. 9Takeaways & Limitations
The output is a 3D garment scene with an explicit editable light, while marked shadows are produced by surface displacement rather than dark texture paint.
Source: §1 · p. 2, §7 · p. 9The system assumes one directional light, constant prompt-derived roughness, and no colored or area sources, translucency, or interreflection; moving the light may alter whether folds cast shadows onto their marks.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Sketching is one of the most natural ways for people to express visual ideas. Unlike the text prompts that most generative systems rely on today, a sketch captures spatial structure and intent directly, including where a designer wants highlights and shadows to appear. Yet current sketch-to-3D pipelines have no input for that intent: they bake the drawn shading into the texture as dark paint, and the light cannot be moved. We propose a system that transforms a rough garment sketch, marked with sparse annotations (a star for a highlight and a moon for a shadow), and a short text prompt describing the texture, into a textured 3D mesh with an explicit directional light, as a scene that can be relit and edited after generation. The two symbols are the most frequent proposals in an elicitation study with art students. From the marked sketch the system predicts the scene's directional light and, separately, a displacement field that bends the surface so that the moon-marked regions fall into shadow under that light, rather than being painted dark in the texture.
1. Introduction
Existing sketch-to-3D pipelines lack a way to capture where a designer intends light and shadow to fall. The paper introduces star and moon marks that guide an editable light and surface displacement in a relightable garment scene.
1. Introduction
- Current pipelines treat shading strokes as line art or reproduce them as appearance, leaving no input for the designer’s shading intent.Text prompts also lack an explicit way to tie a lighting phrase to a drawing region.Source: §1 · p. 1
- An elicitation study with art students motivates a two-symbol notation: a star marks a highlight and a moon marks a shadow.The marks are placed directly on the drawing.Source: §1 · p. 2
- The pipeline predicts an editable directional light from the marks and displaces the generated mesh so moon-marked regions fall into shadow.It combines a marked sketch and texture prompt into a relightable 3D scene.Source: §1 · p. 2
2. Related work
Prior work covers sketch-based geometry, lighting specification, and image-conditioned 3D generation, but the paper identifies a gap: existing pipelines do not read shading intent from a drawing as a scene light.
2.1. Sketch-based modeling
- Earlier sketch-based methods infer geometry from strokes or generated images, while shading marks are treated as line work or texture rather than lighting intent.The authors state that, to their knowledge, no existing method reads intended light placement from a drawing and returns it as a scene light.Source: §2.1 · p. 2
2.2. Specifying lighting
- Artists depict light through effects on form, whereas 3D tools expose editable light sources; prior effect-to-light methods work on an existing 3D scene.Here, light must be inferred from a drawing before geometry exists.Source: §2.2 · p. 2
2.3. Text- and image-conditioned 3D generation
- Text- and image-conditioned 3D generators return geometry and material but no light, and any conditioning-image illumination left in the texture cannot be moved.The paper uses such a generator for the mesh and texture, then adds light as a separate scene element.Source: §2.3 · pp. 2–3
3. Shading marks
The paper motivates direct on-sketch shading marks and selects their symbols through an elicitation study. Its notation separates light-direction cues from shadow-location cues, with moon marks also guiding geometry.
3.1. Motivation
- Marks should let designers indicate light and shadow directly on a sketch, using symbols that people can understand without shading instruction.The authors asked participants which symbols they would use rather than inventing symbols themselves.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Elicitation
- The star and crescent moon were the modal proposals: 14 of 25 participants chose a star or sparkle for highlight, while 6 of 22 chose a moon for shadow.The moon’s lead over the circle, 6 to 4, was too small to be reliable at this sample size; most participants had no formal shading training.Source: §3.2 · p. 3, Table 1 · p. 3
3.3. Representation
- Stars indicate where light falls and carry its direction, while moons indicate shadow locations without specifying the light.The system bends the garment so a moon-marked region falls into shadow under the light defined by the stars, rather than painting it dark in the texture.Source: §3.3 · pp. 3–4
- Fixed-size, non-overlapping marks must lie inside the garment, with at least one star and one moon per sketch.The marks are stamped on a separate layer; the marked sketch guides the U-Net and light CNN, while the clean sketch goes to the albedo generator.Source: §3.3 · p. 4
4. Method
The pipeline turns a marked sketch and texture prompt into a garment scene by predicting lighting and folds, synthesizing albedo, and reconstructing and displacing a mesh.
4.1. Overview
- A U-Net predicts irradiance and a signed fold field from the marked sketch and prompt; a light CNN reads its bottleneck for the directional light, and the remaining stages build and export the scene.The U-Net is trained first and frozen before the light CNN is trained on top of it.Source: §4.1 · p. 4, Figure 2 · p. 5
4.2. Predicting irradiance and light direction
- The U-Net predicts an irradiance map supervised by renders of the training scene with the garment made white and matte, while a light CNN maps the bottleneck to a unit light-direction vector.The matte target retains the light reaching each point, including shadows.Source: §4.2 · p. 4
4.3. Predicting where the surface folds
- The U-Net’s signed field predicts fold direction and strength, with supervision around moon marks and zero displacement on training sketches without moons.The target is rescaled per garment, so it ranks crease strength rather than measuring absolute distance; the training dataset places moon marks at negative extrema of the target measure.Source: §4.3 · p. 4
4.4. Albedo synthesis
- FLUX with a ControlNet synthesizes base-color albedo from the clean sketch and prompt, leaving illumination to be applied afterward rather than baked into the color.The method trains a rank-64 LoRA adapter and the double-stream-block output layers while otherwise freezing ControlNet and FLUX.Source: §4.4 · p. 4, Figure 2 · p. 5
4.5. Composition and reconstruction
- The method combines predicted irradiance and albedo in linear space, adds ambient and roughness-dependent highlight terms, and sends the resulting lit image to Hunyuan3D-2.1 for mesh and texture reconstruction.An ambient term of 0.15 preserves albedo color in unlit regions; when more than a quarter of the garment has almost no predicted irradiance, the image first receives a floor based on lit pixels.Source: §4.5 · p. 5
4.6. Displacement and scene assembly
- The combined displacement field requests vertex motion along normals, with inward movement capped by local shell thickness, while an as-rigid-as-possible solve balances those requests against rigidity.The scene is exported with its painted base-color texture, prompt-selected roughness, and predicted directional light.Source: §4.6 · pp. 5–6, Figure 2 · p. 5
- A crest beside each moon mark, on the side facing the predicted light, raises the surface so the mark lies on its shadowed side.The crest secures a rise without imposing one: within its footprint, the method keeps whichever is larger, the crest or predicted field.Source: Figure 2 · p. 5, §4.6 · p. 6
5. Results
The evaluation measures recovery of the marked sketch’s light and shadow intent. The predicted light is closer to the reference than a fixed-direction baseline, and displacement darkens moon-marked regions, including marks placed on initially flat cloth.
5. Results
- Figure 4 varies moon-mark locations while reusing the same reconstruction, and Figure 5 shows marked inputs and displaced outputs with and without texture.Figure 5’s untextured views show added shading more clearly; prints can obscure it.Source: Figure 4 · p. 7, Figure 5 · p. 8
5.1. Setup
- Each render receives roughly nine moon marks on sharp creases and eight star marks at its brightest ground-truth irradiance locations.The sketch marks are placed within the study’s placement constraints.Source: §5.1 · p. 7
- Lighting is measured on all 366 held-out renders before mesh reconstruction; displacement is measured on 340 renders across 121 garments after rejecting 26 reconstructions with wrong viewing side or excessive silhouette mismatch.A separate 112-render test places moon marks on slightly brighter-than-average flat cloth and retains existing reconstructions, isolating the displacement stage.Source: §5.1 · p. 7
5.2. Lighting recovery
- 22.9° mean and 20.0° median angular error, versus 47.5° for the best fixed direction fitted on the training split, on 366 held-out renders.The mean error and baseline are unchanged to one decimal when restricted to 340 renders with successful reconstructions.Source: §5.2 · p. 9
5.3. Mark-driven shading
- 79% of moon-marked discs darken after displacement, versus 44% of unmarked control discs; with an eight-level decrease threshold, the rates are 51% and 13.5%.With the crest disabled, the predicted field alone darkens 61% of marks against 47% of control discs.Source: §5.3 · p. 9
- On 112 renders with moon marks placed on initially flat cloth, 86% of marks darken after displacement, versus 45% of control discs; 9% are darker than garment average before, and 73% after.With the crest disabled, the field alone darkens 57% of marks against 47% of control discs.Source: §5.3 · p. 9
6. Limitations
Displacement is constrained by differences between the training garment surface and the generated reconstruction, shell thickness, and rule-set crest placement. Artifacts, simplified scene assumptions, and limited evaluation data further bound the system’s scope.
Why some marks do not darken
- The predicted field is learned on DeepFashion3D surfaces but applied to Hunyuan3D reconstructions, and inward motion is capped by shell thickness; the cap binds on 314 of 340 renders.Where it binds, the requested depth is typically several times what the shell allows.Source: §6 · p. 9
Crest placement
- Each mark gets one crest whose footprint and distance from the mark follow a rule rather than being predicted from the sketch.The rest of the displaced surface comes from the predicted field.Source: §6 · p. 9
Artifacts
- Displacement leaves a visible artifact on 100 of 340 held-out renders, most often a blob-like distortion or the outer wall passing through the inner one.Those renders are excluded from figures but retained in measurements.Source: §6 · p. 9
Scene and material assumptions
- The scene assumes one directional light, no colored or area sources, no translucency or interreflection, and constant garment roughness; residual input shading can remain baked into texture.Moving the light preserves folds but does not necessarily preserve the shadow each fold casts onto its mark.Source: §6 · p. 9
Coverage
- The elicitation study used art students from one institution, and quantitative evaluation uses synthetic DeepFashion3D sketches with marks placed by rule or by the authors.Hand-drawn sketches appear only in qualitative results; symbol interpretation and how far marks can move were not tested.Source: §6 · p. 9
7. Conclusion
The system turns star and moon marks on garment sketches into a 3D scene with editable directional lighting and shadows formed by surface displacement. One part remains open: predicting each crest’s footprint and distance from its mark.
7. Conclusion
- The system returns a 3D garment with an explicit editable directional light and uses surface displacement to create the requested shadows.The star and moon marks indicate where light falls and where the garment should be in shadow.Source: §7 · p. 9
- The crest footprint and its distance from each mark are still set by a rule; predicting them is identified as future work.Its height comes from the predicted field and its side from the predicted light.Source: §7 · p. 9