Source-linked AI summary
Sketch2CAD: Sequential CAD Modeling by Sketching in Context
Changjian Li, Hao Pan, Adrien Bousseau, Niloy J. Mitra
TL;DR
Existing sketch-based modeling methods can lack CAD precision and editability, while interpreting each sketch is ambiguous because similar strokes can mean different operations in different contexts. Sketch2CAD feeds the current partial model and sketch to neural networks that classify and segment a CAD operation, then fits operation parameters using synthetic CAD-operation sequences for training. The authors’ examples ranged from 2 to 11 operation steps, spanning bolts and nuts to mixers and cameras; evaluation with novices and diverse created shapes also illustrated the approach.
Problem
Existing sketch-based modeling methods can lack CAD precision and editability, while interpreting each sketch is ambiguous because similar strokes can mean different operations in different contexts.
Source: §1 · pp. 1–2Method
Sketch2CAD feeds the current partial model and sketch to neural networks that classify and segment a CAD operation, then fits operation parameters using synthetic CAD-operation sequences for training.
Source: §1 · pp. 2–3, §6 · p. 8Results
The authors’ examples ranged from 2 to 11 operation steps, spanning bolts and nuts to mixers and cameras; evaluation with novices and diverse created shapes also illustrated the approach.
Source: §8 · p. 11, §9 · p. 13Takeaways & Limitations
Sketch2CAD offers an alternate sketch-based way to interact with CAD systems and produces regular CAD models that can be further edited.
Source: §1 · p. 2, §9 · p. 13The current system does not support drawing primitives on curved faces, and the authors also note difficulty with small features and a need for sketching ability.
Source: §8.4 · p. 12
Abstract
from arXiv · showhide
We present a sketch-based CAD modeling system, where users create objects incrementally by sketching the desired shape edits, which our system automatically translates to CAD operations. Our approach is motivated by the close similarities between the steps industrial designers follow to draw 3D shapes, and the operations CAD modeling systems offer to create similar shapes. To overcome the strong ambiguity with parsing 2D sketches, we observe that in a sketching sequence, each step makes sense and can be interpreted in the \emph{context} of what has been drawn before. In our system, this context corresponds to a partial CAD model, inferred in the previous steps, which we feed along with the input sketch to a deep neural network in charge of interpreting how the model should be modified by that sketch. Our deep network architecture then recognizes the intended CAD operation and segments the sketch accordingly, such that a subsequent optimization estimates the parameters of the operation that best fit the segmented sketch strokes. Since there exists no datasets of paired sketching and CAD modeling sequences, we train our system by generating synthetic sequences of CAD operations that we render as line drawings. We present a proof of concept realization of our algorithm supporting four frequently used CAD operations. Using our system, participants are able to quickly model a large and diverse set of objects, demonstrating Sketch2CAD to be an alternate way of interacting with current CAD modeling systems.
1 INTRODUCTION
Sketch2CAD addresses the gap between approximate design sketches and precise, editable CAD models by interpreting each sketch in the context of the model built so far. It translates sketching steps into CAD operations through learned recognition and segmentation, followed by parameter fitting.
1 INTRODUCTION
- Existing sketch-based methods often lack CAD precision and editability, while some require specific drawing techniques or produce meshes rather than editable models.Sketching and CAD are often handled by different experts, making design iterations cumbersome, expensive, and time-consuming.Source: §1 · p. 1
- Sketching and CAD modeling both build complex shapes by progressively refining simple forms, motivating Sketch2CAD’s translation between their operations.The system produces regular CAD models and a trace of sequential operations for further editing or fabrication.Source: §1 · p. 2, Figure 2 · p. 2
- A three-stage pipeline classifies the intended CAD operation, segments the sketch and contextual maps, then fits operation parameters to instantiate it on selected model faces.Depth and normal maps provide context for interpreting strokes drawn over the current model.Source: §1 · p. 2
- Deep convolutional networks are trained on synthetically generated CAD-like objects and operation sequences balanced for complex operations, parameter values, and sequence lengths.Balancing sequence lengths is intended to support recognition at any stage of a modeling session.Source: §1 · p. 2
- The system recognizes existing CAD operations in arbitrary order, supporting diverse human-made objects and precise, regular shapes despite approximate strokes.The authors present it as an alternate CAD interaction approach, while noting it is not as comprehensive as modern CAD software.Source: §1 · pp. 2–3
- The introduction identifies formalized sketch-to-CAD operations, a recognition-and-segmentation pipeline, synthetic training data, and an interactive modeling system as its contributions.The system aims to unify sequential product-design sketching and CAD modeling workflows.Source: §1 · p. 3
2 RELATED WORK
Prior sketch-based modeling methods either constrain the input or produce outputs with limited CAD editability. Sketch2CAD instead recognizes parametric CAD operations from freehand sketches in the context of an existing model, enabling sequential edits.
2 RELATED WORK
- Methods reconstructing complete sketches often require clean drawings or annotations, and may produce curve networks or meshes rather than directly usable CAD models.Interactive primitive-alignment tools can handle approximate inputs, but require annotations for precise alignment and relative positioning.Source: §2 · p. 3
- Deep-learning reconstruction methods are often limited to particular object classes or surface types and output depth maps or voxel grids converted to meshes.Predicting CAD operations instead allows combinations that produce precise, editable parametric shapes across different CAD models.Source: §2 · p. 3
- Earlier online systems focus on specific operations, multiple orthographic views, custom gestures, or hand-crafted stroke rules, whereas Sketch2CAD recognizes operations from freehand sketches in one perspective view.It aims to let users draw complete shape parts without learning a new gesture vocabulary.Source: §2 · pp. 3–4
- Unlike prior context-based sketching systems that use a scene to guide drawing or infer geometric constraints, Sketch2CAD uses the existing scene to recognize an operation intended to modify it.Its networks handle operations with varying parameter counts in any order by using sketch context.Source: §2 · p. 4
3 METHOD OVERVIEW
The method formulates sketch-based CAD modeling as mapping an existing shape and sketch to a parameterized operation. It uses learned operation classification and segmentation to narrow the inverse problem before fitting operation parameters.
3 METHOD OVERVIEW
- A CAD operator maps an input model to a changed shape, with its parameter vector specifying both the parts modified and the modification parameters.The implementation represents models by their boundary triangle meshes.Source: §3 · p. 4
- Given the current shape, sketch curves, and viewpoint, the system seeks an operator whose result projects to match the sketch.The approach uses orthogonal 3D-to-2D projection.Source: §3 · p. 4
- A two-stage neural network predicts the operator type and segments sketch and context maps, after which parameters are fitted to instantiate the operation.This learned approach avoids exhaustive enumeration and inefficient traditional optimization over the diverse operator and parameter space.Source: §3 · p. 4
4 THE OPERATORS
Sketch2CAD formalizes four common CAD operations as parameterized edits specified by sketches over an existing shape. Their parameters and sketches define how each operation modifies the model, and the resulting steps can be saved as reusable modeling protocols.
4 THE OPERATORS
- The system supports face extrusion, beveling, right-polyhedron addition or subtraction, and cylindrical sweeping, chosen for use in sketching and CAD workflows.The four operations can be interleaved to generate complex shapes.Source: §4 · p. 4, Figure 4 · p. 5
- Extrusion offsets a planar face along its normal, while beveling rounds a sharp crease using a profile curve.Extrusion distance can be positive or negative; the bevel profile is specified on the base face.Source: §4 · p. 5
- Add/subtract combines a prism with the base shape by union or difference, while sweep combines a swept cylindrical shape in either way.The sweep is parameterized by two circular rails and a profile curve; its profile is restricted to circular cross sections only in later regression details, not in this section.Source: §4 · p. 5
- New operators can be added by defining minimal but complete parameters and concise sketches that capture the operation’s important features.These design choices affect the machine-learning models used to recover operator instances.Source: §4 · p. 5
- A protocol serializes the sequence of operators and their parameters, and can be saved, loaded, edited, and reused.The sequence specifies the steps used to obtain the final shape.Source: §4 · p. 5
- The implementation represents solids as triangle meshes while maintaining planar polygonal faces, and applies planar-base operations using Boolean operations.For beveling, it constructs and subtracts a prism; curved-face operators are identified as a possible future extension using representations such as NURBS.Source: §4 · pp. 5–6
5 CONTEXT DRIVEN SKETCH INTERPRETATION
Sketch interpretation uses the current model as context to recognize an operation, segment its defining sketch regions, and recover parameters. The approach combines neural classification and segmentation with operation-specific fitting, including ordered curves to resolve add/subtract ambiguity.
5 CONTEXT DRIVEN SKETCH INTERPRETATION
- Each sketch step is interpreted through operator recognition, segmentation of input regions, and parameter recovery before the inferred operation is applied to the existing geometry.The networks receive the sketch together with depth and normal maps rendered from the existing model at the sketched viewpoint.Source: §5 · p. 6
5.1 Operator classification
- The classification CNN predicts the operator type from the sketch and context maps, using weighted cross entropy with inverse-frequency class weights to address operation-frequency imbalance.The input maps are the sketch, depth, and normal maps.Source: §5.1 · p. 6, §5 · p. 6
5.2 Operator regression
- Regression first segments sketches and context maps, then fits operation parameters to detected regions, a design the authors say is more tractable and robust to prediction inaccuracies than direct parameter regression.Segmentation U-Nets output a base-face probability map and a curve-segmentation map; the training loss combines face-map and masked curve-map errors.Source: §5.2 · p. 6
- For extrusion, the system identifies offset curves and searches for the distance that best matches them; for beveling, it uses the segmented profile curves to recover the bevel geometry.The extrusion search includes both positive and negative distances.Source: §5.2 · pp. 6–7
- For add/subtract, regression orders curves along the face normal to avoid ambiguity between base and offset curves, then fits the curves and recovers the union-or-difference choice and prism length.The system treats a case where neither fitted polygon intersects the base face as an erroneous sketch; in the ambiguous case it defaults to addition, which the user can switch manually.Source: §5.2 · p. 7, Figure 5 · p. 7
- Sweep regression reuses the add/subtract curve maps but restricts sweeps to circular cross sections, fitting circular ends and a profile curve to recover the swept shape.The profile curve is fitted in a plane determined from the circles and their intersection points.Source: §5.2 · p. 7
6 TRAINING DATA GENERATION
Sketch2CAD trains its networks on synthetic CAD-operation sequences rendered as sketches and context maps. The data design samples varied local operation contexts and sketch views to cover diverse geometry and drawing variation.
Training data generation
- Training examples pair rendered sketches and local context maps with ground-truth face and curve segmentation maps for the classifier and segmentation network.For each protocol, the sequence is executed through its last operation, whose sketch curves are overlaid on the built shape.Source: §6 · p. 7, Figure 6 · p. 8
Sequence generation
- Enumerating up to three preceding random operations supplies varied local contexts without enumerating all possible models and operation configurations.The approach relies on local context as the dominant cue for interpreting the geometry of an operation's sketch.Source: §6 · p. 8
- 10k protocols generate 40k training shapes, with 10k additional protocols using the same distribution reserved for testing.Protocols span sequence lengths of one to four operations; the final operation types are sampled at a 1:1:4:2 ratio for extrude, bevel, add/subtract, and sweep.Source: §6 · p. 8, Figure 6 · p. 8
Procedurally generated training set
- Random combinations of operations produce complex shapes within four steps in the procedural training set.The examples show synthesized shapes alongside the next-step sketch.Source: Figure 7 · p. 8
Sequence generation
- Operation instances vary through randomized face selection, offsets, primitive placement, and perturbed non-self-intersecting polygons and profiles.Sequences start with a base box and are randomized to cover geometric variation while avoiding degeneracy.Source: §6 · p. 8
Sketch rendering
- Rendered views vary camera angle and zoom, exclude heavily occluded key curves, and perturb and smooth projected curves to mimic rough freehand sketches.View directions are 20°–80° from the base-face normal, and views are filtered when key curves are over 20% occluded.Source: §6 · p. 8
7 THE MODELING SYSTEM
The prototype supports interactive sketching over a 3D model, with recovered operations and parameters available for review and editing. Its tools include curve regularization, parameter tuning, and symmetry-based completion; users completed reference-shape tasks in 5 to 20 minutes.
Prototype tool
- The prototype provides freeform 2D sketching with instant feedback in a freely viewable 3D model.Users can save, load, replay, and edit operation sequences; the interface also supports regularization, parameter tuning, and symmetry-based completion.Source: §7 · p. 9
Regularization
- Curve snapping and rectification align sketch features with model landmarks and enforce approximate parallelism, equal angles, and equal edge lengths.Users can switch off auto-regularization to sketch arbitrary shapes.Source: §7 · p. 9
User gallery
- All six participants reproduced three reference shapes in 5 to 20 minutes and achieved close matches.The user gallery shows reference shapes in green.Source: Figure 11 · p. 9
Tuning operation parameters
- Users can change Boolean choices for ambiguous add/subtract and sweep results and fine-tune geometric parameters such as offsets and circle radii.For swept shapes, profile control points are adjusted proportionally to preserve the overall shape as much as possible.Source: §7 · p. 9, Figure 10 · p. 9
Auto-completion by symmetry
- Symmetry-based completion reflects a sketched primitive and its Boolean operation across a selected oriented-bounding-box cross-section plane.This avoids manually repeating the sketch.Source: §7 · pp. 9, 11
Selected user modeling steps
- Users followed different operation paths and variations toward similar targets, and inaccurate freehand sketches were translated into intended operations.The selected modeling steps illustrate this variation in user workflows.Source: Figure 13 · p. 11
8 RESULTS AND DISCUSSION
The evaluation reports that novices quickly produced models close to reference shapes, while also exploring variations. Ablations, robustness tests, and stated limitations qualify the system’s performance and scope.
8 RESULTS AND DISCUSSION
- The authors sketched models with operation sequences ranging from 2 to 11 steps, spanning simple bolts and nuts to sophisticated mixers and cameras; user evaluation also confirmed ease of sketching CAD models.Source: §8 · p. 11
8.1 User evaluation
- All participants quickly produced models close to the references, and several also explored variations such as adding a camera-lens part or curving a hammer handle.Participants rated sketch translation 4.75/5 and ease of conceiving modeling sequences 5/5; all drew with a mouse, and none disabled the stroke regularizer.Source: §8.1 · p. 11
8.2 Ablation study
- The full network achieved the best accuracy across segmentation tasks; without context, base-face IoU was frequently below 10%, and primitive-only training lowered base-face IoU by more than 10%.In the bevel example, the no-context network produced no output above threshold, while the primitive network confused the front-facing polygons; the full network produced near-perfect maps.Source: Figure 14 · p. 11, Table 1 · p. 12, §8.2 · p. 12
8.3 Robustness of network predictions
- The reported figures were 85.06% for level 2 and 53.20% for level 3; segmentation maps remained high-quality under moderate noise, while high noise degraded them, though fitting still yielded a reasonable shape.Source: §8.3 · p. 12
8.4 Limitations and Future Work
- The current system does not support drawing primitives on curved faces, and building small features such as knobs or screw threads from scratch can be cumbersome.The authors suggest NURBS primitives as one possible curved-face extension and a library of small part features as a possible aid.Source: §8.4 · p. 12
- Training data samples geometric feasibility rather than semantic validity, so some operation combinations are not functionally meaningful; more realistic distributions and CAD traces are future training targets.The authors also identify a need for some sketching ability and suggest that a tablet interface could lower this entry barrier.Source: §8.4 · pp. 12–13
9 CONCLUSION
The conclusion frames sketching and CAD modeling as similar workflows expressed in different languages. Sketch2CAD parameterizes common operations and uses a neural network to turn approximate sketches into regular CAD models.
9 CONCLUSION
- Industrial design sketching and CAD modeling both build complex shapes through sequences of simple operations, despite their different modes of expression.The paper describes sketching as visceral and approximate, in contrast with the perceived tedium and rigidity of 3D modeling.Source: §9 · p. 13
- Sketch2CAD identifies and parameterizes shared operations, then uses a deep neural network to recognize and segment them in sketches.The conclusion reports that novice evaluation and the diversity of created shapes illustrate the system’s conversion of approximate sketches into regular CAD models.Source: §9 · p. 13
- The system is presented as an interactive way to turn approximate sketches of human-made objects into regular CAD models.The conclusion grounds this scope in novice evaluation and the range of shapes created with the approach.Source: §9 · p. 13