PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamCAD: Scaling Multi-modal CAD Generation using Differentiable Parametric Surfaces

Mohammad Sadil Khan, Muhammad Usama, Rolandos Alexandros Potamias, Didier Stricker, Muhammad Zeshan Afzal, Jiankang Deng, Ismail Elezi

arXiv:2603.05607v2cs.CVcs.AI

TL;DR

Existing CAD generation is constrained by small annotated datasets and costly BRep labels, leaving millions of unannotated meshes underused. DreamCAD learns editable parametric surfaces from point-level supervision and introduces CADCap-1M with over 1M GPT-5-generated descriptions. It achieves state-of-the-art results across point-, image-, and text-conditioned tasks, while complete CAD topology remains an open challenge.

  • Problem

    Existing CAD generation methods rely on small design-history datasets or costly explicit BRep annotations, limiting scalable use of unannotated 3D meshes.

  • Method

    DreamCAD represents CAD models as rational Bézier patches, differentiably tessellates them into meshes for point-based supervision, and uses a coarse-to-fine multimodal generation framework.

  • Results

    DreamCAD achieves state-of-the-art performance across point-, image-, and text-conditioned generation, reducing Chamfer Distance by up to 70% for point-to-CAD and exceeding 75% preference in text- and image-to-CAD evaluations.

  • Takeaways & Limitations

    CADCap-1M and point-supervised parametric surfaces support scalable multimodal CAD geometry generation without CAD-specific ground-truth annotations.

  • Takeaways & Limitations

    DreamCAD outputs control points and weights but lack complete CAD topology, which the paper leaves for future work.

Abstract

from arXiv · show

Computer-Aided Design (CAD) relies on structured and editable geometric representations, yet existing generative methods are constrained by small annotated datasets with explicit design histories or boundary representation (BRep) labels. Meanwhile, millions of unannotated 3D meshes remain untapped, limiting progress in scalable CAD generation. To address this, we propose DreamCAD, a multi-modal generative framework that directly produces editable BReps from point-level supervision, without CAD-specific annotations. DreamCAD represents each BRep as a set of parametric patches (e.g., Bézier surfaces) and uses a differentiable tessellation method to generate meshes. This enables large-scale training on 3D datasets while reconstructing connected and editable surfaces. Furthermore, we introduce CADCap-1M, the largest CAD captioning dataset to date, with 1M+ descriptions generated using GPT-5 for advancing text-to-CAD research. DreamCAD achieves state-of-the-art performance on ABC and Objaverse benchmarks across text, image, and point modalities, improving geometric fidelity and surpassing 75% user preference. Code and dataset will be publicly available.

1 Introduction

DreamCAD targets scalable multimodal CAD generation by learning editable parametric surfaces from point supervision rather than CAD-specific annotations. It pairs this framework with CADCap-1M and reports strong results across point, image, and text-conditioned generation.

  • Motivation: Existing CAD generation methods struggle with generalization because design-history models use small datasets, while UV and graph-based methods require costly BRep annotations.Large-scale ABC BReps remain underutilized because these annotation and representation bottlenecks are difficult to scale.
  • Motivation: DreamCAD uses a decoupled pipeline that first learns scalable parametric geometry from unstructured 3D data before future stages recover complete CAD topology.The proposed paradigm separates geometry generation from topology generation.
  • Method: DreamCAD produces editable rational Bézier surfaces from point-level supervision by differentiably tessellating parametric patches into meshes optimized with Chamfer loss.The framework encodes sparse voxels into structured latents, which are decoded into parametric geometry.
  • Method: C0 continuity is enforced structurally by mapping flood-filled sparse-voxel quads to patches whose adjacent boundary points are shared.The VAE decoder then refines control points and weights to match target geometry without relying on geometric continuity optimization.
  • Data: CADCap-1M provides over 1M GPT-5-generated text descriptions, while DreamCAD is trained on over 1M meshes curated from 10 public datasets.The paper evaluates text-, image-, and point-conditioned generation on ABC and Objaverse.
  • Results: Up to 70% lower Chamfer Distance is reported for point-to-CAD, alongside more than 75% preference in expert and GPT-based text- and image-to-CAD evaluations.The paper describes these results as state of the art across point-, image-, and text-conditioned generation tasks.

2 Related Work

Prior CAD generation methods use exact, editable BRep geometry but face scalability constraints from complex topology, limited design-history data, and missing multimodal descriptions. CADCap-1M addresses the captioning gap with over 1M GPT-5-generated descriptions for CAD models.

  • Generative CAD: BRep-based CAD representations provide exact, watertight, editable geometry but are difficult to learn at scale because of complex parametric and topological structure.This creates a tension between scalability and geometric fidelity.
  • Generative CAD: Design-history approaches model CAD creation through sketch-extrude operations but depend on proprietary or scarce design-history logs and small datasets such as Text2CAD-160K.These constraints limit the scale and diversity available for multimodal generation.
  • CAD Captioning: ABC-1M and Automate-440K provide large CAD collections but lack textual or visual descriptions, restricting their use for multimodal generative learning.Existing mesh-captioning resources do not provide a comparable resource for BRep models.
  • CAD Captioning: CADCap-1M supplies over 1M high-quality GPT-5-generated descriptions for scalable CAD text-to-CAD training and evaluation.The dataset is introduced to bridge the gap between large CAD collections and multimodal learning requirements.

3 Preliminaries

DreamCAD represents CAD geometry with rational Bézier surfaces and converts patches into connected meshes through differentiable tessellation. The formulation supports gradient-based optimization with respect to surface control points and weights.

  • Bézier Surface Representation: DreamCAD uses bicubic rational Bézier surfaces, each defined by a grid of control points and non-negative weights.Rational Bézier surfaces are selected for conceptual simplicity, analytical tractability, and compatibility with standard CAD operations.
  • Bézier Surface Representation: The Bézier surface is evaluated over the uv domain using Bernstein basis functions, with bicubic surfaces using n = m = 3.The formulation is differentiable with respect to both control points and weights, whose values must remain non-negative.
  • Differentiable Mesh Generation: Differentiable tessellation samples each Bézier patch on an r × r uv grid, splits grid cells into triangles, and merges neighboring patches along shared boundaries.This produces a locally consistent mesh while preserving C0 connectivity.
  • DreamCAD Overview: DreamCAD’s pipeline encodes meshes into structured latents, decodes parametric surfaces, and supports coarse-to-fine multimodal generation from images or points.The overview includes sparse-voxel processing, flood-fill initialization, and flow-matching-based refinement.

4 Methodology

DreamCAD uses structured sparse-voxel latents and a coarse-to-fine conditional pipeline to generate connected parametric CAD surfaces from multiple modalities. Its differentiable surface representation and CADCap-1M captioning pipeline support scalable training and text-to-CAD generation.

  • Latent Encoding: DreamCAD encodes voxelized 3D shapes into structured latent features before decoding them into parametric surfaces.Sparse voxels are processed by a sparse Transformer, while RGB, normal, coordinate, and signed-distance cues augment voxel features.
  • Parametric Surface Decoding: Each initial parametric quad surface is refined into Bézier patches whose control points and weights define the reconstructed geometry.The decoder predicts local deformations and weight updates for each patch from structured latent features.
  • Parametric Surface Decoding: Shared boundary points are averaged across adjacent patches to enforce C0 continuity during surface refinement.Positive weights are enforced with softplus, while local deformation bounds help avoid degenerate geometries such as spikes or self-intersections.
  • Differentiable Mesh Generation: The tessellated surface is trained against target point clouds with Chamfer distance alongside tangent-continuity, smoothness, and latent-regularization terms.The differentiable tessellation samples each patch on a grid, forms triangles, merges shared boundaries, and supports end-to-end optimization.
  • CADCap-1M Dataset: CADCap-1M provides over 1M metadata-augmented CAD captions generated from rendered views using GPT-5.Prompts include model names, hole counts, and relative dimensions to produce shape-centric, structure-aware descriptions.
  • Conditional Generation: DreamCAD generates CAD models through two stages: coarse voxel structure first, followed by local SLAT features and parametric surface reconstruction.Image and point-cloud conditions use DINOv2 and PointNet++ encoders, while text conditioning uses text-to-image followed by image-to-CAD.

5 Experiments

DreamCAD is evaluated on multimodal CAD generation, caption quality, architectural choices, and topology recovery across ABC and Objaverse. Results report strong performance, while ablations identify the importance of regularization, initialization, and coarse-to-fine generation.

  • Multimodal Generation Evaluation: DreamCAD is evaluated on point-, image-, and text-conditioned CAD generation using ABC and Objaverse, with geometric and perceptual metrics.Objaverse serves as an out-of-distribution evaluation set, while ABC is the in-distribution benchmark.
  • Multimodal Generation Evaluation: DreamCAD achieves state-of-the-art results across all three modalities, including zero invalid outputs in the DeepCAD comparison.On ABC point-to-CAD, it reduces Chamfer Distance by up to 68% and 75% versus Cadrille and CAD-Recode, respectively.
  • Multimodal Generation Evaluation: Over 75% of evaluators prefer DreamCAD for image-to-CAD outputs on both ABC and Objaverse.The image-to-CAD comparison also reports Chamfer Distance improvements of 80% and 58% over BRepDiff on ABC and Objaverse, respectively.
  • Caption Quality: 95.8% of user ratings and 98.31% of GPT-5 ratings judge CADCap-1M captions correct.Evaluators assess geometric and semantic accuracy from four rendered views, metadata, and captions.
  • Ablation Study: Combining G1 and Laplacian regularizers yields the lowest Laplacian loss, 0.0020, while maintaining strong geometric accuracy and smooth surfaces.Without regularization, Chamfer Distance is minimized aggressively but spike artifacts and rough surfaces appear.
  • Ablation Study: Accurate coarse geometry is essential: extreme voxel-grid noise increases Point2CAD Chamfer Distance by 385%, while extreme initialization noise increases it by 91%.The unperturbed coarse structure achieves CD = 0.93× 10^-3, supporting the coarse-to-fine design.
  • Application: CAD Topology Recovery: A topology-recovery feasibility study converts DreamCAD patch outputs into structured NURBS representations, producing 99.2% valid CAD models with CD = 0.17 × 10^-3.Qwen3-4B with LoRA is fine-tuned on 50K samples and evaluated on 600 samples across point-, image-, and text-to-CAD.

6 Conclusion

The conclusion presents DreamCAD as a scalable framework for generating parametric CAD surfaces from point-level supervision and CADCap-1M as a large captioning resource. It frames complete CAD topology recovery as an unresolved next stage.

  • Conclusion: DreamCAD generates parametric surfaces from point-level supervision using a patch-based representation and differentiable mesh generation.This removes the need for CAD-specific ground-truth annotations and supports training on large-scale 3D datasets.
  • Conclusion: CADCap-1M provides over 1M GPT-5-generated CAD descriptions for text-to-CAD research.The dataset accompanies DreamCAD’s multimodal generation framework.
  • Conclusion: DreamCAD demonstrates strong generalization across text-, image-, and point-conditioned generation tasks.The conclusion states this result for models trained on 1M+ 3D meshes curated from 10 public datasets.
  • Conclusion: Complete CAD topology recovery remains a hard and open problem that DreamCAD treats as a subsequent stage beyond geometric reconstruction.The paper positions DreamCAD’s generated geometry as a foundation for this future stage.

1 More on Data Preparation

The data-preparation pipeline renders multi-view visual features for sparse-voxel processing and text-to-image training. It uses several camera trajectories and randomized diffuse colors for textureless meshes.

  • Visual Feature Generation: Meshes are normalized to [−0.5, 0.5]3 before voxelization and rendered from 150 multi-view images.The views are generated using azimuth, elevation, and uniform-hemisphere camera trajectories.
  • Camera Trajectories: The azimuth sweep renders 50 images at fixed elevation ϕ = 30° while rotating the camera around each object.This is one of three complementary trajectories used for full object coverage.
  • Camera Trajectories: The elevation sweep renders another 50 images with fixed azimuth θ = 30° and varying elevation.Together with the other trajectories, it contributes to full coverage of the object.
  • Camera Trajectories: The final 50 views uniformly sample azimuth from [0, 2π] and elevation from [−π/2, π/2].All trajectories use a 40° field of view and camera radius r = 1.5.
  • Rendering: Textureless meshes receive randomly sampled diffuse RGB colors, while dark palette colors are used during Stable-Diffusion 3.5 fine-tuning.The rendering pipeline uses Kaolin’s CUDA-based renderer at 520×520 resolution.

2 More on CADCap-1M

CADCap-1M combines large-scale GPT-5 captioning with metadata augmentation to describe CAD meshes in detail. The resulting captions exhibit broad vocabulary diversity and improved specificity for visually similar parts.

  • Dataset Construction: GPT-5 processes over 1M CAD samples using metadata augmentation, with part names available in 20%, 39%, and 46% of ABC, Automate, and Fusion360 samples.The batch API processes the data over three weeks at a total cost of $1800.
  • Dataset Statistics: CADCap-1M captions average under 20 words and contain over 21k unigrams, 446k bigrams, and 2.3M trigrams.These statistics indicate substantial linguistic variation across the dataset.
  • Caption Content: Captions describe complex shapes with detailed part characteristics, including fin morphology and hole counts.Examples include a stem-shaped connector with six leaf-like fins and an end sheet with 39 circular holes.
  • Metadata Augmentation: Metadata inclusion improves specificity by distinguishing visually similar ring-like structures through their part names.The examples identify Valve Stem Washer, LV Bushing Washer, and Inductive Sensor Washer as distinct components.

3 More Experimental Results

Additional experiments report DreamCAD’s component settings, multimodal qualitative results, comparisons on DeepCAD, and observed failure modes. The results show strong performance under limited-data conditions, while revealing detail-loss and invalid-output cases.

  • Training Details: DreamCAD’s VAE training initializes deformation predictions near zero and control-point weights near one to reduce early surface artifacts.The deformation vector uses N(0, 10^-5) initialization, while the control-point weight layer is zero-initialized with bias 1.
  • Training Details: DreamCAD uses 12 Transformer decoder layers for both coarse and fine-grained Flow Transformers.The models use a logit-normal time schedule and AdamW optimization with learning rate 5 × 10^-5.
  • Comparison on DeepCAD Test Set: DreamCAD outperforms all baselines across point-, image-, and text-to-CAD tasks on DeepCAD with zero invalid outputs.The comparison uses identical training settings and isolates architectural contributions from data scale.
  • Failure Cases: DreamCAD can lose fine geometric details when sparse voxel resolution is insufficient and can produce invalid outputs when ambiguous images activate no coarse-stage voxels.The paper suggests octree-based representations as a future approach for finer geometric capture.

4 More Ablation Studies

The ablations examine flood-fill initialization and coarse-to-fine generation as geometric priors for parametric-surface reconstruction. Both components tolerate small perturbations but degrade sharply under extreme noise.

  • Impact of Initial Parametric Surface: Unperturbed flood-fill initialization achieves CD = 0.034 on 15K ABC test samples.Small perturbations of σ = 10^-4 increase CD by 15%, whereas σ = 10^-2 increases CD by 91%.
  • Importance of Coarse-to-Fine Generation: The VAE decoder generates coarse voxels first and fine-grained per-voxel latent features afterward to produce parametric surfaces.This sequential design supplies the structure and local features required by the decoder.
  • Importance of Coarse-to-Fine Generation: Unperturbed coarse structure achieves CD = 0.93 × 10^-3 for Point-to-CAD reconstruction on 15K ABC test samples.Noise with σ = 10^-4 raises CD by 8.6%, while σ = 10^-2 raises it by 385%, confirming the sensitivity of refinement to coarse geometry.

5 Disscussion on Future Research Directions

DreamCAD’s patch-based surfaces are presented as a tractable foundation rather than the final form of CAD topology recovery. Future work should build from this geometric foundation toward full topology recovery and production-level CAD models.

  • Discussion on Future Research Directions: DreamCAD focuses on accurate geometric reconstruction from large-scale unstructured 3D data without CAD-specific annotations.The paper frames this as an early but challenging stage of scalable CAD generation.
  • Discussion on Future Research Directions: Patch-based parametric surfaces are treated as a necessary foundation, not the final goal, because full CAD topology recovery remains challenging.The paper distinguishes recovering geometry from jointly generating geometry and topology at scale.
  • Discussion on Future Research Directions: DreamCAD outputs can be converted toward production-level CAD models, while future work is needed for full CAD topology recovery.The discussion points to topology recovery as a subsequent research direction building on the generated surfaces.
  • Additional Qualitative Results: The qualitative results cover captions for complex and simple parts, fasteners, letter-based and engraving-style models, and multimodal CAD generations.Additional examples span mechanical components, furniture, utensils, consumer objects, and free-form designs.
Loading 2603.05607v2…