PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Human Intent-to-CAD: Parametric CAD Model Generation via Cooperative Multi-Task Alignment and Spatial-Aware Reinforcement Learning

Qingwang Zhang, Jiahao Li, Xiangdong Zhou

arXiv:2610.15681v1cs.CVcs.AI

TL;DR

The modalities are highly complementary in terms of information density and geometric precision. HiCAD constructs an aligned sketch, text, and CAD-code dataset and uses Cooperative Multi-Task Alignment followed by Spatial-Aware Reinforcement Learning. HiCAD outperforms existing baselines across multiple tasks and reports superior geometric fidelity.

  • Problem

    The modalities are highly complementary in terms of information density and geometric precision.

    Source: §1 · p. 1

  • Method

    HiCAD constructs an aligned sketch, text, and CAD-code dataset and uses Cooperative Multi-Task Alignment followed by Spatial-Aware Reinforcement Learning.

    Source: §1 · p. 2

  • Results

    HiCAD outperforms existing baselines across multiple tasks and reports superior geometric fidelity.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The HiCAD dataset provides a unified benchmark for studying human intent-to-CAD transformation under realistic design scenarios.

    Source: §6 · p. 9

    Generated results may contain geometric deviations or logical errors, so practical engineering use should include human review and constraint verification.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Parametric Computer-Aided-Design (CAD) modeling from human intent remains challenging, particularly during the conceptual design stage, where design goals are expressed through incomplete and unstructured modalities (e.g., hand-drawn sketches and textual descriptions). In this work, we rethink the human intent-to-CAD pipeline and propose a unified method that directly maps multi-level human intents to executable codes, without assuming the prior existence of target CAD models. To support our study, we construct HiCAD, the first large-scale dataset aligning hand-drawn sketches, textual descriptions, and parametric CAD codes. Based on this, we introduce HiCAD, a two-stage framework comprising Cooperative Multi-Task Alignment to bridge the representational gap between heterogeneous inputs, and Spatial-Aware Reinforcement Learning to enforce geometric and topological consistency. Extensive experiments demonstrate that our method significantly outperforms existing baselines across multiple tasks, validating its effectiveness and robustness in transforming heterogeneous human intents into high-fidelity parametric CAD models. Our project page: https://zqwlearning.github.io/HiCAD.

1. Introduction

The paper targets CAD generation from incomplete, unstructured human intent at the conceptual design stage, where target CAD models may not yet exist. It introduces a dataset and two-stage framework for mapping sketches and text to parametric CAD code.

1. Introduction

  • The paper identifies representational gaps between sketches and text, CAD code sensitivity to syntax and topology, and sparse feedback for 3D geometric fidelity.Source: §1 · pp. 1–2
  • HiCAD combines Cooperative Multi-Task Alignment for intent–code alignment with Spatial-Aware Reinforcement Learning for geometric and topological consistency.The framework coordinates tasks to promote positive transfer, then uses a spatial-aware reward for reinforcement learning.Source: §1 · p. 2
  • The proposed four tasks generate parametric CAD models from combinations of hand-drawn sketches and textual descriptions, without assuming an existing target model.The setting focuses on conceptual design and creating designs from scratch.Source: §1 · pp. 1–2
  • The HiCAD dataset aligns hand-drawn sketches, textual descriptions, and parametric CAD code as a foundation for studying intent-to-CAD mapping.Source: §1 · p. 2
  • Experiments show that HiCAD outperforms existing baselines across multiple tasks.Source: §1 · p. 2

2. Related Work

Prior work represents CAD through parametric command sequences and executable CadQuery, while learning-based approaches use supervised fine-tuning or reinforcement learning to generate CAD.

2.1. Parametric CAD Sequence Modeling

  • Parametric command sequences support incremental, sketch-driven modeling and produce editable, executable programs; CadQuery is adopted as the paper’s expression format.The cited work describes CadQuery as expressive and supportive of industrial standards.Source: §2.1 · p. 2

2.2. LLM and VLM for CAD

  • SFT-based CAD models learn grammar and basic geometric patterns but can struggle with geometric consistency and topological validity under token-level objectives.The related-work discussion attributes this limitation to failure to capture global structural constraints and precise design intent.Source: §2.2 · p. 2
  • Prior RL methods use reconstruction quality, code inspection, or GRPO, while the paper proposes GSPO-based spatial-aware RL to address training instability and reward-design challenges.The discussion links instability in existing RL frameworks to vanilla token-level importance sampling.Source: §2.2 · p. 3

3. Building HiCAD Dataset

HiCAD expands DeepCAD into a dataset aligning executable CadQuery code with hand-drawn sketches and textual descriptions, addressing the lack of a publicly available dataset covering all three modalities. Its construction combines synthetic and volunteer-drawn sketches, generated descriptions, duplicate removal, and a DeepCAD-compatible split.

3. Building HiCAD Dataset

  • HiCAD restructures and expands DeepCAD into a dataset whose samples pair executable CadQuery code with a hand-drawn-style sketch and a textual description.The authors report that no publicly available dataset covered all three modalities simultaneously.Source: §3 · p. 3

3.1. CAD Model Represented by CadQuery

  • CadQuery is HiCAD's unified CAD representation, using code created for DeepCAD by the CAD-Recode project.The paper cites its modularity and interpretability, and the availability of public CadQuery datasets, as reasons for this choice.Source: §3.1 · p. 3

3.2. Hand-Drawn Sketch

  • Sketch data include synthetic sketches generated from CAD models and volunteer drawings of 100 randomly selected test-set models.Synthetic sketches retain visible contours without rendering hidden lines, a choice intended to reduce drawing burden and better reflect real sketching behavior.Source: §3.2 · p. 3

3.3. Textual Description

  • Complete textual descriptions are generated with a VLM from CadQuery code, a rendered model image, and geometric attributes, and describe geometric rather than color or material properties.Partial descriptions are made by randomly removing sentences from complete descriptions; the remaining sentences are concatenated in original order to simulate incomplete user input.Source: §3.3 · pp. 3–4

3.4. Dataset Integration and Splitting

  • 82,075 training, 4,536 validation, and 4,015 test samples make up HiCAD's split, following DeepCAD for comparison with prior work.Samples with sketch-feature similarity scores exceeding 0.99 were removed as duplicates.Source: §3.4 · p. 4

4. Methodology

HiCAD maps sketch and text intents to executable CadQuery code through Cooperative Multi-Task Alignment (CMTA) followed by Spatial-Aware Reinforcement Learning (SARL). CMTA aligns multiple intent tasks, while SARL uses spatial rewards and sequence-level policy updates to refine generated CAD models.

4.1. Problem Formulation

  • The task maps a multimodal query to a CadQuery Python script that generates a parametric B-Rep or 3D mesh when executed.The trainable policy is represented as πθ(q) → C.Source: §4.1 · p. 4

4.2. HiCAD Framework

  • The HiCAD framework first aligns a pre-trained VLM across tasks with CMTA, then initializes a policy for SARL rollouts and spatial-reward updates via GSPO.The framework accepts sketches, text, or both and outputs executable CadQuery code.Source: §4.2 · p. 4, Figure 4 · p. 5

4.2.1. COOPERATIVE MULTI-TASK ALIGNMENT

  • CMTA trains on four input forms—sketch, complete text, sketch with partial text, and sketch with complete text—and interleaves their data to support cross-task alignment.These tasks span unimodal and multimodal inputs with varying information density.Source: §4.2.1 · p. 5

4.2.2. SPATIAL-AWARE REINFORCEMENT LEARNING

  • SARL evaluates executed CAD outputs for topology and geometry: face-adjacency graph edit distance measures topology, while 3D intersection over union measures geometric overlap.The geometric reward uses IoU directly to provide a continuous signal.Source: §4.2.2 · p. 6
  • SARL assigns a fixed penalty to invalid code; valid code receives a linear combination of topological and geometric rewards.The validity check executes the code and tests whether it generates a feasible CAD model.Source: §4.2.2 · p. 6
  • SARL uses group sequence policy optimization with sequence-level importance ratios and clipping to align reward assignment with updates for long CadQuery code generation.The shared sequence-level weight is intended to mitigate instability from variable token-level ratios.Source: §4.2.2 · p. 6

5. Experiment

HiCAD outperforms baselines across tasks, while ablations report gains from cooperative multi-task alignment and further improvements from spatial-aware reinforcement learning.

5.2. Performance Comparison

  • 9.41% to 0.37% IR on HDS-to-CAD: HiCAD reduces invalid outputs versus the strongest baseline and outperforms all baselines across every metric and task.The evaluation covers four human-intent tasks.Source: Table 1 · p. 7, §5.2 · p. 8
  • Qualitatively, general VLMs produce topological errors and Cadrille misses fine details or complex Boolean operations, while HiCAD better preserves intended features.The comparison spans four tasks.Source: Figure 5 · p. 7, §5.2 · p. 8

5.1. Experimental Setup

  • Evaluation covers four tasks and compares 2-shot general-purpose VLMs with CAD-specific models trained on HiCAD, using IoU, Chamfer Distance, and Invalid Rate under the Cadrille protocol.The tasks are HDS-to-CAD, CTD-to-CAD, HDS & PTD-to-CAD, and HDS & CTD-to-CAD.Source: §5.2 · p. 7, §5.1 · p. 7

5.3. Ablation Study

  • 66.61% to 72.39% IoU on HDS & PTD-to-CAD: CMTA improves over independent single-task fine-tuning across all tasks.The authors connect joint training to a more robust shared CAD-geometry representation and better cross-modal generalization for incomplete inputs.Source: Table 2 · p. 8, §5.3 · p. 8
  • Across tasks, SARL brings Invalid Rate to approximately 0.2%–0.4% while increasing IoU; CMTA and SARL together achieve the framework’s optimal performance.The authors describe SARL as rewarding spatial accuracy and valid CAD command sequences.Source: Table 2 · p. 8, §5.3 · pp. 8–9

5.2. Performance Comparison

  • As inputs expand from single-modal to multimodal, HiCAD’s performance improves, while competing methods fluctuate across modality changes.The authors attribute the pattern to complementary sketch–text fusion and report instability for general-purpose VLMs and CAD-specific methods.Source: §5.2 · p. 8
  • On real volunteer sketches, HiCAD achieves the best IoU with a low invalid rate, and adding textual descriptions consistently improves performance.The reported comparison covers HDS-to-CAD and HDS & CTD-to-CAD.Source: Table 3 · p. 8, §5.2 · p. 8

5.3. Ablation Study

  • Joint SARL training across multiple tasks substantially outperforms applying SARL to individual tasks across all evaluation metrics.Individual-task SARL yields varying degrees of improvement.Source: Figure 6 · p. 9, §5.3 · p. 9

6. Conclusion

The paper presents HiCAD as an intent-driven approach to parametric CAD generation for conceptual design. Its experiments report improved geometric fidelity and logical consistency across multiple intent configurations.

6. Conclusion

  • HiCAD integrates cooperative multi-task alignment and spatial-aware reinforcement learning to generate consistent CAD code from human intent, and its dataset provides a unified benchmark for this task.The conclusion frames the work as shifting from reconstruction to intent-driven generation using sketches and text with varying completeness.Source: §6 · p. 9

Impact Statement

The impact statement situates the work in conceptual-design CAD modeling and cautions that generated results require review and constraint verification. The appendix describes dataset and text-prompt resources.

Impact Statement

  • Generated models may contain geometric deviations or logical errors, so practical engineering use should include human review and constraint verification.The statement also notes potential misuse in designing harmful or regulated objects.Source: Impact Statement · p. 9

Appendix

  • The appendix identifies its materials as resources for understanding the main paper.It lists the HiCAD dataset among the appendix items.Source: Appendix · p. 13

A.1. Hand-Drawn Sketch

  • Synthetic hand-drawn sketches are generated by exporting a CAD model’s B-rep, extracting visible edges, and projecting them onto a fixed-size canvas; human and synthetic sketches use 512×512 inputs.The process uses CadQuery and OpenCascade.Source: §A.1 · p. 13

A.2. Textual Description

  • The complete prompt template for text-description generation is provided in Figure 7 to support replication and further research.Source: §A.2 · p. 13, Figure 7 · p. 13

B. Training Details

The training details specify supervised fine-tuning for CMTA and a resource-limited SARL stage trained on selected samples.

B. Training Details

  • CMTA fine-tunes Qwen3-VL-4B-Instruct with the visual encoder frozen and the multimodal projector and language model updated, using Adam, a cosine schedule, 10% warm-up, bfloat16, and DeepSpeed ZeRO-3.The initial learning rate is 1 × 10−5.Source: B. Training Details · p. 13
  • Because of computational resource constraints, SARL training uses 20k samples per task, with GSPO hyperparameters left at their defaults.Source: B. Training Details · p. 13

C. Implementation Details for Baselines

Baseline comparisons use different strategies for general-purpose and CAD-specific VLMs, while keeping data conditions uniform.

C. Implementation Details for Baselines

D. Qualitative Analysis

The qualitative analysis compares methods across four intent-to-CAD tasks and illustrates both successful and failed generations on real hand-drawn sketches.

D. Qualitative Analysis

  • Figure 8 presents qualitative comparisons of different methods across four human intent-to-CAD tasks.Source: D. Qualitative Analysis · p. 14
  • On real hand-drawn sketches, HiCAD’s successful examples largely match input geometry and topology, while failures show geometric deviations or topological inconsistencies.Figure 9 marks clearly incorrect areas with red dashed boxes.Source: D. Qualitative Analysis · p. 14

E.1. Limitation

The paper identifies limits in handling complex structures, sketch quality, and the scope of human-intent representations, and outlines future work on refinement, alignment, and the synthetic-to-real sketch gap.

E.1. Limitation

  • HiCAD still performs poorly on highly complex structures despite performing well compared with existing methods.Source: §E.1 · p. 14
  • Cluttered sketches or non-standard perspectives can sometimes lead to misaligned spatial layouts, and performance depends partly on sketch quality.Source: §E.1 · p. 14
  • The paper’s four tasks are not exhaustive, and how people express intent and models understand it remains open for further exploration.Source: §E.1 · p. 14

E.2. Future Work

  • Future work plans to let users refine generated CAD models through multi-turn feedback in a conversational agent.Source: §E.2 · p. 14
  • The method relies on automatic implicit sketch-text alignment, which the authors consider insufficient; they plan to develop explicit alignment strategies.Source: §E.2 · p. 14
  • Although trained only on synthetic sketches, HiCAD significantly outperforms existing approaches on real sketches, but some performance degradation remains.Future work targets more photorealistic sketch synthesis or stronger cross-domain generalization.Source: §E.2 · p. 14
Loading 2610.15681v1…