Source-linked AI summary
InstructMesh: Selective Refinement of Generative 3D Models for Fabrication
Faraz Faruqi, Ahmed Katary, Demircan Tas, Theresa Hradilak, Ning Zhang, Jiaji Li, Fabian Manhardt, Martin Nisser, Vrushank Phadnis, Ruofei Du, Federico Tombari, Megan Hofmann, Stefanie Mueller
TL;DR
Generative 3D models often prioritize visual plausibility over fabrication-critical geometry, leaving flaws that emerge during physical use. InstructMesh enables selective region-based repair through latent-space operations controlled by natural language or sliders. User studies show that novices can identify and repair visually identifiable flaws, with hybrid workflows and preview visualizations preferred for usability and transparency.
Problem
Image-based supervision can prioritize visual plausibility over fabrication-relevant geometry, producing flaws that compromise downstream physical use.
Method
InstructMesh maps region selections and user instructions to targeted operations on the model’s intermediate latent representation, using natural-language and slider interfaces with preview feedback.
Results
User studies show that novices can identify and repair visually identifiable fabrication-relevant flaws using InstructMesh, with hybrid workflows and preview visualizations improving usability and transparency.
Takeaways & Limitations
InstructMesh supports post-generation refinement of generative 3D models without requiring mesh-manipulation expertise.
Takeaways & Limitations
The evaluation focuses on a single generative model, Trellis, and the current 64^3 voxel grid cannot repair sub-voxel defects.
Abstract
from arXiv · showhide
Recent advances in generative AI allow users to create 3D models from text or images. However, these models prioritize visual plausibility over geometric accuracy, often generating results with flaws that compromise their intended use post-fabrication. We present InstructMesh, an interactive post-generation refinement tool that enables selective repair of generative 3D models through region selection and targeted operations, such as opening or sealing voids, or adjusting local thickness. Users can invoke edit operations via natural language prompts or slider controls. By operating directly on the intermediate latent representation, InstructMesh allows users to apply robust geometric corrections without requiring expert modeling skills. To inform our design, we first analyze common fabrication-related failure modes in outputs from state-of-the-art generative tools. We then conduct two user studies, demonstrating that novices can identify and perform fabrication-relevant repairs on generative outputs using InstructMesh, and revealing user preference for hybrid interfaces that combine slider controls with natural language input.
1 INTRODUCTION
Generative 3D models can look plausible while containing fabrication-critical geometric flaws. InstructMesh addresses this gap through selective latent-space repair with region selection, natural-language or slider controls, and preview feedback.
- Image-based supervision prioritizes visual plausibility over fabrication-critical geometry, producing flaws such as sealed openings, thin walls, and missing structural connections.
- Existing mesh editors demand substantial expertise, while text-driven editing and re-prompting may target appearance or produce unreliable, unintended changes.
- InstructMesh maps region-based user edits to canonical operations on an intermediate latent representation, then decodes the edited representation into a refined mesh.
- The interface combines an LLM-based natural-language mode with parameterized slider controls, allowing users to express geometric corrections through complementary interaction styles.
- The work analyzes fabrication flaws in 120 models and develops canonical latent-space operations paired with natural language, sliders, and preview visualization.
- User studies show that novices can successfully identify and repair visually identifiable flaws, while hybrid workflows with preview visualizations improve usability and transparency.
2 RELATED WORK
Related work spans fabrication support, generative 3D workflows, and human-in-the-loop refinement. InstructMesh extends region-level control from visual appearance to fabrication-relevant geometry.
- Digital fabrication tools support mesh manipulation, parametric customization, and mechanical retargeting, but existing workflows can constrain novice modification or exploration.
- Generative 3D methods broaden access to bespoke models from image and text prompts, alongside research on text-driven 3D editing.
- Generative models trained with image-based supervision may prioritize appearance over physical correctness, leaving functional geometric features absent until downstream use.
- Under-specified inputs create ambiguity, and novices may struggle to craft prompts or debug faulty generated outputs.
- Human-in-the-loop systems support iterative refinement through regional prompting, visual conditioning, inspectable intermediate steps, and evaluator-guided revision.
- InstructMesh gives users region-level control over fabrication-relevant geometry through selection, preview, and iterative revision of generated models.
3 FORMATIVE STUDY
The formative study characterizes fabrication-relevant flaws in Trellis reconstructions of Thingiverse models. Flaws were widespread, diverse, and often co-occurring, motivating targeted post-generation repair.
- 3 FORMATIVE STUDY: The study examines functional geometry details including wall thickness, openings, and manifold integrity, hypothesizing that generative models reproduce aesthetics while missing local geometric requirements.
- 3.1 Dataset Selection and Generative 3D Reconstruction: Researchers sampled 100 popular Thingiverse things, producing 120 unique models, rendered one image per model, and reconstructed each with Trellis.
- 3.2 Inductive Taxonomy Development: Two expert annotators independently compared each generated model with its original, described individual issues and required changes, and used iterative qualitative coding to develop a flaw taxonomy.
- 3.3 Iterative Categorization of Issues: Annotators repeatedly grouped, discussed, merged, and refined issue descriptions until consensus stabilized the taxonomy at nine flaw categories.
- 3.4 Deductive Classification: The dataset was classified deductively by mapping previously collected annotations to one or more of the nine taxonomy categories.
- 3.4 Deductive Classification: 94 models (78.3%) exhibited more than one issue, with 2.4 issues per model on average (SD = 1.10), indicating that geometric flaws commonly co-occur.
- 3.4 Deductive Classification: The most frequent issues were Extraneous Artifacts (65.8%) and Missing Openings (48.6%), with the passage indicating additional frequent flaw types.
- 3.4 Deductive Classification: The prevalence and diversity of flaws motivate post-generation tools that support targeted, user-guided geometric correction.
4 SYSTEM OVERVIEW
InstructMesh refines generated 3D models by mapping localized user edits onto an intermediate latent voxel representation before decoding a new mesh. It combines canonical additive and subtractive operations with natural-language or slider controls and previews.
- System pipeline: The system encodes a generated model into a latent voxel representation, applies localized edits there, and decodes the result into a refined textured mesh.The representation separates coarse structure from decoded appearance and topology.
- Scope: The scope is static fabrication functionality; dynamic mechanisms requiring precise geometric and mechanical constraints are not addressed.Fused Joints are excluded because they involve articulated or mechanical components.
- Interaction workflow: Users inspect generated meshes, highlight flawed regions, specify edits through natural language or direct operation selection, preview changes, and confirm regeneration.The workflow supports continued refinement, undo, side-by-side comparison, and downloading the repaired model.
- Canonical operations: Additive operations increase voxel occupancy through extrusion, expansion, or filling to extend surfaces, thicken regions, or close gaps.These operations support tasks such as lengthening handles, reinforcing thin walls, and solidifying hollow parts.
- Canonical operations: Subtractive operations reduce occupancy through trimming, erosion, or flattening to remove material, hollow interiors, or prepare flat contact areas.Examples include cleaning artifacts, carving channels, and preparing flat bases.
- Design rationale: The operation vocabulary is compact, composable, and parameterized, allowing users to express high-level design intent without manipulating mesh topology directly.Latent grids are coarser than meshes, and decoder processing supplies learned appearance details after editing.
5 EVALUATING AND CALIBRATING OPERATIONS
The technical evaluation calibrates latent operations on a development split and tests them independently on held-out flaw instances. Results show high repair success, accurate natural-language operation selection, and sub-minute editing despite runtime overhead.
- Evaluation protocol: The dataset was randomized and split 80%-20%, using 20% for operation calibration and 80% for evaluation.An independent reviewer judged each flaw-level repair as successful or unsuccessful.
- Calibration: Twelve calibrated presets were defined across six operations to balance repair flexibility, interface complexity, and LLM prediction accuracy.Presets were manually tuned on the development set for different flaw contexts.
- Repair accuracy: Missing Openings achieved 96.3% success and Wall Thickness Issues 95.65%, while Fused / Duplicated Features had the lowest success rate at 83.33%.The evaluation covered a high proportion of repairs across all flaw categories.
- Runtime efficiency: 43.7s average generation time with latent operations exceeded the 31.7s baseline, but edits remained under one minute.The operation pipeline increased runtime from 31.7s (SD = 3.7s) to 43.7s (SD = 5.2s).
- LLM operation selection: 92.1% accuracy was achieved by GPT-4 in predicting the correct canonical operation on the 80% test split without fine-tuning.The result used the described in-context-learning setup.
6 USER EVALUATION I: NOVICE FLAW IDENTIFICATION AND REPAIR
A study with novices evaluated flaw identification, repair effort, and repair correctness across five flawed generated models. Participants identified and repaired most fabrication-relevant flaws, although model and flaw complexity affected performance and time.
- Study design: The study recruited 12 participants without prior 3D modeling, CAD, or digital fabrication experience to inspect five flawed models.Each model contained one to four annotated fabrication flaws.
- Flaw identification: Participants identified 90.4% of fabrication-relevant flaws, with detection significantly affected by model and flaw type.Flaws in M1 were easier to detect than those in M5, while topological issues were the hardest category to identify.
- Repair effort: Repair time varied significantly by model and flaw type, with simpler models repaired faster than more complex ones.Truncated features took 3.00 ± 0.95 min, compared with 1.50 ± 0.52 min for topological errors and 1.84 ± 0.75 min for fused features.
- Repair correctness: 89.7% of 156 flaws were successfully repaired according to an independent expert, with 16 failures.Correctness did not significantly vary by model, although flaw type showed a significant overall effect without significant post-hoc category differences.
- Self-assessment: Novices’ self-assessments closely matched expert ratings, with disagreement in 11 cases, or 7.1%.The comparison found no significant difference between participant and expert judgments.
- User response: Participants described InstructMesh as easy to use and suitable for rapid prototyping, hobbyist printing, and education.Comments emphasized undo support, text-based editing, and avoiding the need to learn CAD software.
7 USER EVALUATION II: INTERACTION MODES AND PREVIEW VISUALIZATION
The second study compares natural-language and slider interfaces with previews during localized repair tasks. LLMs reduced effort and supported expressive edits, sliders increased perceived precision, and all participants preferred a hybrid workflow.
- Study design: The study used 12 new participants without prior 3D modeling or CAD experience, each completing 12 tasks across counterbalanced LLM and slider blocks.Both conditions showed a preview before participants committed an edit.
- Usability and control: Both interfaces were usable: LLMs required less effort, while sliders were perceived as slightly more accurate and control ratings were equal.Effort was M = 2.67 for LLMs versus M = 3.25 for sliders; accuracy was M = 5.00 versus M = 5.17, and control was M = 5.58 for both.
- Preview visualization: Previews received high ratings in both conditions and helped participants understand edits and decide whether to accept or revise them.Preview clarity was M = 6.67 for LLMs and M = 6.75 for sliders on the 7-point scale.
- Interface trade-offs: Overall interface preference was evenly split between sliders and LLMs, with 6 participants choosing each.All 12 participants favored a hybrid design combining both modes.
- Interface trade-offs: Participants associated natural language with broad creative edits and sliders with precise local refinements.Examples included initial sketches and fundamental changes for LLMs versus filling gaps or making holes with sliders.
8 APPLICATIONS
InstructMesh was applied to six fabrication scenarios spanning home objects, medical devices, personal accessories, and robotics, converting flawed generative models into functional printable designs.
- Six application scenarios covered home decor, medical devices, personal accessories, and robotic enclosures, with all examples printed on a Stratasys J55.
- A dragon-handled mug had its sealed lid removed, producing a printable functional version.
- Localized edits added strap holes to a denim-aesthetic knee brace, producing a wearable functional design ready for printing.
- Eyeglass edits hollowed filled lenses, strengthened a thin bridge, and connected temples, while whistle edits added acoustic chambers and refined resonant geometry.
- For a bristle-bot enclosure, InstructMesh shortened obstructing legs, created an electronics cavity, and added a motor tunnel to enable directed walking.
9 DISCUSSION AND FUTURE WORK
The discussion reports strong novice performance on visually identifiable flaws, support for hybrid interaction, and limitations involving generalization, resolution, diagnosis, evaluation scope, and dynamic functionality.
- 9.1 Insights and Design Guidelines: Novices identified 90.4% of fabrication-relevant flaws and successfully repaired 89.7% in Study 1.Detectability and repair difficulty varied across flaw types, motivating future automated flaw detection and operation recommendations.
- 9.1 Insights and Design Guidelines: Study 2 found an even split between LLM and slider preferences (6/6), while all participants preferred a hybrid workflow (12/12).
- 9.1 Insights and Design Guidelines: The design guidelines recommend complementary input modes, previews for opaque latent edits, and calibrated parameterization bridging user intent and operations.
- 9.2 Heuristics vs. Learned Approaches for Latent-Space Editing: Heuristic operations are interpretable, composable, and require no training data, providing an inspectable foundation for fabrication-relevant edits.
- 9.2 Heuristics vs. Learned Approaches for Latent-Space Editing: Learned approaches could support broader and novel repair scenarios, but may reduce the transparency and controllability valued by novices.
- 9.3 Limitations: The evaluation focused on a single generative model, Trellis, limiting direct evidence across additional backbones.
- 9.3 Limitations: The 643 voxel grid makes sub-voxel defects unrepairable because a single voxel spans roughly 1.5% of a shape’s extent.
- 9.3 Limitations: The system assumes users can visually identify and select flaws, whereas internal thickness or tolerance defects may emerge only during slicing or after failed printing.
10 CONCLUSION
InstructMesh enables novices to repair fabrication-relevant flaws in generative 3D models through region selection, natural language, and canonical latent-space operations. Studies found that novices could repair visually identifiable flaws, preferred hybrid workflows, and completed repairs in under ten minutes per model.
- InstructMesh maps region selections and natural language instructions to canonical latent-space operations for fabrication-relevant repair.
- A formative study motivated composable editing operations triggered through text or slider controls, with preview visualization supporting transparency and trust.
- User studies showed that novices could detect and repair visually identifiable flaws in under ten minutes per model and preferred hybrid workflows.
- Applications spanned home decor, assistive devices, personal accessories, and robotics.