PaperlayerView exact revision on arXiv

Source-linked AI summary

Programming-by-Example for Batch-Editing Collision Meshes in 3D Software

Gengyang Xu, Dongwei Xiao, Hengcheng Zhu, Yiteng Peng, Wei Meng, Shuai Wang, Shing-Chi Cheung

arXiv:2607.08804v1cs.GRcs.ROcs.SE

TL;DR

Collision meshes often need manual, task-specific edits because automatically generated geometry may not capture intended interactions, and editing many heterogeneous meshes is difficult to scale. MeshForge uses programming by example and neuro-symbolic synthesis to create reusable batch-editing programs from a few demonstrations. It successfully synthesized programs for 23 of 24 tasks in a 600-mesh benchmark, with 2.2 demonstrations and 3.5 seconds of synthesis time on average.

  • Problem

    Automatically generated collision meshes may not capture intended interactions, leaving developers to manually edit many heterogeneous meshes.

    Source: §I · p. 1

  • Method

    MeshForge uses programming by example and neuro-symbolic synthesis to turn a few demonstrated edits into reusable programs for other collision meshes.

    Source: §I · p. 1

  • Results

    23/24 tasks were successfully synthesized across 24 tasks and 600 collision meshes, requiring 2.2 demonstrations and 3.5 s of synthesis time on average.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors present collision-mesh batch editing as a software-artifact maintenance task that can be automated through program synthesis rather than repeated manual effort.

    Source: §IX · p. 11

    MeshForge depends on the underlying neural model and its DSL does not support conditional logic, proportion-based geometric constraints, or vertex-level editing.

    Source: §VII · p. 10

Abstract

from arXiv · show

As 3D software proliferates, software artifacts now extend beyond code and 2D user interfaces to include 3D assets. Among these assets, collision meshes are critical as they define the geometry used by physics engines for collision detection and physical interaction. Although existing tools can automatically generate collision meshes from visual meshes, they often fail to capture the intended interaction behavior. As a result, developers need to manually edit many heterogeneous collision meshes, a process that is time-consuming and challenging to scale. To address this problem, we present a neuro-symbolic program synthesis approach for batch-editing collision meshes. We formulate the task as a programming-by-example problem: given a family of collision meshes with the same editing intent and a small number of user demonstrations, our approach synthesizes a reusable program that captures the editing intent and applies it to non-demonstration meshes. We implement this in a tool named MeshForge, and evaluate it across 24 tasks on 600 collision meshes. MeshForge successfully synthesizes 23/24 tasks, requiring 2.2 demonstrations and 3.5 seconds of synthesis time on average.

I. INTRODUCTION

Collision meshes often need task-specific edits that existing geometry generation and whole-mesh tools do not provide, while per-mesh fixes are labor-intensive across heterogeneous assets. MeshForge frames batch editing as programming by example and synthesizes reusable editing programs from a few demonstrations.

I. INTRODUCTION

  • Collision meshes may fail task-specific interactions even when geometry is well approximated, such as blocking drawer placement or providing an unsuitable grasp proxy.Collision meshes are simplified convex-hull representations used to reduce physics-simulation overhead.Source: §I · p. 1
  • Manual editing is labor-intensive and redundant across assets with shared intent, while existing whole-mesh tools cannot identify task-relevant hulls across heterogeneous meshes.Higher decomposition fidelity is not a reliable fix because it can overfragment regions that were already adequate.Source: §I · p. 1
  • MeshForge formulates batch editing as programming by example: users demonstrate edits on representative meshes, and a neuro-symbolic system synthesizes reusable extractor-action programs for other meshes.The demonstrations record actions directly while the system synthesizes target-selection logic.Source: §I · p. 1
  • The task requires extractors to generalize beyond low-level geometry, refine coarse semantic labels with structural and spatial cues, and tolerate perception errors that can invalidate otherwise-correct candidates.A single mislabeled hull can cause an extractor consistent with all demonstrations to be rejected.Source: §I · p. 2
  • The authors introduce a symbolic collision-mesh representation and neuro-symbolic DSL, and use ABI to mitigate synthesis bottlenecks caused by neural perception noise.ABI infers minimum label corrections that make a near-miss extractor consistent with the demonstrations, then asks users to confirm them.Source: §I · p. 2

II. MOTIVATING EXAMPLE

The table-asset example illustrates why editing must target task-relevant hulls across geometric variants rather than apply uniform mesh operations. MeshForge uses demonstrations to synthesize reusable selection-and-action rules, then applies them to remaining assets for inspection.

II. MOTIVATING EXAMPLE

  • The table task involves five edits: remove residual hulls, stabilize the tabletop, simplify legs individually, clear the drawer interior, and make attached handles usable for robotic interaction.The asset variants differ in geometry and orientation, making manual editing tedious and error-prone.Source: §II · p. 2
  • MeshForge assigns task-specific semantic labels that users can inspect and correct, then records demonstrated actions while synthesizing target-selection logic across hulls that vary in number, shape, attachment, and orientation.Users may specify one or two positive coordinate axes as orientation landmarks for directional operations.Source: §II · p. 2
  • The resulting extractor-action rules are applied to the remaining assets, whose size, appearance, and decomposition granularity may vary; users can inspect results, correct isolated errors, or add demonstrations for systematic errors.The example connects the representation and DSL to the later synthesis process.Source: §II · p. 3, Figure 3 · p. 3

III. SYMBOLIC REPRESENTATION AND DSL

MeshForge represents collision meshes symbolically so reusable target selection can use semantic, spatial, and attachment information while retaining links to editable hulls. Its DSL builds on these signals to express extractor rules.

III. SYMBOLIC REPRESENTATION AND DSL

  • The symbolic collision mesh is an intermediate representation that augments raw hulls with selection-relevant information while preserving a one-to-one link to editable hulls.The section frames the representation and DSL as the basis for synthesizing target-selection logic that generalizes across meshes.Source: A. Symbolic Collision Mesh · p. 3, §III · p. 3

A. Symbolic Collision Mesh

  • Each symbolic hull corresponds uniquely to a raw hull and carries a task-specific semantic label and an axis-aligned bounding box.The symbolic mesh is formalized as a graph of symbolic hulls.Source: A. Symbolic Collision Mesh · p. 3
  • The contact graph connects nearby hulls using a distance threshold scaled by the mesh bounding-box diagonal, while labels, bounding boxes, and graph relationships support semantic, geometric, and attachment-based selection.The label is annotated by a pretrained model, and users can customize the task-specific label vocabulary.Source: A. Symbolic Collision Mesh · p. 3

B. DSL for Batch-Editing Programs

MeshForge’s DSL combines layered extractors to select target hulls with hull-level actions that edit them. Extractors compose semantic, topological, and geometric criteria to generalize across heterogeneous meshes.

B. DSL for Batch-Editing Programs

1) Extractors: What to Edit:

  • Extractors select hulls by semantic label, refine selections by contact-graph attachment or spatial extremity, and compose them with set operators.Attachment and geometric refinement operate on base selections; restricting them helps avoid overfitting to particular decomposition patterns.Source: 1) Extractors: What to Edit · pp. 3–4

2) Actions: How to Edit:

  • Five hull-level actions—Delete, Replace, Add, Merge, and Scale—transform selected hulls while preserving convexity.Replace aligns inserted geometry to the selected set’s bounding box; Add places a template flush against it in a specified direction.Source: 2) Actions: How to Edit · p. 4

3) Program Structure:

MeshForge programs normalize orientation and then execute ordered rules that combine hull selection with editing actions. The pipeline constructs symbolic meshes, formalizes demonstrations, synthesizes reusable rules, and applies them to other meshes.

3) Program Structure:

  • Programs begin with an orientation directive and execute rules sequentially; component-wise rules can apply an action independently to connected selected hull groups.User-provided landmark pairs can establish canonical axes so directional extractors and actions operate consistently across assets.Source: 3) Program Structure · p. 4

C. Running Example

  • The running example normalizes the direction from leg-hull centers to panel-hull centers as positive Z, then expresses five edits in a synthesized DSL program.It combines orientation normalization, layered extractors, component-wise application, and hull-level actions.Source: C. Running Example · p. 4

A. Overview of the Pipeline

  • MeshForge constructs symbolic collision meshes, records demonstrations as formal examples, synthesizes reusable rules, and applies the program to non-demonstration meshes.The pipeline is modular, inspectable, legality-preserving, and allows user intervention at multiple points.Source: A. Overview of the Pipeline · pp. 4–5, Figure 7 · p. 5

B. Stage I: Symbolic Collision Mesh Construction

  • Stage I labels hulls from orthographic views using a user-specified vocabulary, adds bounding boxes and contact-graph edges, and lets users inspect and correct annotations.The resulting symbolic collision mesh becomes the input to Stage II.Source: B. Stage I: Symbolic Collision Mesh… · p. 5

C. Stage II: Demonstration Formalization

Users demonstrate desired edits on a few representative meshes through a GUI, and MeshForge formalizes those edits as examples for synthesis. The examples preserve target hulls and actions while leaving reusable extractor logic to be inferred.

C. Stage II: Demonstration Formalization

  • Users typically demonstrate on 2–3 representative meshes, with the number depending on task complexity.If the synthesized program proves insufficient, they can add demonstrations and re-synthesize through iterative refinement.Source: C. Stage II: Demonstration Formalization · p. 5
  • GUI demonstrations let users show edits directly rather than express requirements in natural language or write DSL programs.The interactions resemble standard 3D modeling tools developers already use, reducing cognitive burden.Source: C. Stage II: Demonstration Formalization · p. 5
  • Each demonstrated rule records the target hulls and editing action, while synthesis infers the reusable extractor.Multiple editing steps are recorded as an ordered sequence of rules.Source: C. Stage II: Demonstration Formalization · p. 5
  • For Replace, a demonstration records the inserted geometry; for Add, it records the template and placement direction.Source: C. Stage II: Demonstration Formalization · p. 5
  • A demonstration rule pairs an intermediate symbolic mesh with the user-selected target hull set and demonstrated action.The intermediate mesh reflects orientation and any previously applied rules.Source: C. Stage II: Demonstration Formalization · p. 5

D. Stage III: Synthesis-Driven Batch-Editing

  • Stage III searches for extractor logic consistent with the demonstrations, applies a matching program to remaining meshes, and lets users inspect the results.If results are unsatisfactory, users may correct failed cases or add demonstrations and trigger iterative refinement.Source: D. Stage III: Synthesis-Driven… · p. 5
  • Program synthesis composes the orientation directive with ordered rules, each pairing an extractor with an action that selects the demonstrated target set.Since orientation and actions are recorded from demonstrations, synthesis searches for the extractor of each rule sequentially.Source: D. Stage III: Synthesis-Driven… · p. 5

1) Batch-Editing Program Synthesis:

MeshForge searches the DSL for extractors consistent with demonstrations, using abductive inference to test whether sparse label corrections explain mismatches. The inferred corrections are presented for user confirmation.

1) Batch-Editing Program Synthesis:

  • Top-down enumerative search explores DSL extractor programs conditioned on intermediate meshes produced by previously synthesized rules.Partial expressions contain holes that remain to be instantiated.Source: 1) Batch-Editing Program Synthesis · p. 5
  • The search dequeues the lowest-complexity candidate, tests complete candidates against all demonstrations, and expands incomplete ones with feasible refinements.It stops when it finds a valid or repaired extractor, or reaches a timeout.Source: 1) Batch-Editing Program Synthesis · p. 6

2) Abductive Correction of Neural Labels:

  • A single mislabeled hull can reject an otherwise correct near-miss program and push synthesis toward more complex alternatives.Source: 2) Abductive Correction of Neural Labels · p. 6
  • Abductive inference treats a near-miss extractor’s mismatch with selected hulls as evidence of suspect annotations, then infers minimum label corrections consistent with demonstrations.Users confirm inferred corrections rather than supplying them directly.Source: 2) Abductive Correction of Neural Labels · p. 6
  • ABI forms its suspect set from the symmetric difference between the candidate extractor’s output and the demonstrated target set, then enumerates candidate relabelings.The candidate repairs change at most η labels per demonstration; the algorithm retains a minimum-cost repair that matches all demonstrations.Source: 2) Abductive Correction of Neural Labels · p. 6
  • The correction search relies on sparse annotation errors and an η threshold; if too many labels appear suspect, it returns no repair.A proposed repair is surfaced for user confirmation, not applied automatically.Source: 2) Abductive Correction of Neural Labels · p. 6

V. IMPLEMENTATION

MeshForge uses GPT-4o for hull annotation after comparing open-world segmentation and vision-language models. The authors calibrate adjacency, geometric-boundary, and ABI correction thresholds on a 200-mesh sample.

Neural Perception

  • GPT-4o achieved 0.92 annotation accuracy, exceeding Find3D (0.61) and Qwen3-VL (0.76), and was selected at an average cost of $0.0224 per mesh.Annotations were evaluated per hull on 200 meshes; repeated GPT-4o queries showed no significant variation (Friedman test, p = 0.54).Source: §V · pp. 6–7

VI. EVALUATION

The evaluation tests MeshForge across a benchmark of collision-mesh editing tasks, measuring program-selection fidelity, functional outcomes, and user effort. MeshForge achieves 95.8% synthesis success and 0.92 ± 0.08 functional pass rate, with performance declining on harder tasks.

A. Experimental Setup

  • The benchmark covers 24 easy, medium, and hard tasks across eight asset categories, using 200 assets and 600 collision meshes with coarse, medium, and fine decomposition variants.The tasks were derived from concrete editing needs and instantiated using assets from public datasets.Source: A. Experimental Setup · p. 7
  • Five experienced developers independently selected up to five demonstrations per task, and each synthesis round had a 120-second timeout; predefined hull-template construction was outside the evaluation.The protocol evaluates programs on non-demonstration meshes selected by the developers rather than imposing a fixed train/test split.Source: A. Experimental Setup · p. 7
  • Synthesis success measures agreement with reference hull selections on non-demonstration meshes, while functional pass rate checks whether edited meshes satisfy task-specific criteria.The study also reports demonstration count and time as effort metrics.Source: A. Experimental Setup · p. 7

B. RQ1 (Effectiveness)

  • 95.8% of tasks (23/24) achieved synthesis success, with 2.2 demonstrations and 3.5 s synthesis time per task on average.The result is reported alongside Table I; Hard tasks achieved 7/8 success (87.5%), compared with 8/8 for both Easy and Medium tasks.Source: B. RQ1 (Effectiveness) · p. 8, Table I · p. 8
  • 0.92 ± 0.08 was MeshForge’s overall functional pass rate, with rates of 1.00 ± 0.01, 0.92 ± 0.05, and 0.85 ± 0.06 for Easy, Medium, and Hard tasks.When decomposition merges task-critical parts into one hull, hull-level editing cannot recover the missing structure; representative successes and failures appear in Fig. 8.Source: B. RQ1 (Effectiveness) · p. 8, Figure 8 · p. 8
  • MeshForge reached 91.7% SSR with k=3 demonstrations; Medium tasks reached 100.0% SSR at k=3, while Hard tasks had one unresolved failure and 87.5% at k=5.MeshForge achieves 95.8% SSR (one failure) and 0.92±0.08 FPR.Source: B. RQ1 (Effectiveness) · p. 8

C. RQ2 (Comparison)

The comparison frames MeshForge against tools that differ in physical legality, cross-instance generalization, interpretability, and required user input. Its symbolic programs generalize across heterogeneous meshes while operating on convex hulls.

1) Qualitative Comparison:

  • MeshForge operates at the hull level, ensuring each output hull remains convex, whereas Vinedresser3D and TextDeformer do not enforce convexity and ParSEL only partially preserves legality under basic affine operations.Manual editing can preserve legality when developers explicitly maintain convexity.Source: 1) Qualitative Comparison · p. 8
  • Unlike existing approaches that cannot synthesize reusable editing logic across heterogeneous collision meshes, MeshForge synthesizes a symbolic DSL program for cross-instance generalization.The comparison identifies this as a capability specific to the batch-editing setting.Source: 1) Qualitative Comparison · p. 8
  • MeshForge produces an inspectable symbolic DSL program and uses GUI demonstrations, while other approaches offer varied artifacts or inputs such as modified geometry, Python scripts, prompts, or reference images.ParSEL exposes a parameterized DSL, BlenderAlchemy outputs executable Python, and Vinedresser3D and TextDeformer provide no human-readable record of changes.Source: 1) Qualitative Comparison · p. 9

2) Quantitative Comparison:

MeshForge outperforms two adapted VLM baselines on functional pass rate. Ablations indicate that semantic, topological, and geometric extraction layers contribute complementary selection power, while ABI addresses sparse label errors.

D. RQ3 (Ablation Study)

  • Removing geometric extraction reduced SSR to 0.833, removing topological extraction reduced it to 0.750, and semantic-only extraction reduced it to 0.625, versus 0.958 for the full design.The ablation results are shown in Fig. 11; the layers refine selections by semantics, attachment, and spatial boundaries.Source: D. RQ3 (Ablation Study) · p. 9, Figure 11 · p. 10
  • ABI was triggered in an average of 10.4 of 24 tasks per developer and required 3.6 confirmation decisions per task; all evaluated confirm/reject decisions matched reference labels, with zero false-accepts.The user-confirmation step applies a correction only after inspection and confirmation.Source: D. RQ3 (Ablation Study) · p. 9
  • Without noise correction, SSR fell from 1.00 to 0.70 and average end-to-end time rose from 183.2 s to 228.2 s in the fixed set of 10 ABI-triggered instances.Near-miss programs were unconditionally rejected when ABI was disabled, leading to additional demonstration rounds.Source: D. RQ3 (Ablation Study) · p. 9, Figure 12 · p. 10
  • Compared with Standard PBE, MeshForge achieved the same SSR while reducing time and receiving more favorable ratings for cognitive burden and interaction efficiency.The comparison is summarized in Table III, whose ratings use a 1–5 scale.Source: D. RQ3 (Ablation Study) · pp. 9–10, Table III · p. 10

VII. THREATS TO VALIDITY

MeshForge’s evaluation is bounded by neural-model dependence, DSL expressiveness, human judgment, and a limited self-constructed benchmark. The related work positions it as collision-mesh batch editing, extending program synthesis to a 3D software maintenance task.

VII. THREATS TO VALIDITY

  • MeshForge was evaluated on a self-constructed benchmark with limited scale and diversity, and the authors call for broader validation and larger user studies.Its neural component also relies on the GPT-4o API, bringing API costs, latency, and data-privacy risks; open-source VLMs underperform on hull annotation.Source: §VII · p. 10
  • MeshForge’s effectiveness depends on neural recognition, its DSL omits conditional, proportion-based, and vertex-level edits, and users’ demonstration choices can vary.Topological and geometric extractors and ABI reduce—but do not eliminate—reliance on labels and sparse annotation errors.Source: §VII · p. 10

VIII. RELATED WORK

  • Unlike visual or geometric editing systems and physics-property tools, MeshForge preserves convexity constraints while synthesizing reusable programs for collision-mesh editing.Prior PBE work covers code, images, apps, and data; MeshForge extends neuro-symbolic PBE to symbolic collision meshes in 3D software.Source: §VIII · p. 10

IX. CONCLUSION

  • MeshForge frames collision-mesh batch editing as 3D software artifact maintenance that can be automated through program synthesis rather than repeated manual effort.The authors present it as addressing a research gap and hope it inspires further 3D software engineering research.Source: §IX · p. 11
Loading 2607.08804v1…