PaperlayerView exact revision on arXiv

Source-linked AI summary

ParSEL: Parameterized Shape Editing with Language

Aditya Ganeshan, Ryan Y. Huang, Xianghao Xu, R. Kenny Jones, Daniel Ritchie

arXiv:2405.20319v2cs.CVcs.AIcs.GRcs.HCcs.SC

TL;DR

Natural language can express 3D edit intent but is poorly suited to specifying precise edit magnitude. ParSEL turns requests into parameterized editing programs using LLMs and Analytical Edit Propagation, and participants greatly preferred its edits over alternative formulations in the reported perceptual study.

  • Problem

    Natural language communicates edit intent but is poorly suited to specifying precise geometric edit magnitudes.

    Source: §1 · p. 2, §4.3 · p. 5

  • Method

    ParSEL combines LLM prompting with CAS-based Analytical Edit Propagation to extend seed edits into parameterized programs using inter-part geometric relationships.

    Source: §1 · p. 2, §3 · p. 4, §4.3 · p. 5

  • Results

    Participants greatly preferred ParSEL’s edits over two alternative formulations in the CoMPaT3D++ perceptual study.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Adjustable program parameters let users explore shape variations and control edit magnitude for natural-language requests.

    Source: §1 · p. 2

    LLMs often reliably infer only an initial seed edit, struggling to infer secondary edit types and magnitudes for cohesive programs.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

The ability to edit 3D assets from natural language presents a compelling paradigm to aid in the democratization of 3D content creation. However, while natural language is often effective at communicating general intent, it is poorly suited for specifying precise manipulation. To address this gap, we introduce ParSEL, a system that enables controllable editing of high-quality 3D assets from natural language. Given a segmented 3D mesh and an editing request, ParSEL produces a parameterized editing program. Adjusting the program parameters allows users to explore shape variations with a precise control over the magnitudes of edits. To infer editing programs which align with an input edit request, we leverage the abilities of large-language models (LLMs). However, while we find that LLMs excel at identifying initial edit operations, they often fail to infer complete editing programs, and produce outputs that violate shape semantics. To overcome this issue, we introduce Analytical Edit Propagation (AEP), an algorithm which extends a seed edit with additional operations until a complete editing program has been formed. Unlike prior methods, AEP searches for analytical editing operations compatible with a range of possible user edits through the integration of computer algebra systems for geometric analysis. Experimentally we demonstrate ParSEL's effectiveness in enabling controllable editing of 3D objects through natural language requests over alternative system designs.

1 INTRODUCTION

Natural language communicates geometric edit intent but poorly specifies magnitude, while manual modification of high-quality 3D assets is laborious. ParSEL combines language-based requests with adjustable parameters, using LLMs and analytical edit propagation to produce controllable editing programs; participants preferred its edits over alternatives in the reported study.

1 INTRODUCTION

  • Natural language conveys edit intent but poorly specifies magnitude, while manually modifying high-quality 3D assets is tedious and requires substantial training.Terms such as “moderately widen” are subjective, and numerical amounts may not align with an object’s scale.Source: §1 · pp. 1–2
  • ParSEL lets users specify how to edit with language and how much with adjustable parameters, enabling exploration of shape variations.It takes a semantically labeled mesh and an edit request, then exposes an adjustable parameter for exploring variations.Source: §1 · p. 2
  • ParSEL represents edits as closed-form analytical functions of control parameters and propagates proxy deformations to mesh geometry using cage-based methods.The parameterized functions are intended to keep geometry consistent across parameter variations and support solver-less exploration.Source: §1 · p. 2
  • Its LLM-and-CAS module translates language requests into programs, while AEP extends partial programs using geometric relations between object parts.AEP searches for analytical editing functions compatible with a range of possible edits and can dynamically modify shape structure based on the request.Source: §1 · p. 2
  • Participants greatly preferred ParSEL’s edits over two alternative formulations in the CoMPaT3D++ perceptual study.The alternatives asked an LLM to author the program directly or used LLM and AEP without dynamically altering shape structure.Source: §1 · p. 2
  • The evaluation also compares inferred programs with expert-designed annotations and explores composing requests for approximate parametric models and generating shape variation families.The study reports ablations of the LLM prompting workflow and CAS solver alongside two extensions.Source: §1 · p. 2

2 RELATED WORK

Prior work spans analyze-and-edit tools, language-based 3D editing, and parameterized editing. The section contrasts these approaches’ requirements and scope with ParSEL’s intent-driven analytical programs for geometric edits.

Analyse-and-edit approaches

  • Analyze-and-edit approaches cover tasks from non-homogeneous resizing [27] and articulated objects to architecture and SVG patterns, but typically require an initial UI edit and repeated numerical optimization.ParSEL instead precomputes analytical programs for real-time magnitude adjustments.Source: §2 · p. 3

Language based shape editing

  • Language-based 3D systems have handled texturing and stylization, while geometric editing remains limited by weak spatial understanding, low-resolution representations, magnitude ambiguity, or entangled edits.The cited trained approaches often manipulate low-resolution point clouds or implicit functions.Source: §2 · p. 3

Parameterized editing of Shapes

  • Parameterized editing work spans SVG icons, image editing [10] [17], and shape abstractions [22] [23], but data-driven controls may miss user intent or remain unlabeled.ParSEL infers controls from the user’s edit request and searches for analytical edits to maintain inter-part relationships.Source: §2 · p. 3
  • ParSEL makes parameterized editing of non-parametric models easier and uses natural language to explore procedural models for the same geometry, unlike approaches focused on editing parametric models.Its inverse parametric modeling application relates to prior work, but uses language in addition to geometric analysis.Source: §2 · p. 3

3 OVERVIEW

ParSEL turns a segmented mesh and language request into a parameterized editing program, using an LLM to infer an initial edit and Analytical Edit Propagation (AEP) to extend it.

3 OVERVIEW

  • ParSEL takes an instance-segmented mesh and edit request, then exposes parameters for interactively exploring variations in edit magnitude.The LLM infers a seed edit; AEP adds further operations to complete the program.Source: §3 · p. 3
  • The LLM can identify an initial operation but often misses interdependent operations needed for a complete editing program.AEP uses geometric reasoning to extend the seed edit, producing programs intended to preserve geometric fidelity and essential shape features.Source: §3 · pp. 3–4, Figure 2 · p. 4
  • LLM Augmented Inference dynamically modifies shape structure based on edit requests to support different editing intents.The paper also describes a prompting workflow and techniques intended to improve response robustness and accuracy.Source: §3 · p. 4

4 PARAMETERIZED SHAPE EDITING

This section presents a graph-based shape abstraction, a DSL for parameterized edits, and AEP to address the limits of direct LLM inference.

4.1 Structured Shape Abstraction

  • The shape is modeled as a graph of semantically labeled parts abstracted by hexahedral cages, with edges representing relations such as symmetry or attachment.Cage edits deform the underlying mesh using harmonic coordinates; attachment relations constrain relative movement to preserve physical connections.Source: §4.1 · p. 4

4.2 A Language for parameterized shape editing

  • The DSL parameterizes part and feature transforms and symmetry edits with symbolic Amount expressions, supporting composable, parallel shape editing.Its operations include translation, rotation, scaling, and shearing; feature targeting also supports non-affine transforms such as tapering.Source: §4.2 · pp. 4–5
  • A seat-scale seed triggers leg shifts and back scaling to repair attachments, potentially breaking bar links before propagation stops.The process continues until no relation remains broken or all parts have been edited.Source: Figure 3 · p. 5

4.3 Limitations of Direct LLM Inference

  • LLMs often infer the seed operation but miss secondary edit types and magnitudes; CAS-based AEP adds operations to align the program with intent.The paper identifies complex mathematical expressions for operation magnitudes as a challenge for direct LLM inference.Source: §4.3 · p. 5

5 ANALYTICAL EDIT PROPAGATION

AEP represents parts, edits, and inter-part relations algebraically, then searches for parameterized corrective edits that preserve relations across the control range.

5.1 Edit Propagation

  • Propagation iteratively repairs broken inter-part relations by editing previously unaltered parts until no relations remain broken or all nodes have been edited.New edits may break further relations, prompting additional corrections.Source: §5.1 · p. 6
  • Searching once for parameterized corrective edits, rather than optimizing each instantiated seed edit, trades initial computation for smooth real-time parameter adjustment.Traditional numerical optimization can be computationally intensive and depend on graph size.Source: §5.1 · p. 6

5.2 Expressing the Shape Algebraically

  • Symmetry and attachment relations become parameterized algebraic constraints on part hexahedra, checked at uniformly sampled control values because analytical checking can be expensive.Each part hexahedron is represented as a function of the control parameter.Source: §5.2 · p. 6

5.3 Analytical Edit Solver

  • The solver samples feasible DSL operator parameters, then uses CAS to find Amount functions that satisfy broken constraints across the input range.Figure 4 depicts broken-constraint detection, candidate-edit sampling, and the search for valid Amount expressions.Source: §5.3 · p. 7, Figure 4 · p. 7
  • Because mixed constraint sets may defeat CAS, AEP solves constraints independently, numerically checks combined validity, then favors lower ARAP energy and more intrinsic symmetry planes.The selection criterion treats lower ARAP deformation energy and more intrinsic symmetry planes as preferable.Source: §5.3 · p. 7

5.3.1 Improving Solver Robustness.

  • Robustness features add candidates based on other edited parts when the initial set fails and, for unsatisfiable constraints, select the edit breaking the fewest constraints.The added candidates are called nhbd-edits; minimally constraint-breaking choices are called breaking-edits.Source: §5.3.1 · p. 7

6 LLM AUGMENTED INFERENCE

Naive propagation can conflict with edits that intentionally break relations or require non-minimal secondary changes. ParSEL uses LLM-inferred intent to modify shape structure and guide AEP's candidate search.

6.1 A Limitation of Naive Edit Propagation

  • Naive propagation assumes users want inter-part relations preserved and the simplest secondary edits, but intentional symmetry breaking or preferred leg tilts can violate those assumptions.For example, a user may want to edit only a chair’s back legs or tilt legs when widening the seat.Source: §6.1 · pp. 7–8
  • Traditional propagation can require manual structure changes, such as removing a reflection relation or adding a floor attachment, to support different edits.Users must adjust the shape’s structure so generated edits align with their intent.Source: §6.1 · p. 8

6.2 Dynamic Structure Modification Using LLMs

  • ParSEL has the LLM infer a seed edit, symmetry-relation validity, and part type-hints, then uses those outputs to modify constraints and filter edit candidates.This supports requests that conflict with symmetry or specify secondary-part operations.Source: §6.2 · p. 8

6.2.1 Prompting Workflows.

  • Separate prompting workflows handle the three inferences; relation validity is prompted per symmetry relation, and type-hints use translate, rotate, or scale.Each workflow receives part descriptions, the request, and task-specific instructions, then returns executable Python for parsing and execution.Source: §6.2.1 · p. 8
  • CoT, in-context examples, reminders, and majority voting improve LLM responses; five samples suffice for reported voting gains, while parallel prompts keep response times comparable to one call.The prompts have no interdependence, allowing the three tasks to run in parallel.Source: §6.2.1 · p. 8

7 EXPERIMENTS

The experiments compare ParSEL with alternative program-inference designs on segmented 3D assets, using human preference, ground-truth program metrics, and component ablations to assess performance.

7.1 Datasets

  • The test set contains 50 CoMPaT3D++ models from 21 categories, while the PartNet development set contains 51 segmented meshes from five furniture categories.Development models range from 5 to 81 parts; each pair also has a manually written request, and development pairs have expert-derived ground-truth programs.Source: §7.1 · p. 8

7.2 Language-based Parameterized Editing of 3D Assets

  • One-shot LLM produces unrealistic part intersections, Seed Only can miss edit intent, and Ours closely matches requests.The comparison illustrates the qualitative differences among the three system variants.Source: Figure 5 · p. 9, §7.2.2 · p. 10

7.2.1 Analysis of Human Preference.

  • In a two-alternative forced-choice perceptual study comparing our system with two alternate realizations, Ours was preferred over One-shot LLM in 81.1% of judgments and over Ours Seed Only in 75.6% of comparisons.Source: Table 1 · p. 9, §7.2.1 · p. 9
  • The perceptual study compares method variants through pairwise videos of edits whose control parameter varies across its range.The program comparison uses programmatic, geometric, and structural criteria against expert-derived ground truth.Source: §7.2.1 · p. 9, §7.2.2 · p. 9

7.2.2 Analysis of Inferred Programs.

  • Ours performs best across Programmatic, Geometric, and Structural metrics, while One-shot LLM fails across all three and Seed Only has high geometric distance.Seed Only can restore inter-part relations while still deviating geometrically from the intended shape.Source: Table 2 · p. 9, §7.2.2 · p. 10

7.3.1 Analysis of the Prompting Workflow.

  • Voting and in-context examples have the strongest prompting-ablation effects, while removing Chain-of-Thought causes a smaller accuracy decline.The best workflow fully matches human annotations for approximately 35% of pairs, although annotators judged Ours to match intended edits in 72.5% of inputs.Source: §7.3.1 · pp. 10–11, Table 3 · p. 10

7.3.2 Analysis of the AEP Solver.

  • Removing nhbd-edits increases geometric distance, whereas removing breaking-edits increases structural distance.These effects correspond to parameterization quality and the number of broken relations, respectively.Source: Table 4 · p. 10, §7.3.2 · p. 11

8 APPLICATIONS

The applications use ParSEL’s parameterized editing programs to build approximately procedural models and sets of distinct interpretations of an under-specified edit request.

8.1 Automatic Proxydural Modeling

  • Independent editing programs can be composed into a slider-driven Proxydural Model for exploring variations of a shape.Each program has one independent control parameter, and the edited parts are represented through bounding proxies.Source: §8.1 · p. 11
  • LLMs can generate compatible edit requests for a segmented shape, enabling multiple Proxydural Models for one input geometry.The models support exploration of different shape variations, unlike prior inverse procedural modeling approaches.Source: §8.1 · p. 11

8.2 Creating a Shape Variation Family

  • Shape Variation Families preserve a requested primary edit while varying secondary effects to offer distinct realizations of an under-specified request.Users can explore the resulting programs and choose the interpretation that best matches their intent.Source: §8.2 · p. 11

9 CONCLUSIONS

ParSEL combines language-based requests and adjustable parameters with LLM prompting and CAS solvers to infer edits that account for inter-part relationships. The paper reports closer alignment with user intent than baselines, while noting limits in search efficiency, supported deformations, and shape structures.

9 CONCLUSIONS

  • ParSEL infers parameterized editing programs from semantically labeled meshes and language requests, with AEP extending programs using inter-part geometric relationships.Experiments report closer alignment with user intent than other baselines, and the system is demonstrated for approximately procedural modeling.Source: §9 · p. 12

9.1 Limitations and future work

  • Analytical edit search takes a median of about 30 seconds and can exceed 500 seconds for very complex shapes.The authors attribute the cost to generous candidate sampling and suggest smarter sampling or early exit as possible mitigations.Source: §9.1 · p. 12
  • Supported deformations are limited to affine transforms of bounding proxies or their features; additional deformation types require analytically specified inter-part relations.The system does not support iterative non-analytic deformations such as ARAP energy-based deformation.Source: §9.1 · p. 12
  • Bounding boxes may not suit every part, and the solver’s response to other control cages is unclear; the modeled symmetry relations are also limited.The paper identifies joined points on two curves and two-axis wallpaper symmetry as structures for future support.Source: §9.1 · p. 12

Related papers

Loading 2405.20319v2…