PaperlayerView exact revision on arXiv

Source-linked AI summary

Arbor: Explicit Geometric Conditioning for Controllable 3D Asset Generation

Jan-Niklas Dihlmann, Andreas Engelhardt, Simon Donne, Hendrik P. A. Lensch, Mark Boss

arXiv:2606.23514v1cs.CVcs.GR

TL;DR

Current text- and image-conditioned 3D generators offer limited direct control over occupied and empty space. Arbor adds typed 3D constraint meshes to a frozen text-conditioned generator, improving constraint adherence while preserving variation under fixed constraints.

  • Problem

    Text- and image-conditioned 3D generators lack explicit author control over the occupied and empty space an asset should satisfy.

  • Method

    Arbor encodes typed hull, touch, and avoidance meshes into tokens and routes them through an adapter attached to a frozen 3D generator.

  • Results

    Arbor improves constraint adherence over geometry-free, sampling-time, and trained baselines while preserving variation under fixed constraints.

  • Takeaways & Limitations

    Arbor provides an explicit geometric conditioning interface for controlling hull, touch, and avoidance regions during text-conditioned 3D generation.

  • Takeaways & Limitations

    Arbor does not guarantee that constrained regions receive their intended semantic function and does not directly control later refinement of surface detail or materials.

Abstract

from arXiv · show

Text and image conditioned 3D models now generate convincing assets, but they still offer little direct control over the space an object should occupy or avoid. In authoring, this spatial intent is often known before generation starts. A chair should fit a seating envelope, a prop should leave clearance for motion, or a part should expose a contact surface. Prompts and image views are poor carriers for such constraints, requiring the need for an explicit control interface. We present Arbor, a trainable attachment for text conditioned latent 3D generation. Arbor introduces constraint meshes as a native 3D control interface. The interface uses hull regions where geometry should exist, avoidance regions that should remain empty, and touch regions the object should contact. Unlike completion or whole object scaffold control, these meshes are not target evidence. They are local typed requirements and can include regions where no surface should appear. Arbor keeps this signal as geometry by converting constraint meshes into tokens and learning a routed attachment inside a frozen denoiser. Each latent region can therefore receive the part of the constraint that matters for its spatial location. We evaluate Arbor on automatic and artist curated control benchmarks with hull, avoidance, and touch constraints, and compare the metric trends to a user preference study. Even without dedicated compliance losses, Arbor improves constraint obedience while preserving object quality and variation under fixed constraints.

1 Introduction

Arbor addresses the lack of explicit control over occupied and empty space in text- and image-conditioned 3D generation. It uses typed constraint meshes and a latent-space attachment to steer generation while evaluating both constraint adherence and generation quality.

  • Motivation: Current 3D generators produce convincing assets but provide little direct control over occupied and empty space.Prompts describe semantics, while image conditions depend on viewpoint and appearance.
  • Method: Arbor introduces a unified 3D control interface with hull, touch, and avoidance regions specified as meshes.These regions encode where geometry should exist, what it should contact, and what should remain empty.
  • Method: Arbor converts dense constraint meshes and typed signals into compact latent tokens using frozen geometric encoders.The encoding preserves local structure while providing the constraints as model input.
  • Method: A geometry router assigns local constraint evidence to the latent regions where it matters.This addresses the mismatch between dense constraint meshes and compressed 3D latents.
  • Evaluation: Arbor is evaluated against the backbone without geometry and sampling-time steering baselines using constraint adherence and generation quality.The evaluation introduces Ctrl Score and compares its trends with a user preference study.

2 Related work

Related work spans optimization-based, reconstruction-based, and native latent 3D generation, alongside methods for adding spatial control, editing existing assets, and completing partial geometry. Arbor adapts attached-control ideas to native 3D generation while conditioning before an asset exists rather than modifying or completing one.

  • 3D generation: 3D generation follows optimization-based, reconstruction-based, and native latent 3D approaches with different representations and practical tradeoffs.Optimization methods lift text or image supervision into NeRFs or Gaussian splats but can be slow and objective-sensitive; reconstruction methods infer 3D from images.
  • Spatial control: ControlNet and Adapter methods add spatial conditions through a control branch alongside a frozen text-to-image diffusion backbone.Arbor retains this attached-control intuition but rebuilds it for native 3D generation, where conditions must remain meaningful across viewpoints, compressed latents, and output representations.
  • Asset editing: Editing methods modify a complete asset or scene in a selected region, whereas Arbor conditions a generator before any asset exists.Prior methods include depth-conditioned NeRF editing, multiview-image inpainting mapped back to 3D, and feed-forward voxel-flow editing.
  • Reconstruction: Reconstruction methods rely on inputs describing global structure, including skeleton tokens, coarse primitive proxies, retrieved reference shapes, images, and editable point clouds.These structural or image-based inputs serve as priors for reconstructing or generating the asset.
  • Inpainting and completion: Inpainting and completion methods begin with sparse or partial geometry and grow it into a finished asset, while SpaceControl steers a pretrained 3D denoiser during sampling.The cited approaches use partial geometry, coarse guidance shapes, or sampling-time control to influence completion.

3 Method

Arbor adds explicit 3D geometric control to text-conditioned generation through typed constraint meshes and compact geometry tokens. A learned routed attachment injects local constraint evidence and global object context into a frozen TRELLIS 1 denoiser while preserving the text-conditioning path.

  • Constraint encoding: Arbor repurposes TRELLIS 2’s frozen surface encoder to encode constraint signals through aligned normal channels and binary typed-control channels, leaving the TRELLIS 1 backbone unchanged.The binary channels mark hull, avoid, and touch signals on the surface-aligned representation.
  • Constraint representation: Arbor represents desired spatial behavior with hull, avoidance, and touch meshes encoded as compact latent tokens for the text-conditioned generator.Hull regions indicate where geometry should exist, avoidance regions where it should not exist, and touch regions surfaces the object should contact.
  • Constraint encoding: Unlike native TRELLIS 1 encoding, Arbor’s constraint representation supports interactive regional control without voxelizing and rendering constraints from 150 views.The native route is described as slow and designed for complete objects rather than interactive regional conditioning.
  • Constraint encoding: A learned projection and 3D positional embedding map finer-grid geometry tokens into the TRELLIS 1 latent width while preserving each token’s OVoxel position.The router addresses the mismatch between the constraint tokens’ finer 32^3 grid and TRELLIS 1’s 16^3 latent state by selecting relevant geometry tokens.
  • Routed attachment: Arbor inserts a separate geometry grounding branch after frozen text cross-attention and before the feed-forward update in each TRELLIS 1 block.This placement keeps the pretrained text path intact while allowing geometry to affect the hidden state before the FFN.
  • Routed attachment: The router selects bounded local geometry memory for each query group and supplements it with 96 learned summary tokens that retain object-level context.TopK selection keeps routing cost fixed and ties local context to the relevant constraint region.

4 Evaluation

Arbor is evaluated as a typed geometric control interface for text-conditioned 3D generation, measuring constraint obedience, object quality, variation, and conditioning mechanisms. Across benchmark comparisons, user preference, routing ablations, and continuous sweeps, Arbor follows local constraints while preserving recognizable and varied objects.

  • Evaluation setup: Arbor is evaluated against TRELLIS, Gradient, SpaceControl, Spice-E, and Arbor variants on automatic and manual Toys4K benchmarks with hull, avoidance, and touch constraints.The automatic split contains 128 procedurally generated constraints, while the manual split contains 32 hand-authored constraints; training uses roughly 50k objects with frozen TRELLIS backbones.
  • Metrics: The evaluation reports Hull Hit, Avoidance Violation, Touch Hit, Volume Match, multiview CLIP, and Control Score, a harmonic mean requiring simultaneous performance across geometry and appearance terms.Geometry metrics use a shared 643-voxel grid, and higher Control Score values are better.
  • Benchmark results: Arbor preserves readable objects while following typed local roles, outperforming non-Arbor approaches that either match guides accidentally, distort geometry, or fail to distinguish hull, avoidance, and touch.Arbor and Compliance are essentially tied on the manual split, while all Arbor variants separate clearly from the non-Arbor baselines.
  • User study: 59.2% of pairwise user choices favored the Arbor family across 404 trials from 27 participants.The reported parenthesized preference value merges the three Arbor variants, while the single Arbor row was preferred most often before merging variants.
  • Routing ablation: 84.3% of constraint IoU was retained by local routing alone, identifying routed local evidence as the main conditioning mechanism while global summaries add useful object-level context.Removing the signal stream is reported as most damaging, whereas global summaries alone are insufficient.
  • Smooth control and variation: With a fixed prompt, Arbor follows hull movement through position, scale, and orientation, reacts separately to hull, touch, and avoidance signals, and changes details and proportions across seeds under a fixed constraint.The sweeps do not show snapping to a small set of canonical layouts or loss of object identity; image-conditioned baselines remain close to their fixed input.

5 Conclusion

Arbor provides explicit geometric conditioning for a frozen text-conditioned 3D generator by routing typed constraint-mesh information into denoising. The conclusion reports improved constraint adherence on Toys4K while noting limits in semantic function and later refinement control.

  • 5 Conclusion: Arbor converts constraint meshes into compact tokens and injects them through a routed residual branch into the denoising blocks.This makes hull, touch, and avoidance regions part of the generator input rather than only a sampling-time correction.
  • 5 Conclusion: On Toys4K benchmarks, Arbor improves constraint adherence over the backbone without geometry and sampling-time baselines.
  • 5 Conclusion: Constraint regions provide geometry and typed signals but not full semantic function, so a seat region does not guarantee seat usage when the prompt conflicts.The semantic variant did not yet outperform the routed geometry path itself.
  • 5 Conclusion: Arbor acts only at the sparse structure stage and does not directly control later refinement, where surface detail is produced.

A Supplementary Material

The supplementary material provides supporting methodological, qualitative, variant, configuration, and evaluation details that complement the main paper. It is organized into Sections A.1–A.5.

  • A.1: Section A.1 describes data creation, typed constraint synthesis, PartSAM preprocessing, and semantic annotation generation.
  • A.2: Section A.2 presents qualitative coverage beyond the main figures.
  • A.3: Section A.3 summarizes secondary Arbor variants and internal design choices that were not included in the final method.
  • A.4–A.5: Section A.4 records the paper configuration, while Section A.5 defines the evaluation.

A.1 Data Creation

Arbor’s data pipeline trains on a large multi-dataset object corpus with automatically generated typed constraints sampled online. Evaluation uses fixed automatic and artist-authored benchmark splits, while offline part segmentation and semantic labeling support constraint selection and semantic extensions.

  • Training data: Arbor trains on about 50k objects from ABO, HSSD, and Objaverse XL, while ABO and HSSD alone provide roughly 10k objects for development comparisons.The full corpus remains smaller than TRELLIS’s pretraining scale.
  • Constraint creation: The orchestra of constraints automatically creates hull, avoidance, and touch meshes online during data loading, allowing every training batch to sample fresh control geometry.This system addresses the lack of a directly usable dataset of typed constraint meshes.
  • Constraint creation: Constraint families include semantic-part union hulls, random surface patches, layout blockers, surface clearance regions, and touch supports paired with forbidden half spaces.Training rows can combine nearby hull, avoidance, and touch regions to better represent authoring signals.
  • Evaluation data: Because Toys4K lacks typed constraints, Arbor builds two fixed benchmarks: an automatic split with 128 rows and a manual artist-intent split with 32 rows.Main-paper qualitative figures use only the manual split, while appendix examples also cover the automatic split.
  • Segmentation and semantics: Offline PartSAM segmentation provides semantic object parts for constraint selection and semantic extraction, while separately constructed labels form the basis of Arbor Semantics.Part segmentation is performed as offline preprocessing rather than during training-time loading.

A.2 Extended Results

Extended results show that Arbor generalizes across object categories and varied combinations of hull, avoidance, and touch constraints. Supplementary sweep states further illustrate constraint-following behavior in motion.

  • Qualitative Results: Arbor generalizes across object categories and different mixes of hull, avoidance, and touch constraints in manual and automatic Toys4K splits.Additional qualitative rows are provided for both benchmark splits.
  • Sweep States: Supplementary sweep states are easier to assess in motion, revealing whether objects move smoothly with constraints or begin to collapse.The supplementary media provides additional sweep states beyond the still-frame results.

A.3 Arbor Variants

Arbor variants test decoded constraint losses, semantic text injection, inference-time gradient guidance, architectural alternatives, and control encodings. Arbor remains preferred because the alternatives introduce brittleness, artifacts, weaker validation, or inadequate spatial resolution.

  • Compliance: Arbor Compliance adds decoded hull, avoidance, touch, and margin losses during finetuning, but direct constraint pressure can overpower prompt-driven object plausibility.The reported weights are 0.10 for hull, 0.06 for avoidance, 0.06 for touch support, 0.06 for touch forbidden space, and 0.04 for the margin term.
  • Semantics: Arbor Semantics injects labeled semantic prompts through the text route while preserving routed geometry updates, but remains less validated and not preferred over Arbor.Only text cross attention and its input normalization are reopened; self attention, FFN, and time modulation remain frozen.
  • Gradient baseline: The Gradient baseline guides only the final denoising steps by backpropagating a hull loss into frozen TRELLIS latents, but can cause artifacts and guide copying.It often steers samples better than SpaceControl, while potentially reconstructing the guide or damaging the rest of the object.
  • Variant comparison: Late Adapter and Fused Attention+FFN are viable internal alternatives, but Arbor remains the best-balanced model under the full Toys4K protocol.Late Adapter moves the geometry branch later, whereas Fused Attention+FFN merges text and geometry and reopens the following FFN.
  • Encoding alternatives: Direct voxel encoding failed because it required learning a control representation first, while a coarse 163 control field was too coarse for the final 643 output scale.The coarse version could also copy the signal instead of integrating it, supporting TRELLIS.2 encoding as Arbor’s geometry representation.

A.4 Implementation Details

Arbor is implemented as a routed geometry-conditioning attachment on frozen TRELLIS components, using sparse constraint tokens and localized query groups. Training specifies the optimizer, regularization, constraint-family sampling, and compact segmented source geometry.

  • Paper model: Arbor uses a 12-block, width-768 TRELLIS denoiser over a 16^3 latent lattice with 8 channels per site.Constraint meshes are voxelized at 512^3, encoded into sparse 32^3 tokens, projected to model width, and capped at 2048 local geometry tokens per query group.
  • Trainable scope: Only geometry-facing modules and semantic part layers are trained, while TRELLIS self-attention, text cross-attention, and feed-forward weights remain frozen.The trainable modules include geometry projection, geometry position embedding, routed grounding adapters, global summary modules, and small semantic part layers.
  • Training recipe: The final Arbor run uses 8 GPUs, batch size 4 per GPU, AdamW at learning rate 10^-4, EMA rate 0.9999, fp16 training, and adaptive gradient clipping.Classifier-free dropout is applied independently to text and geometry with probabilities 0.1 and 0.1; no explicit compliance loss is used.
  • Constraint family balance: Every training sample contains one hull family, while avoidance and touchable families are independently activated with probability 0.5 each.Part-based hulls have weight 0.6, and the listed alternative hull types each have weight 0.0667; avoidance weights include 0.35 for layout blockers and 0.35 for inverted carvings.
  • Constraint source geometry: Arbor creates part-based constraints from reduced segmented meshes while preserving exact PartSAM identities for semantic joins.The compact geometry source supports stable training while semantic annotations continue to reference the same part identities.

A.5 Evaluation Details

Evaluation uses frozen benchmark manifests, a shared mesh interface, and a common 643 voxel-grid protocol to compare methods. Metric trends are complemented by an anonymized preference study and responsible-research safeguards.

  • Evaluator protocol: Frozen manifests fix each prompt, typed constraint meshes, and canonical frame before methods are aligned to one mesh-returning evaluation interface.Meshes are voxelized or rerendered under one shared evaluator protocol, enabling comparison across latent generators and training-free guidance baselines.
  • Metric definitions: All control metrics use a shared 643 voxel grid, including Hull Hit for required support, Avoid Viol. for forbidden-volume occupancy, and Touch Hit for touch constraints.Very thin contact errors still require qualitative inspection.
  • Metric definitions: MV-CLIP provides a coarse prompt-semantic check, while the size guard uses source Toys4K occupancy and is not a reconstruction metric.The size guard prevents methods from appearing complete by simply overfilling the guide.
  • Metric definitions: Ctrl. Scr. is a per-sample harmonic mean of applicable positive control terms, averaged over the split so severe failures cannot be hidden by one strong metric.It summarizes, but does not replace, the component columns reported in Tab. 1.
  • User study: The user study collected 404 unlabeled pairwise choices from 27 participants, who judged anonymized outputs by combined control following and object plausibility.Reported percentages are pairwise win rates over trials in which a method appears.
  • Responsible research notes: The study was classified as a low-risk visual preference task using anonymized rendered outputs and asking participants to balance constraint following with prompt-plausible object quality.No paid crowd work dataset was collected for this study.

A.6 Extended Limitations

Arbor’s control remains limited by its sparse-stage operation and by conflicts between constraint meshes and the text prior. Later refinement may soften local details, while extreme constraints can force compromises when semantic guidance is weak.

  • Stage limitations: Arbor operates only at the sparse structure stage, so later SLAT refinement and decoding can soften local geometric details.This can occur even when the coarse support is correct.
  • Constraint conflicts: Very small hulls can conflict sharply with the text prior and force the model into compromises.
  • Constraint conflicts: Weak semantic cues or aggressive keep out regions can also force compromises between the constraints and the prompt.
Loading 2606.23514v1…