PaperlayerView exact revision on arXiv

Source-linked AI summary

SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling

Xiaokun Sun, Jian Yang, Zhenyu Zhang

arXiv:2611.02568v1cs.CV

TL;DR

Whole-asset regeneration makes 3D generation, editing, and completion separate tasks even when only a local region needs to change. SeMo3D unifies them by synthesizing selected regions while preserving the rest, and reports stronger performance than specialized state-of-the-art methods across all three tasks.

  • Problem

    Treating generation, editing, and completion as whole-asset regeneration can isolate the tasks, alter content that should remain unchanged, and require task-specific paired data.

    Source: §1 · p. 2

  • Method

    SeMo3D uses selective modeling, region-selective denoising, and Omni3DiT to synthesize selected regions while retaining context, training without task-specific paired data and combining image, text, and point guidance.

    Source: §1 · pp. 2–3

  • Results

    SeMo3D outperforms specialized state-of-the-art methods across generation, editing, and completion.

    Source: §1 · p. 3, §5 · p. 8

  • Takeaways & Limitations

    One selective-modeling framework supports multimodal 3D creation across the three tasks while preserving unchanged content by construction.

    Source: §1 · pp. 2–3

    Trellis2 structured latents limit capture of very fine structures, and the CLIP text encoder struggles with complex compositional prompts.

    Source: §5 · p. 9

Abstract

from arXiv · show

3D generation, editing, and completion are typically treated as regenerating the complete 3D asset under task-specific constraints, even when only a local region needs modification. This leads to task isolation, inconsistency in content that should remain unchanged, and dependence on task-specific paired data. We address these issues with a key observation: the three tasks all synthesize a designated region while keeping the rest intact---the full region for generation, the target region for editing, and the missing region for completion. We term this paradigm Selective Modeling and build SeMo3D upon it; it naturally unifies the three tasks, preserves unchanged content by construction, and enables training without paired editing or completion data, since any 3D asset can be partitioned into a selected region and its context. For training, we propose region-selective denoising, which randomly selects regions of varying sizes on standard 3D assets and applies noise and loss only to them, jointly learning all three tasks. For architecture, we propose Omni3DiT, a multi-condition 3D diffusion transformer that distinguishes regions to create from those to retain and fuses image, text, and point guidance for fine-grained control. Experiments across three benchmarks show SeMo3D surpasses state-of-the-art methods on all tasks, delivering multimodal controllability in generation and strong consistency and accuracy in editing and completion. SeMo3D further serves as a unified modeling tool for agents, enabling efficient 3D creation through natural-language interaction. Project page: https://xiaokunsun.github.io/SeMo3D.github.io/.

1. Introduction

Generation, editing, and completion are usually modeled as whole-asset regeneration, despite sharing a structure: synthesize a designated region while preserving the rest. SeMo3D builds on this selective-modeling view to unify the tasks, train without task-specific paired data, and combine multiple guidance modalities.

1. Introduction

  • Here, editing means local addition, deletion, or replacement, rather than global style transfer.The paper distinguishes these local operations from global edits.Source: §1 · p. 2
  • Whole-asset regeneration leaves generation, editing, and completion isolated, can alter regions meant to stay intact, and makes editing and completion depend on costly paired supervision.The paper gives PartFlow’s 100K editing pairs, curated through a seven-stage pipeline, as an example of this data burden.Source: §1 · p. 2
  • Selective modeling synthesizes only the designated region—the full, target, or missing region for generation, editing, or completion—and reuses the rest.The common structure motivates treating the tasks as variations in which region is selected.Source: §1 · pp. 2–3
  • Because context is reused rather than regenerated, selective modeling preserves unchanged content by construction and needs no task-specific paired data.Any ordinary 3D asset can be partitioned into a selected region and its context.Source: §1 · pp. 2–3
  • Region-selective denoising trains across all three tasks by selecting regions of varying sizes on ordinary 3D assets and applying noise and denoising only to selected tokens.The selected region ranges from local areas to the full asset, spanning editing, completion, and generation.Source: §1 · p. 3
  • Omni3DiT jointly processes clean context and noisy selected tokens while combining image, text, and point guidance for fine-grained control.The introduction reports that SeMo3D outperforms specialized state-of-the-art methods on all three tasks and supports natural-language agent interaction.Source: §1 · p. 3

2. Related Work

Prior work often treats 3D generation, editing, and completion separately or unifies different output modalities rather than these creation tasks. SeMo3D instead frames all three as selective modeling on native 3D representations, with joint image, text, and point conditioning.

2. Related Work

  • Most native 3D generation methods condition on a single modality, while recent pairwise approaches combine image-text or image-point guidance but not all three modalities.The cited work describes images as visually detailed, text as semantically expressive, and points as geometrically precise.Source: §2 · p. 3
  • Native 3D editing and completion methods either repurpose pretrained generators through slow, multi-step inversion or optimization, or use fast feed-forward models that rely on paired datasets and may alter unchanged regions.Examples cited include 100K pairs in PartFlow [72] and 116K in 3DEditFormer [76].Source: §2 · p. 3
  • Existing multimodal 3D models typically unify understanding with generation or bridge 2D and 3D generation, rather than unifying generation, editing, and completion.The passage also notes their reliance on VLM-based architectures and typically one or two conditioning modalities.Source: §2 · p. 3
  • SeMo3D recasts generation, editing, and completion as one selective-modeling problem on native 3D representations, without task-specific architectures or per-task paired supervision.It preserves remaining content by construction and supports joint image, text, and point conditioning.Source: §2 · p. 3

3. Methodology

SeMo3D synthesizes selected 3D regions while retaining clean context, unifying generation, editing, and completion through region-selective denoising and Omni3DiT. The method uses structured latent tokens and combines image guidance with optional text and point conditions.

3.1. Preliminaries

  • Structured latents encode assets as spatially localized tokens across sparse structure, fine-geometry shape, and PBR-texture stages.Rectified-flow models generate each stage's tokens by integrating a learned ODE from noise to data.Source: §3.1 · pp. 3–4

3.2. Overview

  • Generation selects all tokens, while editing and completion select target and missing regions, respectively, so the three tasks differ only in their selected/context partition.Context tokens stay clean; the task-specific partition determines what is synthesized.Source: §3.2 · p. 4

3.3. Region-Selective Denoising

  • Region-selective denoising adds noise and applies loss only to selected tokens, keeping context tokens clean.The selected-set loss is normalized by its size.Source: §3.3 · pp. 4–5
  • Random full-region or local-box partitions train denoising on selected tokens only, supporting all three tasks without paired task-specific supervision.At inference, selected tokens start from noise while context remains clean; free-form target and missing regions are supported despite box-only training.Source: §3.3 · pp. 4–5

3.4. Omni3DiT

  • Omni3DiT jointly processes clean context and noisy selected tokens, using role embeddings and image guidance with optional text and point conditions.Text enters through a parallel cross-attention branch, while sparse points are included in the SS-stage token sequence for spatial reasoning.Source: Figure 3 · p. 4, §3.4 · p. 5
  • Compositional condition dropout trains Omni3DiT with varying subsets of conditions, and its new modules are zero-initialized or additive to largely preserve pretrained behavior at initialization.Image guidance is required at inference; text and sparse points are independently optional.Source: §3.2 · p. 4, §3.4 · pp. 5–6

4. Experiments

Experiments evaluate SeMo3D on Toys4K, Edit3D-Bench, and Omni-Comp, where it reports strong results across generation, editing, and completion. Ablations examine multimodal guidance and model components, and agent integration compares with direct Blender coding.

4.1. Experimental Settings

  • The benchmarks are Toys4K for generation, Edit3D-Bench for editing, and Omni-Comp for completion, with task-specific metrics and standardized 512-resolution outputs.Editing mask-based methods receive the same ground-truth masks, and completion derives the selected region by inverting the partial input occupancy mask.Source: §4.1 · p. 6

4.2. Evaluation

  • SeMo3D leads across the reported generation, editing, and completion evaluations, with qualitative comparisons showing multimodal generation, preserved unedited regions, and coherent completions.For generation, it achieves the best LPIPS, FID, DINO, and CD and second-best KID; it leads in editing metric groups and achieves the best completion CD, EMD, and F-Score.Source: §4.2 · pp. 6–7, Figure 6 · p. 7, Figure 7 · p. 7

4.3. Ablation Study

  • Adding text improves perceptual quality and semantic fidelity, while adding points yields larger geometry gains; the full model combines these cues.With a fixed image, varying text and points produces diverse outputs, demonstrating fine-grained multimodal control.Source: §4.3 · pp. 7–8, Figure 8 · p. 8
  • Compared with direct Blender coding by the same agent, SeMo3D integration yields higher modeling fidelity, faster execution, and lower agent cost.The agent workflow uses GPT-6 Astra with Codex.Source: §4.1 · p. 6, Figure 9 · p. 8
  • Removing role embeddings degrades editing and completion, while point cross-attention and standard CFG dropout weaken point and text guidance.The point cross-attention variant doubles generation CD and degrades completion; CFG dropout retains all available conditions or none.Source: §4.3 · p. 8, Table 4 · p. 8
  • The reported failure cases include coarse palm fronds and missing fruits.The conclusion attributes these limitations to structured-latent resolution and difficulty with complex compositional prompts for the CLIP text encoder.Source: Figure 10 · p. 8, §4.3 · p. 8

5. Conclusion

The conclusion presents SeMo3D as a unified selective-modeling framework for generation, editing, and completion, with region-selective denoising and Omni3DiT. It reports strong results across all three tasks and identifies limits in latent resolution and text-prompt handling.

5. Conclusion

  • SeMo3D synthesizes only designated regions while preserving the rest, and region-selective denoising with Omni3DiT supports all three tasks without task-specific paired data.The model accepts complementary image, text, and point guidance.Source: §5 · pp. 8–9
  • Experiments show SeMo3D surpasses specialized state-of-the-art methods across generation, editing, and completion, while agent integration supports natural-language 3D creation.The conclusion characterizes the agent workflow as intuitive and efficient.Source: §5 · p. 9
  • Structured latents limit capture of very fine structures, while the CLIP text encoder struggles with complex compositional prompts.The conclusion names thin palm fronds and orange fruits rendered as texture rather than objects as examples.Source: §5 · p. 9

6. LLM Usage Statement

The authors report using Codex for manuscript polishing, reviewer-style feedback, code development, experimental organization, and reference checking, and GPT-Image for agentic-experiment reference images.

7. LLM Usage Statement

  • Codex assisted with writing, feedback, code, experiment organization, and reference checking; GPT-Image generated and edited reference images for agentic modeling experiments.GPT-6 Astra with Codex served as the agent, and the authors reviewed the AI-assisted content and take responsibility for the paper’s scientific content and results.Source: §6 · p. 9

7. Training Details

Training uses Trellis-500K assets with randomly selected image and text conditions, while region-selective denoising varies the selected region and keeps context clean. Point conditions are randomly subsampled, and the training schedule uses specified optimizer settings.

7. Training Details

  • Training uses Trellis-500K assets filtered for aesthetic score, with a randomly chosen rendered view and dataset caption as image and text conditions; no paired editing or completion data is used.Assets below aesthetic score 4.5 are filtered out.Source: §7 · p. 13
  • Point training conditions use at most 32 points on the 163 SS grid, with a random count from 1 to 32 selected from spatially distributed points and unused slots masked.Random subsampling is intended to improve robustness to variation in the number and spatial distribution of user-provided points.Source: §7 · p. 13
  • Training settings are listed per stage, and the reported optimizer uses AdamW with learning rate 5×10^-5, weight decay 0.01, EMA rate 0.9999, and bfloat16 mixed precision.All three stages are initialized from Trellis2.Source: §7 · p. 13, Table 5 · p. 13
  • Full-region and local-region modes are sampled equally; local boxes are sampled on the condition-camera-facing half, rejected when sparse, and shrunk to fit occupied voxels.The SS grid has resolution N=16, and its inference partition is transferred to the 323 SLat grid by nearest-neighbor upsampling.Source: §7 · p. 13
  • Compositional condition dropout samples different condition combinations, including individual conditions and the unconditional branch; the standard CFG ablation instead keeps all available conditions or drops them all.For the SS stage, image is withheld in 30% of steps under the compositional scheme.Source: §7 · p. 13

8. Experimental Details

The experiments evaluate generation, editing, and completion on separate benchmarks with task-specific conditions and metrics. They also specify baseline settings, inference procedures, and an agent setup for interactive modeling.

8.1. Benchmarks and Metrics

  • Generation uses 3,180 annotated Toys4K test objects, editing uses 300 Edit3D-Bench cases, and completion uses 180 Omni-Comp partial inputs.The evaluation uses rendered-view and geometry metrics for generation, consistency, quality, and accuracy metrics for editing, and whole-shape geometry metrics for completion.Source: §8.1 · p. 13
  • ULIP and Uni3D scores can disagree with visual quality, so the study relies on metrics computed against ground truth.Trellis2 is described as closer to ground truth than Trellis-I despite receiving lower ULIP and Uni3D scores.Source: §8.1 · p. 14, Figure 11 · p. 14

8.2. Baselines and Inference

  • Baselines use official settings and supported conditions; editing and completion baselines differ in whether they receive edit masks or point guidance.The inference setup uses Euler flow sampling with classifier-free guidance, with different step counts for generation versus editing and completion.Source: §8.2 · p. 14
  • With image and text fixed, point-count and point-distribution tests assess whether overall geometry remains consistent across changed point conditions.The point-count comparison includes 48 points, above the training maximum of 32; changes in spatial distribution may alter local details.Source: Figure 13 · p. 14, Figure 14 · p. 14

8.3. Agentic Interactive 3D Creation

  • The agent invokes SeMo3D for generation, completion, and editing, while a direct-coding baseline creates and modifies assets through Blender code.For editing, it supplies an edited image, target-region mask, and cached source latent; output latents are cached for subsequent edits.Source: §8.3 · pp. 14–15

9. Robustness

Robustness tests vary edit-mask size and point-condition count and placement while holding other conditions fixed. The results report stable overall structure, with local variations under different point distributions.

Mask Size

  • Edits remain nearly identical when the edit region is enlarged by up to 40%, without spurious geometry.The authors attribute this to the SS stage deciding occupancy within the region, so a looser mask does not force content into it.Source: §9 · p. 15

Number of Condition Points

  • With image and text fixed, overall geometry remains consistent across 16, 32, and 48 condition points, including beyond the training maximum of 32.Source: §9 · p. 15

Spatial Distribution of Condition Points

  • Across three point distributions, overall structure remains consistent while local geometry and appearance may vary.Source: §9 · p. 15

10. Additional Results

Additional results show successive agent-directed changes across varied assets and qualitative comparisons across generation, editing, completion, and multimodal control. The examples report preservation of unedited or partial content alongside task-specific changes.

Agentic Interactive 3D Modeling

  • The agentic examples span generation or completion followed by two edits across varied asset categories, with successive user-directed changes maintaining consistency in unedited regions.Examples include accessory addition, part removal or replacement, pose adjustment, and local geometric refinement.Source: §10 · p. 15, Figure 15 · p. 16, Figure 16 · p. 17

3D Generation, Editing, and Completion

  • In additional comparisons, SeMo3D recovers hidden parts in generation, preserves non-target regions in editing, and fills missing parts while retaining completion inputs.The passage contrasts these examples with image-only baselines missing a stein handle and HY3D-Omni duplicating a bicycle frame.Source: §10 · p. 15, Figure 19 · p. 20, Figure 20 · p. 20

Multimodal Control

Related papers

Loading 2611.02568v1…