PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Prompts: Unconditional 3D Inversion for Out-of-Distribution Shapes

Victoria Yue Chen, Emery Pierson, Léopold Maillard, Maks Ovsjanikov

arXiv:2604.14914v2cs.CV

TL;DR

Native text-to-3D models can become insensitive to out-of-distribution prompts through latent sink traps, undermining text-driven inversion and editing. The paper analyzes sampling trajectories and leverages the unconditional generative prior to stabilize inversion, then demonstrates high-fidelity editing of complex out-of-distribution shapes using only a native 3D model and target text. Its method remains bounded by the model distribution, and large semantic shifts on rare poses can yield implausible geometries.

  • Problem

    Out-of-distribution source prompts can misalign with 3D shapes, causing prompt-guided inversion to fail and undermining downstream editing.

  • Method

    The paper analyzes sampling trajectories and performs inversion with the unconditional prior, followed by target-prompt resampling for editing.

  • Results

    The approach achieves high-fidelity reconstruction and text-driven editing of complex, non-rigid, out-of-distribution shapes without auxiliary models or image-based priors.

  • Takeaways & Limitations

    Separating geometric representation from linguistic sensitivity provides a framework for editing shapes that standard text conditioning cannot handle.

  • Takeaways & Limitations

    Large semantic shifts on rare poses can produce unrealistic geometries because editing trajectories may become unstable and geometric plausibility is not explicitly enforced.

Abstract

from arXiv · show

Text-driven inversion of generative models is a core paradigm for manipulating 2D or 3D content, unlocking numerous applications such as text-based editing, style transfer, or inverse problems. However, it relies on the assumption that generative models remain sensitive to natural language prompts. We demonstrate that for state-of-the-art native text-to-3D generative models, this assumption often collapses. We identify a critical failure mode where generation trajectories are drawn into latent "sink traps": regions where the model becomes insensitive to prompt modifications. In these regimes, changes to the input text fail to alter internal representations in a way that alters the output geometry. Crucially, we observe that this is not a limitation of the model's \textit{geometric} expressivity; the same generative models possess the ability to produce a vast diversity of shapes but, as we demonstrate, become insensitive to out-of-distribution \textit{text} guidance. We investigate this behavior by analyzing the sampling trajectories of the generative model, and find that complex geometries can still be represented and produced by leveraging the model's unconditional generative prior. This leads to a more robust framework for text-based 3D shape editing that bypasses latent sinks by decoupling a model's geometric representation power from its linguistic sensitivity. Our approach addresses the limitations of current 3D pipelines and enables high-fidelity semantic manipulation of out-of-distribution 3D shapes. Project webpage: https://daidedou.sorpi.fr/publication/beyondprompts

1 Introduction

Text-driven inversion promises simple 3D editing, but out-of-distribution prompts expose a mismatch between geometric expressivity and text reasoning. The paper analyzes unstable sampling trajectories and uses unconditional priors to enable high-fidelity inversion and text-driven editing without auxiliary models.

  • Motivation: Text-driven inversion maps an input into a pretrained model’s noise space, enabling prompt-based editing, style transfer, and inverse problems.Users can modify a prompt and denoise the latent representation to obtain an edited asset.
  • Problem: Current 3D pipelines often require masks, 2D in-painting models, or image-based priors because text-only inversion and editing have not produced reliable native 3D results.The paper attributes this gap to a mismatch between geometric expressivity and text-reasoning capacity.
  • Approach: Unconditional priors stabilize internal sampling trajectories and enable faithful inversion of complex, non-rigid shapes that standard text conditioning cannot reconstruct.The approach leverages the unconditional distribution rather than relying on mismatched prompts during inversion.
  • Outcome: The proposed framework enables high-fidelity 3D editing driven solely by a target text prompt and a native 3D generative model.It removes the need for auxiliary image-based priors.
  • Failure mode: Sink traps occur when diverse textual conditions fail to trigger geometric changes, despite the model’s ability to represent diverse shapes.For example, diverse prompts for a rabbit can yield nearly identical geometries.

2 Related Work

Related inversion methods originated in image-based generative models and have expanded to flow and 3D editing settings. However, existing 3D frameworks commonly rely on 2D priors or auxiliary models rather than exclusively using native 3D generative capabilities.

  • Image-based inversion: Image-based inversion maps data samples back to latent codes, with Null-Text Inversion optimizing unconditional embeddings to preserve fidelity during prompt-based editing.Direct DDIM inversion can suffer from numerical error accumulation and classifier-free guidance challenges.
  • Flow and latent-space methods: Flow-based and inversion-free methods still require a latent representation and a source prompt describing the original content.The applicability of these latent-space techniques to native 3D latent spaces remains largely unexamined.
  • 3D editing paradigms: 3D asset manipulation commonly follows SDS optimization, multiview-consistent image generation, or direct 3D manipulation coupled with 2D inpainting.These paradigms use image generative or multiview priors to guide editing and preserve geometric coherence.
  • 3D editing paradigms: Early SDS-based 3D editing applies gradients from pretrained 2D diffusion models but is computationally expensive and slow because each edit requires optimization.The approach can operate on NeRFs and explicit meshes.
  • 3D editing paradigms: Multiview editing methods improve efficiency by generating consistent edited views and then reconstructing or refining globally coherent geometry.Their workflow relies on multiview-consistent diffusion models.
  • Native 3D editing: Recent native-3D editing methods modify conditioning inputs or use 2D inpainting, while relying on internal voxel representations to maintain structural fidelity.They do not primarily manipulate the model’s internal latents.
  • Motivation: Existing 3D editing frameworks rarely rely exclusively on native generative capabilities, motivating an isolated study of native 3D latent spaces.The paper positions its analysis as examining the inherent properties of the 3D generative model.

3 Preliminaries

The paper formulates 3D inversion as mapping a shape latent back to noise through a text-to-3D flow model. It then describes Euler inversion, classifier-free guidance, and null-text optimization as the foundations for reconstruction and editing.

  • Problem formulation: The inversion problem takes a 3D-VAE latent, a source prompt, and a trained text-to-3D flow model, returning a noise latent whose reintegration reconstructs the input.A successful inversion places the mesh in a representation that the model can manipulate with an edit prompt.
  • Rectified flow and inversion: Rectified flow transports a simple prior to the data distribution by integrating an ODE, while inversion integrates the process forward from the data latent to noise.The paper uses forward Euler steps as a deterministic inversion baseline.
  • Rectified flow and inversion: Euler inversion updates each latent by adding a timestep-scaled guided velocity evaluated at the current latent and time.The guided velocity is modified by classifier-free guidance.
  • Rectified flow and inversion: Classifier-free guidance combines conditional and null-condition velocities using a guidance scale to modify the sampling trajectory.The null condition is denoted by ∅ in the formulation.
  • Null-text optimization: Null-Text Inversion optimizes unconditional embeddings along the forward trajectory so sampling follows the inverted reference path while preserving shape identity.This improves reconstruction when text-shape alignment is imperfect and supports later prompt-based editing.
  • 3D generative model: The study uses TRELLIS, whose first rectified-flow stage generates a low-dimensional coarse structure and whose second stage generates sparse latents for texture details.The first stage defines global geometry, while the second refines the resulting representation.

4 Expressivity of Language and Geometry

Native text-to-3D models can lose sensitivity to prompt variation in localized sink regions, even though their unconditional latent spaces retain substantial geometric expressivity. The paper analyzes this mismatch and uses unconditional inversion to stabilize reconstruction and enable editing of out-of-distribution shapes.

  • 4.1 Language Diversity: Localized sink regions make native 3D generators converge toward generic geometries despite varied semantic prompts.TRELLIS exhibits near-identical topology and can produce nearly identical geometry and texture for different prompt attributes in some semantic categories.
  • 4.1 Language Diversity: The diversity ratio R = ∆vis/∆txt measures responsiveness to linguistic shifts, and most evaluated subjects across TRELLIS, LN3Diff, and GaussianAnything have R < 1.The ratio compares visual and textual diversity and is related to the model’s text-conditioned Lipschitz behavior.
  • 4.2 Out-of-Distribution Prompts: Approximate out-of-distribution prompts destabilize inversion, producing poor reconstructions and substantially higher predicted velocity norms in TRELLIS.The resulting joint embedding fails to align with the initial geometry, while approximate prompts can drive trajectories toward sparse density regions.
  • 4.3 Geometric Expressivity and Text-Editing: Empty prompts stabilize sampling by removing out-of-distribution classifier-free guidance, yielding smaller velocity norms and more reliable inversion trajectories.The unconditional strategy supports high-fidelity reconstruction where approximate text prompts fail, including complex non-rigid shapes.
  • 4.3 Geometric Expressivity and Text-Editing: Unconditional inversion separates geometric representation from linguistic sensitivity and supports open-vocabulary retargeting of arbitrary 3D shapes.The pipeline inverts a shape with a null source prompt, then resamples its noisy latent with an edit prompt, optionally applying null-text optimization to preserve structure.

5 Experiments

Experiments show that unconditional inversion reconstructs complex out-of-distribution shapes more reliably and supports text-guided 3D edits with structural and multiview consistency.

  • 5.2 Evaluation of geometric expressivity: Approximate prompts can harm reconstruction, whereas Euler and null-text inversion with an empty prompt achieve similarly high accuracy.The comparison evaluates Euler inversion and null-text optimization with approximate versus empty prompts.
  • 5.2 Evaluation of geometric expressivity: Null-prompt Euler inversion captures complex, non-rigid DT4D topologies, while approximate prompts often produce implausible geometries or artifacts.The reported explanation is trajectory drift into sparse density regions during inversion.
  • 5.3 Text-Guided Editing Evaluation: VoxHammer fails on 67% of DT4D samples, while TRELLIS native editing primarily changes texture and makes negligible geometric updates.VoxHammer failures are attributed to inversion drift and latent value explosion; it also requires manual 3D masking when successful.
  • 5.3 Text-Guided Editing Evaluation: Our method consistently outperforms all baselines on SigLIP, CLIP, and LLaVA evaluation of edited assets.The evaluation samples 30 rendered views per edited sample and uses a 1–5 scale for LLaVA.
  • 5.3 Text-Guided Editing Evaluation: The pipeline averages 9s per edit using mesh voxelization and three generative sampling passes, while maintaining higher structural integrity.Compared methods require manual masks, extensive view rendering, or over an hour of per-scene optimization.
  • 5.3 Text-Guided Editing Evaluation: The edits remain strictly 3D-consistent, preserving structural coherence and rendering stability across viewing angles.The paper presents multiview renders as evidence of this consistency.

6 Limitations and future work

The method remains bounded by the generative model’s distribution: extreme rare poses can destabilize editing and yield implausible geometries despite successful unconditional inversion.

  • 6 Limitations and future work: Unconditional inversion recovers highly out-of-distribution shapes, but subsequent editing trajectories can become unstable under simple prompts.The limitation concerns editing after recovery, not the unconditional reconstruction itself.
  • 6 Limitations and future work: Large semantic shifts on rare poses may produce unrealistic results because optimization does not explicitly enforce geometric plausibility.The paper illustrates these failure cases with a dancing pose that is likely geometrically out of distribution.
  • 6 Limitations and future work: Future work includes automatically detecting out-of-distribution and low-diversity states to improve editing robustness.The paper also proposes adapting image morphing techniques to 3D shape analysis.

7 Conclusion

The paper identifies prompt-guided inversion drift in native 3D models and addresses it with unconditional inversion, enabling high-fidelity reconstruction and text-driven editing of complex out-of-distribution shapes.

  • 7 Conclusion: Weak text-shape alignment causes standard prompt-guided inversion to drift into sparse learned-distribution regions, producing poor reconstructions and implausible geometries.The conclusion presents this as a fundamental difference from established image-domain inversion methods.
  • 7 Conclusion: Euler inversion with an empty prompt and null-text optimization of unconditional embeddings achieves high-fidelity reconstruction of complex out-of-distribution shapes.The strategy leverages the unconditional generative prior during inversion.
  • 7 Conclusion: Unconditional inversion enables meaningful text-driven editing while preserving the source shape’s global structure and pose.The conclusion connects reconstruction quality with subsequent semantic editing.

Supplementary Material

The supplementary material details the native 3D implementation, including voxel preprocessing, Euler inversion, null-text optimization, and target-prompt editing from recovered latents.

  • Supplementary Material: The supplementary material includes additional prompt-sensitivity comparisons with FLUX and Stable Diffusion to contextualize 3D inversion susceptibility.It also documents algorithms and parameters for Euler inversion and null-text optimization.
  • Supplementary Material: The pipeline uses the TRELLIS text-to-3D flow model, voxelizes input meshes, and records intermediate latents throughout inversion.The recorded trajectory provides the reference latent sequence used by later optimization and editing.
  • Supplementary Material: Null-text optimization introduces a timestep-specific unconditional embedding and minimizes predicted-latent deviation from the reference trajectory.The optimized embeddings are updated during sampling from t = 1 to t = 0.
  • Supplementary Material: Classifier-free guidance combines conditional and unconditional velocities before a backward Euler step computes the next latent.The displayed equations define the predicted velocity and its backward Euler update.
  • Supplementary Material: Editing reuses the recovered noise latent and optimized unconditional embeddings while replacing the conditional embedding with the target edit prompt.Sampling these components produces the edited asset.
  • Supplementary Material: Inversion runs with an empty conditioning prompt using forward Euler updates, storing the final noise latent and reference trajectory.The supplementary algorithm specifies inversion from t0 = 0 to tK = 1.

B Evaluation of Geometric Expressivity

The evaluation links approximate prompts to poorer DT4D inversion, while TRELLIS’s editing strategy preserves coarse structure and therefore offers limited geometric control.

  • Approximate prompts degrade DT4D inversion performance for both standard Euler inversion and null-text inversion.Table 4 reports notably higher L1 and LPIPS scores with approximate prompts, even after null-text optimization.
  • Null-text optimization with an empty inversion prompt provides the strongest DT4D reconstruction combination.
  • TRELLIS’s original editing method retains the coarse structure and reruns only second-stage refinement with a new prompt.Because coarse structure largely determines the final shape, the method mainly changes appearance or texture and only minor geometric details.
  • The DT4D evaluation uses 13 sequences from different characters, with 16 randomly sampled frames per sequence.

D.2 TRELLIS Generated Samples

The supplementary evaluation specifies TRELLIS sampling settings and reports additional inversion and editing evidence across 3D and image generative models.

  • TRELLIS experiments use 50 sampling steps with CFG strength 5 over the t = 0 to t = 1 interval.The prompts are supplied in supplementary experiment files, with the same sampling parameters used for editing experiments.
  • Additional 3D editing results are presented in Figures 11 and 12.
  • For FLUX, empty prompts recover the main image structure, while approximate and ground-truth prompts produce similarly faithful reconstructions.
  • Across COCO and DiffusionDB, empty prompts often yield improved Stable Diffusion reconstructions compared with ground-truth captions.The experiment inverts and resamples 100 randomly selected images from each dataset.

G VLM-based Evaluation

The evaluation uses LLaVA-1.5-7B to score prompt–asset agreement and supplements these scores with visual comparisons of reconstruction and editing behavior.

  • LLaVA-1.5-7B rates how well each resulting asset matches its edit prompt on a 1–5 scale.
  • Scores range from 1 for no match to 5 for a perfect, error-free match.Intermediate scores represent weak, partial, or good prompt–asset correspondence.
  • Average VLM scores are reported over the entire edit dataset for each compared model.
  • The visual comparisons show that the proposed method can alter overall geometry, whereas TRELLIS mainly preserves coarse structure and changes texture.Additional examples vary character descriptions while preserving pose and global structure; image figures compare reconstruction under different prompt types.
Loading 2604.14914v2…