PaperlayerView exact revision on arXiv

Source-linked AI summary

Position Forcing: Self-Conditioning 3D Generation

Ziheng Ouyang, Zeqiang Lai, Jiarui Chen, Jiangshan Wang, Yuhao Wan, Jingbo Gong, Xiangyu Yue, Hengshuang Zhao, Qibin Hou, Chunchao Guo

arXiv:2610.10342v1cs.CV

TL;DR

Single-stage VecSet models must infer token positions alongside geometry, whereas multi-stage approaches provide explicit spatial guidance through a separate structure-generation stage. Position Forcing recovers positions from predicted clean latents and feeds back progressively finer positional encodings during denoising. The paper reports competitive performance against both single-stage and multi-stage methods.

  • Problem

    VecSet generation starts from unordered noise tokens and must infer their spatial organization alongside geometry, while multi-stage explicit guidance requires a separate structure-generation model and sampling stage.

    Source: §1 · p. 1

  • Method

    Position Forcing recovers token positions from predicted clean latents and feeds them back through progressively finer quantization as denoising proceeds.

    Source: §1 · p. 2

  • Results

    Position Forcing achieves competitive performance against both single-stage and multi-stage methods.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The framework provides coarse-to-fine spatial guidance for single-stage 3D generation without a separate position-generation stage.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recent single-stage 3D generative models commonly adopt VecSet representations, encoding 3D shapes as unordered sets of latent tokens. However, compared with two-stage methods that provide explicit positional guidance, these models must implicitly infer token positions throughout denoising, limiting their generation quality. We observe that, despite the absence of explicit positional conditioning, VecSet tokens retain recoverable spatial correspondences. Building on this observation, we propose Position Forcing, a position-based self-conditioning framework. During denoising, Position Forcing recovers token positions from the current clean latent estimate, quantizes them at progressively finer resolutions according to the denoising stage, and feeds the resulting positional encodings back into the diffusion Transformer. This progressively refined positional feedback provides spatial guidance at a granularity appropriate to each denoising stage, guiding shape generation along a coarse-to-fine trajectory and substantially improving generation quality without a separate position generation stage. Experiments demonstrate that Position Forcing achieves strong performance among single-stage 3D generative methods and outperforms several competitive multi-stage approaches.

1 Introduction

Single-stage VecSet generation must infer token positions and geometry together, while multi-stage methods add separate structure-generation and sampling stages for explicit spatial guidance. Position Forcing recovers spatial correspondences from clean latent estimates and progressively refines positional feedback, enabling coarse-to-fine single-stage generation.

1 Introduction

  • VecSet generation starts from unordered noise tokens and must establish spatial organization and geometric representations within one denoising process.Multi-stage methods instead first generate coarse spatial structure but require a separate structure-generation model and sampling stage.Source: §1 · p. 1
  • VecSet latent tokens retain recoverable correspondences with their spatial query positions, so spatial information can be decoded from the latent representation.Figure 1 illustrates decoding both shape geometry and token positions from latents.Source: §1 · p. 2, Figure 1 · p. 2
  • Recovering positions from noisy latents with full-resolution encodings can produce incorrect geometry when high-noise predictions supply unreliable fine-grained relationships.This motivates recovering positions from predicted clean latents instead.Source: §1 · p. 2
  • Position Forcing quantizes positions coarsely at high noise and progressively increases resolution as noise decreases, refining guidance from global layout to local relationships.The granularity of positional conditioning is matched to the denoising step.Source: §1 · p. 2
  • During inference, the model recovers positions from the predicted clean latent and feeds them through RoPE for subsequent denoising; training uses accurate query positions with the same quantization.These updates provide spatial guidance within a single denoising trajectory.Source: §1 · p. 2, Figure 2 · p. 3
  • Position Forcing guides generation along a coarse-to-fine spatial trajectory, enabling high-quality single-stage 3D generation without a separate position-generation stage.It combines position recovery from predicted clean latents with progressively refined positional feedback.Source: §1 · pp. 2–3

2 Related Work

Related work covers diverse 3D shape-generation approaches, including compact latent-token models and methods operating on spatially structured grids. It also reviews generation ordering and self-conditioning as ways to organize information emergence and reuse prior predictions or latent states.

Shape Generation

  • 3D shape-generation research spans GAN/VAE models, 2D diffusion priors, feed-forward reconstruction, optimization-based methods, autoregressive meshes, and direct 3D representations.Direct 3D methods broadly use compact VecSet latent tokens or spatially structured sparse grids.Source: §2 · p. 3
  • VecSet methods generate in compressed latent space and recover geometry through a learned decoder, while LATTICE adds voxel anchoring and layout-first generation.Examples include 3DShape2VecSet (Zhang et al., 2023), Michelangelo (Zhao et al., 2023), and LATTICE (Lai et al., 2026).Source: §2 · p. 3

Generation Ordering and Self-Conditioning

  • Generation ordering structures information emergence through noise schedules, hierarchies, spatial scales, token granularity, or structured intermediate states.Self-conditioning methods such as Analog Bits (Chen et al., 2022) and RIN (Jabri et al., 2022) reuse clean-sample predictions or internal latent states for subsequent refinement.Source: §2 · p. 3

3 Method

Section 3 builds Position Forcing around latent-token position recovery: it trains a position-aware VAE and feeds progressively quantized positional conditions into a diffusion Transformer.

3.1 Preliminaries

  • VoxSet anchors tokens to voxel-center queries; latent rectified flow trains a Transformer on interpolated noise–geometry latents to predict velocity and generate clean geometry latents.Inference starts from Gaussian noise and integrates the predicted velocity from t = 1 to t = 0.Source: §3.1 · p. 4

3.2 Position VAE

  • Position VAE first trains a position decoder on frozen latent tokens using corresponding jittered queries as direct supervision, then jointly finetunes the encoder and both decoders.Frozen-latent training alone yields limited position accuracy that degrades under noise, while joint training improves recovery under noise and downstream generation quality.Source: §3.2 · pp. 4–5

3.3 Position-Forced DiT

  • Position-Forced DiT recovers positions from predicted clean latents, progressively quantizes them, and injects positional conditions through 3D RoPE for subsequent denoising.Coarse grids at high noise preserve global layout and reduce sensitivity to position errors; finer grids at lower noise provide more precise local relationships. Full-resolution conditioning on noisy-state position predictions often produces incorrect geometry.Source: §3.3 · pp. 5–6
  • Training retains single-timestep flow matching instead of unrolling sampling history, using quantized ground-truth query positions with perturbations; inference conditions on positions from the preceding step.Perturbations improve robustness to position-prediction errors and coordinate jumps between quantization levels.Source: §3.3 · p. 6

4 Experiments

Position Forcing performs strongly in image-conditioned 3D generation, and the ablations examine reconstruction, positional conditioning, position recovery, and joint VAE training. The results support progressively refined spatial feedback and position-aware latent representations.

4.1 Reconstruction

  • Increasing latent size consistently improves reconstruction, with the largest configuration achieving the best results on the held-out mesh set.The evaluation reports Chamfer Distance and F1 against TripoSG and Hunyuan3D-2.1; lower CD and higher F1 indicate better reconstruction.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 Generation

  • Position Forcing achieves the best or tied-best performance across all four metrics, outperforming both single-stage and multi-stage baselines.Qualitative results also show improved structural consistency and finer geometric details.Source: §4.2 · p. 7

4.3 Ablations and Analysis

  • Progressive quantization reduces fragmented house structures and irregular character surfaces compared with fixed resolution 128 during denoising.The results support coarse positional conditions early, followed by progressively finer spatial guidance.Source: §4.3 · p. 9
  • Using predicted clean latents for position recovery improves all four metrics over noisy-latent recovery, while inaccurate training-time positions limit the gains.Ground-truth query positions during training provide a more accurate spatial reference; under this training setup, clean-latent recovery improves geometry completeness and all four metrics over noisy-latent recovery.Source: §4.3 · pp. 9–10
  • Joint VAE training reduces fragmentation, preserves details, and scores higher on all four metrics than replacing it with the Stage-I VAE.Together with improved position recovery, the results indicate that joint geometry and position learning yields a more effective spatially aware representation for downstream generation.Source: §4.3 · p. 10
  • Progressive quantization improves positional cell accuracy across sampling, while clean-latent positions align with final positions earlier than noisy-latent positions.These designs provide coarse-to-fine spatial guidance that better aligns with final generated positions.Source: §4.3 · p. 10, Figure 5 · p. 10

5 Conclusions

Position Forcing recovers spatial guidance from VecSet geometry latents and uses it during denoising. The paper reports competitive performance against single-stage and multi-stage methods.

5 Conclusions

  • Position Forcing is a self-conditioning framework for single-stage 3D generation that recovers positional guidance from VecSet latents and uses it during denoising.Position VAE enables position recovery from geometry latents, while Position-Forced DiT uses the recovered positions as spatial guidance.Source: §5 · p. 10
  • Experiments demonstrate competitive performance against both single-stage and multi-stage methods.Source: §5 · p. 10

A.1 Implementation Details

The Position VAE uses a staged training procedure, followed by joint finetuning, with specified architecture and loss settings.

A.1 Implementation Details

  • The Position VAE uses 64-channel latent tokens, an 8-layer geometry encoder, a 16-layer geometry decoder, and a 4-layer position decoder.Training first freezes the pretrained VAE while training the position decoder, then jointly finetunes the encoder and both decoders; query offsets are sampled from U(−1/64, 1/64), with λpos = 1 and λKL = 10−6.Source: Position VAE · p. 14

Position VAE.

The image-conditioned generator uses a FLUX-based Transformer with DINOv2-Giant image features and 3D rotary positional encoding.

Position VAE

  • The FLUX-based Transformer has 12 double-stream and 24 single-stream blocks, with 1536 hidden dimensions and 16 attention heads.It uses 3D RoPE with 32 channels per spatial axis and frequency base 10,000; DINOv2-Giant image features are concatenated with geometry tokens, and training images are 518 × 518 with image-condition dropout probability 0.1.Source: Position-Forced DiT · p. 14

Position-Forced DiT.

The section specifies DiT training, positional quantization and perturbation, and inference settings for Position Forcing.

Position-Forced DiT.

  • DiT training keeps the Position VAE frozen, increases training from 4096 to 6144 latent tokens, and uses uniformly sampled flow-matching timesteps with AdamW and a warmup-cosine learning-rate schedule.The base learning rate is 2 × 10^-5; gradients are clipped to a maximum norm of 1.0.Source: Position-Forced DiT · p. 14
  • The position-resolution schedule progresses from a single cell at high noise to resolution 128, while training-time perturbations are applied after quantization and mapping with probability 0.5.Perturbations are disabled at resolution 1 and during inference.Source: Position-Forced DiT · p. 14
  • Inference uses 12288 latent tokens and 50-step Euler sampling with classifier-free guidance at scale 5.0, then decodes latents and extracts meshes at resolution 512.The initial step uses resolution 1; later positions come from the preceding step’s clean latent prediction.Source: Position-Forced DiT · p. 14

A.2 Training and Inference Algorithms

Algorithm 1 summarizes training with quantized encoder query positions and inference with positions recovered from predicted clean latents.

A.2 Training and Inference Algorithms

  • During training, quantized query positions from the frozen VAE encoder provide spatial guidance to the DiT.The pseudocode omits training-time positional perturbations and image-condition dropout.Source: §A.2 · p. 14
  • During inference, positions recovered from predicted clean latents are quantized at the next step’s resolution to condition subsequent denoising.The conditional and unconditional CFG branches use shared positional conditions.Source: §A.2 · p. 14
  • The quantized coordinates, named rope_idx, are used to construct 3D RoPE within the DiT.Source: §A.2 · p. 14

A.3 Additional Qualitative Results

The appendix shows additional image-conditioned generations across diverse categories, compares geometry from multiple viewpoints, and visualizes a coarse-to-fine generation trajectory.

A.3 Additional Qualitative Results

  • Position Forcing captures overall shape and local details across plants, stylized characters, buildings, and mechanical objects.Examples include thin leaves, character ornaments, architectural elements, and mechanical components.Source: §A.3 · p. 15

A.4 Multi-view Qualitative Comparison

  • In rear views, Position Forcing produces smoother surfaces and more geometrically plausible details than the baseline, including cleaner character backs and better-preserved mechanical components.The comparison assesses regions not visible in the conditioning images.Source: §A.4 · p. 15, Figure 8 · p. 18

A.5 Generation Trajectory Visualization

  • Across a fixed 50-step sampling trajectory, generation first establishes global structure and rough appearance, then gradually refines local details.The predicted clean sample is visualized at selected steps.Source: §A.5 · p. 15, Figure 6 · p. 15

Related papers

Loading 2610.10342v1…