PaperlayerView exact revision on arXiv

Source-linked AI summary

UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space

Wenjue Chen, Wenyue Chen, Heliang Zheng, Rongfei Jia, Yuan Liu, Meng Li

arXiv:2610.15357v1cs.CV

TL;DR

Generating complete 3D assets from sparse, unposed images requires preserving observed geometry, but image conditioning and joint reconstruction-generation methods face limits in geometric fidelity and coordinate alignment. UniForge3D uses a shared anchor-guided space and structure-guided view selection to jointly predict point maps and a complete sparse structure, then guide detailed shape generation. It achieves state-of-the-art appearance and geometry results on Toys4K and GSO; severe occlusion, strong reflections, and extreme lighting remain challenging.

  • Problem

    Multi-view image conditioning does not strictly constrain generated surfaces to match observed geometry, while joint reconstruction and generation can involve mismatched coordinate spaces and indiscriminate view fusion.

    Source: §1 · p. 2

  • Method

    UniForge3D jointly predicts point maps and a complete sparse structure in a shared anchor-guided space, then uses structure-guided correspondences to select views for detailed shape generation.

    Source: §1 · p. 3

  • Results

    UniForge3D achieves state-of-the-art performance across appearance and geometry metrics on Toys4K and GSO.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The method generates complete 3D assets with faithful geometry and appearance and is reported to generalize well to real-world images.

    Source: §5 · p. 12

    Training uses synthetic renderings, and severe occlusion, strong reflections, and extreme lighting remain challenging.

    Source: §5 · p. 12

Abstract

from arXiv · show

Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity. Project webpage: https://xishuxishu.github.io/uniforge3d-project/

1 Introduction

UniForge3D addresses the challenge of generating complete 3D assets from sparse, unposed images while preserving observed geometry. It jointly models reconstruction and generation in a shared anchor-guided space and uses structure-guided view selection; experiments report state-of-the-art appearance and geometry results on Toys4K and GSO.

1 Introduction

  • The task is to generate complete 3D assets from a few images with unknown camera poses while preserving geometry and appearance observed in every input view.Such assets are relevant to digital content creation, gaming, augmented and virtual reality, and embodied intelligence.Source: §1 · p. 2
  • Multi-view image conditioning does not strictly constrain generated surfaces to match observed geometry, so generative priors may override input-specific structures.Directly conditioning generation on reconstructed geometry or features can also propagate upstream reconstruction errors.Source: §1 · p. 2
  • Joint reconstruction and generation face coordinate mismatch, while indiscriminate view fusion can mix useful evidence with unrelated content and hinder local detail recovery.A region may be clear in one view but difficult to distinguish in another because of oblique angles or occlusion.Source: §1 · p. 2
  • The method jointly models reconstruction and generation in a shared anchor-guided space, with structure-guided view conditioning for local shape detail recovery.The shared space follows the anchor view’s azimuth while keeping the object upright; the branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure.Source: §1 · p. 3
  • UniForge3D achieves state-of-the-art performance on Toys4K and GSO across appearance and geometry metrics.The generated assets preserve structures and visual details observed in sparse, unposed input views.Source: §1 · p. 3

2 Related Work

Related work spans single-view 3D generation, multi-view reconstruction, and multi-view generation. The paper positions its approach against methods that condition generation on reconstructed geometry and joint methods whose branches retain different coordinate spaces.

2.1 Single-View 3D Generation

  • VecSet methods offer compact shape representations and effective global modeling, while sparse-voxel methods retain spatial organization and support local geometric detail.Single-view generation also includes optimizing 3D representations under 2D generative priors and aggregating synthesized multi-view information.Source: §2.1 · p. 3

2.2 Multi-View 3D Reconstruction and Generation

  • Feed-forward multi-view reconstruction estimates geometry from unposed images, but its reconstructions primarily cover observed surfaces and may leave unseen regions incomplete.Prior methods include classical Structure-from-Motion and multiview stereo, learned depth estimation, neural rendering, and feed-forward geometric prediction.Source: §2.2 · p. 4
  • Multi-view generation methods select or fuse image evidence, but image conditioning alone does not ensure that generated surfaces match observed geometry.The approaches include training-free extensions of pretrained single-view generators and training-based feature conditioning or view routing.Source: §2.2 · p. 4
  • Reconstruction-conditioned generation supplies geometric and cross-view information, but errors in reconstruction estimates may propagate into generation.ReconViaGen uses reconstruction features, while RecGen3D additionally incorporates canonicalized point clouds.Source: §2.2 · p. 4
  • Mix3R enables bidirectional interaction between reconstruction and generation, but retains camera-relative reconstruction and canonical-space generation, requiring cross-space alignment.UniForge3D instead places both branches in a shared anchor-guided space and uses structure-guided reconstruction to identify informative views.Source: §2.2 · p. 4

3 Method

UniForge3D jointly reconstructs observed geometry and generates a complete sparse structure in a shared anchor-guided space. Structure-guided point maps establish pixel-to-voxel correspondences, enabling informative views to guide detailed shape generation.

3 Method

  • The pipeline jointly predicts multi-view point maps and a complete sparse structure, then uses structure-conditioned point maps to select views for detailed shape generation.The overview depicts bidirectional branch interaction in the shared space followed by voxel-to-pixel correspondence and selected-view conditioning.Source: §3 · p. 4, Figure 3 · p. 5

3.1 Anchor-Guided Joint Reconstruction and Generation

  • Both branches operate in a shared anchor-guided space and exchange features bidirectionally, allowing observed geometry to guide structure generation and complete-shape context to improve reconstruction.This avoids additional rigid alignment between the branches.Source: §3.1 · p. 5, §3.1.2 · p. 6

3.1.1 Anchor-Guided Shared Object Space

  • The shared object space follows the anchor view’s azimuth while preserving canonical vertical orientation, and both branches’ targets are constructed from geometry in that space.The visible surface points form per-view point-map targets, while complete geometry forms the sparse-structure target.Source: §3.1.1 · p. 5

3.1.2 Bidirectional Joint Modeling

  • Reconstruction features condition sparse-structure generation, while updated generation tokens inject global shape context into reconstruction.The branches exchange features at selected layers, with noise-level gating for generation-to-reconstruction interaction.Source: §3.1.2 · p. 6

3.2 Structure-Guided View Selection and Conditioning

  • Given a generated sparse structure, structure-guided reconstruction identifies observations relevant to each voxel so their image features can guide fine-grained shape generation.The predicted point maps share the voxels’ coordinate system and establish explicit pixel-to-voxel correspondences.Source: §3.2 · p. 6

3.2.1 Structure-Guided Reconstruction

  • The generated structure conditions point-map prediction, allowing image observations and current geometry to inform per-pixel 3D positions in the structure’s coordinate system.The reconstruction network attends to tokens encoding the sparse structure and also predicts confidence maps and reconstruction features.Source: §3.2.1 · p. 7

3.2.2 Voxel-Wise View Selection and Conditioning

  • For each voxel, the selected view is based on valid pixel support; unsupported voxels inherit a view from a directly supported 26-neighbor or use the anchor view.The shape denoiser combines global multiview attention for shape completion with selected-view attention for local evidence.Source: §3.2.2 · p. 7

4 Experiments

Experiments compare UniForge3D with prior methods on held-out Toys4K and GSO objects, then ablate its joint modeling, reconstruction, and view-conditioning choices. Across the benchmarks and ablations, the reported findings favor the full method and its proposed components.

4.1 Experimental Setup

  • Evaluation uses 1,000 held-out objects, with four unposed RGB input views per object and appearance and geometry metrics computed after mesh alignment.The objects comprise 500 Toys4K and 500 GSO examples; appearance metrics include PSNR, SSIM, and LPIPS, while geometry metrics include CD and F-score.Source: §4.1 · pp. 7–8

4.2 Comparison with State-of-the-Art Methods

  • Qualitative comparisons report that UniForge3D better preserves observed geometry and appearance, while competing methods often omit parts or distort local structures.Figure 4 compares aligned meshes rendered from the input-view cameras; Figure 5 presents real-world sparse, unposed image comparisons from common viewpoints.Source: §4.2 · p. 8, Figure 4 · p. 9, Figure 5 · p. 10
  • UniForge3D achieves the best performance across appearance and geometry metrics on both Toys4K and GSO, compared with five prior methods and a Stream3D + TRELLIS.2 baseline.The authors attribute the geometry gains to more accurate geometry and the appearance gains to better preservation under novel viewpoints.Source: §4.2 · p. 8, Table 1 · p. 10

4.3 Ablation Study

  • On GSO sparse-structure generation, reconstruction-feature conditioning improves on DINO conditioning, and anchor-guided joint training achieves the best results across all metrics.Both canonical-space and cross-space joint-training variants perform worse than the anchor-guided formulation.Source: §4.3 · pp. 10–11, Table 2 · p. 11
  • On GSO, jointly trained VGGT outperforms image-only VGGT, while structure-conditioned VGGT produces the most accurate and complete point maps.The comparison evaluates point-map reconstruction distances in normalized object coordinates.Source: §4.3 · p. 11, Table 3 · p. 11
  • For detailed shape generation on GSO, voxel-wise view selection and conditioning improves geometric reconstruction performance over full multi-view conditioning.Both strategies use the same predicted sparse structure.Source: §4.3 · p. 11, Table 4 · p. 11

5 Limitation and Conclusion

UniForge3D jointly models reconstruction and generation in a shared anchor-guided space, using the generated structure to establish pixel-to-voxel correspondences and select views for detailed shape generation. The reported Toys4K and GSO results show faithful geometry and appearance, but severe occlusion, strong reflections, and extreme lighting remain challenging.

5 Limitation and Conclusion

  • The shared-space branches exchange features to predict per-view point maps and a complete sparse structure, which guides reconstruction and selection of informative views for detailed shape generation.Quantitative and qualitative results on Toys4K and GSO, along with the supplementary video, are reported to demonstrate complete assets with faithful geometry and appearance and real-world generalization.Source: §5 · p. 12
  • UniForge3D’s reported benchmarks and comparisons include Toys4K and GSO results and qualitative real-world images.The conclusion describes the real-world image results as evidence of generalization to real-world images.Source: §5 · p. 12
  • The model is trained on synthetic renderings, and severe occlusion, strong reflections, and extreme lighting remain challenging.The authors identify training with simulations of these conditions as future work for improving real-world robustness.Source: §5 · p. 12

A.1.1 Training Details

Training adapts pretrained reconstruction and generation models, then uses generated structure and reconstruction features to condition shape and texture generation.

A.1.1 Training Details

  • The joint model adapts pretrained TRELLIS.2 and VGGT branches with LoRA, fully trains interaction modules, and keeps DINOv3 frozen.The setup uses 32 NVIDIA A800 GPUs, a global batch size of 128, and a peak learning rate of 5 × 10−5.Source: §A.1.1 · p. 19
  • The joint objective combines latent flow matching with point-map supervision, including robust point regression, confidence, and multi-scale normal terms.Robust averaging uses a 98% quantile filter.Source: §A.1.1 · p. 19
  • Separate reconstruction, shape, and texture stages use pretrained initializations; the reconstruction network is frozen to provide point maps and features for generation.Shape and texture training also keep the reconstruction network and DINOv3 frozen; texture generation keeps the shape latent fixed.Source: §A.1.1 · pp. 19–20

A.1.2 Detailed Network Structure

The network couples reconstruction and generation through shared-space feature exchange, then uses structure-conditioned point maps to route image evidence into shape and texture generation.

A.1.2 Detailed Network Structure

  • VGGT reconstruction and DiT generation exchange features bidirectionally in the shared anchor-guided space to predict per-view point maps and a complete sparse structure.Generation features condition reconstruction through time-conditioned cross-attention and gating.Source: §A.1.2 · p. 20
  • The generated structure conditions VGGT point-map prediction, producing shared-space points, confidence maps, and reconstruction features for subsequent generation.Voxel centers are encoded as structure features, which reconstruction features query through cross-attention.Source: §A.1.2 · p. 21
  • Structure-conditioned point maps establish pixel-to-voxel correspondences, while fused reconstruction and image features provide image-aligned conditions for shape generation.The structure also defines the spatial locations of the shape latent.Source: §A.1.2 · p. 21
  • For each voxel, the model scores views by the largest connected component of assigned pixels, favoring the anchor view and then higher mean confidence in ties.It retains the highest-confidence 95% of valid foreground predictions before voxel quantization.Source: §A.1.2 · p. 21
  • Unsupported voxels inherit a selected view from the strongest supported voxel in their 26-neighborhood, or the anchor view if none is supported.Shape generation alternates global multi-view attention with voxel-specific selected-view attention; texture generation keeps the shape latent fixed while updating texture.Source: §A.1.2 · pp. 21–22

A.2.1 Qualitative Ablation of Structure Conditioning

The qualitative analyses compare reconstruction conditioning, view selection, rendered assets, and shared-space alignment, illustrating how the components behave on examples.

A.2.1 Qualitative Ablation of Structure Conditioning

  • Structure-conditioned VGGT produces more complete and coherent geometry than image-only or jointly trained VGGT in the point-map examples.Rows show four input views, the three predicted point-map variants, and ground-truth visible point maps.Source: §A.2.1 · p. 22, Figure 6 · p. 23

A.2.2 Qualitative Ablation of View Selection and Conditioning

  • With the same predicted sparse structure, alternating multi-view and selected-view conditioning better preserves local geometric details than full multi-view conditioning.The figure pairs input details with corresponding generated-shape regions.Source: §A.2.2 · p. 22, Figure 7 · p. 23

A.2.3 Additional Qualitative Results on Rendered Assets

  • Rendered examples show generated meshes aligned to ground truth and viewed from the input cameras, illustrating preservation of observed geometry and appearance.The figure presents four input views alongside generated-mesh renderings.Source: §A.2.3 · p. 22, Figure 8 · p. 25

A.2.4 Visualization of Voxel-Wise View Selection

  • In inference examples, selected views expose queried surface regions more directly and have stronger pixel support than alternatives.The visualization marks query voxels, pixel-to-voxel correspondences, and selected views.Source: §A.2.4 · p. 23, Figure 9 · p. 26

A.2.5 Shared-Space Consistency between Reconstruction and Generation

  • Direct overlays show reconstructed point clouds and generated meshes closely correspond in the shared model coordinate system without object-specific alignment or registration.The examples compare Structure-Conditioned VGGT and Jointly Trained VGGT.Source: §A.2.5 · p. 24, Figure 10 · p. 26, Figure 11 · p. 27
Loading 2610.15357v1…