PaperlayerView exact revision on arXiv

Source-linked AI summary

TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models

Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, Yan-Pei Cao

arXiv:2502.06608v3cs.CVcs.AI

TL;DR

3D shape generation remains constrained by limited data scale, complex processing, and insufficient adaptation of advanced generative techniques, affecting quality and input alignment. TripoSG combines a rectified-flow transformer, geometry-aware VAE supervision, and large-scale curated data, achieving state-of-the-art 3D generation with detailed shapes and strong input-image fidelity.

  • Problem

    3D shape generation faces limited high-quality data, insufficiently explored advanced techniques, and challenges in output quality, generalization, and alignment with input conditions.

  • Method

    TripoSG uses a rectified-flow transformer with scalable architecture, a VAE trained with SDF, normal, and eikonal supervision, and a curated large-scale 3D data pipeline.

  • Results

    TripoSG achieves state-of-the-art 3D generation performance with finer geometry and improved fidelity to input images.

  • Takeaways & Limitations

    The framework indicates that scaling model capacity together with high-quality 3D data and geometry-aware representation learning supports high-fidelity image-to-3D generation.

Abstract

from arXiv · show

Recent advancements in diffusion techniques have propelled image and video generation to unprecedented levels of quality, significantly accelerating the deployment and application of generative AI. However, 3D shape generation technology has so far lagged behind, constrained by limitations in 3D data scale, complexity of 3D data processing, and insufficient exploration of advanced techniques in the 3D domain. Current approaches to 3D shape generation face substantial challenges in terms of output quality, generalization capability, and alignment with input conditions. We present TripoSG, a new streamlined shape diffusion paradigm capable of generating high-fidelity 3D meshes with precise correspondence to input images. Specifically, we propose: 1) A large-scale rectified flow transformer for 3D shape generation, achieving state-of-the-art fidelity through training on extensive, high-quality data. 2) A hybrid supervised training strategy combining SDF, normal, and eikonal losses for 3D VAE, achieving high-quality 3D reconstruction performance. 3) A data processing pipeline to generate 2 million high-quality 3D samples, highlighting the crucial rules for data quality and quantity in training 3D generative models. Through comprehensive experiments, we have validated the effectiveness of each component in our new framework. The seamless integration of these parts has enabled TripoSG to achieve state-of-the-art performance in 3D shape generation. The resulting 3D shapes exhibit enhanced detail due to high-resolution capabilities and demonstrate exceptional fidelity to input images. Moreover, TripoSG demonstrates improved versatility in generating 3D models from diverse image styles and contents, showcasing strong generalization capabilities. To foster progress and innovation in the field of 3D generation, we will make our model publicly available.

1. Introduction

TripoSG addresses quality, geometric-detail, and input-alignment challenges in image-to-3D generation with a rectified-flow model trained on curated large-scale data. Its framework combines scalable generation, geometry-aware VAE supervision, and a data pipeline emphasizing both data quality and quantity.

  • Existing 3D generation methods struggle with artifacts, limited geometric detail, suboptimal quality, and significant alignment gaps between generated models and input images.
  • The data-building pipeline scores, filters, fixes, augments, and converts diverse 3D sources into standardized Image-SDF training pairs.
  • The largest TripoSG model achieves new state-of-the-art performance in 3D generation, with design components validated through mini-setting experiments.
  • TripoSG introduces a large-scale rectified flow transformer for 3D shape generation, leveraging extensive high-quality data to improve fidelity.
  • A hybrid 3D VAE training strategy combines SDF, surface-normal, and eikonal losses to achieve state-of-the-art reconstruction performance.
  • 2M high-quality 3D samples demonstrate the reported importance of data quality and quantity, since improperly processed data can substantially impede training.

2. Related Work

Related work spans deterministic reconstruction and diffusion-based 3D generation, but both face limitations involving consistency, artifacts, representation quality, data scale, or computational cost.

  • Reconstruction-based methods learn 3D geometry directly from images using representations such as point clouds, voxels, meshes, or implicit fields.
  • Some optimization-based approaches require several minutes or hours, whereas newer methods seek faster image-conditioned geometry generation.
  • Diffusion-based methods use voxel, point-cloud, triplane, occupancy, SDF, or VAE-compressed latent representations for 3D generation.
  • Limited large-scale, high-quality 3D datasets have constrained diffusion methods, which often rely on simple datasets and have limited generation ability and practical effectiveness.

3. TripoSG

The TripoSG framework is organized around flow-based generation and sampling, scaling-up strategies, and VAE architecture with supervision.

  • 3.1. Flow-based generation architecture and sampling schedule: The flow-based generation architecture and sampling schedule form the first main part of the TripoSG framework.
  • 3.2. Scaling-up strategy: The scaling-up strategy forms the second main part of the TripoSG framework.
  • 3.3. VAE architecture and supervision: The VAE architecture and supervision form the third main part of the TripoSG framework.

3.1. Rectified Flow Transformer

TripoSG uses a rectified-flow transformer to generate VAE-compressed 3D shape latents conditioned on timestep and image features. Its design combines skip-connected transformer blocks, global and local image cross-attention, and sampling choices intended to improve efficiency, stability, and detail consistency.

  • Architecture: TripoSG encodes 3D shapes into multi-scale latent representations and trains a rectified-flow model to decode them into semantically consistent 3D shapes.The VAE supports latent lengths of 512 and 2048 tokens with 64 channels.
  • Architecture: The flow backbone uses encoder, middle, and decoder transformer blocks linked by long skip connections to improve feature fusion and representational capacity.The backbone contains 2N + 1 blocks, with N = 10, hidden dimension 2048, and approximately 1.5B parameters.
  • Conditioning: Timestep and VAE latent features are projected to a shared hidden dimension, concatenated, and passed into the flow backbone.The concatenated representation has shape (L + 1) × W.
  • Conditioning: Global CLIP and local DINOv2 image features are injected through separate cross-attention mechanisms in every flow block.This design attends to both global and local image information while maintaining detail consistency between generated shapes and inputs.
  • Sampling: Rectified Flow models a linear trajectory from noise to data, simplifying training compared with the curved trajectories of DDPM and EDM.The experiments compare DDPM, EDM, and Rectified Flow, with Rectified Flow selected for the final model.
  • Sampling: Logit-normal sampling increases the weight of intermediate timesteps, where Rectified Flow predictions are more challenging.Resolution-dependent timestep shifting is also introduced to preserve comparable uncertainty as latent resolution increases.

3.2. Model and Resolution Scale-up Strategy.

TripoSG scales generation through larger latent resolutions and a Mixture-of-Experts transformer while controlling computational cost. Its VAE can extrapolate beyond training resolutions, enabling higher-resolution encoding and decoding without retraining.

  • Scale-up: Latent resolution increases from 2048 to 4096, while model parameters scale from 1.5B to 4B through a Mixture-of-Experts architecture.The scale-up targets improved generation quality while minimizing training and inference costs.
  • Resolution: The VAE generalizes beyond training resolutions because it uses no additional input positional encoding and samples queries from a fixed surface-point set.This extrapolation enables direct encoding and decoding at 4096 resolution without VAE retraining.
  • Resolution: Higher query-point counts increase latent resolution and improve the VAE’s modeling capacity.The reported extrapolation ability supports using higher-resolution latent representations directly.
  • Training stability: Q and K are normalized with learnable RMSNorm during high-resolution flow fine-tuning to reduce instability and potential loss divergence in mixed-precision training.The normalization is applied before attention operations within transformer blocks.
  • Mixture-of-Experts: MoE scaling replaces a single FFN with gated parallel experts, routing each token to top-K experts and then restoring the original sequence length.Sparse activation increases parameters while keeping resource usage and inference latency nearly constant.
  • Mixture-of-Experts: The MoE uses 8 experts, activates the top 2 per block, shares one FFN, and expands the final 6 decoder layers.This raises total parameters from 1.5B to about 4B.

3.3. 3D Variational Autoencoder (VAE)

TripoSG replaces occupancy-based 3D representation with neural SDFs to encode finer geometry and reduce aliasing. Its VAE combines SDF, surface-normal, eikonal, and KL supervision, using transformer-based latent-token encoding and decoding.

  • 3D MODEL REPRESENTATION: Neural SDF provides more precise geometric encoding than occupancy, whose reconstructions can show aliasing artifacts and require post-processing.TripoSG uses TSDF for efficiency in the VAE.
  • GEOMETRY LEARNING WITH SURFACE NORMAL GUIDANCE: Surface-normal supervision targets fine-grained geometry at surface points, while eikonal regularization complements the SDF loss.The method also includes KL regularization in the latent space.
  • NETWORK ARCHITECTURE: A latent vector set encodes point clouds and decodes an SDF geometry function through a transformer-based encoder-decoder architecture.The architecture uses cross-attention, self-attention, and linear layers.
  • NETWORK ARCHITECTURE: Marching Cubes extracts the final mesh from the decoded signed distance field at a chosen resolution.
  • NETWORK ARCHITECTURE: Multi-resolution VAE training uses 512 and 2048 tokens with shared weights, enabling inference at higher resolutions such as 4096 tokens.The position-encoding-free transformer supports extrapolation to higher-resolution latent representations.

4. Data-Building System.

TripoSG builds standardized Image-SDF training pairs from heterogeneous 3D sources through scoring, filtering, fixing and augmentation, and field-data production. The pipeline addresses inconsistent source quality and prepares meshes for neural implicit supervision.

  • Data-Building System: The data-building system has four stages: scoring, filtering, fixing and augmentation, and field data production.
  • Data scoring: A learned scoring model selects high-quality models using four rendered normal-map views rated by 10 professional 3D modelers on a 1–5 scale.The scoring model is trained from manually annotated examples.
  • Data filtering: Filtering removes models with large planar bases, animation rendering errors, and multiple objects.
  • Data fixing and augmentation: An orientation model fixes character poses by comparing DINOv2 features from six orthogonal views across 24 candidate orientations.The process also addresses untextured models through additional processing.
  • Field data producing: Because many Objaverse models remain unsuitable after preprocessing, TripoSG converts non-watertight meshes into watertight geometry for occupancy or SDF supervision.

5. Experiments

Experiments progressively scale TripoSG’s data, latent resolution, and model size while evaluating VAE reconstruction and image-conditioned generation. The evaluation combines geometric metrics, Normal-FID, and LMM-based assessment.

  • Experimental setup: The main experiment trains a 1.5B-parameter model on 2M data, progresses from 512 to 2048 latent tokens, and then scales to 4B parameters using MoE.
  • Experimental setup: Ablations use a 180K dataset and a 975M-parameter model to test non-scaling and scaling-up design choices.The non-scaling ablations use 512 latent tokens and approximately 300k training steps.
  • Experimental setup: Foreground resizing to 90% and random center rotation within [−10°, 10°] are used during training to support varied input images.During inference, foreground detection and matching resizing are applied.
  • VAE training: The VAE uses an 8-layer encoder and 16-layer decoder, with surface-normal, eikonal, and KL weights set to 10, 0.1, and 0.001.
  • Dataset: After preprocessing Objaverse(-XL), TripoSG obtains 2 million high-quality 3D objects for training.
  • Metrics: Normal-FID compares generated and ground-truth normal maps rendered from matching viewpoints, while GPTEval3D provides an additional evaluation metric.
  • Metrics: VAE reconstruction quality is measured using Chamfer distance, F-score at a 0.02 threshold, and normal consistency.

5.3. Quantitative and Qualitative Evaluation1

TripoSG is compared with image-to-3D methods using matched-view normal-map visualizations across complex cases. The reported evaluations emphasize semantic consistency, detail, generalization, and multiple human-relevant quality dimensions.

  • COMPARISON WITH DIFFERENT METHODS IN VISUALIZATION: Figure 7 compares TripoSG and prior image-to-3D methods using the same processed input images and normal maps rendered from the same viewpoint.The comparisons include complex and widely varying cases rather than only simple standard images.
  • COMPARISON WITH DIFFERENT METHODS IN VISUALIZATION: The visual evaluation examines semantic consistency, detail, and generalization across the compared generations.Reported examples include clothing textures, accessories, and varied cases.
  • SOTA PERFORMANCE OF TRIPOSG: GPTEval3D uses a Large Multimodal Model to provide evaluation results intended to align more closely with human preferences.The authors adapt the evaluation script and use Claude3.5 instead of GPT-4Vision.
  • SOTA PERFORMANCE OF TRIPOSG: Showcased results cover complex structures, diverse styles, imaginative designs, multi-object compositions, thin surfaces, and richly detailed scenarios.The generated models are presented without post-processing such as smoothing or floater removal.

6. Texture Generation

TripoSG uses rendered normal maps as conditions for multiview texture generation, then projects the generated texture images onto the geometric surface to produce detailed texture maps.

  • Rendered normal maps from TripoSG geometry condition existing multiview generation methods to produce consistent multiview texture images.These texture images are subsequently projected onto the geometric surface.
  • The resulting textured 3D outputs are shown in Fig.12.

7. Ablation and Analysis

Ablation studies show that rectified-flow sampling, skip connections, scaling model resolution and capacity, and high-quality data improve 3D generation. VAE reconstruction benefits from SDF, normal guidance, eikonal regularization, and more training data, while data quality and quantity both matter.

  • 7.1. Ablation for Flow Model: R-Flow sampling outperforms EDM and DDPM, while skip connections produce the most obvious improvement among the flow-model ablations.CLIP conditioning also slightly improves generation results.
  • 7.1. Ablation for Flow Model: Increasing latent resolution consistently improves generation, with the largest gain occurring from 512 to 2048 tokens.MoE increases model parameters, and larger high-quality datasets improve performance more than higher resolution in the reported comparisons.
  • 7.2. Ablation for VAE: SDF with surface-normal guidance and eikonal regularization improves VAE reconstruction geometry over occupancy-based representations.The improvements include lower Chamfer distance, higher normal consistency, and fewer aliasing artifacts; thin-shell reconstruction remains challenging.
  • 7.3. Ablation for Data-Building System: Reducing raw Objaverse data from 800K to 180K while improving quality yields better generation results, indicating that quality outweighs quantity for in-the-wild data.
  • 7.3. Ablation for Data-Building System: Expanding high-quality data from 180K to 2M produces a significant performance boost and greater improvement than quality enhancement alone.Performance continues improving with increased data size without a bottleneck at the current training scale.
  • 7.4. The Visualization for Flow Model Ablation: Visualization results agree that high-quality data scaling improves performance more than raw-data size or resolution increases.The comparison is presented in Fig.10 using rows from Tables 2 and 4.

8. Conclusion and Discussion

TripoSG is an image-to-3D model built around a rectified-flow transformer, a data-building system, geometry-aware VAE training, and scale-up strategies. The paper concludes that these components enable high-fidelity 3D shape generation and identifies further scaling and downstream tasks as future directions.

  • 8.1. Conclusion: TripoSG combines a rectified-flow transformer, processed high-quality training data, SDF representation, surface-normal guidance, eikonal regularization, MoE, and high-resolution training.
  • 8.1. Conclusion: Filtered and fixed high-quality data improves training performance compared with using all in-the-wild 3D models.
  • 8.1. Conclusion: Experiments show that TripoSG generates high-fidelity 3D shapes and achieves state-of-the-art performance.
  • 8.2. Discussion: Future work includes scaling model parameters and training data, finer-grained condition injection, 3D super-resolution, scene generation, and stylization.
  • 8.2. Discussion: TripoSG provides both texture-free and textured 3D shape examples across diverse generated outputs.
Loading 2502.06608v3…