PaperlayerView exact revision on arXiv

Source-linked AI summary

TripoSR: Fast 3D Object Reconstruction from a Single Image

Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, Yan-Pei Cao

arXiv:2403.02151v1cs.CV

TL;DR

Single-image 3D reconstruction needs a faster alternative to optimization-heavy methods while maintaining reconstruction quality and controllability. TripoSR builds a transformer-based feed-forward model on LRM, improving data processing, model design, and training. On public benchmarks, it achieves state-of-the-art performance with inference under 0.5 seconds on an A100 GPU.

  • Problem

    Optimization-based 3D generation methods can be slow and difficult to control, while feed-forward reconstruction offers a more efficient direction.

    Source: §1 · p. 2

  • Method

    TripoSR extends the LRM transformer architecture with improvements in data curation, rendering, model design, and training for single-image 3D reconstruction.

    Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 2

  • Results

    TripoSR achieves state-of-the-art reconstruction performance on two public benchmarks and produces a 3D mesh in less than 0.5 seconds on an A100 GPU.

    Source: §1 · p. 2, §3 · p. 4, §4 · p. 5

  • Takeaways & Limitations

    TripoSR is released as an open-source MIT-licensed model with source code, pretrained weights, and an interactive demo.

    Source: §1 · p. 2

Abstract

from arXiv · show

This technical report introduces TripoSR, a 3D reconstruction model leveraging transformer architecture for fast feed-forward 3D generation, producing 3D mesh from a single image in under 0.5 seconds. Building upon the LRM network architecture, TripoSR integrates substantial improvements in data processing, model design, and training techniques. Evaluations on public datasets show that TripoSR exhibits superior performance, both quantitatively and qualitatively, compared to other open-source alternatives. Released under the MIT license, TripoSR is intended to empower researchers, developers, and creatives with the latest advancements in 3D generative AI.

1. Introduction

Single-image 3D generation has advanced through large-scale datasets and generative architectures, but optimization-heavy methods remain slow and difficult to control. TripoSR addresses this gap with fast feed-forward generation and reports superior quantitative and qualitative performance among open-source alternatives.

Motivation

  • 2D diffusion-guided 3D generation can create detailed objects but typically requires slow optimization and offers limited output control.These methods use 2D priors to compensate for scarce 3D training data.Source: §1 · p. 2
  • Feed-forward 3D reconstruction improves computational efficiency through rapid inference and may provide more precise control over generated outputs.Source: §1 · p. 2

Contribution

  • TripoSR generates a 3D object from one image in less than 0.5 seconds on an A100 GPU.The model is introduced as a fast feed-forward approach built on the LRM architecture.Source: §1 · p. 2
  • TripoSR combines improvements in data curation, rendering, model design, and training techniques.Source: §1 · p. 2

2. TripoSR: Data and Model Improvements

TripoSR is a transformer-based single-image reconstruction model that converts image features into a compact triplane-NeRF representation. Its improvements target robustness, data quality, representation efficiency, and training efficiency while preserving detailed reconstruction.

2.1. Model Overview

  • TripoSR uses an image encoder, an image-to-triplane decoder, and a triplane-based NeRF to map one RGB image to a 3D object representation.Source: §2.1 · p. 2
  • The decoder uses self-attention and cross-attention to model triplane relationships and incorporate global and local image features.The NeRF predicts color and density for 3D points.Source: §2.1 · p. 2
  • The model estimates camera extrinsics and intrinsics instead of receiving them explicitly, improving robustness to in-the-wild images.Source: §2.1 · p. 2

2.2. Data Improvements

  • Curated Objaverse data and diverse rendering techniques improve training-data quality and better approximate real-world image distributions.Source: §2.2 · p. 2

2.3. Model and Training Improvements

  • 40 triplane channels balance reconstruction quality with computational efficiency, enabling larger batch sizes and higher resolution.Source: §2.3 · p. 3
  • 128 × 128 foreground-biased random patches replace costly high-resolution supervision, preserving surface-detail learning while reducing computational and GPU-memory demands.Source: §2.3 · pp. 3–4

3. Results

TripoSR is evaluated against feed-forward reconstruction baselines on curated GSO and OmniObject3D subsets using shape metrics and qualitative comparisons. It achieves state-of-the-art quantitative performance, fast inference, and higher-quality shape and texture reconstruction.

Evaluation

  • GSO and OmniObject3D evaluations use manually filtered subsets of around 300 diverse objects from each dataset to reduce bias toward simple shapes.Source: §3 · p. 4

Quantitative Comparisons

  • TripoSR significantly outperforms One-2-3-45, TriplaneGaussian, ZeroShape, and OpenLRM on both Chamfer Distance and F-score.Source: §3 · p. 4

Performance vs. Runtime

  • Around 0.5 seconds are required to produce a 3D mesh from one image on an NVIDIA A100 GPU, placing TripoSR among the fastest networks.Source: §3 · p. 4

Visual Results

  • TripoSR’s qualitative reconstructions show higher-quality shapes, textures, and intricate details than previous state-of-the-art methods.Source: Figure 3 · p. 5, §3 · p. 5
  • Some baselines produce inaccurate estimated shapes, whereas TripoSR captures better overall 3D structure and detailed geometry.Source: §3 · p. 5

4. Conclusion

TripoSR is an open-source, transformer-based feed-forward 3D reconstruction model developed upon LRM. Across two public benchmarks, it achieves state-of-the-art reconstruction performance with high computational efficiency.

  • TripoSR is an open-source feed-forward 3D reconstruction model based on a transformer architecture developed upon LRM.Source: §4 · p. 5
  • The model incorporates substantial technical improvements across multiple axes.The passage identifies data curation, model design, and training strategy as improvement areas.Source: §4 · p. 5
  • Evaluated on two public benchmarks, TripoSR demonstrates state-of-the-art reconstruction performance with high computational efficiency.Source: §4 · p. 5

Related papers

Loading 2403.02151v1…