PaperlayerView exact revision on arXiv

Source-linked AI summary

ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model

Fukun Yin, Xin Chen, Chi Zhang, Biao Jiang, Zibo Zhao, Jiayuan Fan, Gang Yu, Taihao Li, Tao Chen

arXiv:2311.17618v3cs.CV

TL;DR

Comprehensive pretrained models for 3D shapes integrated with other modalities remain underexplored. ShapeGPT combines discretized shape representations and multimodal inputs in an instruction-driven language-model framework, reporting competitive performance across diverse shape tasks.

  • Problem

    Comprehensive pretrained models that handle 3D shapes together with other modalities remain underexplored, despite their potential relevance to virtual construction and network-aided design.

    Source: §1 · p. 2

  • Method

    ShapeGPT converts shapes into discrete tokens and combines shape, image, and text inputs in a unified instruction-driven language-model framework.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ShapeGPT achieves competitive performance across diverse shape tasks, including image-to-shape, text-to-shape, shape-to-text, multi-modal-to-shape, completion, and editing.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors expect instruction-based multimodal shape generation to benefit virtual construction and network-aided design.

    Source: §1 · p. 2

    Current scope leaves textured-shape generation and voice or video inputs for future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

The advent of large language models, enabling flexibility through instruction-driven approaches, has revolutionized many traditional generative tasks, but large models for 3D data, particularly in comprehensively handling 3D shapes with other modalities, are still under-explored. By achieving instruction-based shape generations, versatile multimodal generative shape models can significantly benefit various fields like 3D virtual construction and network-aided design. In this work, we present ShapeGPT, a shape-included multi-modal framework to leverage strong pre-trained language models to address multiple shape-relevant tasks. Specifically, ShapeGPT employs a word-sentence-paragraph framework to discretize continuous shapes into shape words, further assembles these words for shape sentences, as well as integrates shape with instructional text for multi-modal paragraphs. To learn this shape-language model, we use a three-stage training scheme, including shape representation, multimodal alignment, and instruction-based generation, to align shape-language codebooks and learn the intricate correlations among these modalities. Extensive experiments demonstrate that ShapeGPT achieves comparable performance across shape-relevant tasks, including text-to-shape, shape-to-text, shape completion, and shape editing.

1. Introduction

ShapeGPT targets the limited availability of unified models for instruction-guided 3D shape tasks by integrating shapes, images, and text in a multimodal language-model framework. It uses shape tokens and staged training to support diverse shape tasks with one model.

1. Introduction

  • Comprehensive pretrained models for 3D shapes and other modalities remain underexplored, despite potential relevance to virtual construction, gaming, network-aided design, and 3D printing.The paper identifies multimodal representation and unified multi-task modeling as challenges for such a framework.Source: §1 · p. 2
  • Prior systems often solve shape tasks independently and use specific datasets, limiting task versatility and diverse input combinations.The paper links these constraints to insufficient modeling of relationships among shapes, modalities, and datasets.Source: §1 · p. 2
  • A shape-specific VQ-VAE converts 3D shapes into tokens for a pretrained language model, while a visual transformer and visual-language perceiver integrate image inputs.The language model learns relationships between encoded shapes and textual descriptions.Source: §1 · p. 2
  • Training first teaches shape generation and captioning on 3D datasets, then aligns text, rendered images, and shapes using a multimodal dataset.The authors describe this as a three-stage scheme, with the third stage not detailed in this passage.Source: §1 · p. 2
  • ShapeGPT is a single model for diverse shape tasks, with competitive performance across image-to-shape, text-to-shape, shape-to-text, multi-modal-to-shape, completion, and editing.The paper also proposes a general shape benchmark for multi-task evaluation.Source: §1 · p. 2

2. Related Work

Prior work spans several shape-generation and representation approaches, but many systems handle multiple tasks with a single model only to a limited extent. Multimodal language models for 3D shapes and instruction-driven shape tasks remain comparatively underexplored.

3D Shape Generation and Representation

  • ShapeCrafter uses CLIP text encoding for text-conditioned recursive generation, while SDFusion fuses text, images, and partial shapes in a diffusion framework.Other work uses image conditions with latent-vector shape representations; many prior methods remain limited to a single model for multiple tasks.Source: §2 · p. 2

Shape-language Pre-training

  • ShapeGPT's overview pairs multimodal corpus construction with a shape-aware language model for image-to-shape, text-to-shape, shape-to-text, completion, and editing.The corpus stage tokenizes multimodal inputs, while the model stage handles vision-shape-language grammar.Source: Figure 2 · p. 3

Language Models and Multi-Modal

  • Multimodal language-model research covers images, audio, motion, and video, while models handling 3D shapes remain at an early stage.The cited background describes language models as converting language-based tasks into text-to-text formats.Source: §2 · pp. 2–3

Shape-language Pre-training

  • Most text-to-shape methods use tag-to-shape inputs and typically do not interpret user-specific instructions, leaving shape-task applications of advanced language models largely unexplored.The paper presents ShapeGPT as a unified shape-language model addressing this gap.Source: §2 · p. 3

3. Method

ShapeGPT unifies shape, image, and text processing in an instruction-based language framework, using discrete shape representations and a staged training strategy to support multiple shape-related tasks. Its pipeline constructs multimodal inputs, aligns shape-aware language with pretrained language models, and then fine-tunes for broader instruction-based generation.

3. Method

  • ShapeGPT formats multimodal data as instructions and expected outputs, enabling generation of shape and text sequences for tasks including shape generation, captioning, reasoning, and editing.The framework combines multimodal corpus construction, a shape-aware language model, and multi-task natural language generation.Source: §3 · pp. 3–4

3.1. Multimodal Corpus Construction

  • A 3D VQ-VAE maps continuous SDF shapes into discrete codebook sequences, which its decoder maps back to continuous SDF representations.The codebook is learnable, and the VQ-VAE loss combines reconstruction, embedding, and commitment terms.Source: §3.1 · p. 4
  • A pretrained CLIP image encoder supplies penultimate-layer features, flattened by height and width into localized image embeddings for the multimodal corpus.The corpus also includes discretized shape representations and caption tokens.Source: §3 · p. 3, §3.1 · p. 4

3.2. Shape-aware Multimodal Language Model

  • ShapeGPT converts discrete shape token IDs into explicit shape words and brackets serialized shape sentences with boundary symbols to preserve shape-representation integrity in instructions.Shape-aware grammar is aligned with pretrained natural-language models using a word-sentence-paragraph progression.Source: §3.2 · p. 4
  • The framework uses T5 for sequence-to-sequence processing of multimodal language paragraphs, concatenating shape-aware tokens and image embeddings when an image placeholder is present.The model uses a transformer-based language encoder and decoder.Source: §3.2 · p. 4

3.3. Training on Shape-Image-Text Dataset

  • Training first learns and then freezes the shape VQ-VAE, aligns the language model using four tasks on annotated chair and table data, and finally fine-tunes diverse instructions across more shape categories.The alignment stage uses Text-to-Shape, Image-to-Shape, Multimodal-to-Shape, and Shape-to-Text pairs; the instruction stage covers generation, captioning, editing, completion, and inference.Source: §3.3 · p. 5
  • The initial alignment stage has limited scope and simple tasks, and the paper reports that it still has inadequate data generalization before instruction-based training.The authors identify monotonous instructions, simplistic tasks, and inadequate data generalization as limitations at this stage.Source: §3.3 · p. 5

4. Experiments

Experiments compare ShapeGPT on ShapeNet across common shape-generation tasks and instruction-driven extensions, then test token length, model size, and pretraining. Results are comparable on image- and text-to-shape tasks and stronger on multimodal-to-shape generation, with ablations favoring 512 tokens, T5-base, and pretraining.

4.1. Experimental Setup

  • Evaluations use ShapeNet and Objaverse, with ShapeNet text-to-shape comparisons limited to annotated chair and table classes and tags used for other classes.The setup reports reconstruction metrics for image or partial-shape conditions, ULIP for caption-conditioned generation, and CLIP similarity for generated captions.Source: §4 · p. 5, §4.1 · p. 6

4.2. Comparisons on Shape-relevant Tasks

  • ShapeGPT is comparable to recent methods on image-to-shape and close to the previous best on text-to-shape, while outperforming the latest methods across multimodal-to-shape metrics.The multimodal comparison combines a single rendered image with a caption; qualitative results report closer alignment with multimodal prompts.Source: Figure 4 · p. 6, §4.2 · pp. 7–8
  • ShapeGPT also handles shape captioning, completion, reasoning, and editing in one architecture, with completion tested after removing 0%–50% of each original shape.Editing uses an original shape and instruction; reasoning prompts describe function, application, or user needs without specifying the shape's class or appearance.Source: §4.2 · p. 8, Table 4 · p. 8

4.3. Ablation Studies

  • 512 shape tokens outperform shorter sequences; the 512-token setting comes from an 8×8×8 grid, while larger voxel configurations exceed GPU memory limits.The study also tests 64 tokens, flattened from a 4×4×4 grid.Source: §4.3 · p. 8, Table 5 · p. 8
  • T5-base (220M parameters) performs better than T5-small (60M), while larger models are infeasible under the available GPU memory.The paper reports that T5-base basically meets the requirements of the shape-related tasks.Source: §4.3 · p. 8, Table 5 · p. 8
  • Pretraining improves performance; without it, outputs often miss the expected modality or fail to form coherent shape language.The training setup first uses partial data and simple instructions before expanding to the full dataset and complex instructions.Source: §4.3 · p. 8, Table 5 · p. 8

5. Conclusion and Limitation

ShapeGPT unifies instruction-based shape-related tasks by mixing shapes, captions, images, and instructions into a language-model sequence-to-sequence format, with performance comparable to state-of-the-art methods on common shape generative tasks. The authors identify textured shapes, additional shape-centered tasks, and modalities such as voice and video as planned extensions.

  • Planned extensions include more shape-centered tasks, textured shapes, voice and video, multi-object shapes, 3D scenes, dynamic shapes, and video generation.Source: §5 · p. 8
  • ShapeGPT performs comparably to state-of-the-art methods on common shape generative tasks.Source: §5 · p. 9
  • ShapeGPT intermixes instructions with shapes, captions, and images in a natural-language format for sequence-to-sequence generation.Source: §5 · p. 9
Loading 2311.17618v3…