PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models

Zhengyi Wang, Jonathan Lorraine, Yikai Wang, Hang Su, Jun Zhu, Sanja Fidler, Xiaohui Zeng

arXiv:2411.09595v1cs.LGcs.AIcs.CLcs.CV

TL;DR

LLMs have primarily generated text, and integrating meshes raises a tokenization challenge. LLaMA-Mesh represents OBJ geometry as text and fine-tunes a pretrained model on mesh and dialogue data. It achieves mesh-generation quality comparable to models trained from scratch, with a stated boundary on mesh detail and complexity.

  • Problem

    LLM generation has primarily been textual, while integrating meshes raises tokenization challenges and common vocabulary-expanding autoencoders add training cost and lose information.

    Source: §1 · pp. 1–2

  • Method

    LLaMA-Mesh encodes OBJ vertex coordinates and face definitions as plain text and fine-tunes a pretrained LLM on text–mesh pairs and interleaved dialogue data.

    Source: §1 · p. 2

  • Results

    Mesh-generation quality is comparable to specialized models trained from scratch, while language ability shows slight degradation after fine-tuning.

    Source: §5.2 · p. 8, §5.1 · p. 8

  • Takeaways & Limitations

    The fine-tuned model generates meshes from prompts, produces interleaved text and mesh outputs, and describes meshes in natural language.

    Source: §1 · p. 2

    Coordinate quantization can reduce geometric detail, and context length limits complexity; the model currently supports at most 500 faces.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.

1. Introduction

LLAMA-MESH brings 3D mesh generation into a language model by representing meshes as text and fine-tuning on text–mesh data. It supports mesh generation alongside text and mesh understanding, with generation quality comparable to models trained from scratch.

1. Introduction

  • LLM generation has largely remained textual, and unifying text and mesh generation presents a tokenization challenge; added vocabularies raise training cost and autoencoding loses information.Source: §1 · pp. 1–2
  • The proposed language-based mesh generation is intended to support workflows in computer graphics, engineering, robotics, and virtual or augmented reality.Source: §1 · p. 1
  • LLAMA-MESH represents OBJ vertex coordinates and face definitions as plain text, letting LLMs process meshes without tokenizer or vocabulary changes.The design draws on spatial knowledge embedded in pretrained LLMs.Source: §1 · p. 2
  • Fine-tuning on text–3D pairs and interleaved dialogues supports prompted mesh generation, mixed text-and-mesh conversational output, and natural-language mesh descriptions.Source: §1 · p. 2
  • LLAMA-MESH unifies text and 3D in one model, achieving mesh-generation quality comparable to models trained from scratch while maintaining strong text generation.Source: §1 · p. 2

2. Related Work

Related work spans multimodal LLMs, 3D object generation, and autoregressive mesh generation. LLaMA-Mesh differs by directly generating mesh text rather than relying on a new visual tokenizer or scene-generation tools.

2.1. Enabling LLMs to be Multi-Modal

  • Prior multimodal LLM work includes visual understanding and image–text generation, while LLaMA-Mesh outputs OBJ text without modifying tokenization.Source: §2.1 · p. 2
  • Unlike scene-generation tools that compose predefined objects from layouts, LLaMA-Mesh directly generates 3D meshes as text.Source: §2.1 · p. 2

2.2. 3D Object Generation

  • Text-to-3D methods use score distillation or feed-forward generation, but often represent objects as numerical fields and extract meshes with marching cubes or variants.Source: §2.2 · p. 2

2.3. Auto-Regressive Mesh Generation

  • Autoregressive mesh methods model objects as tokenized coordinate sequences; some condition on point clouds, and these methods are trained from scratch without language capabilities.Source: §2.3 · p. 3

3. Method

LLaMA-Mesh represents meshes as plain-text OBJ sequences that LLMs can process directly, then fine-tunes a pretrained LLaMA model on mesh and conversation data. The method combines coordinate quantization with generated text-mesh dialogues to support mesh generation, understanding, and conversational use.

3.1. 3D Representation

  • OBJ represents vertices as coordinate lines and faces as vertex-index lists, letting LLMs process meshes as sequential text without tokenizer or vocabulary changes.OBJ files from the internet may vary in format; the method adopts a widely used, straightforward standard.Source: §3.1 · p. 3, Figure 4 · p. 3
  • Quantizing coordinates into 64 bins per axis reduces token counts while slightly lowering precision, making longer mesh sequences feasible without sacrificing geometric fidelity.Coordinates are scaled to [0, 64] and rounded to the nearest integer.Source: §3.1 · p. 4, Figure 5 · p. 4

3.2. Pretrained Models

  • LLaMA 3.1-8B-Instruct is the base model, but it performs poorly on mesh generation without fine-tuning on specialized text-mesh data.Before fine-tuning, pretrained models can produce simple OBJ objects, but mesh quality and complexity are often unsatisfactory.Source: §3.2 · p. 4, Figure 6 · p. 4

3.3. 3D-task Finetuning

  • The SFT dataset combines rule-based mesh-description and mesh-generation examples with LLM-augmented interleaved text-mesh dialogues.UltraChat general-conversation data is also included to preserve language capabilities.Source: §3.3 · pp. 4–5

4. Experiments

Experiments assess mesh generation, language retention, and comparisons with existing methods. LLaMA-Mesh produces meshes of comparable quality to specialized methods while retaining language abilities and using fewer GPU hours than MeshXL.

4.1. Implementation Details

  • Training used 32 A100 GPUs for 21k iterations over about 3 days, with rapid convergence and no observed loss instability.The SFT data combines mesh generation, mesh understanding, and general conversation data.Source: §4.1 · p. 6, Figure 9 · p. 6, Table 1 · p. 6

4.2.1. Mesh Generation Results

  • Repeated prompts yield varied meshes that satisfy the prompt, while generated meshes exhibit high quality and artist-like topology learned during training.The paper evaluates diversity by generating meshes multiple times from the same text prompt.Source: Figure 10 · p. 6, §4.2.1 · p. 6

4.3. Comparison with Existing Methods

  • LLAMA-MESH generates meshes of comparable quality to MeshXL and Unique3D while retaining robust language understanding in a single model.The comparison uses the same text prompt; Unique3D receives an SDXL-generated input image because it was designed for image-to-3D generation.Source: Figure 11 · p. 7, §4.3 · p. 7
  • LLAMA-MESH uses far fewer GPU hours than MeshXL despite its larger model size, benefiting from pretrained LLM weights.Table 2 compares training time, computational resources, and model sizes across methods.Source: Table 2 · p. 7, §4.3 · p. 7

4.2.2. Language and Conversational Abilities

  • Dialogues remain coherent and contextually appropriate, while MMLU, PIQA, HellaSwag, and GSM8K performance remains comparable to base models.The benchmarks assess general knowledge, commonsense reasoning, and mathematical problem-solving.Source: §4.2.2 · p. 7, Table 3 · p. 8

5. Discussion

The discussion reports limitations in geometric detail, context length, language ability, training data, and model scale, then summarizes the approach and suggests future directions. It also envisions conversational 3D creation across several application areas.

5.1. Limitations

  • LLAMA-MESH has limitations: quantizing vertex coordinates into limited bins can reduce geometric detail and mesh fidelity, context length constrains highly complex or large-scale 3D generation, and support currently tops out at 500 faces.Source: §5.1 · p. 8

5.2. Conclusion

  • The conclusion reports mesh quality comparable to specialized models trained from scratch while describing the approach as preserving base-model language understanding.The limitations section separately notes slight language-ability degradation after fine-tuning.Source: §5.2 · p. 8, §5.1 · p. 8

5.3. Future Work

  • Future work could improve 3D encoding efficiency, context length, and geometric precision, and add textures, physical properties, or dynamic-scene capabilities.The discussion envisions conversational 3D creation for areas including virtual reality, gaming, education, and manufacturing.Source: §5.3 · p. 8
  • The discussion envisions conversational 3D content creation as a more intuitive approach with potential applications in virtual reality, gaming, education, and manufacturing.It also imagines language models producing text, images, and 3D structures.Source: §5.3 · p. 8

Reproducibility

The paper identifies where implementation and evaluation information is documented for reproducibility.

Reproducibility

Ethics Statements

The paper notes that unifying mesh generation with LLMs may inherit ethical concerns from both language models and generative models.

Ethics Statements

Loading 2411.09595v1…