Source-linked AI summary
MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh
Shuangkang Fang, I-Chao Shen, Yufeng Wang, Yi-Hsuan Tsai, Yi Yang, Shuchang Zhou, Wenrui Ding, Takeo Igarashi, Ming-Hsuan Yang
TL;DR
LLM-based 3D mesh modeling faces limited training data and difficulty retaining mesh structure in text sequences. MeshLLM decomposes meshes into localized units and trains structural and task-specific objectives; the paper reports better generation and understanding than LLaMA-Mesh. Its findings also describe mesh dialogue with structural explanations.
Problem
LLMs' token-length limits constrain mesh training data, and text sequences can fail to convey intrinsic 3D structure.
Source: §3.2 · p. 3Method
MeshLLM decomposes meshes into localized Primitive-Mesh units and progressively trains LLMs on vertex-face prediction, mesh assembly, mesh understanding, and mesh generation.
Source: §1 · p. 2, §3.2 · p. 3Results
MeshLLM significantly outperforms LLaMA-Mesh in text-serialized mesh generation and understanding.
Source: §1 · p. 2Takeaways & Limitations
In dialogue scenarios, MeshLLM generates meshes matching user descriptions and provides explanatory feedback on mesh details and topology.
Source: §4.2 · p. 6Mesh data remains scarce relative to language corpora, and imprecise text-to-geometry alignment limits fine-grained mesh generation.
Source: §5 · p. 8, §8.3 · p. 15
Abstract
from arXiv · showhide
We present MeshLLM, a novel framework that leverages large language models (LLMs) to understand and generate text-serialized 3D meshes. Our approach addresses key limitations in existing methods, including the limited dataset scale when catering to LLMs' token length and the loss of 3D structural information during mesh serialization. We introduce a Primitive-Mesh decomposition strategy, which divides 3D meshes into structurally meaningful subunits. This enables the creation of a large-scale dataset with 1500k+ samples, almost 50 times larger than previous methods, which aligns better with the LLM scaling law principles. Furthermore, we propose inferring face connectivity from vertices and local mesh assembly training strategies, significantly enhancing the LLMs' ability to capture mesh topology and spatial structures. Experiments show that MeshLLM outperforms the state-of-the-art LLaMA-Mesh in both mesh generation quality and shape understanding, highlighting its great potential in processing text-serialized 3D meshes.
1. Introduction
LLM-based 3D mesh understanding and generation remain underexplored, while text-serialized mesh methods face limits in dataset scale and structural awareness. MeshLLM uses localized mesh units and structure-focused training to address these challenges, and the paper reports improved performance over LLaMA-Mesh.
1. Introduction
- Existing text-serialized mesh training uses only 31k samples and can overlook connectivity and semantic segmentation in 3D meshes.The introduction identifies limited token length and insufficient structural awareness as central challenges.Source: §1 · p. 2, Figure 2 · p. 2
- KNN decomposition creates 1500k+ localized Primitive-Mesh training samples, while semantic segmentation adds over 100K semantically detailed samples.The semantic-level samples are intended to provide accurate structural and semantic details.Source: §1 · p. 2
- Vertex-face prediction infers face connectivity from vertices, while local mesh assembly reconstructs complete meshes from local structures.The paper describes these tasks as targeting topological reasoning and global modeling, respectively.Source: §1 · p. 2
- MeshLLM significantly outperforms LLaMA-Mesh in text-serialized mesh generation and understanding.The paper presents this as an experimental finding without giving a metric in this passage.Source: §1 · p. 2
2. Related Work
Prior work extends language models to multimodal inputs and develops separate approaches for mesh generation and understanding. MeshLLM builds on text-serialized mesh modeling while addressing reported dataset-use and structural-information challenges.
2. Related Work
- Multimodal LLM approaches such as LLaVA, Video-LLaMA, and SpeechGPT use customized tokenizers to embed image, video, or speech data.The related-work section connects this need to difficulties in textualizing those modalities or the length of their textualized data.Source: §2 · p. 2
- Mesh-generation methods include sequential representations, while mesh-understanding methods use approaches such as multi-view semantics, part detection, or encoder-based embeddings.The paper notes that existing methods generally treat mesh generation and understanding as separate tasks.Source: §2 · p. 3
- LLaMA-Mesh explores unified generation and understanding using text-serialized meshes, but the paper identifies inefficient dataset use and structural-information loss as limitations.MeshLLM is proposed as an improvement to these issues.Source: §2 · p. 3
3. Method
The method converts meshes into text and decomposes them into localized Primitive-Mesh units, then progressively trains LLMs on structural and task-specific mesh objectives. It also uses supervised fine-tuning data tailored to those tasks.
3.1. Preliminaries: Text-serialized Mesh
- Meshes are represented in OBJ format, with coordinates quantized to integers from 0 to 64, vertices and faces sorted, and the result unfolded into text.The coordinate range avoids modifying the LLM tokenizer or vocabulary.Source: §3.1 · p. 3
3.2. Primitive-Mesh
- Primitive-Mesh decomposition addresses token-length constraints, weak structural signaling, and the difficulty of learning from long text sequences.Localized components retain spatial information and form shorter, more numerous training sequences.Source: §3.2 · p. 3
3.3. Training Task Design
- Progressive training first uses KNN-derived Primitive-Meshes, then semantically segmented units for vertex-face prediction and mesh assembly, and finally mesh understanding and generation tasks.The initial structural tasks familiarize the model with mesh structure and local semantics; later tasks target complex structures and high-level semantics.Source: Figure 4 · p. 4, §3.3 · pp. 4–5
3.2. Primitive-Mesh
- KNN-based decomposition yields over 1500k training samples, while semantic segmentation supplies over 100k high-quality samples with well-defined semantic boundaries.KNN is computationally efficient and applicable to existing mesh datasets, but its parts may lack semantic coherence.Source: §3.2 · p. 4
3.3. Training Task Design
- Vertex-face prediction teaches connectivity from vertices, and mesh assembly trains reconstruction of a complete mesh from Primitive-Mesh components.Assembly is described as capturing geometric relationships between local units and mitigating spatial-information loss from textual serialization.Source: §3.3 · p. 4
3.4. SFT Data Curation
- Mesh understanding trains descriptions from mesh data, while mesh generation trains plausible mesh structures from textual descriptions.The SFT data packages the four tasks as high-quality input-output pairs using standard language-modeling objectives.Source: §3.3 · p. 5, §3.4 · p. 5
4. Experimental Results
MeshLLM is evaluated for dialogue, mesh generation and understanding, and component ablations. It surpasses LLaMA-Mesh on generation and understanding, with generation performance comparable to mesh-generation methods.
4.1. Implementation Details
- Training uses Objaverse-XL and ShapeNet data, finetunes LLaMA-8B-Instruct, and samples prior-phase and UltraChat data with 30% probability to retain conversational ability.The dataset construction also yields over 1500k KNN-based and over 100k semantic-level Primitive-Meshes.Source: §4.1 · pp. 5–6
4.2. Dialogue Ability
- MeshLLM supports natural-language mesh interactions, explanatory feedback about mesh details and topology, and coherent multi-turn dialogue while retaining language-generation ability.The evaluation uses interactive dialogue scenarios simulating natural-language instructions.Source: §4.2 · p. 6
4.3. Performance Evaluation
- MeshLLM surpasses LLaMA-Mesh in mesh generation and understanding, with generation performance comparable to MeshXL and PolyGen.The supplied tables and evaluation prose report these comparisons without providing cell values.Source: §4.3 · p. 7, Table 1 · p. 8, Table 2 · p. 8
4.4. Ablation Studies
- Ablations show that removing KNN-based Primitive-Meshes, semantic Primitive-Meshes, vertex-face prediction, or mesh assembly reduces performance, with pronounced declines for structural components.The table reports generation results for chairs and mesh understanding; the prose describes component-specific effects.Source: §4.4 · p. 8, Table 3 · p. 8
5. Limitation and Future Work
The authors identify data scale and text-geometry alignment as continuing limitations, alongside opportunities for improving complex-mesh handling and geometric refinement.
5. Limitation and Future Work
- Available mesh data remains vastly smaller than the corpora used in natural language processing.The authors identify constructing larger, higher-quality datasets as important for fully leveraging LLM abilities.Source: §5 · p. 8
- Imprecise text-to-geometry alignment constrains fine-grained mesh generation and refinement, especially when data is scarce.The authors suggest additional modalities such as images could encode structural information and enhance performance in that setting.Source: §5 · p. 8
- Complex-mesh handling may benefit from compact serialization and larger LLM token capacities; external agents and reinforcement learning are proposed for geometric accuracy and aesthetic objectives.The authors describe serialization and token-capacity optimizations as orthogonal to this work.Source: §5 · p. 8
6. Conclusions
MeshLLM addresses limited dataset use and structural disruption from serializing meshes as text. It combines Primitive-Mesh data construction with hierarchical training to support mesh understanding and generation.
6. Conclusions
- MeshLLM targets insufficient use of available datasets and disruption of 3D structure during 2D serialization.The conclusion identifies these as two key limitations of existing approaches.Source: §6 · p. 8
- Primitive-Mesh decomposition expands the trainable dataset to over 1500k samples and adds a curated collection of more than 100k semantically segmented meshes.The curated meshes are described as high-quality and semantically segmented.Source: §6 · p. 8
- A hierarchical training paradigm models meshes from vertices to faces and then through mesh assembly to support LLM perception of 3D structure.The conclusion presents this as the approach for mesh understanding and generation.Source: §6 · p. 8
7. Additional Implementation Details
MeshLLM constructs KNN-based and semantic Primitive-Mesh datasets, then evaluates mesh generation with point-based metrics.
7.1. Construction of Primitive-Mesh
- KNN segmentation uses farthest-point sampling and KNN clustering to create localized mesh parts, each with a limited number of faces that fit LLM token constraints.The strategy takes 0.2 seconds per mesh and supports rapid dataset construction.Source: §7.1 · p. 13
- Semantic-based parts come from SamPart3D segmentation and GPT-4o labeling of more than 25k high-quality meshes, producing semantically meaningful components with textual annotations.The authors report that dataset construction took over three days using 128 A800 GPUs and incurred API query costs.Source: §7.1 · p. 13
7.2. Metric Details
- Point-based measures are computed from 2048 randomly sampled points from all baseline results.Source: §7.2 · p. 13
- Generation evaluation uses MMD, COV, and 1-NNA: MMD measures closeness, COV measures reference-set coverage, and 1-NNA assesses whether generated and real meshes are evenly distributed.Lower MMD, higher COV, and 1-NNA near 50% indicate better results on these respective measures.Source: §7.2 · p. 13
8. Additional Results
MeshLLM’s additional analyses compare training order, examine generated-mesh novelty, and describe a limitation in fine-grained generation.
8.1. Shape Novelty Analysis
- KNN-based Primitive-Mesh parts preserve meaningful spatial structure, while semantic-based parts include annotations that better aid LLM understanding and generation.Source: Figure 10 · p. 14
- Generated meshes retain overall structural resemblance to training examples but show significant local differences, which the authors interpret as generalization rather than replication.Figure 11 compares each generated mesh with its three closest training meshes by Chamfer Distance.Source: §8.1 · pp. 13–14, Figure 11 · p. 15
8.2. Training Strategy Analysis
- Large-scale pretraining followed by high-quality semantic fine-tuning performs better than reversing the Primitive-Mesh training order.The authors attribute the reverse-order challenge to initial learning difficulty and possible loss of semantic knowledge when later exposed to 15 times more KNN-based samples.Source: Table 4 · p. 15, §8.2 · p. 15
8.3. Failure Case Analysis
- Relatively scarce mesh data leads to imprecise text-geometry alignment, limiting fine-grained mesh generation; the authors suggest incorporating modalities such as images.Source: §8.3 · p. 15, Figure 12 · p. 15