Source-linked AI summary
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, Kaisheng Ma
TL;DR
ShapeLLM addresses the under-explored problem of precise-geometry 3D object understanding for embodied interaction. It combines ReCon++, a multi-view-enhanced 3D point-cloud encoder, with an LLM and instruction-following data, achieving state-of-the-art recognition and 3D interaction benchmark results.
Problem
Precise geometry-aware embodied interaction remains under-explored when LLMs are applied to 3D object understanding.
Method
ShapeLLM combines ReCon++, a 3D point-cloud encoder using multi-view distillation and representation learning, with instruction-tuned language modeling and constructed interaction data.
Results
42.7% and 49.3% Total accuracy were achieved by ShapeLLM-7B and ShapeLLM-13B on 3D MM-Vet, surpassing previous 3D-point-cloud results by +2.1% and +5.1%.
Takeaways & Limitations
The work establishes a 3D MLLM and benchmark spanning recognition, language generation, spatial awareness, and embodied interaction.
Abstract
from arXiv · showhide
This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/
1 Introduction
ShapeLLM addresses the need for geometry-aware, interaction-oriented 3D object understanding through a universal language interface. It combines point-cloud geometry, embodied interaction knowledge, instruction-following data, and an improved encoder, achieving strong recognition and benchmark results.
- Real-world embodied agents require sufficient 3D geometry, embodied interaction knowledge, and a universal interface connecting information encoding with agent instructions.
- LLMs offer foundational knowledge and unified reasoning, but precise-geometry embodied interaction remains under-explored in 3D object understanding.
- ShapeLLM uses 3D point clouds rather than rendered images to represent physical environments and support geometry-dependent interactions.
- ShapeLLM is trained with approximately 45K instruction-following samples and 30K embodied part-understanding samples, then evaluated with the 3D MM-Vet benchmark.
- 95.25% and 95.0% fine-tuned accuracy were achieved by ReCon++ on ScanObjectNN and ModelNet40, while zero-shot accuracy reached 53.7% and 65.4% on Objaverse-LVIS and ScanObjectNN.
- 42.7% and 49.3% Total accuracy were achieved by ShapeLLM-7B and ShapeLLM-13B on 3D MM-Vet, exceeding prior 3D-point-cloud results by +2.1% and +5.1%.
2 ShapeLLM
ShapeLLM combines a pretrained 3D encoder with an LLM and addresses interactive-data scarcity through instruction-following data. ReCon++ improves geometry representation using multi-view information and matching across views.
- Overall Architecture: ShapeLLM combines a pretrained 3D encoder with an LLM to learn and understand 3D representations through a universal language interface.
- How to alleviate interactive 3D understanding Data Desert?: The training data addresses the interactive-data desert by constructing question-answer pairs from multi-view object images and annotated object parts.
- How to alleviate interactive 3D understanding Data Desert?: Embodied part questions pair part semantics with 3D positions to support object grasping and interaction.
- ReCon++: Scaling Up 3D Representation Learning: ReCon++ uses multi-view image and depth information with bipartite matching to improve semantic and pose-aware 3D representation learning.
- ReCon++: Scaling Up 3D Representation Learning: The matching objective pairs each multi-view image feature with a query through a minimum-cost permutation, using cosine similarity in practice.
- ReCon++: Scaling Up 3D Representation Learning: Concatenating local encoder and global decoder features provides comprehensive multimodal information for 3D understanding.
3 3D MM-Vet: Benchmarking 3D Comprehension
3D MM-Vet evaluates whether multimodal models can move from recognition and knowledge use to spatial reasoning and embodied action planning. The benchmark uses held-out 3D models and questions designed around distinct capabilities.
- The evaluation framework covers four levels: General Recognition, Knowledge and Language Generation, Spatial Awareness, and Embodied Interaction.
- General Recognition: General Recognition tests coarse attributes such as color and shape alongside fine-grained details such as subparts and counting.
- Knowledge Capability & Language Generation: Knowledge and Language Generation evaluates natural, social, physical, sequential, and mathematical reasoning together with detailed response generation.
- Embodied Interaction: ShapeLLM supports general VQA, embodied task and action planning, and 6-DoF pose estimation in qualitative multimodal dialogue examples.
- Spatial Awareness: Spatial Awareness probes spatial relations between object parts using the location information contained in point clouds.
- The benchmark uses 59 Turbosquid models and 232 questions, with each question targeting one specific capacity and paired with a model-specific ground-truth answer.
4 Experiments
Experiments evaluate ReCon++ for transfer and zero-shot recognition, then assess ShapeLLM across multimodal tasks, corruptions, ablations, and qualitative interaction settings. Results show strong representation transfer, broad task performance, robustness, and gains from both the encoder and instruction-following data.
- 3D Representation Transferring with ReCon++: ReCon++ achieves 95.25% accuracy on ScanObjectNN PB_T50_RS, improving over the Transformer baseline by +16.14%.
- 3D Representation Transferring with ReCon++: ReCon++ demonstrates superior or on-par zero-shot recognition across benchmarks, including gains of +0.6% on Objaverse-LVIS and +7.2% on ScanObjectNN over Uni3D-L.
- Multimodal Comprehension with ShapeLLM: ShapeLLM significantly outperforms PointLLM across 3D MM-Vet metrics, particularly on embodied tasks.
- Multimodal Comprehension with ShapeLLM: ShapeLLM shows significantly superior robustness to single-view, jitter, and rotation corruptions on 3D MM-Vet-C.
- Baseline Improvement: Replacing PointLLM’s encoder with ReCon++ yields a +4.20% improvement, while the constructed SFT data closes the remaining performance gap to ShapeLLM.
- Qualitative Analysis: Qualitative examples show support for general VQA, embodied planning, action planning, 6-DoF pose estimation, and robust dialogue from single-view point clouds.
5 Discussions
ShapeLLM is evaluated for geometry-dependent referring expression grounding and generalization beyond its articulated-furniture training domain. The results emphasize the value of 3D inputs for 6-DoF spatial understanding and suggest open-world generalization potential.
- Is ShapeLLM grounded in physical worlds?: ShapeLLM significantly outperforms image-only methods on 6-DoF referring expression grounding, including methods using fine-tuning or in-context prompting.
- Is ShapeLLM grounded in physical worlds?: Image-only methods cannot perform zero-shot geometry-necessary 6-DoF pose estimation, highlighting the need for geometric input.
- Generalization to Unseen Objects: Although trained on GAPartNet’s indoor articulated furniture, ShapeLLM demonstrates promising spatial-understanding generalization to unseen open-world objects.
6 Related Works
Related work frames interaction-oriented 3D understanding and multimodal LLMs as complementary lines of research. Prior studies cover 3D perception, semantic parsing, scene interaction, and LLM-based downstream systems.
- Interaction-Oriented 3D Understanding: Interaction-oriented 3D understanding includes concept-only perception and semantic parsing as well as physical-grounded interaction with 3D objects.
- Interaction-Oriented 3D Understanding: Prior 3D language-enabled work supports scene question answering, grounding, and captioning, while multimodal LLM research integrates LLMs with downstream systems as agents.
7 Conclusions
ShapeLLM combines the ReCon++ 3D encoder with projected local, global, and absolute-position representations, learnable visual prompts, and instruction-following data. Ablations show that these architectural components and prompt selection materially affect multimodal comprehension and grounding.
- Architecture: ShapeLLM uses ReCon++ as its 3D encoder and projects local and global representations into the language model.Absolute position encoding supplies additional low-level geometric information for geometry-dependent tasks such as 6-DoF pose estimation.
- Architecture: Absolute position encoding is projected from 3D coordinates to provide precise geometric information.
- Architecture: Learnable visual prompts adaptively modulate semantic information from absolute-position, local, and global representations.
- Ablation studies: All dedicated input designs are necessary for strong performance on 3D comprehension and real-world grounding.The ablation evaluates ShapeLLM-13B on 3D MM-Vet and GAPartNet referring-expression grounding.
- Ablation studies: 32 visual prompts are optimal among the evaluated settings of 1, 8, 16, 32, and 64.The prompt-count ablation reports varied improvements across settings.
A.2 Multimodal Comprehension with ShapeLLM
ShapeLLM is evaluated across generative recognition, captioning, hallucination resistance, language-only baselines, representation transfer, and multi-view spatial understanding. The reported evidence highlights broad task performance while cautioning that traditional captioning metrics may not reflect response quality.
- Generative recognition and captioning: ShapeLLM consistently outperforms other methods across generative 3D recognition and captioning metrics.Evaluation includes classification accuracy and captioning scores from GPT-4, Sentence-BERT, and SimCSE.
- Generative recognition and captioning: Traditional BLEU-1, ROUGE-L, and METEOR scores are highly unreliable for accurately revealing 3D captioning response quality.
- Evaluation: GPT-4 evaluation has a standard deviation below 0.1 on 3D MM-Vet, supplemented by human win-rate evaluation.
- Visual hallucination: ShapeLLM is qualitatively assessed for resisting visual hallucinations caused by detailed captions and misleading questions.The comparison concerns mis-recognition and false identity association in previous methods.
- Language-only baseline: Language-only baselines perform poorly when questions use pronouns for unknown objects; LLaMA hallucinates while ChatGPT often refuses without specific inputs.
- Representation transfer: ReCon++ outperforms plain and hierarchical Transformer methods in Linear SVM evaluation and achieves state-of-the-art few-shot recognition results.The cited evaluations use ModelNet40 features and few-shot classification.
- Multi-view alignment: Multi-view alignment produces geometrically informed spatial understanding in attention maps, including evidence relevant to object pose and 3D spatial relationships.
Multi-view Alignment visualization analysis.
The multi-view alignment analysis visualizes which image queries local patches attend to and isolates scaling and alignment as key ReCon++ modifications. The ablation attributes gains to stronger 3D representations and further improvements from multi-view distillation.
- Visualization: Attention-map visualizations show which multi-view image query each local patch primarily attends to.Distinct colors denote distinct image queries.
- Ablation: Scaling up 3D representation is critical for representation learning and improves 3D multimodal comprehension, while multi-view distillation further improves results.
B.1 Evaluation System
The 3D MM-Vet evaluation system uses GPT-based few-shot scoring, repeated assessments, and capability-level aggregation across five question categories. GPT-4 is selected for greater scoring stability, while dataset construction and API costs are also documented.
- Scoring procedure: Each answer is scored K times and averaged, with K set to 5 for the reported evaluation.
- Score aggregation: Capability-level averages sum scores across the answer set for each category and normalize by the number of answers in that capability set.
- Scoring stability: GPT-4 is used to reduce scoring variability, achieving a standard deviation below 0.1 across rounds.The paper contrasts this stability with greater variability from GPT-3.5.
- Benchmark composition: 3D MM-Vet contains five question categories, with knowledge and general visual recognition receiving higher proportions because they contain multiple subparts.
- Implementation: The estimated cost is approximately $900 for ShapeLLM training data and around $12 for 3D MM-Vet inference.
C Implementation details
ReCon++ uses ViT-consistent model variants and point-cloud inputs with coordinate and color channels. Its design also includes masking and stop-gradient choices evaluated through ablations.
- C Implementation details: ReCon++ follows the standard ViT architecture with four model structures consistent with prior work.The model variants are documented in Table 14.
- C Implementation details: Each point-cloud sample contains 10,000×6 values: xyz coordinates followed by rgb values.
- C Implementation details: The point encoder retains the original local-guided stop-gradient strategy.
- C Implementation details: Causal masking improves transfer-learning capability compared with random masking in the ablation experiments.Mask type and stop-gradient settings are evaluated on ReCon++-L and ShapeLLM-13B.
D Training details
Training uses contrastive reconstruction with a two-stage process: masked-modeling reconstruction followed by cross-modal alignment through knowledge distillation. The paper also documents distinct point counts for transfer classification versus zero-shot and ShapeLLM tasks.
- D Training details: ReCon++ training uses contrast with reconstruct and separates reconstruction from cross-modal alignment.The reconstruction stage uses mask modeling, while alignment uses knowledge distillation.
- D Training details: The reconstruction stage is based on mask modeling, and the cross-modal alignment stage is based on knowledge distillation.
- D Training details: ReCon++ is pretrained on 1,024 points for transfer-learning classification and 10,000 points for zero-shot and ShapeLLM tasks.
- D Training details: Training details and hyperparameter settings are documented in Table 16.
- D Training details: The paper identifies future work in larger embodied-understanding datasets, unified 3D-scene comprehension, robotics co-design, and real-time efficiency.Suggested efficiency directions include model compression.