Source-linked AI summary
UniMesh: Unifying 3D Mesh Understanding and Generation
Peng Huang, Yifeng Chen, Zeyu Zhang, Hao Tang
TL;DR
3D generation and understanding are often fragmented across separate architectures, limiting their integration. UniMesh unifies them through latent bridging, iterative mesh editing, and self-reflective caption refinement, achieving competitive benchmark performance while enabling additional interactive capabilities.
Problem
3D generation and understanding typically use isolated architectures and incompatible representations, limiting knowledge transfer and holistic scene modeling.
Method
UniMesh combines BAGEL and Hunyuan3D through a Mesh Head, Chain-of-Mesh editing, and Actor–Evaluator–Self-reflection feedback within one architecture.
Results
UniMesh achieves competitive performance across 3D generation and understanding benchmarks while enabling iterative editing and mutual enhancement between modalities.
Takeaways & Limitations
UniMesh supports 3D systems that generate, understand, critique, and improve their own creations through unified interaction between modalities.
Takeaways & Limitations
Chain-of-Mesh relies on reference image latents rather than direct 3D understanding, while the BAGEL-based evaluator can make incorrect judgments that degrade self-reflection quality.
Abstract
from arXiv · showhide
Recent advances in 3D vision have led to specialized models for either 3D understanding (e.g., shape classification, segmentation, reconstruction) or 3D generation (e.g., synthesis, completion, and editing). However, these tasks are often tackled in isolation, resulting in fragmented architectures and representations that hinder knowledge transfer and holistic scene modeling. To address these challenges, we propose UniMesh, a unified framework that jointly learns 3D generation and understanding within a single architecture. First, we introduce a novel Mesh Head that acts as a cross model interface, bridging diffusion based image generation with implicit shape decoders. Second, we develop Chain of Mesh (CoM), a geometric instantiation of iterative reasoning that enables user driven semantic mesh editing through a closed loop latent, prompting, and re generation cycle. Third, we incorporate a self reflection mechanism based on an Actor Evaluator Self reflection triad to diagnose and correct failures in high level tasks like 3D captioning. Experimental results demonstrate that UniMesh not only achieves competitive performance on standard benchmarks but also unlocks novel capabilities in iterative editing and mutual enhancement between generation and understanding. Code: https://github.com/AIGeeksGroup/UniMesh. Website: https://aigeeksgroup.github.io/UniMesh.
1 Introduction
UniMesh addresses the fragmentation between 3D generation and understanding by coupling both within one architecture. Its Mesh Head, Chain-of-Mesh, and self-reflection mechanisms support latent transfer, iterative editing, and feedback-driven understanding.
- 3D generation and understanding commonly use isolated architectures and incompatible representations, limiting knowledge transfer and holistic scene modeling.
- UniMesh jointly learns and iteratively couples 3D generation and understanding within a single cohesive architecture.
- The Mesh Head bridges BAGEL’s diffusion-based image generation with Hunyuan3D’s implicit shape decoder through latent translation.
- Chain-of-Mesh uses reference image latents and editing prompts to iteratively generate meshes aligned with requested modifications.
- Self-reflection uses an Actor–Evaluator–Self-reflection triad to diagnose captioning failures and guide regeneration.
- Experiments report competitive benchmark performance alongside iterative mesh editing, mutual enhancement, and few-shot error correction capabilities.
2 Related Work
Prior work has advanced feed-forward 3D generation, alternative geometric representations, and multimodal understanding, but these capabilities generally remain separated. UniMesh instead combines creation and understanding in one architecture.
- Feed-forward systems have enabled fast, high-fidelity 3D synthesis from diverse inputs using vision-language models and multi-view diffusion priors.
- Large reconstruction models regress 3D representations directly, while later systems improve processing, speed, geometric fidelity, and textured mesh generation.
- Alternative approaches generate 3D Gaussians as efficient differentiable representations for rapid reconstruction and sparse-view prediction.
- Unified generation pipelines bridge diffusion priors and explicit geometry through reconstruction heads, differentiable supervision, textures, materials, and normal maps.
- Unlike methods that separate understanding from generation, UniMesh jointly supports iterative semantic editing and caption refinement within one coherent architecture.
- 3D understanding research addresses visual appearance, shape reasoning, part awareness, and links between geometry and natural-language semantics.
3 The Proposed Method
UniMesh integrates latent-space generation, Chain-of-Mesh editing, and self-reflective understanding into a bidirectionally communicating 3D framework. The method reuses latent and textual context for inference-time editing and iteratively refines captions through feedback.
- 3.1 Overview: UniMesh jointly optimizes 3D generation and understanding, establishing bidirectional communication between generative and understanding modules.
- 3.2 Mesh Head: The Mesh Head translates BAGEL image latents into Hunyuan3D conditioning latents, bypassing lossy RGB reconstruction and preserving geometric fidelity.
- 3.2 Mesh Head: The Mesh Head is supervised-finetuned on 3D data for one-step shape prediction from single-view inputs.
- 3.3 Chain-of-Mesh: Chain-of-Mesh combines the original image latent with a new editing prompt, regenerating an updated latent and edited mesh through BAGEL, the Mesh Head, and Hunyuan3D.
- 3.3 Chain-of-Mesh: Chain-of-Mesh repeats this latent–prompting–regeneration loop at inference time without parameter updates.
- 3.4 Self-Reflection: The understanding pipeline renders diverse views, selects six informative images, and uses an Actor–Evaluator–Self-reflection loop to refine captions.
- 3.4 Self-Reflection: Figure examples show captions describing object identity, color combinations, and structural elements from four good views.
4 Experiment
UniMesh is evaluated across 3D generation, captioning, semantic editing, and component ablations. It shows competitive generation and balanced captioning, while enabling iterative natural-language mesh modification without retraining.
- 3D Object Understanding: UniMesh achieves balanced captioning performance with CLIP Image-Text 0.297, CLIP Text-Text 0.686, and FID 0.113, while Cap3D leads R@10 at 41.27%.UniMesh reaches R@10 35.97% while outperforming most models in visual-textual consistency and caption realism.
- 3D Object Generation: 0.296 CLIP Image-Text similarity surpasses all compared baselines, establishing UniMesh as state of the art among open models under this protocol.The generated meshes match textual descriptions in shape, color, pose, and compositional details.
- Object Editing: Chain-of-Mesh enables iterative semantic editing that preserves original geometry while applying natural-language changes without manual mesh manipulation or retraining.Examples include changing colors, adding attributes, modifying structure, and removing objects.
- Ablation Study: DiffuRank improves CLIP alignment and retrieval, CoT further improves semantic alignment and FID, and Reflexion adds gains in lexical similarity.The full DiffuRank + CoT + Reflexion configuration performs strongest across most metrics.
5 Limitation and Future Work
UniMesh’s current limitations concern indirect 3D reasoning and imperfect self-evaluation. Future work targets native geometric understanding and more reliable reflection mechanisms.
- Limitations: Chain-of-Mesh relies on reference image latents rather than direct 3D understanding, limiting how edits are grounded geometrically.The stated future direction is to understand 3D objects directly in their native geometric representation.
- Future Work: The BAGEL-based Reflexion evaluator has limited 3D reasoning capability, which can produce incorrect judgments and degrade self-reflection quality.Future work proposes more reliable evaluation and reflection mechanisms.
6 Conclusion
UniMesh unifies 3D generation and understanding through latent-space integration and iterative reasoning mechanisms. The resulting closed loop supports semantic editing, caption refinement, and mutual improvement without task-specific redesign, while retaining limitations from 2D-view reasoning and evaluator imperfections.
- Unified Architecture: A purpose-built Mesh Head integrates BAGEL and Hunyuan3D, transferring semantic-rich image latents directly to high-fidelity 3D shape generation.This bypasses intermediate rendering and preserves geometric integrity.
- Reasoning Mechanisms: Self-reflection critiques and refines understanding outputs, while Chain-of-Mesh converts semantic critiques into actionable geometric refinements for zero-shot text-guided editing.Together, these mechanisms extend UniMesh beyond one-pass synthesis.
- Conclusion: The combined components form a closed-loop system in which generation informs understanding and understanding guides generation without task-specific redesign.The conclusion identifies reliance on 2D views and evaluator imperfections as remaining limitations.
A DETAILED CAPTIONS QUALITY ANALYSIS
Captioning results show distinct strengths across specialized and general-purpose models, while UniMesh combines generation fidelity with retrieval and semantic alignment. Model scale and task-specific or hybrid training affect which metrics are strongest.
- Cap3D achieves the strongest retrieval results among 3D-specialized models, including R@10: 41.27%, R@5: 33.52%, and R@1: 17.51%.Its performance is attributed to large-scale 3D-text pair pretraining for understanding and describing 3D objects.
- Qwen2.5-VL-7B records the highest CLIP Text-Text score, 0.715, among the compared models.This result reflects strong semantic understanding in the captioning evaluation.
- Kimi-VL leads CLIP Image-Text similarity at 0.312, but its Text-Text similarity and lexical similarity are lower at 0.597 and 0.116.The results indicate stronger cross-modal alignment alongside weaker semantic and lexical consistency on these measures.
- UniMesh achieves the best FID score, 0.113, while retaining competitive retrieval and CLIP performance.Its retrieval scores are R@10: 35.97%, R@5: 28.09%, R@1: 13.72%; CLIP scores are Image-Text: 0.297 and Text-Text: 0.686.
- Larger models tend to excel on CLIP metrics, whereas smaller models remain competitive on retrieval and can achieve lower FID.The comparison supports the practicality of smaller models in settings with limited computational resources.
- Task-specific design and hybrid training strategies can deliver strong generation quality while preserving robust retrieval and semantic understanding.