Source-linked AI summary
EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers
Zongyuan Yang, Mingjing Yi, Wanli Ma, Chenzhuo Fan, Bocheng Li, Baolin Liu, Yuke Lou, Yingde Song, Yongping Xiong, Zhengdong Guo, Shimu Wang
TL;DR
Existing 3D reconstruction and MLLM approaches do not natively integrate geometric reasoning with multimodal sequences. EVA01 uses a dual-expert Mixture-of-Transformers architecture to unify 3D understanding, generation, and editing, achieving state-of-the-art text-to-3D fidelity and identity-preserving multi-turn editing.
Problem
Existing approaches separate semantic understanding from geometric reasoning or treat 3D as an external output rather than a native MLLM modality.
Method
EVA01 couples semantic and generation experts through shared attention and hard routing to model 3D meshes within a unified multimodal sequence.
Results
EVA01 achieves state-of-the-art native text-to-3D fidelity and identity-preserving multi-turn 3D editing inaccessible to stateless reconstruction pipelines.
Takeaways & Limitations
The results identify architectural principles for 3D-native MLLMs, including shared semantic-geometric interfaces, curriculum training, and high-fidelity finetuning.
Takeaways & Limitations
Current automatic 3D metrics incompletely assess mesh quality, under-resolving topology, thin structures, sharp creases, watertightness, materials, and edit-preserved identity.
Abstract
from arXiv · showhide
This paper addresses the challenge of integrating 3D meshes as a native modality within Multimodal Large Language Models (MLLMs). Diffusion-based large reconstruction models decouple semantic understanding from geometric reasoning, operating as stateless reconstructors conditioned on dense 2D pixel priors. Recent MLLM-based methods treat the 3D modality as an external output rather than a native component of the multimodal sequence, making incremental adaptations without a systematic analysis of how geometric manifolds align with MLLM feature spaces. We introduce EVA01, a unified framework that extends the modality boundary of MLLMs to natively incorporate 3D mesh understanding, generation, and context-aware editing. Built upon a Mixture-of-Transformers (MoT) architecture, EVA01 decouples the model into a pre-trained Understanding Expert ($E_{\mathrm{und}}$) and a structurally mirrored Generation Expert ($E_{\mathrm{gen}}$), coupled through shared global self-attention with hard modality routing. This design aligns the semantic latent space of the MLLM backbone with the geometric manifold, enabling direct transfer of multimodal priors without intermediate 2D representations. Results show that EVA01 achieves state-of-the-art native text-to-3D generation fidelity and unlocks robust long-context multi-turn geometric editing with identity preservation, a capability fundamentally inaccessible to stateless reconstruction pipelines. Our findings further offer architectural insights for integrating 2D foundation models with 3D tasks, informing the design of 3D-native multimodal systems. Project Page: https://www.seeles.ai/research/pages/EVA01
1 Introduction
EVA01 addresses the semantic-geometric gap by making 3D meshes a first-class modality in a unified MLLM sequence. Its Mixture-of-Transformers design combines semantic anchoring, geometric generation, progressive alignment, and stateful multi-turn editing.
- Motivation: Diffusion-based reconstruction models exploit dense pixel-level priors from multi-view images, while recent MLLM methods primarily use language models as semantic feature extractors or conditioning modules.These approaches lack systematic representation-level analysis of the relationship between MLLM feature spaces and 3D geometry.
- Unified 3D MLLM: EVA01 integrates 3D mesh understanding, generation, and context-aware multi-turn editing as a first-class modality within a unified Mixture-of-Transformers architecture.The framework uses a structurally mirrored generation expert and shared global self-attention for cross-modal knowledge transfer.
- Unified 3D MLLM: The Understanding Expert preserves the pretrained MLLM’s semantic priors, while the Generation Expert is dedicated to geometry synthesis within the unified architecture.This decouples semantic anchoring from geometric generation while retaining their interaction through the shared architecture.
- Semantic Alignment: EVA01 uses curriculum-based semantic alignment to bridge mismatches between semantic representations and geometric structures through progressive modality alignment.The method further uses interleaved text–image–mesh sequences and modality dropout to establish robust cross-modal correspondences.
- Stateful Editing: Its stateful generation formulation treats 3D editing as conditional sequence modeling, enabling identity-preserving geometric modifications across multi-turn interactions.This paradigm provides insights into the training dynamics of 3D-native MLLMs.
2 Related Work
Prior 3D synthesis methods rely on dense 2D diffusion priors and reconstruction-oriented pipelines, while emerging MLLM approaches integrate 3D tokens but retain important architectural and editing limitations. EVA01 is distinguished by combining expert decoupling, native mesh-token processing, and stateful long-context editing.
- 3D generation: High-fidelity 3D generation remains difficult because of data sparsity, cubic volumetric complexity, and strict topological requirements.
- 3D generation: SDS distillation and cascaded multi-view reconstruction use 2D diffusion priors for 3D synthesis but treat 3D as reconstruction conditioned on dense pixel priors.
- MLLMs and 3D: MLLMs have progressed toward unified language-backbone architectures supporting fine-grained control and compositional reasoning across long contexts, but 3D editing requires identity preservation across sequential modifications.
- MLLMs and 3D: ShapeLLM-Omni adds 8,192 learned 3D VQ-VAE tokens and performs autoregressive understanding and generation, but lacks modality-specific expert decoupling.
- EVA01 distinction: EVA01 combines MoT expert decoupling with shared cross-modal attention, native mesh tokens using structured grid latents and flow matching, and stateful long-context multi-turn 3D editing.
3 Methodology
EVA01 natively integrates text, images, and 3D meshes in a unified sequence through sparse-voxel tokenization, dual expert routing, shared attention, and conditional flow matching. Its methodology also introduces spatially grounded 3D positional encoding and a curriculum-supported data and training pipeline for context-aware generation and editing.
- Unified 3D Modeling: EVA01 models p(xmesh | xtxt, ximg, cctx) within a unified sequence stream, extending a pre-trained Qwen3-VL backbone to conditional flow matching for 3D geometry.The context variable cctx encodes historical information from multi-turn interactions.
- 3D Mesh Tokenization: Sparse O-Voxel structures jointly encode geometry and appearance, while a pre-trained VAE compresses them into sparse latent mesh tokens and Flexible Dual Grid conversion avoids iterative field extraction.Each active voxel stores geometric features, PBR material parameters, and grid coordinates; inactive voxels are discarded.
- 3D Mesh Tokenization: EVA01 replaces unordered VecSet tokens with spatially anchored sparse voxels, preserving positional grounding needed for stable geometric correspondences.The sparse representation explicitly retains spatial coordinates, unlike VecSet’s order-invariant latent tokens.
- Mixture-of-Transformers Backbone: The Mixture-of-Transformers backbone separates parameters into Understanding and Generation Experts, with hard routing assigning text and image tokens to Eund and mesh tokens to Egen.Eund preserves pre-trained semantic priors, while Egen undergoes geometric optimization without compromising understanding.
- Generation and Cross-Modal Attention: Egen generates meshes hierarchically through sparse structure, geometry, and material stages, while shared global attention lets it query semantic priors under unified visibility masking.The structure stage predicts active-voxel occupancy, geometry generates shape features in occupied coordinates, and material synthesizes aligned PBR features.
- 3D Positional Encoding: 3D Interleaved MRoPE maps rotary embeddings to sparse voxel coordinates (x, y, z), distributing spatial information across dimensions to prevent geometric drift during long-context editing.The method reuses the original (T, W, H) rotary structure for the three spatial axes.
- Data and Training: The training pipeline combines hierarchical 3D data curation with curriculum learning: approximately 1.2M raw assets yield a premium 400K subset, followed by progressively staged capability alignment.Multi-turn interleaved editing trajectories are synthesized from the static asset foundation, and optimization combines generation and alignment losses.
4 Experiments
EVA01 delivers its strongest gains in text-to-3D generation and supports native mesh understanding and stateful multi-turn editing. Image-to-3D remains a relative weakness against the specialist TRELLIS.2, while captioning results show a shift from reference-style overlap toward instruction-following and human-preferred descriptions.
- Text-to-3D: 35.72 CLIP, 122.48 FD, 1.18 KD, and 70.4% preference establish EVA01’s text-to-3D advantage over TRELLIS and 3DGen-R1.EVA01 surpasses TRELLIS (30.80 CLIP, 238.45 FD, 4.25 KD, 14.8% preference) and 3DGen-R1 (29.35 CLIP, 263.72 FD, 6.85 KD, 7.9% preference).
- Mesh understanding: EVA01-Align achieves the strongest reference-aligned captioning across BLEU-1, BLEU-4, ROUGE-L, Sentence-BERT, and SimCSE, ranking second on METEOR.These results indicate that the first three curriculum stages align geometric input with language, although standard metrics emphasize PointLLM-200 reference-caption style.
- Mesh understanding: EVA01-Final lowers lexical-overlap scores but retains the second-best Sentence-BERT and SimCSE scores and achieves the highest GPT-ref and GPT-img scores.The later stages repurpose mesh grounding for instruction-following, fine-grained, and human-preferred descriptions rather than narrow reference imitation.
- Multi-turn editing: EVA01 achieves mask-free stateful editing, remaining close to VoxHammer in consistency with 0.018 CD and 29.3 PSNR versus 0.015 CD and 32.1 PSNR.Historical mesh tokens, block attention masking, and 3D Interleaved MRoPE condition differential geometric updates on accumulated interaction history, preserving identity across multiple edits.
5 Limitations, Discussion & Future Work
EVA01’s limitations arise from its 2B-scale, 5123-resolution training setting, partial 3D metrics, representational asymmetry, and weaker image-conditioned geometric features. Future work targets larger experts, higher resolution, finer evaluation, unified 3D-native encoders, and improved editing consistency.
- Model scale and resolution: EVA01 is bounded by its 2B-scale training budget and 5123 sparse-voxel resolution, leaving its 3D-native scaling behavior unexamined.The paper proposes scaling each expert toward 4B–8B and resolution toward 10243.
- Evaluation: Existing automatic 3D metrics remain partial proxies because global embeddings and sampled point clouds under-resolve local topology, thin structures, and sharp creases.The discussion specifically identifies ULIP, ULIP-2, and Uni3D as measures of coarse semantic agreement rather than complete mesh quality.
- Failure modes: EVA01 remains imperfect on out-of-distribution compositions, spatial reasoning, exact counting, legible surface text, and context-aware editing consistency.Image-to-3D failures also occur when thin, occluded, or distant components lack sufficient dense pixel evidence.
- Representation: Mesh understanding and generation use different representations: point-cloud features for understanding and structured sparse-voxel latents for generation.This asymmetry is described as pragmatic, with point-cloud encoders supporting mesh captioning and sparse voxels supporting generation.
- Image-conditioned generation: Image-conditioned performance is limited because SigLIP2 features provide weaker dense geometric evidence than DINO-style features, causing EVA01 to trail the strongest image-specialized reconstruction pipeline.The paper attributes this boundary to the quality of dense pixel evidence rather than to a failure of the Mixture-of-Transformers formulation.
- Scope and future work: The paper frames EVA01 as a scalable architectural template combining semantic and generation experts through shared attention, rather than as a saturated solution for all 3D regimes.Identified gaps include larger scale, higher resolution, finer evaluators, unified 3D-native encoders, and stronger dense geometric representations.
6 Conclusion
EVA01 unifies 3D mesh understanding, generation, and multi-turn editing through a dual-expert Mixture-of-Transformers architecture. Its results establish design principles for 3D-native MLLMs while identifying model scale, spatial resolution, evaluation fidelity, encoders, and visual conditioning as future-work directions.
- Contributions: EVA01 integrates 3D mesh understanding, generation, and multi-turn editing within one Mixture-of-Transformers architecture.Its dual-expert design with shared global attention transfers multimodal priors from a pre-trained MLLM backbone to 3D under limited supervision.
- Results: EVA01 achieves state-of-the-art native text-to-3D generation fidelity and enables context-aware, identity-preserving multi-turn editing.The editing capability is inaccessible to stateless reconstruction pipelines.
- Design principles: The experiments identify grid-based sparse latents, modality dropout, and multi-stage curriculum training as practical design principles for 3D-native MLLMs.Grid-based sparse latents support geometric validity, while modality dropout and curriculum training bridge textual and geometric manifolds.
- Limitations and future work: Future work is motivated by limitations in model scale, spatial resolution, evaluation fidelity, unified 3D-native encoders, and dense visual conditioning.These limitations define natural directions for scaling the architectural template for 3D-native multimodal models.
Authors
The paper is authored by a team led by Zhengdong Guo and Shimu Wang, with Zongyuan Yang leading the algorithm development and additional core and contributing authors.
- The full author list also includes Zhengdong Guo and Shimu Wang alongside the named contributors.
- Zhengdong Guo and Shimu Wang are the team leaders, while Zongyuan Yang is the algorithm leader.
- Zongyuan Yang, Mingjing Yi, and Wanli Ma are identified as core contributors.
- Chenzhuo Fan, Bocheng Li, Baolin Liu, Yuke Lou, Yingde Song, and Yongping Xiong are identified as contributors.