PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers

Zongyuan Yang, Mingjing Yi, Wanli Ma, Chenzhuo Fan, Bocheng Li, Baolin Liu, Yuke Lou, Yingde Song, Yongping Xiong, Zhengdong Guo, Shimu Wang

arXiv:2605.16745v1cs.CV

TL;DR

Diffusion-based 3D reconstruction separates semantic interpretation from geometric construction, while integrating meshes as native MLLM modalities remains a representational challenge. EVA01 uses a dual-expert Mixture-of-Transformers to integrate mesh understanding, generation, and editing, achieving state-of-the-art native text-to-3D fidelity and identity-preserving multi-turn editing.

  • Problem

    Integrating 3D meshes as first-class MLLM modalities remains a representational challenge, while diffusion-based reconstruction separates semantic interpretation from geometric construction.

    Source: §1 · p. 2

  • Method

    EVA01 uses a Mixture-of-Transformers with pretrained understanding and mirrored generation experts, linked by shared global attention and hard modality routing.

    Source: §1 · p. 2

  • Results

    EVA01 achieves state-of-the-art native text-to-3D generation fidelity and enables context-aware, identity-preserving multi-turn 3D editing.

    Source: §6 · p. 22

  • Takeaways & Limitations

    The paper identifies grid-based sparse latents, modality dropout and curriculum training, and high-fidelity finetuning as practical design principles for 3D-native MLLMs.

    Source: §6 · p. 22

    Current automatic 3D-generation metrics remain partial proxies, under-resolving mesh properties such as topology, thin structures, watertightness, and edit identity preservation.

    Source: §5 · p. 20

Abstract

from arXiv · show

This paper addresses the challenge of integrating 3D meshes as a native modality within Multimodal Large Language Models (MLLMs). Diffusion-based large reconstruction models decouple semantic understanding from geometric reasoning, operating as stateless reconstructors conditioned on dense 2D pixel priors. Recent MLLM-based methods treat the 3D modality as an external output rather than a native component of the multimodal sequence, making incremental adaptations without a systematic analysis of how geometric manifolds align with MLLM feature spaces. We introduce EVA01, a unified framework that extends the modality boundary of MLLMs to natively incorporate 3D mesh understanding, generation, and context-aware editing. Built upon a Mixture-of-Transformers (MoT) architecture, EVA01 decouples the model into a pre-trained Understanding Expert ($E_{\mathrm{und}}$) and a structurally mirrored Generation Expert ($E_{\mathrm{gen}}$), coupled through shared global self-attention with hard modality routing. This design aligns the semantic latent space of the MLLM backbone with the geometric manifold, enabling direct transfer of multimodal priors without intermediate 2D representations. Results show that EVA01 achieves state-of-the-art native text-to-3D generation fidelity and unlocks robust long-context multi-turn geometric editing with identity preservation, a capability fundamentally inaccessible to stateless reconstruction pipelines. Our findings further offer architectural insights for integrating 2D foundation models with 3D tasks, informing the design of 3D-native multimodal systems. Project Page: https://www.seeles.ai/research/pages/EVA01

1 Introduction

EVA01 treats 3D meshes as a native modality in a unified MLLM for mesh understanding, generation, and context-aware editing. Its Mixture-of-Transformers design couples a semantic understanding expert with a geometry-generation expert, while training and sequence modeling target semantic-geometric alignment and stateful editing.

Motivation

  • Diffusion reconstruction methods use 2D priors but separate semantic understanding from geometry and require full regeneration for edits; recent MLLM methods lack systematic representation-level alignment analysis.The cited reconstruction approaches include SDS Poole et al. (2023); Wang et al. (2023), cascaded multi-view reconstruction Xu et al. (2024a); Team (2025b), and feed-forward models Xiang et al. (2024); cited MLLM approaches include Ye et al. (2025b, 2026); Huang and Xu (2026); Chen et al. (2025b); Huang et al. (2026); Chen et al. (2026).Source: §1 · p. 2
  • Native 3D sequencing must align text semantics, image spatial correlations, and topologically constrained meshes, with autoregressive modeling for text and images and flow matching for geometry.The paper identifies scarce large-scale, high-quality text–3D paired data as a further challenge to transferring multimodal priors.Source: §1 · p. 2

EVA01

  • EVA01 is a native 3D MLLM that combines mesh understanding, generation, and multi-turn editing through mirrored experts and shared global self-attention with hard modality routing.The pre-trained Understanding Expert preserves MLLM semantic priors, while the Generation Expert synthesizes geometry; shared attention enables cross-modal knowledge transfer with optimization independence.Source: §1 · p. 2, Figure 2 · p. 3
  • Progressive modality alignment uses interleaved text–image–mesh sequences and modality dropout to establish robust cross-modal correspondences.This multi-stage post-training strategy addresses the mismatch between semantic representations and geometric structures.Source: §1 · pp. 2–3
  • EVA01 models each geometric state from the current instruction and full interaction history, enabling context-aware, identity-preserving edits across turns.The paper constructs a large-scale interleaved text–image–mesh dataset for this stateful generation formulation.Source: §1 · pp. 2–3

2 Related Work

Related work identifies limitations in 3D generative representations and in integrating 3D with MLLMs. It frames EVA01’s distinct combination of native mesh processing, shared semantic–geometric modeling, and identity-preserving multi-turn editing.

2.1 3D Generative Models and Latent Representations

  • 3D generation remains constrained by data sparsity, topology, and representation trade-offs: global tokens oversmooth detail, sparse tokens cost more, and split geometry–appearance modeling can misalign semantics.Prior 2D-prior reconstruction methods can also produce view inconsistencies and decouple semantic reasoning from geometry.Source: §2.1 · p. 3

2.2 Unified Multimodal Large Language Models

  • Unified MLLMs enable compositional control across long contexts, but 3D unification must preserve spatial consistency and asset identity through sequential edits.Cascaded text-to-image-to-3D pipelines can decouple semantic planning from geometry, leading to identity drift and topological discontinuities.Source: §2.2 · pp. 3–4

2.3 Unified 3D Multimodal Large Models

  • Prior 3D MLLMs variously use autoregressive tokens, dual-stream attention, MoT experts, or external MLLM conditioning, but differ in native mesh processing and persistent edit identity.CG-MLLM is the closest architectural parallel, using dedicated understanding and generation experts with hard routing; the passage distinguishes EVA01 by its representation, native mesh tokens, and post-training curriculum.Source: §2.3 · p. 4
  • EVA01’s defining combination is shared-attention MoT experts, native mesh tokens in a structured grid-based latent with flow matching, and stateful long-context editing with explicit identity preservation.Each geometric update is conditioned on the full interaction history.Source: §2.3 · p. 4

3 Methodology

EVA01 brings mesh understanding and generation into a shared multimodal sequence using sparse voxel latents, specialized transformer experts, curated editing data, and a five-stage training curriculum. Its method combines conditional flow matching with contextual conditioning to support semantic-geometric alignment and multi-turn editing.

3.1 Architecture: Unified 3D Multimodal Mixture-of-Transformers

  • Sparse O-Voxel latents anchor geometry and appearance to grid coordinates, enabling mesh conversion and spatial grounding for long-context sequences.A pre-trained VAE compresses the representation into sparse tokens; the paper contrasts this coordinate-anchored design with unordered VecSet tokens, which it says can contribute to geometric collapse.Source: §3.1 · pp. 5–6
  • Hard routing assigns text and image tokens to Eund and mesh tokens to Egen, while shared global attention lets generation query semantic priors.Generation is staged as sparse structure, geometry, then material: inference runs sequentially, while training parallelizes stages through the unified attention mask.Source: §3.1 · pp. 5–6

3.2 Data: From Static Assets to Contextual Editing

  • The data pipeline curates approximately 1.2M raw assets, reserves 400K premium assets, and synthesizes approximately 3M procedural plus 300K semantic editing trajectories.Procedural sequences cover transformations, topology changes, and animation; semantic sequences use LLM instructions and 2D image editing lifted to 3D, with turns conditioned on accumulated context.Source: §3.2 · pp. 6–7

3.3 Training: Multi-Stage Curriculum Learning

  • Training combines conditional flow matching for mesh generation with autoregressive cross-entropy for mesh-conditioned text, then progresses through five capability-building stages.The first stage trains only a lightweight mesh-feature projector; the second initializes generation with image-conditioned flow matching and mesh-text understanding, and the third aligns text, image, and mesh-text conditioning using triple-batch sampling and image-token dropout.Source: §3.3 · p. 7
  • Context-aware instruction tuning conditions each new geometric state on full interaction history and the previous clean mesh latent, supporting identity-preserving edits across turns.Final finetuning uses the curated 400K high-quality subset at reduced learning rates to refine geometry and surface quality while retaining editing and understanding capabilities.Source: §3.3 · p. 8

4 Experiments

Section 4 evaluates EVA01 across text- and image-conditioned generation, multi-turn editing, and mesh understanding, alongside analyses and ablations of its semantic and geometric representation paths, staged curriculum, spatially anchored tokens, and shared-sequence coupling.

4.1 Experimental Protocol

  • The evaluation covers Toys4K generation, a 400-sample editing set, and prompt-only captioning on 200 held-out PointLLM-200 objects.Generation uses CLIP, FD, and KD; editing measures unedited-region consistency and overall quality, with manually verified masks available to mask-dependent baselines but not used by EVA01 at inference. Caption metrics include lexical overlap, embedding similarity, and GPT-ref and render-grounded GPT-img scores.Source: §4.1 · pp. 9–10

4.2 Single-Turn Generation

  • On Toys4K text-to-3D, EVA01 scores 35.72 CLIP and 70.4% preference versus TRELLIS’s 30.80 CLIP and 14.8% preference.For image-to-3D, TRELLIS.2 leads with 89.34 CLIP, 56.82 FD, 0.49 KD, and 41.7% preference; EVA01 ranks second with 87.28 CLIP, 61.74 FD, 0.63 KD, and 22.2% preference. EVA01 uses one mesh-native model for both settings, and qualitative comparisons report coherent geometry and object-level semantics across text and image inputs.Source: Table 3 · p. 10, §4.2 · pp. 11–12, Figure 4 · p. 11

4.3 Mesh Understanding

  • EVA01-Align leads reference-aligned captioning on five PointLLM-200 metrics and ranks second on METEOR, while EVA01-Final leads both GPT-based judge metrics.After instruction-oriented and high-quality finetuning, EVA01-Final’s lexical overlap falls, but it retains second-best Sentence-BERT and SimCSE scores. The authors caution that single-reference overlap metrics reward matching caption wording and style, whereas GPT-img rewards detailed visual coverage; EVA01-Final’s stronger semantic coverage may occasionally include details not explicit in the renders.Source: Table 4 · p. 12, §4.3 · pp. 12–13

4.4 Multi-Turn Context-Aware Editing

  • EVA01 achieves 93.75% editing preference and improves CLIP from VoxHammer’s 30.45 to 70.18 while reducing FDDINOv2 from 178.62 to 89.37.VoxHammer leads unedited-region consistency with 0.015 CD and 32.1 PSNR, while mask-free EVA01 remains close at 0.018 CD and 29.3 PSNR. EVA01 conditions each new mesh state on the instruction and accumulated text–image–mesh history, supporting sequential edits that preserve identity and prior changes.Source: §4.4 · pp. 13–15, Table 5 · p. 14

4.5 Subjective Evaluation

  • In user studies, EVA01 ranks among the top two in both single-turn modalities and receives 93.75% of multi-turn editing selections.The study recruited 23 participants, each completing at least 25 comparisons per category; judgments covered geometric fidelity, semantic alignment, and, for editing, identity preservation across turns. EVA01 receives 70.4% of text-to-3D selections, while TRELLIS.2 leads image-to-3D with 41.7% and EVA01 is second at 22.2%.Source: §4.5 · p. 16

4.6 Representation Analysis and Visualization

  • CLIP has the highest Objaverse retrieval score (43.50), but DINOv3 ViT-L/16 leads dense geometric probes with 92.74 NAVI R@5cm, 0.8935 NYUv2 δ1, and 0.3126 RMSE.The analysis distinguishes global semantic alignment from dense geometry: CLIP’s high retrieval accompanies weaker spatial probes, while DINO-style features retain smoother, part-aligned structure; SigLIP2 lies between these regimes. Probe protocols differ, and pooled retrieval features are not a general measure of semantic understanding for all model types.Source: §4.6 · pp. 17–18, Table 6 · p. 18
  • Qwen3-VL’s 3D-relevant information varies by path and layer, favoring layer- and path-aware routing over reliance on the final hidden state.Qwen3-VL-8B VT ds16 has the group’s best NAVI score at 80.25, while Qwen3-VL-2B LLM l14 leads its NYUv2 results with 0.7826 δ1 and 0.5246 RMSE; LLM l28 leads retrieval at 15.80 as dense-probe scores weaken.Source: §4.6 · p. 18
  • Semantic retrieval, cross-view correspondence, and depth probing favor different feature families, while generation-side latents preserve appearance but do not provide complete transferable 3D geometry.Qwen2.5-VL final states favor semantic compatibility and retrieval, whereas MMDiT blocks expose denser local cues; VAE latents preserve appearance and reconstruction layout but transfer weakly to geometry probing.Source: §4.6 · p. 19, Figure 9 · p. 19

4.7 Ablation Studies and Critical Insights

  • A 10K alignment warm-up before instruction tuning yields lower CE loss and higher captioning quality than direct mesh-caption tuning, while sparse VAE latents alone fail as language-readable inputs.The sparse Shape-to-Text variant plateaus at high CE loss and produces semantically unreliable descriptions.Source: §4.7 · p. 20
  • Image-conditioned generation warm-up followed by mesh-understanding samples improves convergence and final score over text-only training; shared global attention also outperforms concatenation-based cross-attention and supports context-aware editing.The captioning objective sharpens Eund spatial grounding, which Egen can query through token-level mutual visibility in the unified sequence.Source: §4.7 · p. 20
  • Unlike compact VecSet tokens, sparse voxel tokens bind features to explicit 3D coordinates, enabling meaningful 3D Interleaved MRoPE and usable geometry in the unified sequence.VecSet loss plateaus and its normalized score remains far below grid-based sparse latents because flattened tokens lack intrinsic position anchors.Source: §4.7 · p. 20
  • The ablations identify modality dropout, Mesh-to-Text supervision, and high-quality Stage 5 finetuning as practical requirements for text conditioning, bidirectional alignment, and final fidelity.Stage 3 modality dropout prevents Egen from ignoring weak text conditions; Stage 5 finetuning sets the fidelity ceiling after representation and alignment are stabilized.Source: §4.7 · p. 20

5 Limitations, Discussion & Future Work

EVA01’s 2B experts and 512^3 sparse-voxel resolution validate its decoupled, shared-attention design but leave scaling, evaluation, and visual-conditioning limits. The paper frames these gaps as directions for a more capable 3D-native multimodal architecture.

Limitations and future work

  • EVA01 is limited to 2B experts and 512^3 sparse-voxel resolution; scaling to 4B–8B and 1024^3 is proposed to improve long-context reasoning and fine geometry.The current setting verifies the design hypothesis but does not establish scaling behavior across 3D-native MLLMs.Source: §5 · p. 20
  • Current 3D metrics capture coarse semantic agreement but under-resolve mesh properties, motivating fine-grained benchmarks for geometry, faithfulness, and editing consistency.Metrics based on sampled point clouds or global shape embeddings miss local topology, thin structures, sharp creases, watertightness, material boundaries, and identity preservation in unmodified regions.Source: §5 · pp. 20, 22
  • Mesh understanding uses point-cloud features while generation uses sparse-voxel latents; future work could give both specialized encoders a shared 3D latent substrate.The split is pragmatic: point clouds support mesh captioning, while sparse voxels suit high-resolution synthesis and 3D positional attention.Source: §5 · p. 22
  • EVA01’s image-conditioning path can limit fine-geometry recovery, while text-to-3D still struggles with out-of-distribution compositions and spatial reasoning.The paper attributes image-to-3D weaknesses to insufficient dense pixel evidence: thin, occluded, or distant components may be lost; text or symbol layouts and exact counting also remain imperfect. EVA01 is competitive in image-to-3D but trails the strongest image-specialized reconstruction pipeline; stronger patch-aligned visual features or multi-tower aggregation are proposed.Source: Figure 11 · p. 21, §5 · p. 22
  • EVA01 is presented as a scalable architectural template, not a claim that a 2B, 512^3 model or current metrics covers all 3D-generation regimes.The template combines an MLLM-prior-preserving semantic expert, sparse geometric flow-matching generation, and shared attention; future directions include greater scale and resolution, finer evaluation, unified 3D encoders, and stronger dense visual conditioning.Source: §5 · p. 22

6 Conclusion

EVA01 unifies 3D mesh understanding, generation, and multi-turn editing in a Mixture-of-Transformers framework. It achieves state-of-the-art native text-to-3D fidelity and enables context-aware, identity-preserving editing, while its experiments identify practical design principles and future-work directions.

6 Conclusion

  • EVA01 unifies 3D mesh understanding, generation, and multi-turn editing through dual experts with shared global attention, transferring pre-trained MLLM priors to 3D under limited supervision.The design decouples semantic understanding from geometric generation to bridge their alignment gap.Source: §6 · p. 22
  • EVA01 achieves state-of-the-art native text-to-3D generation fidelity and enables context-aware, identity-preserving multi-turn editing unavailable to stateless reconstruction pipelines.The paper identifies grid-based sparse latents, modality dropout with multi-stage curriculum training, and high-fidelity finetuning as practical design principles.Source: §6 · p. 22
  • The paper identifies grid-based sparse latents as necessary for geometric validity, modality dropout and multi-stage curriculum training for manifold alignment, and high-fidelity finetuning as the final fidelity ceiling.The finetuning ceiling applies after alignment is stabilized.Source: §6 · p. 22
  • Future work is motivated by limitations in model scale, spatial resolution, evaluation fidelity, unified 3D-native encoders, and dense visual conditioning.The authors present these directions as extensions of a scalable architectural template for 3D-native multimodal models.Source: §6 · p. 22

Related papers

Loading 2605.16745v1…