Source-linked AI summary
Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation
Chongjie Ye, Cheng Cao, Chuanyu Pan, Yiming Hao, Yihao Zhi, Yuanming Hu, Xiaoguang Han
TL;DR
Native 3D generation remains constrained by scarce 3D data and indirect 2D-to-3D pipelines. Omni123 unifies text, images, and 3D in one autoregressive model trained through interleaved cross-modal cycles, significantly improving geometric consistency and semantic alignment in text-guided 3D generation and editing.
Problem
Native multimodal 3D generation remains challenging because high-quality 3D data are scarce and existing indirect pipelines can compromise geometric consistency.
Method
Omni123 represents text, images, and 3D as discrete tokens in a shared sequence space and trains them through interleaved semantic–visual–geometric generation cycles.
Results
Interleaved cross-modal generation significantly improves geometric consistency and semantic alignment in text-guided 3D generation and editing tasks.
Takeaways & Limitations
Cross-modal generative consistency provides a scalable pathway toward multimodal 3D world models.
Takeaways & Limitations
The model confines rendered imagery to Image→3D and Text→3D objectives because its synthetic rendering distribution conflicts with in-the-wild photographs.
Abstract
from arXiv · showhide
Recent multimodal large language models have achieved strong performance in unified text and image understanding and generation, yet extending such native capability to 3D remains challenging due to limited data. Compared to abundant 2D imagery, high-quality 3D assets are scarce, making 3D synthesis under-constrained. Existing methods often rely on indirect pipelines that edit in 2D and lift results into 3D via optimization, sacrificing geometric consistency. We present Omni123, a 3D-native foundation model that unifies text-to-2D and text-to-3D generation within a single autoregressive framework. Our key insight is that cross-modal consistency between images and 3D can serve as an implicit structural constraint. By representing text, images, and 3D as discrete tokens in a shared sequence space, the model leverages abundant 2D data as a geometric prior to improve 3D representations. We introduce an interleaved X-to-X training paradigm that coordinates diverse cross-modal tasks over heterogeneous paired datasets without requiring fully aligned text-image-3D triplets. By traversing semantic-visual-geometric cycles (e.g., text to image to 3D to image) within autoregressive sequences, the model jointly enforces semantic alignment, appearance fidelity, and multi-view geometric consistency. Experiments show that Omni123 significantly improves text-guided 3D generation and editing, demonstrating a scalable path toward multimodal 3D world models.
1 Introduction
Omni123 addresses 3D data scarcity by unifying text, images, and 3D geometry as discrete tokens in one autoregressive framework. Its interleaved cross-modal training uses semantic-visual-geometric consistency to transfer 2D knowledge into more geometrically consistent and semantically aligned 3D generation and editing.
- Motivation and challenge: 3D generation is under-constrained by scarce high-quality data, while indirect image-editing and lifting pipelines often compromise geometric consistency.2D imagery is abundant, but existing approaches commonly use it as an intermediate proxy followed by iterative optimization.
- Motivation and challenge: Simply aggregating cross-modal tasks can cause harmful task interference because text-to-image and text-to-3D rely on different appearance and geometric priors.Effective knowledge transfer therefore requires careful coordination across objectives.
- Method: Interleaved X-to-X training treats image–3D generative consistency as an implicit structural constraint for robust 3D representations.Text supplies semantic intent, images provide appearance and partial geometric cues, and 3D provides explicit spatial structure and multi-view information.
- Method: Omni123 tokenizes text, images, and 3D geometry in a shared sequence space, enabling native 3D reasoning and editing without intermediate 2D proxies.The unified autoregressive formulation trains conditional generation across heterogeneous paired datasets rather than requiring fully paired text-image-3D triplets.
- Method: Semantic-visual-geometric cycles let abundant 2D observations serve as a geometric prior, mitigating task interference and strengthening 3D representations under data scarcity.The approach coordinates diverse generative tasks across heterogeneous datasets for beneficial cross-modal transfer.
- Results: Interleaved cross-modal generation significantly improves geometric consistency and semantic alignment in text-guided 3D generation and editing.The experiments examine interactions among multimodal objectives and their effects on learned 3D representations.
2 Related Work
Prior work progressed from GANs to diffusion and flow-based image generation, while 3D synthesis developed through indirect 2D-prior distillation and native 3D modeling. Multimodal language models have also increasingly unified visual generation and understanding, with early efforts extending this direction to 3D.
- Image Generation: Diffusion models and Flow Matching became dominant image-generation paradigms by formulating synthesis as reverse denoising from noise to data.GANs were earlier representative methods for image generation and editing, while later likelihood-based models improved distribution coverage.
- 3D Generation: 3D generation follows indirect 2D-prior distillation or native 3D modeling, with the former suffering slow optimization, inconsistent appearance, and artifacts.Native 3D generative models learn the 3D distribution directly and offer a more efficient, scalable route toward high-quality synthesis.
- Unified Multimodal Models: Multimodal language models evolved from mapping visual tokens into pretrained diffusion text spaces toward end-to-end unified generation and understanding across modalities.Recent systems including GPT-4o and Nano-Banana accelerated this progress, while ShapeLLM-Omni was among the first efforts to unify 3D generation.
3 Data Pre-Processing
Omni123’s data pipeline combines large-scale text, image, and 3D pairwise data with interleaved single- and multi-view generation tasks. It further improves 3D supervision through asset processing, geometry-aligned multi-granularity captions, and synthesized assets with richer structure and detail.
- Image–3D Processing: The 120 M image–3D pairs provide the largest subset and support reconstruction learning through rendering, filtering, and geometric post-processing.Assets are canonicalized, textured, rendered under 2,000 HDRI lighting setups, filtered for geometry and texture defects, then converted to watertight meshes with SDF and point-cloud sampling.
- 3D Captioning: A visual chain-of-thought captioning pipeline generates detailed, multi-granularity descriptions for 5 M filtered 3D assets and filters them for caption–geometry consistency.The process yields 9 M Text–3D training pairs, using VLM analysis, categorization, and few-shot augmentation from 10K human captions.
- Data Synthesis: A Text→Image→3D synthesis pipeline addresses Objaverse’s simple structures by generating more complex and geometrically detailed assets.It curates single-object prompts, generates controlled object-centric images, removes and evaluates backgrounds, and synthesizes meshes with Meshy 4 before standard post-processing.
4 Model Training
Omni123 trains a unified autoregressive model by converting text, images, and 3D shapes into compatible discrete token sequences and coordinating cross-modal generation tasks. Its pre-training uses abundant 2D supervision as a geometric prior, while continued training adds multi-view sequences to enforce consistency between appearance and 3D geometry.
- Unified architecture: Omni123 combines modality-specific tokenizers with a unified autoregressive Transformer and separate output heads for text-conditioned image and 3D generation.Text uses CLIP and Qwen3-0.6B encoders, while images and 3D shapes are represented as compact 1D discrete token sequences.
- 3D tokenization: Cube3D converts 3D shape representations into discrete tokens designed to preserve smooth surfaces, sharp edges, and high-frequency geometric details.The tokenizer is built on the 3DShape2VecSet framework and is compatible with mixed-modal autoregressive foundation models.
- X-to-X pre-training: The X-to-X paradigm unifies text-to-image, text-to-3D, image-to-3D, and 3D-to-image as autoregressive next-token prediction over a shared discrete vocabulary.Traversing semantic–visual–geometric cycles enforces cross-modal generative consistency as an implicit structural constraint.
- Training objective: Text conditioning is randomly dropped with probability pdrop = 0.1 during training, enabling Classifier-Free Guidance at inference time.All four pre-training tasks use a shared autoregressive cross-entropy objective with uniform loss weighting.
- Continued training: Continued training adds viewpoint-aware sequences that combine images, 3D shapes, and novel-view rendering, enforcing consistency between preceding images, generated meshes, and rendered views.This stage addresses the missing viewpoint reasoning needed to connect 2D appearance with 3D geometry.
5 Align to Instruction-Based 3D Editing
Omni123 is adapted for instruction-based 3D editing by conditioning autoregressive target generation on source 3D tokens and edit text. Its editing pipelines target localized, semantically aligned modifications while preserving multi-view consistency and, for appearance edits, the underlying geometry.
- Editing Architecture: Source 3D shape tokens form a non-causal prefix, while the edit instruction enters through cross-attention to autoregressively generate the target token sequence.The target sequence is decoded into an edited mesh using the Cube3D decoder and Marching Cubes.
- Editing Pipelines: The method supports structure and articulation edits through before–after pose pairs and appearance edits through foundation-model pipelines that modify textures, colors, or style.The appearance pipeline preserves the underlying geometry.
- Training: Text conditioning is dropped with probability pdrop = 0.1 during training to enable Classifier-Free Guidance at inference.The model is therefore exposed to conditioned and unconditioned text inputs during training.
- Editing Pipelines: Both editing pipelines enforce edit locality, multi-view consistency, and semantic alignment between the instruction and the applied modification.These properties are shared across structural and appearance editing.
- Training: Training tokenizes each pair into source tokens, target tokens, and text embeddings, computes loss only over target tokens, and fine-tunes the full model from the SFT checkpoint.The source prefix is excluded from loss computation, with uniform weighting across target positions.
6 Experiments
Omni123 is evaluated on text-to-3D shape generation, instruction-based 3D editing, and image-tokenizer reconstruction. Results emphasize cross-modal geometric fidelity, editing precision, and high-quality reconstruction under limited 3D data.
- Experimental setup: Omni123 is evaluated on text-to-3D shape generation, instruction-based 3D editing, and image-tokenizer reconstruction quality.Unless otherwise stated, experiments use greedy decoding with classifier-free guidance scale ω = 5.0.
- Text-to-shape generation: Text-to-3D generation compares Omni123 with two-stage image-to-3D pipelines and native autoregressive models ShapeLLM-Omni and Cube3D.The two-stage workflows can compromise geometric consistency, while native models are constrained by 3D data scarcity and limited cross-modal transfer.
- 3D editing: Instruction-based editing on Edit3D-Bench evaluates semantically faithful addition and removal of geometry using Chamfer Distance and F1 score.The benchmark uses curated source-target mesh pairs and requires localized geometric transformations from natural-language directives.
- Image tokenizer: Qualitative comparisons show superior preservation of high-frequency details, structural integrity, text rendering, and logo reconstruction over competing tokenizers.Figure 7 covers object-centric images, scene-level details, human details, and text rendering; competing methods often exhibit blur and semantic distortion.
7 Conclusion
Omni123 addresses 3D data scarcity by unifying text-to-2D and text-to-3D generation in one autoregressive multimodal model. Its staged training uses abundant 2D observations as an implicit structural constraint for 3D learning and supports text-guided generation and instruction-based 3D editing.
- Omni123 is a unified multimodal generative model designed to address 3D data scarcity.
- Its three-stage recipe combines heterogeneous-pair cross-modal pre-training, learnable view-token training, and Text–Image–3D interleaved fine-tuning.The stages provide cross-modal learning, viewpoint awareness, and a semantic–visual–geometric loop.
- The framework unifies text-to-2D and text-to-3D generation within a single autoregressive framework.Abundant 2D observations serve as an implicit structural constraint for 3D learning.
- Experiments cover text-guided 3D generation and instruction-based 3D editing.