Source-linked AI summary
Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer
Ziyuan Huang, DanDan Zheng, Cheng Zou, Rui Liu, Xiaolong Wang, Kaixiang Ji, Weilong Chai, Jianxin Sun, Libin Wang, Yongjie Lv, Taozhi Huang, Jiajia Liu, Qingpei Guo, Ming Yang, Jingdong Chen, Jun Zhou
TL;DR
Unifying visual understanding and generation is difficult because they favor different representations, while existing approaches may separate the two processes. MingTok and Ming-UniVision use a shared continuous visual space and autoregressive framework, with up to 66% fewer input tokens than prior unified architectures.
Problem
Understanding and generation favor different visual representations, while existing approaches separate their processes into distinct spaces and incur repeated encoding and decoding cycles.
Source: §1 · p. 1, §3.2 · p. 7Method
MingTok uses a three-stage continuous tokenizer, and Ming-UniVision builds on it to model text and images in a shared autoregressive framework.
Source: §1 · p. 3Results
Up to 66% fewer input tokens than prior unified architectures, with support for text-to-image, image-to-text, and multi-round interactions.
Source: §1 · pp. 3–4Takeaways & Limitations
A shared visual representation offers early evidence of mutual enhancement between generation and understanding.
Source: §5.4 · p. 18The model struggles to generalize to editing sequences longer than those seen in training and with freeform interleaved understanding and generation.
Source: §5.4 · p. 18
Abstract
from arXiv · showhide
Visual tokenization remains a core challenge in unifying visual understanding and generation within the autoregressive paradigm. Existing methods typically employ tokenizers in discrete latent spaces to align with the tokens from large language models, where the quantization errors can limit semantic expressiveness and degrade the capability of vision-language understanding. To address this, we introduce MingTok, a new family of visual tokenizers with a continuous latent space, for unified autoregressive generation and understanding. While understanding tasks favor discriminative high-dimensional features, generation tasks prefer compact low-level codes. Thus, to reconcile these competing demands, MingTok adopts a three-stage sequential architecture involving low-level encoding, semantic expansion, and visual reconstruction. Built on top of it, Ming-UniVision eliminates the need for task-specific visual representations, and unifies diverse vision-language tasks under a single autoregrsssive prediction paradigm. By formulating both understanding and generation as next-token prediction in a shared continuous space, it seamlessly supports multi-round, in-context tasks such as iterative understanding, generation and editing. Empirically, we find that using a unified continuous visual representation reconciles the competing requirements on the tokenizers by the understanding and generation tasks, thereby leading to state-of-the-art level performance across both domains. We hope our findings will facilitate unified visual tokenization in the continuous domain. Inference code and model weights are released to benefit community.
1 Introduction
Ming-UniVision combines MingTok’s continuous visual tokenizer with a shared autoregressive model for understanding and image generation. Its unified latent interface supports multi-round interaction while reducing visual input-token counts.
1 Introduction
- Understanding favors high-dimensional semantic features while generation favors compact codes preserving visual detail; discrete shared latents can introduce quantization error that harms fidelity and semantic expressiveness.These competing representational demands have led many systems to use asymmetric tokenization schemes, adding optimization difficulty and architectural complexity.Source: §1 · pp. 1, 3
- MingTok maps images through compact continuous latents, autoregressive semantic expansion, and pixel reconstruction, with masked image modeling supervising intermediate representations and reconstructed pixels.The three stages are a low-level encoder, semantic decoder, and pixel decoder; this design targets efficient generation, expressive semantic features, and high-fidelity reconstruction.Source: §1 · p. 3
- Ming-UniVision uses one autoregressive architecture for joint text-image modeling, consuming a shared semantic space and generating compact continuous latents token by token.The paper says this reduces architectural complexity compared with dual-encoder designs.Source: §1 · p. 3
- Shared text, visual, or mixed-modality prefixes can condition reasoning or generation, supporting multi-round tasks such as super-resolution, region-aware editing, and sequential refinement.The paper frames understanding and generation as interwoven steps in a continuous visual dialogue rather than isolated pipelines.Source: §1 · pp. 3–4
- Ming-UniVision uses up to 66% fewer input tokens than prior unified architectures, enabling faster iteration and lower memory overhead in interactive editing.The token reduction supports multi-round in-context editing without repeated encoding through distinct tokenizers.Source: §1 · p. 4
2 Unified Visual Tokenization without Vector Quantization
MingTok uses a three-stage tokenizer to produce compact generation latents, expand them into semantic features, and reconstruct images. Its training combines masked feature prediction and pixel reconstruction to shape the latent, semantic, and reconstruction capabilities.
2.1 Tokenizer Model Architecture
- MingTok chains a compact low-level encoder, causal semantic decoder, and pixel decoder, using full attention for encoding and reconstruction and pixel unshuffle to improve texture fidelity and edge sharpness.The pixel decoder reconstructs images from the semantic features.Source: §2.1 · pp. 4–5
2.2 Training
- A spatial compression ratio of 32 and masked feature prediction train compact latents to predict features from a pretrained vision model, regularizing them with semantic and structural priors for autoregressive generation.The example target model is DINOv2 (Oquab et al., 2023).Source: §2.2 · p. 5
- Masked compact latents pass through the semantic decoder, which autoregressively expands them into high-dimensional features supervised by text-aligned visual-backbone representations for scalable vision-language tasks.CLIP (Radford et al., 2021) is given as an example backbone.Source: §2.2 · p. 6
- Training pixel reconstruction on masked and unmasked inputs teaches recovery of full images despite missing or noisy latents; adding reconstruction to masked feature prediction improved downstream understanding with multimodal LLMs.Because reconstruction uses semantic features, it also encourages the semantic decoder to preserve low-level details.Source: §2.2 · p. 6
3 Unified Image Understanding and Generation
Ming-UniVision uses MingTok to unify image understanding, generation, and editing through shared continuous visual representations and autoregressive prediction. Its latent-space workflow supports iterative interactions, and training progresses from pre-training to supervised fine-tuning.
3.1 Multimodal Model Architecture
- Ming-UniVision feeds semantic-decoder features to the language model for both perceived and generated images, while predicting text and visual latents autoregressively.Its per-token vision head uses rectified flow instead of diffusion denoising and a SwiGLU feed-forward network instead of a standard MLP; the paper reports better latent prediction accuracy and image quality under the same parameter budget.Source: §3.1 · pp. 6–7
3.2 Multi-round In-context Image Understanding, Generation and Manipulation
- Ming-UniVision reuses semantic features in a shared continuous latent space for iterative understanding, generation, and editing without costly pixel-space re-encoding.Figure 4 reports 66% fewer visual input tokens than hybrid AR-diffusion models (Shi et al., 2024; Deng et al., 2025) and 50% fewer than existing unified autoregressive models (Fan et al., 2025).Source: §3.2 · p. 8, Figure 4 · p. 9
3.3 Training
- Training proceeds from large-scale web-data pre-training to supervised fine-tuning on curated datasets and instruction-following corpora.The stages are intended to establish foundational vision-language capabilities, then improve instruction following and complex multimodal interaction.Source: §3.3 · p. 9
3.3.1 Pre-training
- Pre-training first warms up the MLP and vision head with frozen MingTok and LLM, then unlocks the language model for joint understanding and generation.The warm-up data is approximately 30% understanding and 70% generation. Joint pre-training uses 1024×1024 images for understanding and 512×512 for generation, selectively unlocking the semantic decoder while keeping the low-level encoder fixed.Source: §3.3.1 · p. 9
3.3.2 Supervised Finetuning
- Supervised fine-tuning progresses from standard image understanding and generation to a final stage focused on in-context image manipulation.Stage 1 freezes MingTok and uses approximately 30% understanding, 10% NLP, and 60% text-to-image data. Stage 2 shifts to 15% understanding, 5% NLP, 35% text-to-image generation, and 55% single- or multi-round editing tasks.Source: §3.3.2 · p. 10
4 Evaluations
Evaluations span image understanding, generation, editing, and reconstruction, with comparisons to existing approaches using a Ling-lite model with 2.8 billion activated parameters.
4.1 Multi-modal Understanding
- At comparable scale, Ming-UniVision performs comparably overall and is competitive on semantic reasoning and hallucination tasks, but has gaps on fine-grained recognition.The authors link the OCRBench and MMMU gaps to limited character-level detail from compressed latents and the causal semantic decoder, and leave detail-preserving representations for future work.Source: §4.1 · p. 11
4.2 Visual Generation
- Ming-UniVision achieves state-of-the-art overall GenEval performance and leads Position (0.92), Colors (0.93), and Color Attribute (0.70).The authors attribute the gains to the joint perception–generation representation and report faster training convergence; improving fine-grained detail generation remains future work.Source: §4.2 · pp. 11–12, Table 3 · p. 11
4.3 Visual Editing
- On GEdit-Bench-EN (Liu et al., 2025), Ming-UniVision has competitive single-round editing quality (G_PQ) and strong multi-round success rates (G_SC), but its overall score lags prior work.Results use a base model trained without mixed-resolution strategy; the authors identify absent large-scale multimodal sequence pretraining and continuous-token detail density as limitations under complex instructions.Source: §4.3 · p. 12, Table 5 · p. 12
4.4 Image Reconstruction
- At 32× compression, MingTok records rFID 0.54 and PSNR 30.77 db; joint semantic-decoder training yields LPIPS 0.12 and rFID 0.38.The authors say joint training improves retention of fine textures and global semantics.Source: §4.4 · p. 12, Table 4 · p. 12
5 Analysis
The analysis examines how shared continuous visual representations affect understanding and generation, multi-round editing, and visual reasoning. It also reports limitations in editing and interaction alongside early evidence of mutual enhancement between generation and understanding.
5.1 Unified Representation for Competing Tasks
- Using MingTok for both tokenizer roles achieves the best understanding and generation performance, while its semantic and detail-rich features ease alignment and narrow the generation gap in joint training.MingTok as the generation tokenizer improves GenEval over VAE regardless of understanding tokenizer; generation-only models perform better than joint models, but unified training reduces the gap. VAE’s detail-focused, less semantic features may make alignment harder.Source: Table 6 · p. 13, §5.1 · pp. 13–14, Figure 5 · p. 14
5.2 Multi-round Understanding, Generation, and Editing
- Seg-as-Edit raises average G_SC by +0.41 and G_O by +0.33, improving semantic consistency in 9/11 tasks while generally preserving perceptual quality.The largest G_SC gains are +0.82 for motion_change and +0.52 for background_change. Practical examples include coherent restoration and iterative super-resolution, plus subject matting with fewer halo artifacts; the unified feature space supports these stateful editing workflows.Source: §5.2 · pp. 14–15, Table 7 · p. 15
5.3 Visualized CoT for Visual Reasoning and Image Editing
- Visualized CoT outperforms single-step editing on GEdit-Bench, with a +0.5-point gain in semantic consistency.It first generates an image marking regions to edit, then uses those visual cues to guide editing directly, providing a spatial prior that reduces ambiguity; unlike GoT, it reasons visually rather than in natural language and avoids translating reasoning into an editing mask.Source: §5.3 · pp. 15–17, Table 8 · p. 16
5.4 Discussions
- Ming-UniVision remains limited in fine-grained editing and understanding, long editing sequences, and flexible interleaved interactions such as describing, generating, comparing, and revising.The discussion identifies missing large-scale interleaved pretraining and MingTok’s high compression ratio as factors limiting editing generalization and precision; the model also struggles with sequences longer than training examples.Source: §5.4 · pp. 17–18
- A shared representation may enable mutual enhancement: generation contributes texture synthesis and compositional reasoning to perception, while understanding supplies grounded priors for controllable generation.The paper reports early evidence that sharing representations reduces the performance gap between pure generation and unified training and alleviates task competition.Source: §5.4 · p. 18