Source-linked AI summary
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Zhiheng Liu, Weiming Ren, Xiaoke Huang, Shoufa Chen, Tianhong Li, Mengzhao Chen, Yatai Ji, Sen He, Jonas Schult, Belinda Zeng, Tao Xiang, Wenhu Chen, Ping Luo, Luke Zettlemoyer, Yuren Cong
TL;DR
Pretrained vision encoders and separate visual representations remain common in unified multimodal models, motivating direct raw-pixel modelling. Tuna-2 removes these encoders, uses pixel-space representations and flow matching, and achieves strong cross-task results, with particular advantages for fine-grained understanding.
Problem
Unified multimodal models commonly rely on pretrained vision encoders and separate or shared visual representations, leaving end-to-end modelling directly from raw pixels largely unexplored.
Method
Tuna-2 progressively removes vision encoders, using patch-based pixel embeddings, a unified vision-language backbone, a pixel-space flow matching head, and end-to-end training.
Results
Tuna-2 achieves state-of-the-art performance across multimodal benchmarks, remains competitive for image generation, and surpasses latent-space unified models on fine-grained visual understanding.
Takeaways & Limitations
Pretrained vision encoders are not required for strong unified multimodal modelling, and encoder-free pixel-space learning can produce stronger fine-grained visual representations.
Abstract
from arXiv · showhide
Unified multimodal models typically rely on pretrained vision encoders and use separate visual representations for understanding and generation, creating misalignment between the two tasks and preventing fully end-to-end optimization from raw pixels. We introduce Tuna-2, a native unified multimodal model that performs visual understanding and generation directly based on pixel embeddings. Tuna-2 drastically simplifies the model architecture by employing simple patch embedding layers to encode visual input, completely discarding the modular vision encoder designs such as the VAE or the representation encoder. Experiments show that Tuna-2 achieves state-of-the-art performance in multimodal benchmarks, demonstrating that unified pixel-space modelling can fully compete with latent-space approaches for high-quality image generation. Moreover, while the encoder-based variant converges faster in early pretraining, Tuna-2's encoder-free design achieves stronger multimodal understanding at scale, particularly on tasks requiring fine-grained visual perception. These results show that pretrained vision encoders are not necessary for multimodal modelling, and end-to-end pixel-space learning offers a scalable path toward stronger visual representations for both generation and perception.
1 Introduction
Tuna-2 addresses the challenge of building unified multimodal models without relying on pretrained vision encoders. Its encoder-free, pixel-space design achieves competitive generation and stronger multimodal understanding, especially for fine-grained perception.
- Unified multimodal models must encode images into representations that support both visual understanding and generation.
- Existing designs use either decoupled representations or shared pretrained vision encoders, leaving end-to-end raw-pixel modelling largely unexplored.
- Tuna-2 progressively removes vision encoders, ultimately using a single transformer decoder to process image and video tokens directly from raw pixels.
- A masking-based visual feature learning scheme stabilizes high-dimensional pixel-space training and encourages more robust representations.
- After sufficient multimodal pretraining, Tuna-2 is competitive with encoder-based Tuna-R for generation and consistently stronger for multimodal understanding, especially fine-grained perception.
- The work contributes pixel-space unified modelling, comprehensive ablations, and analyses of training dynamics for native unified multimodal models.
2 Method
The method progressively removes modular vision components, replacing them with patch embeddings and pixel-space generation. Masked feature learning and fully end-to-end training are used to strengthen pixel-space representations across multimodal tasks.
- Architecture: Tuna-2 is derived by progressively simplifying existing unified multimodal architectures toward an encoder-free formulation.
- Architecture: Tuna-R removes the VAE but retains a pretrained representation encoder as an intermediate controlled-comparison design.
- Architecture: Tuna-2 replaces pretrained vision encoders with patch embedding layers that convert images into visual tokens processed jointly with text tokens.
- Pixel-space generation: For pixel-space generation, the unified model predicts clean images from noisy images, transforms predictions into velocity terms, and regresses those terms.
- Masked feature learning: Masking makes generation harder by requiring clean-patch prediction from partially observed noisy inputs and regularizes understanding under partial visual observation.
- Training pipeline: The encoder-free design trains end-to-end without separate connector-layer training, using joint pretraining followed by supervised fine-tuning.
3 Experiments
Tuna-2 remains competitive across generation and editing while outperforming encoder-based and latent-space alternatives on multimodal understanding, especially fine-grained perception. Its encoder-free design learns more robust cross-modal alignment, though encoder priors can improve early training and generation.
- Multimodal understanding: After removing the VAE, Tuna-R and Tuna-2 outperform Tuna and achieve state-of-the-art results among 7B-scale native UMMs.
- Multimodal understanding: Tuna-2 outperforms Tuna-R on image understanding, suggesting unified monolithic training can exceed pretrained encoder inductive biases.
- Fine-grained perception: Both pixel-space variants outperform latent-space UMMs across fine-grained benchmarks focused on small objects and high-resolution visual reasoning.
- Image generation: Both Tuna-R and Tuna-2 achieve state-of-the-art image-generation results, while Tuna-R performs slightly better, consistent with benefits from encoder semantic priors.
- Image generation: Tuna-2 achieves competitive quality and significantly greater diversity than Tuna under LLM-judge evaluation, while outperforming Tuna on judged generation quality.
- Image editing: Tuna-2 remains competitive on ImgEdit and supports instruction-guided editing directly in pixel space, although encoder-based variants retain a small fidelity advantage.
- Training dynamics: Pretrained encoders accelerate early multimodal understanding, but Tuna-2 later catches up and surpasses Tuna-R as training proceeds.
- Training dynamics: Tuna-R consistently outperforms Tuna-2 during GenEval training, although this generation gap weakens as training-data scale increases.
4 Related Works
Related work has moved from modular multimodal systems and decoupled visual representations toward unified and encoder-free pixel-space architectures. Tuna-2 extends this direction by scaling pixel-space flow matching to unified multimodal pretraining.
- Unified multimodal models seek to combine visual understanding and generation within a single framework.
- Traditional multimodal systems combine pixel-space representation encoders with language-model decoders through connectors such as cross-attention or MLPs.
- Pixel-space unified models can integrate representation encoders or monolithic patch embeddings while jointly processing image and language tokens.
- Most visual-generation models use compressed latent spaces with KL- or VQ-regularized VAEs, while pixel-space generation is generally more challenging.
- Prior pixel-space flow-matching studies were mainly small-scale, whereas Tuna-2 demonstrates scaling to large-scale unified multimodal pretraining.
5 Conclusion
Tuna-2 performs multimodal understanding and visual generation directly in pixel space, without VAE encoders or latent diffusion. It surpasses prior latent-space unified models on fine-grained visual understanding while remaining competitive in image generation.
- Tuna-2 performs multimodal understanding and visual generation directly in pixel space without relying on VAE encoders or latent diffusion.
- Both representation encoder-based and encoder-free Tuna-2 variants achieve strong performance across multimodal understanding and generation benchmarks.
- Tuna-2 surpasses prior latent-space unified models such as Tuna and Show-o2 on fine-grained visual understanding benchmarks while remaining competitive on image generation.