Source-linked AI summary
DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, Chong Ruan
TL;DR
DeepSeek-VL2 addresses limitations in multimodal understanding and efficient inference by combining dynamic high-resolution visual encoding, an MoE language architecture with MLA, and improved training data. It achieves similar or better performance with fewer activated parameters across multimodal and grounding benchmarks, while remaining constrained by limited multi-image context, robustness gaps, reasoning limitations, and primarily English-Chinese OCR coverage.
Problem
DeepSeek-VL’s fixed-resolution vision encoder was limited on high-resolution and extreme-aspect-ratio inputs, motivating improved multimodal processing.
Method
DeepSeek-VL2 combines dynamic tiling, DeepSeekMoE with Multi-head Latent Attention, and improved vision-language data in an open-source model series.
Results
DeepSeek-VL2 achieves similar or better performance with fewer activated parameters on multimodal benchmarks and outperforms similarly scaled VLMs on grounding benchmarks.
Takeaways & Limitations
The model series provides efficient multimodal models with capabilities spanning visual question answering, OCR, document/table/chart understanding, and visual grounding.
Takeaways & Limitations
DeepSeek-VL2 currently supports only a few images per chat and remains challenged by blurry images, unseen objects, reasoning, and non-English OCR coverage.
Abstract
from arXiv · showhide
We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, through two key major upgrades. For the vision component, we incorporate a dynamic tiling vision encoding strategy designed for processing high-resolution images with different aspect ratios. For the language component, we leverage DeepSeekMoE models with the Multi-head Latent Attention mechanism, which compresses Key-Value cache into latent vectors, to enable efficient inference and high throughput. Trained on an improved vision-language dataset, DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition, document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively. DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models. Codes and pre-trained models are publicly accessible at https://github.com/deepseek-ai/DeepSeek-VL2.
1. Introduction
DeepSeek-VL2 is an open-source MoE vision-language model series designed to improve performance and efficiency through upgraded visual encoding, language modeling, and training data.
- Dynamic tiling processes high-resolution images with varying aspect ratios and supports fine-grained visual tasks.The strategy dynamically segments images into local tiles processed by a shared vision transformer while maintaining a manageable number of visual tokens.
- Multi-head Latent Attention compresses the Key-Value cache into latent vectors, enabling faster inference and higher throughput.Sparse computation from DeepSeekMoE further improves language-model efficiency.
- Improved vision-language data broadens performance across VQA, OCR, document/table/chart understanding, visual reasoning, and chatbot tasks.The dataset also supports visual grounding and GUI perception.
- DeepSeek-VL2 combines a vision encoder, VL adaptor, and MoE-based LLM in a LLaVA-style architecture.
2. Model Architecture
DeepSeek-VL2 retains a LLaVA-style three-module design while replacing fixed-resolution visual processing with dynamic tiling and using an efficient DeepSeekMoE language model.
- The architecture consists of a vision encoder, vision-language adaptor, and Mixture-of-Experts language model.It builds on a decoder-only LLaVA-style architecture.
- Fixed 1024 × 1024 encoding limited DeepSeek-VL on larger or extreme-aspect-ratio images used in dense OCR, InfographicVQA, and visual grounding.
- Each tile initially yields 27 × 27 = 729 visual embeddings of 1152 dimensions from the SigLIP-SO400M-384 encoder.
- DeepSeekMoE combines Multi-head Latent Attention, KV-cache compression, sparse computation, and a global expert bias for load balancing.
3. Data Construction
DeepSeek-VL2’s data pipeline combines staged alignment, mixed vision-language and text data, quality-controlled annotations, and specialized datasets for grounding, OCR, documents, tables, and reasoning.
- Training data is organized into VL alignment, VL pretraining, and supervised fine-tuning stages.The alignment stage uses approximately 1.2M ShareGPT4V caption and conversation samples to train the connector.
- Pretraining maintains an approximately 70% vision-language to 30% text-only data ratio.
- The collection mixes open datasets, Chinese content, and in-house real-world knowledge to expand coverage and multilingual capability.
- Caption quality is improved through filtering and a DeepSeek Chat scoring pipeline that removes low-quality annotations.The authors report repetition issues in large-scale annotation pipelines.
- Specialized datasets target OCR, general VQA, table/chart/document understanding, web and plot code, visual prompts, grounding, and reasoning.Grounding data includes bounding-box responses, negative samples, and in-context cross-image localization.
- The in-house OCR dataset mainly covers English and Chinese character recognition.The authors plan to expand coverage to other languages.
- Public VQA responses are regenerated using questions, images, and OCR information to address short answers, poor OCR, and hallucinated content.
4. Training Methodology
DeepSeek-VL2 is trained through staged alignment, large-scale multimodal pretraining, and supervised fine-tuning, with infrastructure and hyperparameters tailored to its vision-language architecture.
- Training proceeds through alignment, vision-language pretraining, and supervised fine-tuning.The stages progressively connect visual and language features, optimize the full model, and improve instruction following.
- Alignment adapts the vision encoder for dynamic high-resolution images while connecting visual features with pretrained DeepSeekMoE language models.
- Vision-language pretraining unfreezes the vision encoder, adaptor, and language model and uses approximately 800B image-text tokens.The stage aims to improve multimodal understanding while maintaining most language capabilities.
- Supervised fine-tuning optimizes all parameters while masking system and user prompts and combining multimodal data with pure-text dialogue.
- Training uses HAI-LLM and fine-grained vision-encoder layer division to balance pipeline workloads across GPUs.
- Training completed in 7/10/14 days using 16/33/42 nodes, each equipped with 8 NVIDIA A100 GPUs.
5. Evaluation
DeepSeek-VL2 demonstrates broad multimodal capabilities across benchmarked tasks and qualitative settings, including question answering, multi-image reasoning, storytelling, and visual grounding. It also achieves competitive performance with fewer activated parameters and supports in-context and grounded interaction.
- Benchmark comparisons: DeepSeek-VL2 achieves similar or better multimodal-understanding performance with fewer activated parameters, while outperforming other VLMs at similar scales on grounding benchmarks.The evaluation covers document, chart, OCR, general reasoning, and other multimodal tasks.
- Qualitative capabilities: The model demonstrates improved general visual question answering, including dense image description, landmark and text recognition, chart understanding, and meme-humor explanation.The reported capabilities span English and Chinese rich text as well as culturally informed meme interpretation.
- Qualitative capabilities: DeepSeek-VL2 analyzes associations and differences across multiple images and performs simple reasoning by integrating their contents.One example involves planning how to prepare a dish from images of ingredients.
- Qualitative capabilities: The model can generate creative stories from several images, drawing on visual abilities such as landmark recognition and OCR.The authors note that broader storytelling genres and plot types remain challenging alongside safety requirements.
- Visual grounding: DeepSeek-VL2 generalizes visual grounding beyond natural scenes to memes, animes, celebrities, and abstract concepts.It can locate objects using category names, descriptions, or abstract concepts.
- Visual grounding: In-context visual grounding lets the model use an object reference in one image to locate a corresponding object in a second image.The ability works with visual prompts and textual descriptions, and grounded conversation can identify relevant objects with accurate locations.
6. Conclusion
DeepSeek-VL2 combines efficient MoE scaling with dynamic tiling for high-resolution images and is designed for deployment on single GPUs. The authors identify limited multi-image context, robustness, and reasoning as future improvement areas.
- Model series: 1.0B, 2.8B, and 4.5B activated parameters correspond to the 3B, 16B, and 27B total-parameter model variants.These configurations facilitate efficient computational consumption during training and inference.
- Deployment: The model variants can be deployed on single GPUs with 10 GB, 40GB, and 80GB memory, respectively.
- Core design: DeepSeek-VL2 uses dynamic tiling vision encoding to process high-resolution images with varied aspect ratios.
- Limitations and future work: The current context window supports only a few images per chat session, limiting richer multi-image interactions.
- Limitations and future work: Future work targets robustness to blurry images and unseen objects, along with stronger reasoning capabilities.