Source-linked AI summary
DeepSeek-VL: Towards Real-World Vision-Language Understanding
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, Chong Ruan
TL;DR
DeepSeek-VL addresses the difficulty of building open-source multimodal models that perform well in real-world scenarios without sacrificing language ability. It combines diverse real-world data, taxonomy-based instruction tuning, a hybrid high-resolution vision encoder, and modality-balanced pretraining, achieving competitive visual-language and robust language performance across 1.3B and 7B models.
Problem
Open-source multimodal models remain behind proprietary systems in real-world performance and user experience, while multimodal training can degrade language capabilities.
Method
DeepSeek-VL uses diverse real-world pretraining data, taxonomy-based instruction tuning, a hybrid vision encoder, and modality-balanced joint vision-language pretraining.
Results
DeepSeek-VL delivers state-of-the-art or competitive visual-language benchmark performance and robust language-centric performance at comparable model sizes.
Takeaways & Limitations
The publicly released 1.3B and 7B models provide an open foundation for real-world multimodal applications and further innovation.
Takeaways & Limitations
The planned expansion to larger models with Mixture of Experts is future work rather than part of the reported released model family.
Abstract
from arXiv · showhide
We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse, scalable, and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content, aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model's user experience in practical applications. Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024), while maintaining a relatively low computational overhead. This design choice ensures the model's ability to capture critical semantic and detailed information across various visual tasks. We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.
1. Introduction
DeepSeek-VL targets the persistent gap between open-source and proprietary multimodal models in real-world performance and user experience. It combines real-world data curation, high-resolution visual processing, and modality-balanced training to support broad multimodal and language capabilities.
- Motivation: Open-source multimodal models still lag proprietary systems especially in real-world performance and user experience.The paper identifies challenges in comprehensive pretraining, authentic instruction tuning, high-resolution processing, and language-skill preservation.
- Approach: DeepSeek-VL is built on the DeepSeek language model series with extensive pretraining, use-case-based data curation, high-resolution architecture, and modality-balanced training.The approach also supports scaling from 1B to 7B parameters.
- Approach: The hybrid vision encoder combines coarse semantic extraction at 384 × 384 with detailed visual processing at 1024 × 1024.This design addresses demanding scenarios such as OCR and tiny-object recognition while managing inference costs.
- Approach: Maintaining at least 70% language data and using modality warm-up preserves language knowledge while developing multimodal capability.The training strategy gradually incorporates more vision-language data to balance both modalities.
- Results: DeepSeek-VL achieves state-of-the-art or competitive performance across visual-language benchmarks while maintaining robust language-centric performance.The 1.3B and 7B models also provide superior real-world chatbot user experiences and are publicly accessible.
2. Data Construction
DeepSeek-VL constructs a large, diverse dataset organized into vision-language pretraining and supervised fine-tuning data. The collection covers multimodal understanding, practical visual tasks, language preservation, and real-world instruction scenarios.
- Data organization: The dataset is divided into vision-language pretraining data and smaller supervised fine-tuning data across three training stages.Pretraining data warm up the vision-language adaptor and jointly pretrain the model, while supervised data support vision-language fine-tuning.
- Vision-Language pretraining Data: Pretraining sources span publicly accessible and proprietary data, including interleaved image-text, captions, tables, charts, web code, OCR, and text-only corpora.These sources are intended to improve cross-modal understanding and preserve language-centric proficiency.
- Vision-Language pretraining Data: The document OCR dataset includes paired images and texts from 1.4 million arXiv articles, plus 860K English and 180K Chinese e-books.The collection also includes educational exam questions and rendered document material.
- Supervised Fine-tuning Data: The supervised fine-tuning data combine open-source multimodal instruction datasets, table and chart data, and text-only DeepSeek-LLM data.This mixture covers multimodality and language tasks in the joint supervised fine-tuning stage.
- Supervised Fine-tuning Data: The in-house fine-tuning data are organized by a taxonomy whose categories comprehensively represent high-quality multimodal use cases.The taxonomy is also used to align evaluation tasks with the intended use-case coverage.
3. Approach
DeepSeek-VL combines a hybrid high-resolution vision encoder with staged vision-language training to support detailed visual understanding while preserving language capabilities. Its training balances multimodal learning with language data to mitigate language forgetting.
- Architecture: The system combines a hybrid vision encoder, a vision adaptor, and a language model.The vision adaptor bridges visual features and the language model.
- Architecture: The hybrid encoder combines high-resolution SAM-B processing at 1024 x 1024 with low-resolution SigLIP-L processing to preserve semantic and detailed information.SAM-B produces a 64 x 64 x 256 feature map, while SigLIP-L processes 384 x 384 inputs.
- Architecture: The VL adaptor transforms and concatenates high- and low-resolution features into visual tokens for the language model.High-resolution features are interpolated, convolved, reshaped, and combined with SigLIP-L features.
- Training pipeline: Training proceeds through adaptor warmup, joint vision-language pretraining, and supervised fine-tuning.The stages progressively involve the adaptor, language model, and selected vision encoders.
- Joint vision-language pretraining: Direct multimodal pretraining improves multimodal metrics but causes a severe decline in language metrics.The 100% multimodal and 0% language mixture exposes a trade-off between multimodal improvement and linguistic proficiency.
- Joint vision-language pretraining: Including language data and selecting roughly a 7:3 language-to-multimodal ratio mitigates language loss while retaining multimodal abilities.The authors use this balance for the final model.
4. Evaluation
DeepSeek-VL is evaluated across multimodal, language, manual, and ablation settings, where it generally performs strongly against similarly sized open-source models while preserving language capability and improving training efficiency.
- Public Multimodal Benchmarks Evaluation: DeepSeek-VL-7B surpasses most open-source models of similar size across a wide range of multimodal benchmarks.The evaluation covers comprehensive understanding, chart and table understanding, hallucination, and scientific problem datasets.
- Public Multimodal Benchmarks Evaluation: 70.4 vs. 71.6 on SEEDbench places DeepSeek-VL close to GPT-4V, while 36.1 vs. 47.8 on MathVista shows a larger gap on mathematical reasoning.The reported comparisons are DeepSeek-VL versus GPT-4V.
- Tiny Multimodal Models Evaluation: 1.3B vs. 2.7B parameters, DeepSeek-VL-1.3B outperforms comparable models on MMB and achieves results comparable to 7B open-source models on MathVista.These findings support the model’s performance at a smaller scale.
- Public Language Benchmarks Evaluation: 68.4 vs. 68.5 on HellaSwag shows comparable language performance to DeepSeek-7B, while DeepSeek-VL outperforms it on MMLU and AGIEval but declines on GSM8K.The language evaluation uses perplexity-based, generation-based, and language-modeling-based procedures across different benchmark types.
- Manual Evaluation: DeepSeek-VL-7B achieves better overall manual-evaluation results than other open-source models and is preferred in the majority of GPT-4V comparisons.The manual evaluation uses a 100-question dataset and a separate 99-sample pairwise comparison.
- Ablation Study: 20% training-efficiency improvement is achieved by grouping language and multimodal data into separate batches without compromising performance.The ablations also report benefits from modality warm-up, complete three-stage training, and mixed adaptor designs.
5. Conclusion, Limitation, and Future Work
The conclusion presents DeepSeek-VL as an efficient, high-resolution multimodal model with strong visual and language performance, publicly released to support further research, while noting planned scaling to larger models.
- Conclusion: 1024 x 1024 images are processed within a constrained token budget while preserving nuanced details and semantic integrity.This capability comes from the hybrid vision encoder.
- Conclusion: DeepSeek-VL achieves strong performance across visually focused benchmarks while sustaining proficiency on language-centric evaluations.The conclusion characterizes this performance relative to generalist models in its class.
- Conclusion: Public release of DeepSeek-VL is intended to provide a foundation for further multimodal research and innovation.The stated goal is to facilitate collective advancement in handling multimodal data.
- Future Work: Future work plans to scale DeepSeek-VL to larger sizes and incorporate Mixture of Experts technology.The planned expansion is described as targeting efficiency and effectiveness.
A. Appendix
The appendix illustrates DeepSeek-VL’s real-world understanding through programming diagrams, code and charts, graphs, coins, and tables.
- Visualization Results: DeepSeek-VL explains children’s real-world programming diagrams in a detailed and organized manner.The appendix presents this capability as a visualization result.
- Graph Interpretation: The appendix identifies the blue curve as a Multi Step Learning Rate Scheduler with an 80% + 10% + 10% schedule.The graph compares this scheduler with a Cosine Learning Rate Scheduler over processed tokens and training loss.
- Visualization Results: DeepSeek-VL demonstrates understanding of real-world code and charts.This capability is reported in a visualization result.
- Object Understanding: The appendix identifies a pictured coin as a 10 Baht coin from Thailand.The answer is given directly alongside the coin-identification question.
- Visualization Results: DeepSeek-VL is presented as possessing extensive real-world knowledge and accurately reading real-world table contents.These claims appear as separate visualization results.