PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, Chong Ruan

arXiv:2403.05525v2cs.AI

TL;DR

Prior vision-language models often used relatively low image resolutions, while multimodal pretraining could degrade language skills. DeepSeek-VL combines curated vision-language pretraining and instruction-tuning data, a hybrid high-resolution vision encoder, and joint vision-language pretraining with a warm-up data ratio. DeepSeek-VL achieves state-of-the-art or competitive performance across visual-language benchmarks at the same model size while maintaining robust language-centric performance.

  • Problem

    Prior vision-language models often used relatively low image resolutions, while multimodal pretraining could degrade language skills.

    Source: §1 · p. 3

  • Method

    DeepSeek-VL combines curated vision-language pretraining and instruction-tuning data, a hybrid high-resolution vision encoder, and joint vision-language pretraining with a warm-up data ratio.

    Source: §1 · p. 3, §2 · p. 6, §5 · pp. 22–23

  • Results

    DeepSeek-VL achieves state-of-the-art or competitive performance across visual-language benchmarks at the same model size while maintaining robust language-centric performance.

    Source: §1 · p. 5, §5 · p. 23

  • Takeaways & Limitations

    Publicly accessible 1.3B and 7B versions are intended to support applications across varying computational capabilities.

    Source: §1 · p. 5

    The report plans to scale DeepSeek-VL to larger sizes incorporating Mixture of Experts (MoE) technology.

    Source: §5 · p. 23

Abstract

from arXiv · show

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse, scalable, and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content, aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model's user experience in practical applications. Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024), while maintaining a relatively low computational overhead. This design choice ensures the model's ability to capture critical semantic and detailed information across various visual tasks. We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.

1. Introduction

DeepSeek-VL addresses the real-world performance and user-experience gap in open-source multimodal models through broad pretraining, real-use-case instruction tuning, high-resolution vision, and language-preserving training.

Motivation and challenges

  • Open-source models trail closed-source models in real-world performance and user experience; evidence identifies need for broad pretraining, academic-dataset tuning’s real-world shortfall, 336×336/448×448 resolution, and language degradation after prolonged multimodal training.Source: §1 · p. 3

Data and use-case taxonomy

  • Pretraining draws on sources such as Common Crawl, Web Code, e-books, educational materials, and arXiv to cover practical content including screenshots, PDFs, OCR, charts, and knowledge.For instruction tuning and evaluation, authentic GPT-4V and Gemini test cases are organized into a use-case taxonomy that guides prompt selection and dataset construction.Source: §1 · p. 5

High-resolution vision architecture

  • A hybrid vision encoder combines a text-aligned 384 × 384 encoder with a 1024 × 1024 detail encoder, condensing images to 576 tokens within a fixed budget.The 576-token representation is intended to support text-image interleaving and multi-turn inference while managing inference costs.Source: §1 · p. 5

Scaling and evaluation

  • For scaling experiments, DeepSeek-VL modifies evaluation to compare option perplexities and mixes a small amount of instruction-tuning data into pretraining so a 1B model can measure iteration effects.The paper notes that 1B models may not perform reasonably on benchmarks or faithfully reflect model performance under standard evaluation.Source: §1 · p. 5

Results and model release

  • DeepSeek-VL achieves state-of-the-art or competitive visual-language benchmark performance at the same model size, maintains robust language-centric performance, and is publicly available in 1.3B and 7B versions.The paper also reports superior user experiences in real-world applications.Source: §1 · p. 5

2. Data Construction

DeepSeek-VL’s data construction separates broad vision-language pretraining data from smaller supervised fine-tuning data, assigning them distinct roles across three training stages. The datasets span diverse visual and language tasks, while in-house instruction tuning is organized around real-world use cases.

2. Data Construction

  • Pretraining data warms up the vision-language adaptor in stage 1 and jointly trains the model in stage 2, while supervised fine-tuning data teaches downstream tasks in stage 3.The paper describes supervised fine-tuning data as relatively smaller than pretraining data.Source: §2 · p. 6

2.1. Vision-Language pretraining Data

  • Pretraining data includes interleaved image-text, image captions, table and chart data, and text-only data to support multimodal learning and language proficiency.The text-only pretraining corpus is the same one used in DeepSeek-LLM (DeepSeek-AI, 2024).Source: §2.1 · pp. 6, 8
  • The web-code collection contains 2 million image-code pairs, filtered to 1.1 million instances with a single image and at least 5 lines of code.The pairs were curated from diagrams and preceding code in approximately 1.46 million Jupyter notebooks; the data also uses Websight for UI inverse rendering and a MATCHA-like strategy for visual plots.Source: §2.1 · p. 6
  • Document OCR pretraining data pairs images and text from 1.4 million arXiv articles, 860K English and 180K Chinese e-books, and K–12 exam questions.Articles were rendered from source code and compiled PDFs, while e-books and educational materials were converted from HTML using multiple templates.Source: §2.1 · p. 7
  • Scene-text OCR data targets text embedded in environmental images and draws on multiple public datasets.The listed sources include ArT, MLT-17, LSVT, UberText, Coco-text, RCTW-17, ReCTS, TextOCR, OpenVINO, and HierText.Source: §2.1 · pp. 7–8

2.2. Supervised Fine-tuning Data

  • In-house multimodal fine-tuning data is organized using a taxonomy of real-world tasks, including recognition, conversion, analysis, reasoning, evaluation, and safety.The taxonomy is built by analyzing authentic GPT-4V and Gemini test cases, guides representative prompt selection, and also informs a balanced evaluation dataset.Source: §2.2 · p. 8
  • Supervised fine-tuning combines shared open-source multimodal datasets and table, chart, and UI-code data with curated in-house multimodal examples, including Chinese data.The paper also includes the text-only supervised fine-tuning data used in DeepSeek-LLM (DeepSeek-AI, 2024).Source: §2.2 · pp. 8, 10

3. Approach

DeepSeek-VL combines a hybrid vision encoder, a vision-language adaptor, and a language model, trained through staged visual-language alignment, joint language-multimodal pretraining, and instruction tuning. The approach pairs high-resolution visual detail with semantic features and uses language data during joint pretraining to balance multimodal learning with language retention.

3.1. Architecture

  • The hybrid encoder combines SAM-B features from 1024 x 1024 inputs with SigLIP-L features from 384 x 384 inputs to preserve detailed and semantic information.SAM-B produces a 64 x 64 x 256 feature map; the adaptor combines its transformed features with SigLIP-L features into 576 visual tokens of 2048 dimensions.Source: §3.1 · p. 10

3.2. Training Pipelines

  • Training proceeds through adaptor warmup, joint vision-language pretraining, and supervised fine-tuning, with next-token loss calculated only on the language part.Figure 3 specifies that stage 1 freezes the encoder and language model; stage 2 trains the adaptor and language model; stage 3 trains SigLIP-L, the adaptor, and language model.Source: Figure 3 · p. 12, §3.2 · p. 12

3.2.1. Stage 1: Training Vision-Language Adaptor

  • Adaptor warmup freezes the vision encoder and LLM while training the adaptor on 1.25 million image-text captions and 2.5 million Document OCR rendering pairs.Increasing the data scale at this stage provides no benefit and may worsen performance, motivating unfreezing the LLM in stage 2.Source: §3.2.1 · p. 12

3.2.2. Stage 2: Joint Vision-Language pretraining

  • Stage 2 freezes the vision encoder and optimizes the language model and adaptor; for 1.3B training, Multi-choice PPL monitoring and minimal SFT data stabilize metrics.The authors attribute unstable generative metrics to limited model capacity and absent SFT data, which hinder instruction following.Source: §3.2.2 · pp. 13–14
  • A roughly 7:3 language-to-multimodal training ratio is selected to retain language capabilities while improving multimodal pretraining.Direct multimodal-only training improved multimodal metrics but sharply reduced language metrics; adding language data alleviated this decline without significant multimodal loss.Source: §3.2.2 · p. 13, Figure 4 · p. 14

3.2.3. Stage 3: Supervised Fine-tuning

  • Supervised fine-tuning creates DeepSeek-VL-Chat by training the language model, adaptor, and hybrid encoder while keeping SAM-B frozen.It supervises answers and special tokens, masks system and user prompts, and blends multimodal data with pure-text dialogue data.Source: §3.2.3 · pp. 14–15

3.3. Hyperparameters and Infrastructures

  • Training uses HAI-LLM and pipeline parallelism, recomputes visual encoder passes across tensor-parallel ranks, and rebalances layers for throughput.DeepSeek-VL-7B took 5 days on 64 nodes with 8 Nvidia A100 GPUs each; DeepSeek-VL-1B took 7 days on 16 nodes.Source: §3.3 · p. 15

4. Evaluation

DeepSeek-VL performs competitively across public multimodal and language benchmarks and human evaluations, while ablations identify training strategies that improve efficiency and help preserve performance across modalities.

4.1. Public Multimodal Benchmarks Evaluation

  • DeepSeek-VL-7B surpasses most similarly sized open-source models, scoring 70.4 vs. 71.6 on SEEDbench and 36.1 vs. 47.8 on MathVista against GPT-4V; DeepSeek-VL-1.3B exceeds comparable models on MMB with nearly half the parameters (1.3B vs. 2.7B).DeepSeek-VL-1.3B also achieves MathVista results comparable to 7B open-source models.Source: §4.1 · p. 16, Table 5 · p. 17, Table 6 · p. 17

4.2. Public Language Benchmarks Evaluation

  • DeepSeek-VL performs comparably to or better than DeepSeek-7B on most language benchmarks, including higher MMLU and AGIEval results, but declines on GSM8K; HellaSwag scores are 68.4 vs. 68.5.The authors attribute the mathematics decline to competition between modalities under the 7B model-capacity limit, and report that DeepSeek-VL aims to minimize language-capability declines.Source: §4.2 · p. 18, Table 7 · p. 18

4.3. Human Evaluation

  • In manual evaluation, DeepSeek-VL-7B achieves better overall performance than the compared open-source models and comes close to GPT-4V in recognition, conversion, and commonsense reasoning.The evaluation dataset contains 100 questions across seven categories; all open-source models remain far behind GPT-4V in logical reasoning.Source: §4.3 · p. 18, Figure 6 · p. 19
  • GPT-4V-based assessment judged DeepSeek-VL superior in over 60% of instances against Fuyu-8B, CogVLM-17B, and InternLM-XComposer2-VL, and comparably strong against GPT-4V.GPT-4V compared answers over 99 test samples, choosing the better response or declaring a tie.Source: §4.3 · p. 19, Figure 7 · p. 19

4.4. Ablation Study

  • Scaling stage 1 data does not improve performance, while combining stages 1, 2, and 3 outperforms stages 1 and 3 alone and slightly exceeds stages 2 and 3.The authors attribute the lack of benefit from stage 1 data expansion to the projector’s limited capacity, and conclude that both multimodal pretraining and adaptor warmup contribute to performance.Source: §4.4 · pp. 19–20, Table 8 · p. 20, Table 9 · p. 20
  • Grouping language and multimodal data into separate batches improves training efficiency by 20% without compromising model performance.Grouping avoids making faster language samples wait for slower multimodal samples within mixed batches.Source: §4.4 · pp. 20–21
  • Modality warmup prevents an early decline in language capability and yields comparatively stronger final outcomes in both language and multimodal tasks.The strategy starts with a language-data ratio of 1 and gradually decreases it to the target ratio, such as 0.7.Source: §4.4 · p. 21, Figure 9 · p. 21
  • The model uses a hybrid SigLIP-and-SAM vision encoder and a mixed adaptor strategy, which delivers stable, improved performance in adaptor comparisons.Vision-only self-supervised encoders improve training loss; reducing feature sequence length by stacking along image dimensions generally underperforms merging along the embedding dimension, while the mixed adaptor combines separate encoder-specific MLPs with shared fusion.Source: §4.4 · pp. 21–22, Figure 10 · p. 22, Table 10 · p. 22

5. Conclusion, Limitation, and Future Work

DeepSeek-VL combines joint vision-language pretraining and a hybrid vision encoder to process high-resolution images while preserving linguistic capabilities. The report also highlights its benchmark performance, public release, and plans for larger Mixture of Experts models.

  • Joint vision-language pretraining, a strategic warm-up data ratio, and a hybrid vision encoder aim to preserve LLM linguistic capabilities while efficiently processing semantically rich, high-resolution images.The encoder handles 1024 x 1024 images within a constrained token budget.Source: §5 · pp. 22–23
  • DeepSeek-VL performs exceptionally across visually-centric benchmarks while sustaining strong language-centric proficiency.The report says it meets or exceeds the standards of generalist models in its class.Source: §5 · p. 23
  • DeepSeek-VL is publicly available as a foundation for further research and innovation in multimodal data.Source: §5 · p. 23
  • Future work plans to scale DeepSeek-VL to larger sizes and incorporate Mixture of Experts technology.The report anticipates this expansion will further enhance efficiency and effectiveness.Source: §5 · p. 23

A. Appendix

The appendix presents visual examples of DeepSeek-VL understanding real-world diagrams, code, charts, knowledge, and tables.

  • Figure 11 shows DeepSeek-VL understanding children’s programming diagrams and providing detailed, organized explanations.Source: Figure 11 · p. 30
  • Figure 12 illustrates DeepSeek-VL’s strong understanding of real-world code and charts.Source: Figure 12 · p. 31
  • Figure 13 illustrates DeepSeek-VL’s extensive knowledge of the real world.Source: Figure 13 · p. 32
  • Figure 14 shows DeepSeek-VL accurately reading the contents of real-world tables.Source: Figure 14 · p. 33

Related papers

Loading 2403.05525v2…