Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

• Open-source LVLMs lag behind proprietary models and commonly lack fine-grained abilities such as object grounding and text reading.Source: §1 · p. 2
• Qwen-VL adds visual capacity to Qwen-7B through a language-aligned visual encoder, position-aware adapter, concise interface, and three-stage training pipeline.Source: §1 · p. 3
• Qwen-VL performs vision-language tasks including image captioning, question answering, text-oriented question answering, and visual grounding, while Qwen-VL-Chat adds instruction-tuned interaction.Source: §1 · p. 3
• Qwen-VLs achieve top-tier accuracy across captioning, question-answering, grounding, and dialogue benchmarks against similarly scaled counterparts.Source: §1 · p. 3
• Multilingual image-text training supports English, Chinese, and multilingual instructions, while interleaved inputs enable comparison and analysis across multiple images.Source: §1 · p. 3
• Higher-resolution inputs and fine-grained training data yield competitive grounding, text-reading, text-oriented question answering, and fine-grained dialogue performance.Source: §1 · p. 3

• Qwen-VL’s architecture consists of a large language model, visual encoder, and position-aware vision-language adapter.Source: §2.1 · p. 3
• The language component is initialized from pretrained Qwen-7B weights, while the visual encoder uses OpenCLIP’s ViT-bigG and processes images into patch features.Source: §2.1 · p. 3
• The adapter compresses image features with cross-attention using trainable queries and incorporates 2D absolute positional encodings.Source: §2.1 · pp. 3–4
• Compressed image features form a fixed-length sequence of 256 vectors that is fed into the language model.Source: §2.1 · p. 4
• Special tokens <img> and </img> mark the beginning and end of image-feature sequences, distinguishing visual from text inputs.Source: §2.2 · p. 4
• Bounding-box training uses region descriptions, questions, and detections to teach accurate generation of localized visual descriptions.Source: §2.2 · p. 4

• The training pipeline has two pre-training stages followed by instruction fine-tuning.Source: §3 · p. 5

Pre-training

• Stage one cleans 5 billion weakly labeled image-text pairs to 1.4 billion examples, comprising 77.3% English and 22.7% Chinese text data.Source: §3.1 · p. 5
• Stage one freezes the language model and optimizes the vision encoder and adapter on 224 × 224 images.Source: §3.1 · p. 5

Multi-task Pre-training

• Stage two introduces high-quality fine-grained annotations, larger-resolution inputs, interleaved image-text data, and seven simultaneous tasks.Source: §3.2 · pp. 5–6, Table 3 · p. 6
• The multi-task stage raises visual-encoder resolution from 224 × 224 to 448 × 448 and trains the whole model to reduce down-sampling information loss.Source: §3.2 · p. 6

Instruction Fine-tuning

• Instruction fine-tuning enhances instruction following and dialogue capabilities, producing the interactive Qwen-VL-Chat model.Source: §3.3 · p. 6

Image Caption and General VQA

• Qwen-VL and Qwen-VL-Chat achieve better results than previous generalist models on image captioning and general VQA.The evaluation covers image captioning and five general VQA benchmarks.Source: §4.1 · p. 7
• 79.5, 58.6 and 59.3 accuracy on VQAv2, OKVQA and GQA respectively surpass recent LVLMs by a large margin.Qwen-VL also shows strong zero-shot performance on ScienceQA and VizWiz.Source: §4.1 · p. 7

Text-oriented Visual Understanding

• Qwen-VL models outperform previous generalist models and recent LVLMs on most text-oriented VQA benchmarks, often by a large margin.The benchmarks include TextVQA, DocVQA, ChartQA, AI2Diagram and OCR-VQA.Source: §4.2 · p. 7

Referring Expression Comprehension

• Qwen-VL models obtain top-tier results on all evaluated referring expression comprehension benchmarks.These benchmarks assess localization of target objects from descriptions.Source: §4.3 · pp. 7–8

Few-shot Learning and Instruction Following

• Qwen-VL performs better in few-shot learning than similarly sized models on OKVQA, VizWiz, TextVQA and Flickr30K, while Qwen-VL-Chat leads three real-world instruction benchmarks.On SEED-Bench, visual capabilities transfer to video tasks by sampling four frames; TouchStone shows a particularly clear advantage in Chinese capabilities.Source: §4.4 · p. 8, §4.5 · p. 9

Vision-Language Research

• Prior work includes vision-language models and multi-task generalist models for image-text retrieval, generation, and related multimodal tasks.The section situates Qwen-VL among a broad line of vision-language research.Source: §5 · p. 9

Limitations of Previous Models

• Previous vision-language models are limited by poor instruction following, weak generalization to unseen tasks, and a lack of in-context abilities.These limitations are presented as remaining challenges despite significant progress.Source: §5 · pp. 9–10

LLM-based LVLMs

• Researchers increasingly build large vision-language models on top of LLMs to address multimodal understanding beyond text-only modeling.The related work describes this development as following the rapid progress of LLMs.Source: §5 · pp. 9–10

Conclusion

• Qwen-VL is released as a multilingual model series supporting multilingual conversations, multi-image interleaved conversations, Chinese grounding, and fine-grained recognition.The release is intended to facilitate multimodal research.Source: §6 · p. 10

Future Work

• Future work includes integrating Qwen-VL with additional modalities such as speech and video.This is identified as one dimension for continued capability enhancement.Source: §6 · p. 10
• Future scaling targets model size, training data and input resolution to handle more complex relationships within multimodal data.The authors frame these as planned augmentations to Qwen-VL.Source: §6 · p. 10
• Future work also targets multimodal generation of high-fidelity images and fluent speech.This extends the model’s planned capabilities beyond understanding.Source: §6 · p. 10

Corpus construction

• The pre-training corpus combines web-crawled image-text pairs from LAION-en, LAION-zh, LAION-COCO, DataComp, and Coyo, followed by extensive cleaning.Cleaning removes problematic aspect ratios, small images, harsh CLIP matches, unsupported characters, emojis, unsuitable text lengths, HTML tags, and irregular patterns.Source: §A.1 · p. 16

Task data

• VQA data uses maximum-confidence answers for VQAv2, while other VQA datasets receive no special answer processing.Source: §A.2 · p. 16
• Grounding data concatenates noun or phrase text with bounding-box coordinates, while GRIT captions are greedily cleaned to remove recursive boxes.Source: §A.3 · p. 16

OCR data

• Synthetic OCR data renders text over COCO scenery using 41 English and 11 Chinese fonts, retaining generated text locations as quadrilateral labels.Source: §A.4 · p. 16

Document data

• PDF and HTML collections are rendered into images with text annotations and bounding boxes, using PyMuPDF for PDFs and Puppeteer for HTML pages.Source: §A.4 · pp. 16, 18

Data formats

• The multi-task format mixes captioning, VQA, OCR, grounding, and related image-text tasks, while ChatML marks image turns and dialogue interactions with special tokens.Training supervises answers and special tokens but not role names or question prompts.Source: §B.1 · pp. 18–19

Stage one

• The first pre-training stage uses AdamW, cosine learning rates, 500-step warm-up, weight decay, gradient clipping, and layer-wise learning-rate decay for the ViT.Source: C Hyperparameters · p. 20

Stage two

• The second multi-task stage increases visual input resolution from 224 × 224 to 448×448 and unlocks the language model for whole-model training.Source: C Hyperparameters · p. 20

Benchmark summary

• Table 9 provides a summary of the evaluation benchmarks and corresponding metrics used for Qwen-VL.Source: D Summary of the evaluation benchmarks · p. 20, Table 9 · p. 21

E.1 Convergence of the Pre-training Stage

• Training loss decreases steadily as the number of training pictures increases during the first pre-training stage.The stage uses BFloat16 mixed precision, batch size 30720, learning rate 2e−4, and one epoch over all images.Source: §E.1 · p. 21, Figure 6 · p. 21
• Zero-shot VQA scores increase amid fluctuations even though no VQA data is added during the first pre-training stage.Source: §E.1 · p. 21

E.2 Number of Learnable Queries in the Vision-Language Adapter

• The adapter uses 256 learnable queries because too few queries lose information at 448×448 resolution, while too many or too few slow convergence.Source: §E.2 · pp. 21–22, Figure 7 · p. 22

E.3 Window Attention vs Global Attention for Vision Transformer

• Vanilla Attention is selected over Window Attention because Window Attention produces significantly higher loss while training speeds remain similar.Source: §E.3 · p. 22, Table 10 · p. 22, Figure 8 · p. 23
• At 896 × 896 resolution, Window Attention is not used because training takes almost 2.5 times longer than with 448 × 448 input.Source: §E.3 · p. 23

• Qwen-VL is initialized from an intermediate Qwen-7B checkpoint, supporting comparability with many text-only LLMs on pure-text tasks.Source: §E.4 · p. 23
• Qwen-VL does not compromise pure-text ability after multimodal training, according to the reported results.Source: Table 11 · p. 23, §E.4 · p. 24
• Qwen-VL incorporates pure-text data during multi-task training and SFT to prevent catastrophic forgetting of text comprehension.This supplements the visual and language-related training data.Source: §E.4 · pp. 23–24
• Qwen-VL even improves on pure-text capability after multi-task training.Source: §E.4 · p. 24