Source-linked AI summary
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou
TL;DR
Qwen-VL addresses the limited fine-grained visual understanding of open-source LVLMs. It adds visual perception, localized grounding, and text reading to Qwen-7B through a visual receptor, multilingual corpus, and three-stage training pipeline. Qwen-VL and Qwen-VL-Chat achieve strong results across visual and dialogue benchmarks while supporting multilingual and multi-image interaction.
Problem
Open-source LVLMs lag behind proprietary models and often lack fine-grained capabilities such as object grounding and text reading.
Method
The Qwen-VL series equips Qwen-7B with a visual receptor, concise multimodal interface, multilingual cleaned corpus, and three-stage training pipeline using fine-grained multimodal data.
Results
Qwen-VLs achieve top-tier performance across visual-centric benchmarks and dialogue benchmarks, with fine-grained grounding, text-reading, and visual-understanding capabilities.
Takeaways & Limitations
The released models support multilingual conversations, interleaved multi-image conversations, Chinese grounding, and fine-grained recognition.
Takeaways & Limitations
Future work includes integrating speech and video and scaling model size, training data, and resolution for more complex multimodal relationships.
Abstract
from arXiv · showhide
In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal cleaned corpus. Beyond the conventional image description and question-answering, we implement the grounding and text-reading ability of Qwen-VLs by aligning image-caption-box tuples. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales on a broad range of visual-centric benchmarks (e.g., image captioning, question answering, visual grounding) and different settings (e.g., zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to existing vision-language chatbots. Code, demo and models are available at https://github.com/QwenLM/Qwen-VL.
1 Introduction
Qwen-VL addresses open-source LVLMs’ limited fine-grained visual understanding by adding visual capacity to Qwen-7B through a visual receptor, concise interface, and three-stage training. The resulting Qwen-VL and Qwen-VL-Chat support multilingual and multi-image interaction while achieving strong performance across visual and dialogue benchmarks.
- Open-source LVLMs lag behind proprietary models and commonly lack fine-grained abilities such as object grounding and text reading.
- Qwen-VL adds visual capacity to Qwen-7B through a language-aligned visual encoder, position-aware adapter, concise interface, and three-stage training pipeline.
- Qwen-VL performs vision-language tasks including image captioning, question answering, text-oriented question answering, and visual grounding, while Qwen-VL-Chat adds instruction-tuned interaction.
- Qwen-VLs achieve top-tier accuracy across captioning, question-answering, grounding, and dialogue benchmarks against similarly scaled counterparts.
- Multilingual image-text training supports English, Chinese, and multilingual instructions, while interleaved inputs enable comparison and analysis across multiple images.
- Higher-resolution inputs and fine-grained training data yield competitive grounding, text-reading, text-oriented question answering, and fine-grained dialogue performance.
2 Methodology
Qwen-VL combines a Qwen-7B language model with a ViT-based visual encoder and a position-aware adapter. Images become fixed-length visual-token sequences, while specialized interfaces support image input and bounding-box grounding.
- Qwen-VL’s architecture consists of a large language model, visual encoder, and position-aware vision-language adapter.
- The language component is initialized from pretrained Qwen-7B weights, while the visual encoder uses OpenCLIP’s ViT-bigG and processes images into patch features.
- The adapter compresses image features with cross-attention using trainable queries and incorporates 2D absolute positional encodings.
- Compressed image features form a fixed-length sequence of 256 vectors that is fed into the language model.
- Special tokens <img> and </img> mark the beginning and end of image-feature sequences, distinguishing visual from text inputs.
- Bounding-box training uses region descriptions, questions, and detections to teach accurate generation of localized visual descriptions.
3 Training
Qwen-VL uses two pre-training stages followed by instruction fine-tuning. The pipeline progresses from cleaned web-scale image-text data to high-resolution, fine-grained, multi-task training and interactive dialogue tuning.
- The training pipeline has two pre-training stages followed by instruction fine-tuning.
- Pre-training: Stage one cleans 5 billion weakly labeled image-text pairs to 1.4 billion examples, comprising 77.3% English and 22.7% Chinese text data.
- Pre-training: Stage one freezes the language model and optimizes the vision encoder and adapter on 224 × 224 images.
- Multi-task Pre-training: Stage two introduces high-quality fine-grained annotations, larger-resolution inputs, interleaved image-text data, and seven simultaneous tasks.
- Multi-task Pre-training: The multi-task stage raises visual-encoder resolution from 224 × 224 to 448 × 448 and trains the whole model to reduce down-sampling information loss.
- Instruction Fine-tuning: Instruction fine-tuning enhances instruction following and dialogue capabilities, producing the interactive Qwen-VL-Chat model.
4 Evaluation
Qwen-VL and Qwen-VL-Chat are evaluated across conventional, fine-grained, few-shot, and real-world instruction-following tasks. They achieve strong results across captioning, VQA, text-oriented understanding, grounding, and dialogue benchmarks.
- Image Caption and General VQA: Qwen-VL and Qwen-VL-Chat achieve better results than previous generalist models on image captioning and general VQA.The evaluation covers image captioning and five general VQA benchmarks.
- Image Caption and General VQA: 79.5, 58.6 and 59.3 accuracy on VQAv2, OKVQA and GQA respectively surpass recent LVLMs by a large margin.Qwen-VL also shows strong zero-shot performance on ScienceQA and VizWiz.
- Text-oriented Visual Understanding: Qwen-VL models outperform previous generalist models and recent LVLMs on most text-oriented VQA benchmarks, often by a large margin.The benchmarks include TextVQA, DocVQA, ChartQA, AI2Diagram and OCR-VQA.
- Referring Expression Comprehension: Qwen-VL models obtain top-tier results on all evaluated referring expression comprehension benchmarks.These benchmarks assess localization of target objects from descriptions.
- Few-shot Learning and Instruction Following: Qwen-VL performs better in few-shot learning than similarly sized models on OKVQA, VizWiz, TextVQA and Flickr30K, while Qwen-VL-Chat leads three real-world instruction benchmarks.On SEED-Bench, visual capabilities transfer to video tasks by sampling four frames; TouchStone shows a particularly clear advantage in Chinese capabilities.
5 Related Work
Related vision-language models have advanced multi-task generalist modeling, but previous systems retain weaknesses in instruction following, generalization to unseen tasks, and in-context abilities. The emergence of LLM-based LVLMs motivates more capable multimodal systems.
- Vision-Language Research: Prior work includes vision-language models and multi-task generalist models for image-text retrieval, generation, and related multimodal tasks.The section situates Qwen-VL among a broad line of vision-language research.
- Limitations of Previous Models: Previous vision-language models are limited by poor instruction following, weak generalization to unseen tasks, and a lack of in-context abilities.These limitations are presented as remaining challenges despite significant progress.
- LLM-based LVLMs: Researchers increasingly build large vision-language models on top of LLMs to address multimodal understanding beyond text-only modeling.The related work describes this development as following the rapid progress of LLMs.
6 Conclusion and Future Work
The paper releases Qwen-VL as a multilingual vision-language model series supporting broad multimodal capabilities. Future work targets additional modalities, larger and higher-resolution training, and multimodal generation.
- Conclusion: Qwen-VL is released as a multilingual model series supporting multilingual conversations, multi-image interleaved conversations, Chinese grounding, and fine-grained recognition.The release is intended to facilitate multimodal research.
- Future Work: Future work includes integrating Qwen-VL with additional modalities such as speech and video.This is identified as one dimension for continued capability enhancement.
- Future Work: Future scaling targets model size, training data and input resolution to handle more complex relationships within multimodal data.The authors frame these as planned augmentations to Qwen-VL.
- Future Work: Future work also targets multimodal generation of high-fidelity images and fluent speech.This extends the model’s planned capabilities beyond understanding.
A Dataset details
Qwen-VL uses cleaned web, academic, grounding, OCR, PDF, and HTML data, formatted as multi-task image-text sequences and ChatML dialogues. The datasets include captioning, VQA, grounding, and OCR examples with spatial annotations.
- Corpus construction: The pre-training corpus combines web-crawled image-text pairs from LAION-en, LAION-zh, LAION-COCO, DataComp, and Coyo, followed by extensive cleaning.Cleaning removes problematic aspect ratios, small images, harsh CLIP matches, unsupported characters, emojis, unsuitable text lengths, HTML tags, and irregular patterns.
- Task data: VQA data uses maximum-confidence answers for VQAv2, while other VQA datasets receive no special answer processing.
- Task data: Grounding data concatenates noun or phrase text with bounding-box coordinates, while GRIT captions are greedily cleaned to remove recursive boxes.
- OCR data: Synthetic OCR data renders text over COCO scenery using 41 English and 11 Chinese fonts, retaining generated text locations as quadrilateral labels.
- Document data: PDF and HTML collections are rendered into images with text annotations and bounding boxes, using PyMuPDF for PDFs and Puppeteer for HTML pages.
- Data formats: The multi-task format mixes captioning, VQA, OCR, grounding, and related image-text tasks, while ChatML marks image turns and dialogue interactions with special tokens.Training supervises answers and special tokens but not role names or question prompts.
C Hyperparameters
Qwen-VL training uses AdamW-based optimization across pre-training and multi-task stages, with cosine schedules, warm-up, and large-scale batches. The second stage raises image resolution and trains the whole model.
- Stage one: The first pre-training stage uses AdamW, cosine learning rates, 500-step warm-up, weight decay, gradient clipping, and layer-wise learning-rate decay for the ViT.
- Stage two: The second multi-task stage increases visual input resolution from 224 × 224 to 448×448 and unlocks the language model for whole-model training.
D Summary of the evaluation benchmarks
The paper summarizes the evaluation benchmarks and their corresponding metrics in a dedicated benchmark table.
- Benchmark summary: Table 9 provides a summary of the evaluation benchmarks and corresponding metrics used for Qwen-VL.
E.1 Convergence of the Pre-training Stage
Pre-training loss decreases steadily as training images increase, while zero-shot VQA improves despite the absence of VQA data. Ablations select 256 adapter queries and Vanilla Attention based on convergence and efficiency trade-offs.
- E.1 Convergence of the Pre-training Stage: Training loss decreases steadily as the number of training pictures increases during the first pre-training stage.The stage uses BFloat16 mixed precision, batch size 30720, learning rate 2e−4, and one epoch over all images.
- E.1 Convergence of the Pre-training Stage: Zero-shot VQA scores increase amid fluctuations even though no VQA data is added during the first pre-training stage.
- E.2 Number of Learnable Queries in the Vision-Language Adapter: The adapter uses 256 learnable queries because too few queries lose information at 448×448 resolution, while too many or too few slow convergence.
- E.3 Window Attention vs Global Attention for Vision Transformer: Vanilla Attention is selected over Window Attention because Window Attention produces significantly higher loss while training speeds remain similar.
- E.3 Window Attention vs Global Attention for Vision Transformer: At 896 × 896 resolution, Window Attention is not used because training takes almost 2.5 times longer than with 448 × 448 input.
E.4 Performance on Pure-text Tasks
Qwen-VL incorporates pure-text data during multi-task training and supervised fine-tuning to preserve text comprehension. On pure-text benchmarks, it does not degrade and can improve after multi-task training.
- Qwen-VL is initialized from an intermediate Qwen-7B checkpoint, supporting comparability with many text-only LLMs on pure-text tasks.
- Qwen-VL does not compromise pure-text ability after multimodal training, according to the reported results.
- Qwen-VL incorporates pure-text data during multi-task training and SFT to prevent catastrophic forgetting of text comprehension.This supplements the visual and language-related training data.
- Qwen-VL even improves on pure-text capability after multi-task training.