Source-linked AI summary
CogVLM: Visual Expert for Pretrained Language Models
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang
TL;DR
CogVLM addresses the challenge of adding strong visual understanding to pretrained language models without sacrificing their NLP capabilities. It uses trainable visual experts in attention and FFN layers for deep vision-language fusion, and achieves state-of-the-art performance across 17 classic multimodal benchmarks.
Problem
Pretrained language models need visual understanding, but shallow alignment lacks deep fusion and direct multimodal training can compromise NLP capabilities.
Method
CogVLM adds trainable visual experts with separate QKV and MLP components in each layer while keeping the original language-model parameters fixed.
Results
CogVLM-17B achieves state-of-the-art performance across 17 classic cross-modal benchmarks.
Takeaways & Limitations
Deep fusion retains the pretrained language model’s capabilities while adding visual understanding across multimodal tasks.
Takeaways & Limitations
VLM training remains in its infancy, with better supervised alignment, RLHF, and anti-hallucination identified as directions for further work.
Abstract
from arXiv · showhide
We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular shallow alignment method which maps image features into the input space of language model, CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. As a result, CogVLM enables deep fusion of vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X 55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.
1. Introduction
CogVLM addresses the challenge of combining visual understanding with a pretrained language model by replacing shallow alignment with deep visual-language fusion. It aims to retain NLP capabilities while achieving strong performance across multimodal tasks.
- Motivation: Shallow alignment maps image features into the language model’s input space but lacks deep fusion between visual and linguistic representations.Visual features can diverge from the expected input distributions of deeper language-model layers, limiting task-specific encoding.
- Motivation: Directly training language-model parameters during multimodal training can compromise generalizability because image-text data distributions differ from text-only pretraining data.The discussion identifies this distribution shift as a source of catastrophic forgetting.
- Approach: CogVLM adds a trainable visual expert with new QKV and MLP components for image features while freezing the original language-model parameters.The visual expert doubles parameters without increasing FLOPs, and image-free inputs preserve the original language model’s behavior.
- Contributions: CogVLM-17B, trained from Vicuna-7B, achieves state-of-the-art performance across 17 classic cross-modal benchmarks.The model is presented as deeply integrating visual and linguistic features while retaining the pretrained language model’s full capabilities.
- Contributions: Ablation studies validate the visual expert and deep fusion, while examining visual-encoder scale, attention masks, influential parameters, and self-supervised image loss.The studies also compare alternative visual-expert placements and configurations.
- Contributions: The authors publicly release CogVLM weights and the dataset used for supervised fine-tuning.They anticipate this open sourcing will support visual-understanding research and industrial application.
2. Method
CogVLM combines a pretrained vision encoder, an MLP adapter, a frozen GPT-style language model, and trainable visual experts inserted throughout its Transformer layers. Its training uses large-scale image-text pretraining, visual grounding data, and instruction alignment for conversational and grounding capabilities.
- Architecture: CogVLM comprises a ViT encoder, MLP adapter, pretrained GPT-style language model, and visual expert module.The ViT and adapter map image information into the language model’s feature space, while the expert supports deeper fusion.
- Architecture: The visual expert supplies separate QKV matrices and FFN components for image features while the original language-model components process text features.These expert components are added in each layer and initialized with shapes matching the pretrained language model.
- Architecture: All visual tokens share one position id because the ViT already encodes positional information and image sequences may contain hundreds to thousands of tokens.This design is intended to mitigate remote attenuation and excessive positional-encoding lengths.
- Pretraining: Pretraining uses publicly available LAION-2B and COYO-700M image-text data, leaving about 1.5B images after filtering.A separate 40M-image visual-grounding dataset associates caption nouns with bounding boxes.
- Pretraining: The first pretraining stage optimizes image-captioning loss, while the second mixes captioning with referring-expression comprehension.The second stage uses normalized bounding-box coordinates and increases image resolution during its final 30,000 iterations.
- Alignment: Instruction alignment trains CogVLM-Chat for natural-language interaction and CogVLM-Grounding for inputs and outputs containing bounding boxes.Chat alignment combines visual question-answering and dialogue datasets with prompts distinguishing concise and extended answers.
- Alignment: The grounding data covers grounded captioning, referring-expression generation, referring-expression comprehension, and grounded visual question answering.These datasets can be adapted across tasks, although some conversions can introduce ambiguity between similar regions.
3. Experiments
CogVLM is evaluated across captioning, VQA, LVLM, grounding, and ablation settings, showing broad benchmark strength and benefits from deep visual fusion. The experiments also identify effects from initialization, masking, image self-supervision, encoder scale, and EMA.
- Image Captioning: CogVLM achieves state-of-the-art or competitive performance across image-captioning benchmarks, including NoCaps, Flickr, COCO, and TextCaps.On NoCaps, it surpasses GIT2 by up to 5.7 points in the out-of-domain split while using 1.5B versus 12.9B pretraining pairs; on Flickr, it scores 94.9, 9.1 points above Qwen-VL.
- VQA and LVLM Benchmarks: CogVLM demonstrates strong generalization on VQA and LVLM benchmarks, including daily-life, text-intensive, knowledge-demanding, recognition, OCR, and reasoning tasks.The model achieves state-of-the-art results on all 7 reported LVLM benchmarks and outperforms several models using larger language models.
- Visual Grounding: CogVLM achieves state-of-the-art visual grounding across the reported benchmarks and surpasses specialized models on 5 of 9 splits.It scores 92.76 on RefCOCO val, 92.91 on RefCOCO+ test-A, and 90.79 on RefCOCOg test, exceeding the cited comparison models on each split.
- Ablation Study: Ablations show that shallow adapter-only alignment performs significantly worse, while visual experts outperform training the language model and benefit from inclusion in attention and FFN layers.Adding experts only every fourth layer or removing attention causes some performance decline, though the decline remains acceptable for computational trade-offs.
- Ablation Study: Initializing visual experts from language-model weights consistently outperforms random initialization across datasets.The results indicate that language-pretrained transformer architecture provides a more effective starting point for processing visual tokens.
- Ablation Study: Image self-supervised loss provides no downstream-task improvement, a smaller visual encoder causes only slight declines on most benchmarks, and EMA often improves performance.The notable encoder exception is TextVQA, where replacing EVA2-E with EVA2-L reduces performance by 2.5.
4. Conclusion
CogVLM is introduced as an open visual language foundation model that shifts VLM training from shallow alignment to deep fusion. The authors identify substantial opportunities for future work in VLM training.
- CogVLM shifts VLM training from shallow alignment to deep fusion and achieves state-of-the-art performance on 17 classic multimodal benchmarks.
- Future directions include better SFT alignment, RLHF, and anti-hallucination.
A.1. Details of Training Settings
The paper reports parameter settings for pre-training and multitask training in dedicated tables. These settings are presented as implementation details for the training procedures.
- Parameter settings during pre-training are reported in Table 5.
- Parameter settings during multitask training are reported in Table 6.
- The reported settings cover both pre-training and multitask finetuning of CogVLM.
A.2. Details of Associated Datasets
This section introduces the datasets and explains their use in evaluating the associated benchmarks. It frames the dataset details as part of the evaluation process.
- The section introduces details of datasets used in the evaluation process.
- The dataset descriptions are associated with the paper’s benchmark evaluations.
- Dataset use is documented together with the details of the associated benchmarks.
A.2.1. IMAGE CAPTIONING •
The image-captioning evaluation draws on COCO, NoCaps, Flickr30K, and TextCaps, covering conventional captions, novel objects, everyday scenes, and text-rich images. The benchmark summary is provided in Table 7.
- COCO: COCO contains 330K images with separate train, validation, and test caption collections.
- NoCaps: NoCaps evaluates novel object captioning with nearly 400 object classes absent from COCO.
- NoCaps: NoCaps validation and test sets contain 4,500 and 10,600 images, respectively.
- NoCaps: NoCaps is divided into “in”, “near”, and “out” domains, with out-domain objects never appearing in COCO.
- Flickr30K: Flickr30K contains 31,783 images and 158,915 captions describing everyday activities, events, and scenes.
- TextCaps: TextCaps contains 145k captions for 28k images and requires OCR and captioning abilities to integrate textual and visual information.
A.2.2. GENERAL VQA •
These datasets evaluate visual question answering across broad question types, external knowledge, science, text in images, and fine-grained visual reasoning.
- VQAv2 contains over 200,000 images and more than 1.1 million questions spanning yes/no, counting, and open-ended queries.
- OK-VQA tests visual question answering that requires external knowledge or common sense beyond image content.It contains 14,055 open-ended questions with five ground-truth answers per question.
- ScienceQA comprises 21,208 multimodal multiple-choice questions across natural science, language science, and social science.Questions include explanations linked to relevant lectures.
- TDIUC contains 1.6M questions across 170K images and 12 question types, including object, color, counting, and positional reasoning.
- OCR-VQA contains 207,572 book-cover images paired with over 1 million question-answer pairs.
- TextVQA evaluates detecting, reading, and reasoning about text within images to answer questions.It includes 45,336 questions on 28,408 images.
A.3. LVLM Benchmarks
These benchmarks assess visual-language models across open-ended capability integration, multimodal perception and reasoning, instruction following, hallucination, and mathematical understanding.
- MM-Vet evaluates six core visual-language capabilities and 16 integrations using an LLM-based unified scoring metric for open-ended outputs.
- SEED-Bench contains 19K human-annotated multiple-choice questions covering 12 evaluation dimensions for image and video understanding.
- MMBench contains approximately 3000 multiple-choice questions across 20 capability dimensions spanning perception, reasoning, localization, and attribute inference.
- LLaVA-Bench evaluates multimodal instruction following with 60 questions covering varied real-world and artistic scenes.Its questions use detailed, manually curated descriptions and selections.
- POPE evaluates object hallucination through binary classification queries across random, popular, and adversarial subsets totaling 8,910 entries.
- MMMU contains 11.5K multidisciplinary questions across six disciplines, 30 topics, and 183 subfields, including diverse image types.
- MathVista combines 31 existing and three newly created multimodal datasets totaling 6141 examples for seven types of mathematical reasoning.
A.3.1. GROUNDING •
The grounding benchmarks test referring expressions, grounded visual question answering, caption-region alignment, and image relationships across several annotated datasets.
- RefCOCO contains 142,209 expressions across 19,994 images without linguistic constraints, while RefCOCO+ emphasizes appearance descriptions without location terms.RefCOCO+ contains 141,564 expressions over 19,992 images.
- RefCOCOg contains 85,474 natural referring expressions spanning 26,711 images, each with two to four same-category objects.
- Grounded Visual7W presents a “which”-type question with four candidate bounding boxes, requiring selection of the correct region.Its grounded subset contains 25,733 images and 188,068 questions.
- VisualGenome annotates over 100k images with average counts of 21 objects, 18 attributes, and 18 inter-object relationships.These annotations are aligned with standardized WordNet terminology.
- Figure 5 reports performance on TDIUC using fine-grained question classes.
B. Additional Fine-grained Experiments
Additional experiments evaluate CogVLM on fine-grained VQA categories and computational efficiency, reporting broad gains on TDIUC and reduced training resource consumption.
- CogVLM is evaluated on the public TDIUC validation split using official-script VQA accuracy.
- 94.0 accuracy on TDIUC surpasses the previous specialist SOTA MUREL at 88.2 overall.CogVLM consistently outperforms MUREL across 12 specific question types.
- The TDIUC results are reported as demonstrating comprehensive problem-solving skills on general VQA tasks.
- CogVLM demonstrates marked reductions in training resource consumption relative to models with comparable parameter magnitudes.The comparison considers pretraining and finetuning data from datasets such as VQAv2 and TextVQA.
- Table 8 compares model computational efficiency using PFLOPS*days as the metric.