Source-linked AI summary
CogVLM2: Visual Language Models for Image and Video Understanding
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, Jie Tang
TL;DR
The paper addresses limitations in visual language models involving shallow vision-language fusion, high-resolution processing, and video temporal understanding. It proposes the CogVLM2 family, combining visual experts, improved training, high-resolution image processing, timestamped multi-frame video inputs, and automated temporal grounding data. The models achieve state-of-the-art results across multiple image and video benchmarks and are open-sourced.
Problem
Visual language models need deeper vision-language fusion, efficient high-resolution processing, and broader image and video understanding capabilities.
Method
The CogVLM2 family uses visual experts, improved training recipes, high-resolution architecture, timestamped multi-frame inputs, and automated temporal grounding data construction.
Results
CogVLM2 and related models achieve state-of-the-art or top performance across diverse image and video understanding benchmarks.
Takeaways & Limitations
The open-sourced family broadens visual language model applications across image understanding, video understanding, document analysis, GUI comprehension, and temporal video grounding.
Abstract
from arXiv · showhide
Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in pursuit of enhanced vision-language fusion, efficient higher-resolution architecture, and broader modalities and applications. Here we propose the CogVLM2 family, a new generation of visual language models for image and video understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image understanding model, CogVLM2 inherits the visual expert architecture with improved training recipes in both pre-training and post-training stages, supporting input resolution up to $1344 \times 1344$ pixels. As a video understanding model, CogVLM2-Video integrates multi-frame input with timestamps and proposes automated temporal grounding data construction. Notably, CogVLM2 family has achieved state-of-the-art results on benchmarks like MMBench, MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4, contributing to the advancement of the field.
1 Introduction
The CogVLM team develops open-source visual language models to overcome LLMs’ text-only limitation and improve vision-language fusion. The CogVLM2 family extends these efforts across high-resolution images, video understanding, and diverse visual tasks.
- Motivation: LLMs’ text-only input limits access to visual information and motivates the development of visual language models.
- Vision-language fusion: CogVLM addresses shallow image-language alignment with visual experts that deepen fusion without sacrificing natural language performance.
- High-resolution understanding: CogAgent improves text recognition and GUI comprehension by processing images at 1120 × 1120 pixels through a high-resolution cross-module.
- Image understanding: CogVLM2 and GLM-4V use enhanced training recipes and achieve top performance across OCR, document, chart, general, and subject-specific understanding benchmarks.
- Video understanding: CogVLM2-Video extends the family to video with automated temporal grounding data construction and multi-frame inputs with timestamps.
- Research direction: The family’s research emphasizes deeper vision-language fusion, higher-resolution efficiency, broader modalities, and open-source release.
2 The CogVLM2 Family
The CogVLM2 family combines ViT encoding, visual-language adaptation, language modeling, and optional visual experts. Its image and video variants address information loss, efficiency, temporal awareness, and bilingual visual understanding.
- Architecture: CogVLM architectures contain a ViT encoder, adapter, language model, and optional visual expert module.
- Architecture: The ViT encoder converts raw images into semantically rich continuous visual features using EVA-CLIP.
- Architecture: The adapter uses 2×2 convolution followed by SwiGLU to reduce ViT sequence length to one-quarter while preserving critical image information.
- CogVLM2: CogVLM2 uses visual experts in attention and FFN modules, 2×2 downsampling, and LLaMA3-8B to deepen fusion and support efficient higher-resolution input.
- CogVLM2-Video: CogVLM2-Video adds timestamped multi-frame inputs and automated temporal grounding data to support temporal localization and open-domain video question answering.
- GLM-4V: GLM-4V is a bilingual visual language model using image-language co-training instead of visual experts, with a reported slight degradation in language-task performance.
3 Pre-training
CogVLM-family pre-training combines noisy-data refinement, synthetic data generation, progressive dataset construction, and training settings that balance visual integration with language preservation.
- Overview: Pre-training develops multimodal understanding through data processing, synthetic generation, progressive dataset construction, and balanced visual-language settings.
- Data challenges: Public image-text datasets provide scale but contain noise, coarse descriptions, and limited coverage of Chinese text recognition and GUI understanding.
- Data processing: Iterative refinement re-annotates data with an initial model, applies manual correction, and reuses corrected data to improve later model versions.
- Dataset construction: The progressive pre-training datasets include general image-text data, grounding data, digital-world screenshots, and specialized OCR resources.
- Synthetic data: Synthetic OCR data covers generated images, real-world extracted text, academic-paper LaTeX, and rendered tables and formulae.
- Synthetic data: CLAY-1B uses detailed recaptioning to improve captioning abilities in CogVLM2 and GLM-4V.
- Training settings: Training settings include progressive parameter unfreezing, mixed language and visual-language data, and gradually increasing image resolution.
4 Post-training
The post-training pipeline combines broader VQA and alignment data with staged supervised fine-tuning, while video training adds temporal grounding data and multi-frame inputs.
- Image post-training data: Broader VQA data improves model performance, but concise-answer datasets can reduce conversational quality without response-type distinction.CogVLM2 prefixes concise answers with “Short Answer” to separate VQA responses from dialogue responses.
- Image post-training data: Approximately 300K alignment examples and 50K preference-alignment examples support proportional training control and human-preference-oriented outputs.
- Video TQA dataset: The automated TQA process addresses limited existing temporal-grounding coverage and costly manual annotation by using image models and GPT-4o for filtering and generation.
- Image supervised fine-tuning: Two-stage image SFT first builds foundational capabilities with VQA and alignment data, then optimizes output style with selected VQA and preference data.
- Video supervised fine-tuning: CogVLM2-Video starts from a 224 × 224 CogVLM2 variant, processes 24 frames sequentially, and compresses video features with an added 2 × 2 convolution.
- Video supervised fine-tuning: The base video model targets video benchmarks, whereas the chat model is further fine-tuned on the TQA dataset for temporal grounding.
5 Evaluation
The CogVLM2 family is evaluated across diverse image and video understanding tasks. CogVLM2 and GLM-4V-9B achieve state-of-the-art performance on most image tasks, while CogVLM2-Video does so across multiple video benchmarks.
- Evaluation scope: The evaluation covers image and video understanding tasks across diverse visual domains, including the CogVLM2, CogVLM2-Video, GLM-4V-9B and GLM-4V-Plus models.
- Image tasks: CogVLM2 and GLM-4V-9B achieve state-of-the-art performance on most evaluated image tasks against open-source models of similar scale.
- Video tasks: CogVLM2-Video achieves state-of-the-art performance on multiple video question-answering tasks evaluated with MVBench, VideoChatGPT-Bench and LVBench.
6 Conclusion
The CogVLM2 family extends language models beyond text-only input through open-source image and video understanding models. Its visual experts, high-resolution processing, and data-generation techniques support visual-language tasks across several applications.
- Conclusion: The CogVLM Family addresses the limitation of traditional LLMs restricted to text-based inputs by integrating visual and language modalities.
- Conclusion: The family includes models that understand and generate content from images and videos for document analysis, GUI comprehension and temporal video grounding.
- Conclusion: Visual Expert and high-resolution cross-module architectures are presented as enabling visual-linguistic fusion while preserving language capabilities.
- Conclusion: The CogVLM2 family is open-sourced as a set of tools for academic research and practical applications.
A Details for Generating TQA Dataset
The TQA generation pipeline converts videos into temporally grounded question-answer data through frame captioning, scene-change filtering, and few-shot question generation.
- Frame captioning: The pipeline first extracts video frames and uses CogVLM2 to produce image captions for them.
- Scene filtering: GPT-4o evaluates caption sequences to identify significant scene changes and filter out videos with minor changes.
- Question generation: The generation prompt uses timestamp keys and frame descriptions to support questions about when people or scene transitions appear.
- Question generation: GPT-4o then uses a few-shot approach to generate time-related question-answer pairs from the image captions.
- Output constraints: The process directs GPT-4o to output one concise question-answer pair or “None” when the captions cannot support such generation.
B.1 Perception and Understanding
CogVLM2 demonstrates image perception across floor plans, text extraction, world scenes, symbols, and logos. Its outputs include structured details, multilingual descriptions, and recognized visual entities.
- Image perception: CogVLM2 identifies rooms, areas, orientation, and watermark details in a residential floor plan.It reports named rooms with square-meter areas, identifies north from the compass, and reads the Fang.com watermark.
- Visual recognition: CogVLM2 recognizes the twelve zodiac animals and identifies BMW, Infiniti, and Maserati logos with visual and brand descriptions.The logo analysis distinguishes the BMW circular quadrants, Infiniti symbol, and Maserati trident.
- Text and document understanding: CogVLM2 extracts structured identity-card fields, including names, USCIS number, category, birthplace, dates, and sex.The response is formatted as JSON-like key-value information across the supplied passages.
- Image perception: CogVLM2 describes the Golden Pavilion as a gold-covered Buddhist temple in Kyoto surrounded by a reflective pond, trees, and rocks.The description combines visual content with associated world knowledge presented in the model response.
B.2 Reasoning and Analysis
CogVLM2 applies visual reasoning to interpret memes, assess physical quantities, identify salient chart values, and connect images with explanatory context. Its answers combine visual observations with task-specific analysis.
- Meme reasoning: CogVLM2 interprets a voting meme as contrasting enthusiasm for a state-of-the-art model with reluctance to spend 80% of the time preparing data.The explanation attributes the contrast to the complexity and time demands of data preparation.
- Scientific reasoning: Sample A has the higher average kinetic energy and temperature because its particles average 600 m/s versus 540 m/s in Sample B.The response uses KE = 0.5 × m × v2 and equal particle masses to connect speed with kinetic energy.
- Figure and table analysis: The United States has 73% landfilled plastic waste, exceeding the listed percentages for the other regions.The generated table also reports mismanaged, incinerated, and recycled shares for each region.
- Meme reasoning: CogVLM2 explains a gaming meme as a mismatch between a desired 300 FPS experience and a 75 Hz monitor.The response interprets the characters and balloon as a humorous depiction of hardware limitations.
B.3 Video Understanding
CogVLM2-Video describes multi-frame scenes and answers temporal questions about events. Its demonstrations include detailed scene summaries and locating a salient action at a specific second.
- Video description: CogVLM2-Video describes a colorful tiger sculpture on an urban street with Asian signage, passersby, and falling pink petals.The response connects traditional artistry, materials, city surroundings, and human activity across the video.
- Temporal understanding: The video highlight is the player in the green jersey making a basket at the 4th second.This answer combines event identification with temporal localization.
C MVBench Results
The paper presents detailed CogVLM-family video-understanding results on MVBench. The supplied materials identify the benchmark comparison table but do not include its scores.
- MVBench evaluation: MVBench is used to compare video-understanding performance across models.The supplied caption states that the table reports video-understanding performance comparisons and highlights the best results.