PaperlayerView exact revision on arXiv

Source-linked AI summary

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, Jifeng Dai

arXiv:2312.14238v3cs.CV

TL;DR

Vision and vision-language foundation models have lagged behind the rapid growth of LLMs, leaving gaps in scale and representation between vision encoders and language models. InternVL scales its vision encoder to 6 billion parameters and progressively aligns it with an LLM using web-scale image-text data, achieving leading performance across diverse visual-linguistic tasks and multi-modal dialogue.

  • Problem

    Vision and vision-language foundation models have lagged behind LLMs, with gaps in parameter scale and feature representation between vision encoders and language models.

  • Method

    InternVL scales a vision encoder to 6 billion parameters and progressively aligns it with an LLM through contrastive training, generative training, and supervised fine-tuning.

  • Results

    InternVL achieves leading performance across image and video classification, retrieval, captioning, perception, and multi-modal dialogue benchmarks.

  • Takeaways & Limitations

    InternVL demonstrates a broadly capable vision-language foundation model for generic visual-linguistic tasks and multi-modal dialogue.

Abstract

from arXiv · show

The exponential growth of large language models (LLMs) has opened up numerous possibilities for multimodal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foundation model (InternVL), which scales up the vision foundation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image/video-text retrieval, and link with LLMs to create multi-modal dialogue systems. It has powerful visual capabilities and can be a good alternative to the ViT-22B. We hope that our research could contribute to the development of multi-modal large models. Code and models are available at https://github.com/OpenGVLab/InternVL.

1. Introduction

InternVL addresses the scale, representation, and connection gaps between vision encoders and LLMs by scaling the vision component and progressively aligning it with diverse image-text data. The resulting foundation model supports perception, vision-language, and multimodal dialogue tasks, achieving leading performance across broad benchmarks.

  • Motivation: Existing VLLMs rely on lightweight glue layers, but large parameter-scale disparities, representation inconsistencies, and inefficient connections limit vision–language alignment.LLMs reach up to 1000 billion parameters, while widely used VLLM vision encoders remain around one billion.
  • Model design: InternVL scales its vision encoder to 6 billion parameters and aligns it with the LLM to reduce the gap in parameter scale and feature representation.Its design combines parameter-balanced vision and language components with representation alignment.
  • Training strategy: The model uses a progressive alignment strategy that applies contrastive learning to large-scale noisy image-text data before generative learning on fine-grained, high-quality data.A pretrained multilingual LLaMA initializes the middleware and helps maintain representation consistency with the LLM.
  • Capabilities: InternVL can operate as a standalone vision encoder or collaborate with language components for vision-language tasks and multimodal dialogue systems.The language middleware bridges the vision encoder and LLM decoder.
  • Evaluation: InternVL achieves leading performance across generic visual-linguistic tasks including ImageNet classification, ADE20K segmentation, Kinetics video classification, retrieval, and image captioning.The paper reports comparisons spanning image, video, retrieval, captioning, and multimodal dialogue tasks.

2. Related Work

Related work traces vision foundation models from CNNs and residual networks to high-capacity ViT variants, while LLM research motivates multimodal interaction. Recent VLLMs connect visual inputs with language models for tasks including visual question answering and visual instruction tuning.

  • Vision foundation models: CNN-based vision foundation models advanced through architectures such as AlexNet and residual networks, which helped address vanishing gradients and enabled increasingly large and deep models.These developments were accompanied by continual improvements on the ImageNet benchmark.
  • Vision foundation models: ViT and its variants expanded vision-model capacity and performed strongly across important visual tasks.In the LLM era, these models commonly connect to language models through lightweight glue layers.
  • Multimodal language models: Because real-world interactions extend beyond natural language, incorporating vision into LLMs is identified as an important direction for multimodal interaction.The vision modality can provide additional information while leveraging LLM capabilities.
  • Vision-language large models: VLLMs enhance language models with visual processing and interpretation, with Flamingo demonstrating few-shot visual question answering.Later systems including GPT-4, the LLaVA series, and MiniGPT-4 introduced visual instruction tuning.

3. Proposed Method

InternVL combines a 6-billion-parameter vision encoder with an 8-billion-parameter language middleware and progressively aligns them through contrastive learning, generative learning, and supervised fine-tuning. This design supports flexible use across visual perception, retrieval, captioning, VQA, and multimodal dialogue tasks.

  • Model Architecture: InternVL combines InternViT-6B, a 6-billion-parameter vision transformer, with QLLaMA, an 8-billion-parameter language middleware initialized from multilingual-enhanced LLaMA.The two components are designed to bridge substantial modality and structural differences.
  • Flexible Task Composition: InternVL flexibly combines its vision encoder and language middleware for visual perception, image/video-text retrieval, image captioning, visual question answering, and multimodal dialogue.For dialogue, InternVL-Chat connects the visual component to an off-the-shelf LLM decoder through an MLP layer and supports two configurations.
  • Vision Encoder Design: InternViT-6B is tuned for accuracy, speed, and stability by searching over depth, head dimension, and MLP ratio, while smaller depths can improve speed when computation is not saturated.With equal parameter counts, the searched architectural factors have little impact on accuracy according to the reported findings.
  • Language Middleware: QLLaMA provides LLM-aligned image-token representations and uses 8 billion parameters for vision-language alignment, 42 times more than QFormer.Its pretrained initialization enables transformation of InternViT-6B image tokens into representations aligned with LLMs.
  • Progressive Alignment Training: InternVL training proceeds through three stages: vision-language contrastive training, vision-language generative training, and supervised fine-tuning.The stages move from noisy web image-text pairs toward high-quality caption, VQA, and multimodal dialogue data.
  • Progressive Alignment Training: The first stage uses CLIP-style symmetric cross-entropy contrastive learning to align InternViT-6B with text, supporting zero-shot classification, image-text retrieval, and semantic segmentation.The second stage connects InternViT-6B with QLLaMA and uses ITC, ITM, and ITG losses to align visual features with LLMs.

4. Experiments

Experiments evaluate InternVL’s visual perception, zero-shot vision-language capabilities, and multimodal dialogue performance. InternViT-6B shows strong transfer performance, while InternVL-C and InternVL-G achieve robust classification and retrieval results across diverse settings.

  • Visual perception: InternViT-6B consistently outperforms ViT-22B across five few-shot image-segmentation experiments with varying training-data proportions.Under linear probing, it attains 47.2 mIoU.
  • Zero-shot video classification: 76.1%, 75.5%, and 67.5% average accuracy are achieved on Kinetics-400, Kinetics-600, and Kinetics-700, respectively, using one center frame per video.The reported video-classification metrics include top-1 accuracy and the mean of top-1 and top-5 accuracy.
  • Zero-shot image-text retrieval: InternVL-C achieves state-of-the-art performance across most image-text retrieval metrics, while InternVL-G further improves zero-shot retrieval after second-stage pretraining.Multilingual retrieval is evaluated across eight languages using the XTD dataset.
  • Multimodal dialogue: InternVL-Chat is evaluated on MME and POPE, two multimodal dialogue benchmarks covering perception, cognition, and popularity-based hallucination assessment.MME contains 14 perception and cognition subtasks.

5. Conclusion

InternVL is a large-scale vision-language foundation model that scales vision modeling to 6 billion parameters and aligns it for generic visual-linguistic tasks. It combines InternViT-6B, the LLM-initialized QLLaMA middleware, and web-scale image-text data.

  • InternVL scales its vision foundation model to 6 billion parameters for generic visual-linguistic tasks.The model is presented as a large-scale vision-language foundation model.
  • InternViT-6B serves as the large-scale vision foundation model within InternVL.
  • InternVL progressively aligns InternViT-6B with the LLM-initialized language middleware QLLaMA using web-scale image-text data from various sources.The data is used for efficient training and the alignment bridges vision foundation models with LLMs.

A. Supplementary Materials · A.1. More Experiments

The supplementary experiments assess InternVL across zero-shot image classification, multilingual and video-text retrieval, fine-tuned retrieval, and Tiny LVLM evaluation. InternVL averages 78.1% across 20 zero-shot image-classification benchmarks, exceeding EVA-02-CLIP-E+ by 1.0 points.

  • A.1. More Experiments: InternVL achieves 78.1% average performance across 20 zero-shot image-classification benchmarks.The evaluation expands beyond ImageNet and its variants to test effectiveness and robustness across 20 datasets.
  • A.1. More Experiments: InternVL surpasses EVA-02-CLIP-E+ by 1.0 points in average zero-shot image-classification performance.EVA-02-CLIP-E+ is identified as the previously leading method.
  • A.1. More Experiments: The supplementary results examine InternVL on zero-shot multilingual image-text retrieval using the XTD dataset.The comparison covers English, Spanish, French, Chinese, Italian, Korean, Russian, and Japanese, reporting Image-to-Text recall@10.
  • A.1. More Experiments: The experiments compare zero-shot video-text retrieval performance on MSR-VTT.The table includes the number of frames, and some models use temporal attention layers.
  • A.1. More Experiments: Fine-tuned image-text retrieval is evaluated in English and Chinese on Flickr30K and Flickr30K-CN.Separate fine-tuning is used for each dataset to prevent data leakage.
  • A.1. More Experiments: Tiny LVLM results address visual commonsense and object hallucination.The results are reported in Table 17.

A.2. More Ablation Studies

The ablation studies examine InternVL’s compatibility with alternative LLMs, zero-shot generalization, multimodal capabilities, and image-text encoding efficiency. They also note optimization opportunities beyond the reported PyTorch, Flash Attention, and bf16 measurements.

  • Zero-Shot Image Classification: InternVL demonstrates zero-shot image-classification generalization across 20 datasets spanning various domains, beyond ImageNet.The evaluation compares zero-shot image-classification performance across 20 other datasets.
  • Tiny LVLM Evaluation: The Tiny LVLM evaluation covers visual reasoning, visual perception, visual knowledge acquisition, visual commonsense, and object hallucination.These five categories define the reported multimodal capability evaluation.
  • Compatibility with Other LLM: InternVL shows promising compatibility with LLMs other than Vicuna, with InternLM-7B performing slightly better than Vicuna-7B.The comparison uses InternLM-7B as an example under the same experimental setup as the main paper’s Table 9.
  • Efficiency Analysis: InternVL’s image-text encoding efficiency is analyzed by separating image and text encoding across InternVL-C and InternVL-G at image sizes 224, 336, and 448.The total encoding time includes both image and text encoding, with measurements taken using batch size 128 on one A100 GPU.
  • Efficiency Analysis: The efficiency measurements use PyTorch with Flash Attention and bf16 precision, while model quantization and TensorRT are identified as potential optimizations.The passage notes considerable room for further optimization.

A.3. Detailed Training Settings

InternVL training uses fully trainable two-stage optimization with large-scale data and staged resolution changes, followed by task-specific fine-tuning for dialogue and retrieval. The settings specify initialization, optimizer schedules, compute scale, and regularization choices across these procedures.

  • Stage 1: Stage 1 fully trains randomly initialized InternViT-6B and multilingual LLaMA-7B components with AdamW and separate starting learning rates of 1e-3 and 1e-4.The optimizer uses β1 = 0.9, β2 = 0.95, weight decay 0.1, and cosine learning-rate scheduling.
  • Stage 1: 175K iterations process about 28.7 billion samples using a total batch size of 164K across 640 A100 GPUs.Training begins at 196×196 resolution with 50% image-token masking, then switches to 224×224 without masking for the final 0.5 billion samples.
  • Dialogue Fine-tuning: InternVL-Chat without QLLaMA follows LLaVA-1.5 recipes, training MLP layers on LGS-558K and the LLM on LLaVA-Mix-665K for one epoch each.The QLLaMA setup instead uses a two-step procedure with MLP training followed by LLM fine-tuning, and increases batch size to 512 for the expanded dataset.

A.4. Data Preparation for Pre-training

InternVL’s pre-training used web-scale image-text pairs across two stages, with progressively stricter filtering. Evaluation covered multilingual image-text retrieval, zero-shot video-text retrieval, and image captioning datasets.

  • Training Data: Stages 1 and 2 used vast image-text collections including LAION-en, LAION-multi, LAION-COCO, COYO, and Wukong.These datasets were among the sources listed for the first and second training stages.
  • Training Data Cleaning: Stage 1 applied minor filtering using six factors while removing duplicates from ImageNet-1K/22K, Flickr30K, and COCO.The factors were CLIP similarity, watermark probability, unsafe probability, aesthetic score, image resolution, and caption length.
  • Training Data Cleaning: Stage 2 used more stringent caption-based filtering and retained only 1.03 billion entries.Filtering considered caption length, completeness, readability, gibberish or boilerplate, offensive language, placeholder text, and source code.
  • Testing Datasets: Evaluation used five datasets for zero-shot multilingual image-text retrieval, MSR-VTT for zero-shot video-text retrieval, and three image captioning datasets.The passages identify the retrieval and captioning dataset counts, and specifically name MSR-VTT for video-text retrieval.

A.5. Data Preparation for SFT

InternVL’s SFT data preparation collects broad, high-quality instruction data and converts non-dialogue datasets into dialogue-style examples. The resulting InternVL-Chat models are evaluated on image captioning, visual question answering, and multimodal dialogue.

  • Testing Datasets for SFT: InternVL-Chat models are validated on three tasks: image captioning, visual question answering, and multi-modal dialogue.The evaluation datasets use the same response-formatting prompts as LLaVA-1.5 [91].
  • Training Data for SFT: SFT training uses a wide range of high-quality instruction data, with only training sets used for model training.Non-dialogue datasets are converted using response-formatting prompts described in [91].
  • Training Data for SFT: Non-dialogue datasets are converted into dialogue datasets through task-specific response-formatting prompts.For EST-VQA, the prompt requires answering each question with a single word or phrase.
  • Training Data for SFT: The SFT training data includes datasets such as EST-VQA, which pairs questions and images with answers and question-specific bounding boxes.EST-VQA’s bounding boxes identify the image areas informing the answers.
Loading 2312.14238v3…