PaperlayerView exact revision on arXiv

Source-linked AI summary

What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?

Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong

arXiv:2307.02469v2cs.CVcs.CL

TL;DR

Because multimodal LLM studies vary in architecture, data, training recipes, prompts, and benchmarks, it is difficult to identify consequential design choices and quantify progress. The paper systematically studies these choices and evaluates them with an image-and-video benchmark; Lynx performs best among the compared open-source models on multimodal understanding and generation.

  • Problem

    Multimodal LLM studies differ in architecture, training data, recipes, prompts, and benchmarks, while suitable quantitative evaluation benchmarks are lacking.

    Source: §1 · p. 1

  • Method

    The authors compare controlled model variants and introduce an image-and-video benchmark covering multimodal understanding and text generation.

    Source: §1 · p. 2, §2.2 · p. 3, §3.1 · p. 6

  • Results

    Lynx outperforms existing open-source models in multimodal understanding and language generation, while experiments identify prefix-tuning, task and prompt counts, and data quality as important factors.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    The study concludes that multimodal LLM design must balance understanding accuracy and text generation ability.

    Source: §1 · p. 2

    A comprehensive automatic benchmark for multimodal LLMs remains an open problem, and compute constraints prevented extensive searches for optimal data combinations.

    Source: §5.2 · p. 13

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the performance of these models heavily relies on design choices such as network structures, training data, and training strategies, and these choices have not been extensively discussed in the literature, making it difficult to quantify progress in this field. To address this issue, this paper presents a systematic and comprehensive study, quantitatively and qualitatively, on training such models. We implement over 20 variants with controlled settings. Concretely, for network structures, we compare different LLM backbones and model designs. For training data, we investigate the impact of data and sampling strategies. For instructions, we explore the influence of diversified prompts on the instruction-following ability of the trained models. For benchmarks, we contribute the first, to our best knowledge, comprehensive evaluation set including both image and video tasks through crowd-sourcing. Based on our findings, we present Lynx, which performs the most accurate multi-modal understanding while keeping the best multi-modal generation ability compared to existing open-sourced GPT4-style models.

1 Introduction

The paper studies which design choices drive GPT4-style multimodal models, addressing the difficulty of comparing models built with differing architectures, data, recipes, prompts, and benchmarks. It evaluates controlled variants, proposes an image-and-video benchmark, and reports findings that inform Lynx.

1 Introduction

  • Existing multimodal models vary across key design choices, and suitable quantitative benchmarks are lacking, making progress difficult to attribute and measure.The comparison problem spans network structure, training data, training recipes, prompts, and evaluation benchmarks.Source: §1 · p. 1
  • The study implements about 20 controlled variants to examine architectures, LLM backbones, image-text data combinations, and prompt diversity across more than 50 tasks.At least three prompts per task were manually labeled, with additional prompts generated by GPT-4.Source: §1 · p. 2
  • The proposed benchmark evaluates image and video understanding with open-ended questions and assesses image-conditioned text generation separately.Open-VQA covers objects, OCR, counting, reasoning, action recognition, and chronology; GPT-4 judging agrees with human evaluation 95% of the time, while human annotators rank models on OwlEval.Source: §1 · p. 2
  • Prefix-tuning with trainable adapters outperforms cross-attention for multimodal adaptation; data quality, prompt diversity, and balancing understanding with generation also matter.The paper reports that noisy large-scale image-text data can contaminate language generation, while instruction-finetuned backbones such as Vicuna can improve instruction-following.Source: §1 · p. 2
  • Lynx achieves the most accurate multimodal understanding and best multimodal generation ability among the compared open-source models.Its two-stage recipe aligns visual and linguistic embeddings with about 120M image-text pairs, then instruction-finetunes on 20 multimodal tasks and NLP instruction data.Source: §1 · p. 2

2 Lynx

Lynx extends Vicuna with trainable adapters to process visual and instructional tokens and generate text autoregressively. Its training combines image-text pretraining with instruction finetuning across multimodal tasks.

2.1 Formulations

  • The model conditions each response word on all visual and instruction inputs plus previous predictions, generating text autoregressively.Source: §2.1 · p. 3

2.2 Details of Model Architecture

  • Lynx concatenates projected vision tokens with instruction tokens as a decoder prefix and uses small trainable adapters in frozen LLM layers.The adapters are inserted after every block in the reported setup.Source: §2.2 · p. 3
  • Vision features are condensed into a sequence of 32 tokens regardless of the number of raw vision-encoder tokens.Source: §2.2 · p. 4

2.3 Pretraining

  • Pretraining uses more than 120M image-text pairs, moving from 224×224 images to 420×420 after 100k steps for another 10k steps.The stated motivation for higher resolution is that low-resolution pretraining alone was insufficient for tasks such as table reading and OCR.Source: §2.3 · p. 4

2.4 Instruction Fintuning

  • Instruction finetuning draws on 50+ text-only, image-text, and video-text tasks, with at least three manually written prompts per task and additional GPT4-generated prompts.The collected public instruction data supplements these task prompts, and the authors report that data-weight combinations crucially influence final performance.Source: §2.4 · p. 4

3 Experiment

The experiments evaluate GPT4-style models on image and video understanding, open-ended generation, and human-rated scenarios. They report Lynx’s benchmark performance alongside ablations of backbone, prompts, data, architecture, and image resolution.

3.2 Quantitative Experiments

  • Lynx achieves the best Open-VQA image and video understanding results, though InstructBLIP performs better on OCR, color, and action recognition; Lynx also leads OwlEval generation while retaining high Open-VQA performance.The Open-VQA examples show mostly correct answers with concise supporting reasons, while InstructBLIP often gives one-word answers.Source: Table 1 · p. 6, §3.2 · pp. 6–7, Table 2 · p. 7

3.1 Evaluation Protocols

  • Open-VQA manually curates 450 open-ended image/video questions, with GPT-4 judging answers at over 95% human consistency; OwlEval uses human scores for general-purpose generation.OwlEval images are padded by 8 pixels for fairness, and annotators prioritize correctness over richness.Source: §3.1 · p. 6

3.2 Quantitative Experiments

  • Lynx is state of the art on 7 of 14 MME subtasks, especially perception, but appears weaker on three cognition tasks evaluated with only 20 examples each.The authors caution that the small cognition subsets may create high variance across checkpoints.Source: §3.1 · p. 6, Figure 7 · p. 9, §3.2 · p. 9

3.3 Ablation Study

  • Vicuna follows instructions better than LLaMA and gives shorter answers (15.81 vs 20.15 words), but is slightly better on Open-VQA and slightly worse on OwlEval correctness.The comparison illustrates that instruction adherence and benchmark correctness do not move uniformly together.Source: §3.3 · p. 10
  • Diversified user- and GPT-4-written prompts improve open-ended instruction-following, balance abilities, and support generalization to new tasks and instructions.The authors report that these gains extend to final performance.Source: §3.3 · p. 10
  • Large-scale noisy image-text data such as COYO700M and DataComp1B damages language generation and performs no better than a smaller, cleaner dataset in human evaluation.The authors attribute the damage to dataset texts that are not fluent or natural language expressions.Source: §3.3 · p. 10
  • Cross-attention variants produce fluent, relevant responses but answer questions less correctly and receive lower human preference than prefix-tuning with adapters.The authors note that cross-attention may need more hyperparameter search.Source: §3.3 · p. 10
  • Training at 420x420 resolution performs better than training only at 224x224; the increase is made affordable by freezing the vision encoder after 10K first-stage steps.Source: §3.3 · p. 10

4 Related Work

Related work spans language-only models adapted for multimodal interaction and end-to-end multimodal LLMs. The latter differ in how they fuse visual and language inputs and which components they train.

Large-language models

  • Decoder-only LLMs can follow open-ended natural-language instructions after instruction fine-tuning.The passage describes their generality on zero-shot tasks and gives examples including GPT-3, PaLM, BLOOM, and LLaMA.Source: §4 · p. 10

Centralized Multi-modal Interactive System

  • Centralized systems use an LLM as a manager that selects specialized tools, but may lack zero-shot ability for open-ended requests no worker handles.Examples include Visual ChatGPT, MM-REACT, and HuggingGPT.Source: §4 · p. 12

End-to-end Multi-modal Large Language Models

  • End-to-end multimodal LLMs adapt language models using visual embeddings, with designs varying in fusion strategy and trainable components.Examples include Flamingo's cross-attention and methods that concatenate projected visual embeddings with language inputs.Source: §4 · p. 12

5 Discussions and Limitations

The discussion identifies design and data choices associated with multimodal model performance, alongside unresolved limits in evaluation, training data, multilingual ability, and safety.

5.1 Findings and Takeaways

  • Prefix-tuning with adaptors outperforms cross-attention in multimodal adaptation, while cross-attention may need more hyperparameter search.The authors report good open-ended instruction following after training on billions of multimodal tokens.Source: §5.1 · p. 12
  • Multimodal models can miss user instructions, and the authors link zero-shot generality to training across diverse tasks and prompts.They hypothesize that limited high-quality, diversified multimodal instruction data contributes to the instruction-following gap.Source: §5.1 · pp. 12–13
  • A small amount of high-quality, fluent-text data can perform slightly better than large noisy datasets, which may contaminate generated output distributions.The authors attribute this to generative pretraining learning conditional word distributions rather than image-text similarity; useful data should be fluent and well aligned.Source: §5.1 · pp. 12–13
  • Undertraining downstream visual-language tasks can bring hallucinations, while overtraining can impair instruction-following and long-answer generation.The authors therefore emphasize balancing training data to retain both image/video understanding and generation ability.Source: §5.1 · p. 13

5.2 Limitations

  • A comprehensive automatic benchmark for multimodal LLMs remains an open evaluation problem despite the paper's quantitative assessment of understanding and language generation.The authors note that evaluation differs from traditional vision-language tasks.Source: §5.2 · p. 13
  • The authors lack long-text image-text pretraining datasets and could not extensively search for optimal data mixtures because of computational-resource limits.Unexplored choices include sampling ratios, tasks, and prompts.Source: §5.2 · p. 13
  • Because Lynx is built on LLaMA, which is mainly trained on English, its multilingual responses are limited and multilingual multimodal modeling remains unexplored.The paper notes that Lynx can understand and sometimes produce other languages.Source: §5.2 · p. 13
  • The authors conduct no safety checks or output restrictions, so the model may produce inappropriate or toxic content depending on its training data.They do not support harmful language generation with their code or models.Source: §5.2 · p. 13

6 Conclusions

The paper presents Lynx and reports that it outperforms existing open-source models in multimodal understanding and language generation. It identifies future needs in model scale, training tasks, data quality, multilingual ability, and safety.

6 Conclusions

  • Lynx outperforms existing open-source models in multimodal understanding and language generation, and the paper identifies prefix-tuning, task diversity, data quality, and ability balancing as important factors.The model accepts image and video inputs and responds with open-ended natural language.Source: §6 · p. 13
  • Future work could scale Lynx to larger LLaMA models, including 30B and 65B, and use a larger, more diverse set of instructional tasks.The paper also calls for a large-scale, high-quality multimodal dataset.Source: §6 · pp. 13–14
  • The authors identify multilingual ability and safety as important for realistic applications.They also describe collecting a large-scale, high-quality multimodal dataset as worthwhile.Source: §6 · p. 14

A.1 Training Details

Training uses DeepSpeed and BFloat16 by default; Table 5 reports training hyperparameters.

A.1 Training Details

  • DeepSpeed is used to accelerate training.The passage does not specify the acceleration amount.Source: §A.1 · p. 22
  • BFloat16 is the default model precision.Source: §A.1 · p. 22
  • Table 5 lists training hyperparameters and indicates that some parameters do not use a learning-rate decay schedule.Source: Table 5 · p. 22

A.2 Hyper-parameters for Generation

The appendix reports generation settings and additional benchmark comparisons and examples. Most models performed better with a description-first prompt, while MiniGPT4 used its default settings.

A.2 Hyper-parameters for Generation

  • Most models performed better when prompted to describe the image before processing the user’s instructions; MiniGPT4 instead used its default settings.mPLUG-owl also followed the default parameters listed for its deployment.Source: §A.2 · p. 22
  • Table 6 lists hyper-parameters for visual question answering and general-purpose vision-input generation, with short responses encouraged for Open-VQA.The passage refers to the table but does not provide its parameter values.Source: Table 6 · p. 22

B MME Performance

  • Tables 7 and 8 present open-sourced multimodal LLM comparisons and rankings on the MME benchmark.The supplied captions do not state the table values or identify a leading model.Source: Table 7 · p. 23, Table 8 · p. 23

C.1 Image VQA Cases

  • Figures 10 and 11 provide additional cases from the Open-VQA image and video benchmarks, respectively.The captions identify the benchmark modalities but do not describe the cases’ outcomes.Source: Figure 10 · p. 24, Figure 11 · p. 25

D.1 Data & Tasks

This appendix includes training-data and prompt materials alongside examples of benchmark cases and model behavior. In the instruction-following demonstrations, Lynx and mPLUG-owl follow instructions to some extent, while InstructBLIP is not sensitive to different instructions.

D.1 Data & Tasks

  • Table 9 presents the training data used in the study.The supplied caption does not enumerate datasets or quantities.Source: Table 9 · p. 26

D.2 Prompt Examples

  • Prompt examples preserve applicable original instructions and use dataset-specific question, hypothesis, options, or meme inputs where relevant.Dialogs are split into rounds, and the model is prompted to speak next using the dialog history.Source: §D.2 · p. 27, Table 10 · p. 27

E OwlEval Cases

F.1 Multi-turn Dialog

F.2 Multi-lingual Response

F.3 Instruction-following Ability

  • Lynx and mPLUG-owl follow instructions correctly to some extent, whereas InstructBLIP is not sensitive to different instructions.Figure 16 demonstrates instruction-following ability across models.Source: §F.3 · p. 32, Figure 16 · p. 32

Related papers

Loading 2307.02469v2…