PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating the Catastrophic Forgetting in Multimodal Large Language Models

Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, Yi Ma

arXiv:2309.10313v4cs.CLcs.AIcs.LG

TL;DR

Catastrophic forgetting remains insufficiently characterized in MLLMs, especially for fundamental image classification. The paper introduces EMT to evaluate MLLMs as image classifiers and finds that most tested models lose vision-encoder classification performance, while moderate fine-tuning can help before excessive fine-tuning causes hallucination and forgetting.

  • Problem

    MLLMs suffer catastrophic forgetting, but prior evaluations have emphasized visual-language comprehension rather than fundamental image-classification capabilities.

  • Method

    The paper introduces EMT, which prompts MLLMs to classify images and post-processes their outputs into classification accuracy.

  • Results

    Nearly all tested open-source MLLMs fail to match their vision encoders’ classification performance; moderate LLaVA fine-tuning helps other tasks, but excessive fine-tuning causes forgetting and hallucination.

  • Takeaways & Limitations

    MLLM fine-tuning needs improvement because early fine-tuning can improve visual-text alignment, whereas later fine-tuning reduces generalizability through overfitting and hallucination.

  • Takeaways & Limitations

    The study evaluates catastrophic forgetting from the image-classification perspective and leaves analogous evaluation methods for other scenarios to future work.

Abstract

from arXiv · show

Following the success of GPT4, there has been a surge in interest in multimodal large language model (MLLM) research. This line of research focuses on developing general-purpose LLMs through fine-tuning pre-trained LLMs and vision models. However, catastrophic forgetting, a notorious phenomenon where the fine-tuned model fails to retain similar performance compared to the pre-trained model, still remains an inherent problem in multimodal LLMs (MLLM). In this paper, we introduce EMT: Evaluating MulTimodality for evaluating the catastrophic forgetting in MLLMs, by treating each MLLM as an image classifier. We first apply EMT to evaluate several open-source fine-tuned MLLMs and we discover that almost all evaluated MLLMs fail to retain the same performance levels as their vision encoders on standard image classification tasks. Moreover, we continue fine-tuning LLaVA, an MLLM and utilize EMT to assess performance throughout the fine-tuning. Interestingly, our results suggest that early-stage fine-tuning on an image dataset improves performance across other image datasets, by enhancing the alignment of text and visual features. However, as fine-tuning proceeds, the MLLMs begin to hallucinate, resulting in a significant loss of generalizability, even when the image encoder remains frozen. Our results suggest that MLLMs have yet to demonstrate performance on par with their vision models on standard image classification tasks and the current MLLM fine-tuning procedure still has room for improvement.

1. Introduction

MLLM research has largely emphasized multimodal comprehension, leaving catastrophic forgetting in fundamental image classification insufficiently evaluated. This paper introduces EMT and finds widespread classification-performance loss, with moderate fine-tuning initially helping before excessive fine-tuning causes hallucination and forgetting.

  • MLLM evaluations have mainly focused on visual question answering and related capabilities, leaving image-classification forgetting insufficiently explored.
  • EMT treats each MLLM as an image classifier and converts its prompted outputs into classification accuracy.The framework prompts the model to classify an image and post-processes its output.
  • Hallucinated extra labels or unverifiable label descriptions contribute to performance degradation in the tested MLLMs.
  • Nearly all tested open-source MLLMs fail to retain the classification performance of their vision encoders.The comparison is against the vision encoders’ zero-shot performance.
  • Moderate fine-tuning improves non-fine-tuned tasks, whereas excessive fine-tuning ultimately causes catastrophic forgetting.
  • The fine-tuning process still requires improvement to mitigate catastrophic forgetting and reduce hallucinations.

2. Related Works

Prior work documents catastrophic forgetting in fine-tuned language models and develops mitigation strategies, while MLLM research has not thoroughly studied the phenomenon. Related theoretical work connects forgetting in classification to minority collapse and neural collapse.

  • Fine-tuned LLMs can overfit small datasets and lose performance on other tasks, producing catastrophic forgetting.
  • Proposed mitigation approaches include pre-trained weight decay, learning-rate decay, regularization, and adversarial fine-tuning.
  • MLLMs combine multimodal inputs with LLM reasoning to perform complex vision-language tasks.
  • Minority collapse describes supervised-learning behavior with imbalanced data in which minority-class classifiers converge toward one another.

3. Fine-Tuning Image Classification

The paper first studies catastrophic forgetting through sequential image-classification fine-tuning, then extends the investigation to CLIP. Results show rising in-domain accuracy alongside declining out-of-domain performance after fine-tuning.

  • The experiments use ResNet pre-training and fine-tuning, followed by CLIP fine-tuning, to investigate catastrophic forgetting in vision-language models.
  • ResNet18 is pretrained on the first 50% of classes and fine-tuned on the remaining 50%, with no overlap between the class sets.
  • CLIP fine-tuning uses standard cross-entropy loss with text inputs formed by concatenating labels and short descriptions.
  • Significant in-domain gains coincide with declining out-of-domain performance after CLIP fine-tuning.
  • By epoch 15, nearly all in-domain performance metrics approach 99%, while out-of-domain performance suffers.

4. EMT: Evaluating Multimodal Large Language Models

EMT evaluates multimodal large language models as image classifiers, comparing their classification accuracy with the zero-shot performance of their vision encoders. Applying EMT reveals widespread performance degradation and identifies incorrect predictions and hallucinations as important failure modes.

  • EMT prompts an MLLM to classify an input image and post-processes its output to compute classification accuracy.The framework addresses evaluation of catastrophic forgetting from an image-classification perspective.
  • Most tested open-source MLLMs fail to retain classification performance comparable to their vision encoders across standard image-classification tasks.The evaluated models include LLaVA, Otter, InstructBLIP, and LENS; InstructBLIP-7b is a notable exception on CIFAR-10.
  • 4.2. Analyzing Failure Modes of MLLMs: MLLM outputs degrade accuracy through incorrect predictions, intrinsic hallucinations, and extrinsic hallucinations.Intrinsic hallucination contradicts source content, whereas extrinsic hallucination has no verifiable connection to it.
  • LENS achieves the worst performance across the reported tasks, despite its ViT-H-14 vision encoder lacking a comparable performance shortfall.The authors hypothesize that the performance gap may be associated with LENS’s Flan-T5 base language model rather than its vision encoder.

5. EMT on Multimodal Large Language Models Fine-Tuning

The fine-tuning experiments examine how LLaVA’s performance changes across image datasets and training stages. Early or moderate fine-tuning can generalize, but continued training produces hallucinations, overfitting, and reduced performance on non-fine-tuning datasets.

  • 5.1. Experimental Setup and Overview: The experiments fine-tune LLaVA-7b and LLaVA-13b on MNIST, CIFAR-10, CIFAR-100, and miniImagenet, using linear and LoRA fine-tuning.The reported evaluations cover three fine-tuning epochs and track accuracy across training stages.
  • 5.2. Excessive Fine-Tuning Causes Forgetting: Fine-tuning on one dataset decreases performance on another non-fine-tuning dataset, although some non-fine-tuned tasks can improve.For example, LLaVA-7b improves on miniImagenet after three epochs of fine-tuning on CIFAR-10.
  • 5.2. Excessive Fine-Tuning Causes Forgetting: Fine-tuning causes hallucinations by producing text related to the fine-tuning dataset while ignoring the original question.After fine-tuning on other datasets, LLaVA can hallucinate during CIFAR-10 classification; examples include extra labels or unrelated class representations.
  • 5.3. Moderate Fine-Tuning is Beneficial: Early linear fine-tuning on RGB datasets generalizes to other datasets, consistent with improved alignment between visual and textual features.Because linear fine-tuning changes only the projection layer, the authors associate early gains with feature alignment.
  • 5.3. Moderate Fine-Tuning is Beneficial: Subsequent linear fine-tuning epochs lead to overfitting and hallucinated texts, while jointly fine-tuning the LLM and adapter causes performance degradation after one epoch.The experiments also indicate that diverse fine-tuning data supports generalization, whereas MNIST fine-tuning degrades performance on the remaining datasets.

6. Conclusions

The paper proposes EMT to quantify catastrophic forgetting in MLLMs and finds that fine-tuning produces both early generalization benefits and later degradation through overfitting and hallucination.

  • EMT quantitatively evaluates MLLM fine-tuning performance by treating multimodal models as image classifiers.The paper presents EMT as its framework for studying catastrophic forgetting.
  • Almost all tested open-source MLLMs fail to match the classification accuracy of their base vision encoders.
  • Moderate fine-tuning improves LLaVA performance on similar tasks by aligning visual and textual features during early training.
  • Excessive fine-tuning causes LLaVA to overfit and hallucinate, degrading performance on datasets excluded from fine-tuning.

7. Discussions and Future Work

The discussion identifies dataset diversity as important for reducing forgetting and frames EMT as an image-classification starting point whose output evaluation can be extended.

  • Dataset Diversity is Important for Fine-Tuning: Fine-tuning on CIFAR10, CIFAR-100, or miniImagenet for one epoch generalizes to the other two datasets, whereas MNIST fine-tuning degrades performance on all remaining datasets.
  • Catastrophic Forgetting Beyond Image Classifications: The paper initially studies catastrophic forgetting through image classification and proposes developing analogous evaluations for unsafe outputs, visual localization, and hallucinations.
  • Post-processing the Outputs: EMT output correctness can also be evaluated with sentence embedding models rather than only the OpenAI API.

Appendices

The appendix defines the notation for multiclass classification, feature matrices, classifier weights, and cross-entropy analysis.

  • Notations: The appendix represents matrices with bold capital letters and vectors with bold lowercase letters, with v_i denoting vector v's ith entry.
  • Notations: For K-class classification, y_k denotes the kth one-hot label vector.
  • Notations: W denotes the K × d final-layer weight matrix, whose kth row is w_k.
  • Notations: H collects feature vectors for all inputs in R^d × N, with n_k samples in class k and N total samples.
  • Notations: The analysis focuses on cross-entropy loss with respect to one-hot labels.

A.1. Preliminary Results of Neural Collapse and Minority Collapse

The appendix connects catastrophic forgetting to neural-collapse geometry, imbalanced pre-training and fine-tuning classes, and cross-entropy optimization in classification and MLLM token prediction.

  • Neural Collapse: Neural collapse characterizes optimal classifier and feature geometry in balanced multiclass classification and has been studied for imbalanced and incremental learning.
  • Minority Collapse: Minority collapse occurs when minority-class feature vectors converge to a single vector as the majority-to-minority sample ratio increases.
  • Data Imbalance and the SELI geometry: SELI geometry describes optimal logits under unconstrained features, becoming simplex ETF geometry for balanced data and minority collapse under asymptotic imbalance.
  • Theoretical Formulation of NC: Neural collapse predicts maximally contrastive final-layer classifier weights in balanced classification.
  • Theoretical Formulation: The theoretical setting uses disjoint strict subsets of classes for pre-training and fine-tuning, making both phases infinitely imbalanced relative to the full label space.
  • Application to MLLMs: The analysis extends classification-based reasoning to MLLMs by viewing next-token generation as sequential classification over the vocabulary.

B.2. Additional Results of Training ResNet

Additional ablations show that replacing the classifier slightly mitigates catastrophic forgetting, whereas restarting the optimizer and scheduler does not affect it. Forgetting is less severe on MNIST than on CIFAR100 and miniImageNet.

  • Additional ablations: Using a new classifier during fine-tuning mitigates catastrophic forgetting to a certain extent.The experiment replaces the classifier rather than changing criterion weights.
  • Task complexity: Forgetting is less severe on MNIST, while its degree remains similar to the original results on CIFAR100 and miniImageNet.These observations indicate a relationship between catastrophic forgetting and task complexity.
  • Additional ablations: Restarting the optimizer and learning-rate scheduler during fine-tuning does not prevent catastrophic forgetting.The pre-trained-class accuracy curve remains nearly identical to the original result.

C.2. Fine-Tuning Dataset of Tested MLLMs

This section documents the datasets used to fine-tune or evaluate the tested MLLMs and illustrates their classification outputs under EMT. The examples include both correct predictions and outputs that diverge from the supplied labels.

  • Fine-tuning datasets: LLaVA, Otter, InstructBLIP, and LENS use different vision-language data or visual modules in their respective training setups.LLaVA uses CC3M and instruction-following data; Otter uses MIMIC-IT; InstructBLIP transforms public vision-language datasets; LENS uses visual modules with a frozen LLM.
  • Evaluation results: EMT reports classification accuracy for CLIP, Otter, LLaVA, OpenCLIP, LENS, and InstructBLIP across four tasks.The table compares each MLLM with its base vision model and sums performance gaps across all four tasks.
  • Classification outputs: Example outputs show mismatches such as Otter describing a digit as a letter and LLaVA predicting frog or bird for supplied labels.Other LLaVA examples match the labels, including deer and horse.
  • Classification outputs: InstructBLIP examples describe related but incorrect content, including a ladybug for harvestman and a street sign for street_sign.Some examples align more closely with their labels, such as a dog for African_hunting_dog and a coral reef for coral_reef.
  • Data construction: The LLaVA fine-tuning data pairs images with object-identification conversations, and experiments use data from MNIST, CIFAR10, CIFAR100, and miniImagenet.EMT evaluations use 20% randomly selected training data, while fine-tuning uses training data from all datasets.
  • Fine-tuning comparison: Linear fine-tuning preserves some visual-language binding, whereas LoRA fine-tuning directly leads to overfitting in the illustrated examples.The linear model gives several exact class predictions, while LoRA outputs related labels such as garbage truck, aircraft carrier, and rhinoceros beetle.

D.3. More Analysis on Hallucination after Fine-tuning

After fine-tuning, models preserve visual information but often map inputs to visually similar labels from the fine-tuning dataset. Additional CLIP-based post-processing reproduces the catastrophic-forgetting pattern, while each post-processing method has distinct limitations.

  • Hallucination patterns: Fine-tuned models usually predict a few fine-tuning-dataset labels that are visually similar to the ground-truth class.The reported examples include African_hunting_dog mapping to Dog or Deer, and Arctic_fox mapping to Dog or Cat.
  • Hallucination patterns: 75.21% of predictions are “Aircraft Carrier ship” and 24.79% are “The object is an airplane” for the reported example.This example concerns miniImagenet testing after CIFAR-10 fine-tuning.
  • Scope caveat: The reported prediction percentages cover only 10 classes in one miniImagenet-on-CIFAR10 setting because of space limitations.The authors state that the hallucinated outputs follow the visually similar-label pattern more generally.
  • Post-processing analysis: The CLIP text-feature post-processing method yields a similar catastrophic-forgetting phenomenon to the original post-processing method.The paper reports the same pattern across the additional tables, including cross-dataset forgetting and more severe degradation from LoRA than linear fine-tuning.
  • Post-processing limitations: CLIP embeddings select the most similar label but can classify an output such as “The airplane is 8.” as airplane despite ignoring its correctness.ChatGPT instead judges that malformed output as not making a classification.
Loading 2309.10313v4…