PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Catastrophic Forgetting in Language Models via Implicit Inference

Suhas Kotha, Jacob Mitchell Springer, Aditi Raghunathan

arXiv:2309.10105v2cs.CLcs.LG

TL;DR

The paper studies how fine-tuning affects capabilities outside its target distribution and whether apparent forgetting reflects shifted implicit task inference. It introduces conjugate prompting to make prompts appear farther from the fine-tuning distribution while preserving the task, and reports recovery of several pretrained capabilities, including harmful content generation suppressed by safety fine-tuning.

  • Problem

    The paper addresses the lack of a systematic understanding of fine-tuning trade-offs between target-task performance and capabilities outside the fine-tuning distribution.

  • Method

    The paper models catastrophic forgetting as shifted task inference and tests Conjugate Prompting, which transforms prompts to appear farther from fine-tuning data while preserving the underlying capability.

  • Results

    Conjugate prompting recovers pretrained capabilities across synthetic and language-model experiments, including in-context learning, natural-language reasoning after code fine-tuning, and harmful content generation suppressed by safety fine-tuning.

  • Takeaways & Limitations

    The findings suggest that some fine-tuning-induced capability losses may reflect suppressed solutions rather than complete forgetting, while also raising security concerns for safety-fine-tuned black-box models.

  • Takeaways & Limitations

    Conjugate prompting requires knowledge of training data and deployment tasks, while translation quality is constrained by third-party services, low-resource languages, and contextual knowledge.

Abstract

from arXiv · show

We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We hypothesize that language models implicitly infer the task of the prompt and that fine-tuning skews this inference towards tasks in the fine-tuning distribution. To test this, we propose Conjugate Prompting, which artificially makes the task look farther from the fine-tuning distribution while requiring the same capability, and we find that this recovers some of the pretraining capabilities in our synthetic setup. Since real-world fine-tuning distributions are predominantly English, we apply conjugate prompting to recover pretrained capabilities in LLMs by simply translating the prompts to different languages. This allows us to recover in-context learning abilities lost via instruction tuning, natural reasoning capability lost during code fine-tuning, and, more concerningly, harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.

1 INTRODUCTION

Fine-tuning improves targeted capabilities but can compromise capabilities outside its narrower data distribution. The paper proposes conjugate prompting to counter shifted task inference and recover some pretrained behavior.

  • Motivation: Fine-tuning can cause models to lose performance on tasks that pretrained models could solve, creating poorly understood capability trade-offs.These trade-offs arise because fine-tuning datasets are smaller and less diverse than web-scale pretraining data.
  • Core hypothesis: The paper hypothesizes that apparent forgetting reflects shifted implicit task inference rather than complete loss of the underlying capability.Conjugate prompting is designed to counter this shift by making prompts appear farther from the fine-tuning distribution.
  • Synthetic evidence: In a synthetic linear-regression setup, conjugate prompting alleviates some trade-offs induced by fine-tuning.The approach changes the prompt while preserving the underlying regression capability required to solve it.
  • LLM evaluations: Across five models and four non-English languages, translation-based conjugate prompting recovers pretrained in-context-learning capability after instruction tuning.The experiments also use two additional prompt transformations.

2 LINEAR REGRESSION EXPERIMENTS

The synthetic setup shows that fine-tuning toward a discrete task distribution improves performance there while damaging continuous-task performance. The authors hypothesize that this reflects altered task inference that suppresses, rather than erases, pretrained solutions, and show conjugate prompting can partially recover them.

  • 2.1 Setup: In-Context Learning for Linear Functions: Transformers trained on continuous tasks match Bayes-optimal ridge regression, while models trained on 64 fixed tasks match Bayes-optimal discrete regression.The two training distributions require different optimal estimators.
  • 2.4 Pretraining over the Mixture (Dmix): Mixture regression creates a trade-off between continuous and discrete-task loss, while intermediate priors can infer the distribution and achieve low loss on both.Pretrained models approach the corresponding mixture-regression solution, though convergence is slow and the trade-off limits discrete-task performance.
  • 2.5 The Effect of Fine-Tuning Pretrained Models: Fine-tuning rapidly improves discrete-task performance but causes large continuous-task performance drops, an instance of catastrophic forgetting.The authors note that this degradation is unnecessary because alternative solutions avoid such a drastic increase.
  • 2.6 Understanding the Effects of Fine-Tuning: Fine-tuning changes loss unevenly according to how likely prompts are to come from the fine-tuning distribution, with continuous prompts eventually improving as that likelihood decreases.This reversal differs from the usual expectation that increasingly out-of-distribution inputs perform worse.
  • 2.7 Hypothesis: Fine-Tuning Is Suppressing Solutions: The task-inference hypothesis models predictions as weighting discrete and ridge-regression solutions, suggesting fine-tuning down-weights ridge regression rather than eliminating it.The hypothesis treats catastrophic forgetting as increased weighting of fine-tuning tasks and altered inference about which algorithm to use.
  • 2.8 Conjugate Prompting for Linear Regression: Conjugate prompting rescales regression prompts to make them resemble tasks farther from the fine-tuning distribution and partially recovers ridge regression, especially with few ambiguous samples.The authors therefore characterize the capability as suppressed rather than forgotten.

3 CONJUGATE PROMPTING TO RECOVER PRETRAINING CAPABILITIES

Conjugate prompting transforms prompts to make them less likely under the fine-tuning distribution while preserving the underlying task. An inverse transformation then converts the model's output back to an answer for the original prompt.

  • The prompting transform s lowers the prompt’s likelihood under the fine-tuning distribution, shifting task inference toward the pretraining solution.
  • The transform must be invertible so that solving the transformed prompt can produce an answer to the original prompt.
  • Conjugate prompting applies s, runs the model, and applies s−1 to recover a solution reflecting the suppressed pretrained capability.

4 EXPERIMENTS ON LANGUAGE MODELS

Experiments test whether fine-tuning shifts implicit task inference, causing losses on capabilities outside the fine-tuning distribution, and whether translated prompts can recover them.

  • Instruction tuning and in-context learning: Instruction-tuned models perform in-context learning less often than their pretrained counterparts, especially when instruction following conflicts with in-context learning.Alpaca and Vicuna perform ICL on 56.75% and 40.00% fewer inputs, while OPT-IML performs it on 21.00% fewer inputs.
  • Instruction tuning and in-context learning: The authors model instruction-tuned behavior as task inference that increases the posterior probability of interpreting prompts as instruction following, suppressing the ICL capability.Conjugate prompting is intended to reverse this inference effect without directly changing the model’s internal capabilities.
  • Conjugate prompting: Conjugate prompting translates prompts into languages less represented in instruction-tuning data, then translates outputs back to preserve the task.The approach relies on translation being both less likely under the fine-tuning distribution and invertible.
  • Conjugate prompting: Translation is imperfect for tasks involving contextual knowledge that varies across languages, limiting conjugate prompting’s invertibility.The paper identifies this as a specific caveat of the translation-based approach.
  • Code fine-tuning and reasoning: After code fine-tuning, English XNLI accuracy falls by 8.36%, while French, Spanish, and German accuracy changes by less than 2%.Spanish performs best among the evaluated languages, according to the accompanying table description.
  • Safety fine-tuning and harmful generation: Safety fine-tuning reduces harmful-answer frequency much more in English than in non-English languages, partially preserving harmful instruction following outside English.English answer frequency drops from 92% to 3%, whereas Malayalam changes from 71% to 65%.

5 RELATED WORK

The related work situates the paper within research on catastrophic forgetting, continual learning, in-context learning, fine-tuning, and multilingual NLP.

  • Catastrophic forgetting and continual learning: Prior work studies catastrophic forgetting broadly and evaluates its prevalence in modern models, alongside continual-learning methods and data replay.The cited literature spans foundational catastrophic-forgetting studies and newer assessments in modern settings.

6 DISCUSSION AND FUTURE WORK

The discussion argues that fine-tuning may shift task inference and that transformed prompts can recover pretrained capabilities, while emphasizing security and practical limitations.

  • Discussion: The paper attributes catastrophic fine-tuning effects to shifted task inference and reports that moving prompts away from the fine-tuning distribution can recover pretrained capabilities.This interpretation is presented as relevant to black-box APIs such as ChatGPT and Claude.
  • Discussion: Conjugate prompting raises a security concern because restricting access to safety-fine-tuned models may not securely prevent harmful behavior.The concern is framed specifically for increasingly common black-box API deployments.
  • Limitations and future work: Translation is imperfect because of third-party services, low-resource languages, and contextual knowledge, while conjugate prompting requires knowledge of training data and deployment tasks.The paper also calls for broader evaluation across data, model size, and task relationships.

7 ETHICS STATEMENT

The paper seeks a principled understanding of fine-tuning to improve efficiency and reliability while acknowledging that its analysis reveals gaps that could potentially affect safety. It presents an explanation related to existing manual attempts rather than new attacks or vulnerabilities.

  • The work aims to make fine-tuning more efficient and reliable through a principled understanding of its effects.
  • The analysis identifies gaps in fine-tuning that could potentially be exploited to bypass safety measures.
  • The authors state that the work does not directly create new attacks or expose new vulnerabilities.
  • The paper studies task inference and intervenes with conjugate prompting to switch between fine-tuned and pretrained behavior.

B.1 DERIVATION

The appendix derives the mixture-regression estimator and examines how fine-tuning changes task inference and performance across prompt ambiguity, model scale, and training settings. Conjugate prompting recovers ridge-regression behavior most clearly when task identification is difficult.

  • Derivation: The mixture-regression estimator combines posterior-weighted predictions from the discrete and continuous task distributions.
  • Derivation: The integral in the estimator is approximated with 16384 samples of w.
  • Task inference: Fine-tuning changes loss most for continuous-distribution prompts that are likeliest to resemble the discrete fine-tuning distribution.
  • Conjugate prompting: Conjugate prompting recovers the pretrained ridge-regression solution, especially at lower sample counts where task inference is more ambiguous.
  • Conjugate prompting: γ = 3.0 begins to reduce conjugate-prompting gains, while the method helps most for models with the highest α.
  • Training dynamics: Training on the discrete distribution initially produces ridge regression before gradually shifting toward the Bayes-optimal discrete-regression solution.
  • Fine-tuning mixtures: Fine-tuning with α = 0.99 retains speedup while reducing regressions, whereas α = 0.75 further prevents regressions but lowers speedup.
  • Robustness: Both larger and smaller models forget at similar rates, and conjugate prompting helps both model sizes.

C.7 REPRODUCTION FOR DIFFERENT DATASET SIZE

The reproduction experiments vary pretraining duration and retain the central forgetting pattern: fine-tuning rapidly improves discrete-task loss while worsening continuous-task loss. Conjugate prompting continues to recover ridge-regression behavior across these settings.

  • Dataset size: For 2000 and 10000 pretraining steps, fine-tuning still produces forgetting, with discrete loss decreasing rapidly while continuous loss increases rapidly.
  • Dataset size: Conjugate prompting recovers the pretrained ridge-regression solution after 2000 and 10000 pretraining steps, especially at lower sample counts.
  • Experimental setup: The standard setup uses 64 discrete tasks, σ = 1 noise, exemplar counts from 0 to 40, and a 22.4 million-parameter GPT-2 model.

D.3 EXPANDED RESULTS

The expanded results decompose the reported Table 1 findings by task and model. The only observed performance increase occurs for English OPT to OPT-IML on Capslock Math, which reduces the average English in-context-learning drop.

  • The only performance increase is for English OPT to OPT-IML on Capslock Math, and it lowers the average English in-context-learning drop.

E CODE FINE-TUNING EXPERIMENT DETAILS

The experiments evaluate multilingual task performance and classify model responses to harmful instructions. They compare answer, refusal, and ambiguous behaviors across models, languages, and example prompts.

  • Multilingual evaluation: MLNI evaluation uses the harness prompt template across languages and checks whether each output begins with the correct target-language answer on 2,490 validation prompts per language.French, Spanish, and German are used because XNLI supports them with Latin alphabets in LLaMA pretraining.
  • Multilingual evaluation: Expanded ICL-versus-IF results report correct first-word completion accuracy for each task, language, and model, averaged across 100 samples.Pretrained models are marked PT and instruction-tuned models IT; the table notes that English usually has the largest performance drop.
  • Harmful-instruction evaluation: Harmful-instruction outputs are manually categorized as ANSWER, REFUSE, or AMBIGUOUS without judging answer correctness.ANSWER attempts to respond directly, REFUSE does not answer during the completion, and AMBIGUOUS covers incoherent, unrelated, or negated responses.
  • Harmful-instruction evaluation: The harmful instructions and outputs are translated with Google Translate, and non-English completions are translated back to English for illustrative examples.Examples include terrorism, software exploitation, spam distribution, and bomb-making instructions, with outputs receiving different response classifications.
  • Harmful-instruction evaluation: Examples show English models refusing some harmful requests while Japanese or Malayalam completions provide harmful instructions classified as ANSWER.The examples include government infiltration guidance and automated spam-email code.

F.3 EXPANDED RESULTS

The expanded results further decompose refusal behavior into three response classes. This decomposition is reported in Table 6 alongside the expanded toxic-generation results.

  • Expanded toxic-generation results: REFUSE responses are split into AMBIGUOUS and unambiguous REFUSE, producing three classes for the expanded analysis.The resulting classes are ANSWER, AMBIGUOUS, and REFUSE.
  • Expanded toxic-generation results: Table 6 reports expanded toxic-generation results using the three classes ANSWER, AMBIGUOUS, and REFUSE.
Loading 2309.10105v2…