PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at the Limitations of Instruction Tuning

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha

arXiv:2402.05119v5cs.CLcs.AI

TL;DR

The paper investigates underexplored limitations of instruction tuning and studies model changes across open-source datasets, models, and tuning paradigms. It finds that tuning does not reliably add knowledge, and that LoRA outperforms the compared methods for improving instruction tuning.

  • Problem

    Instruction tuning’s limitations and the transformations it induces in LLMs remain underexplored, with comprehensive evaluation and training-data visibility identified as gaps.

    Source: §1 · p. 1

  • Method

    The authors study open-source instruction-tuned models using varied datasets and training paradigms, combining human and GPT-4 evaluation with token-distribution analysis.

    Source: §1 · p. 2, Finding 1. LFT Responses align closely… · p. 3

  • Results

    LoRA mainly teaches response initiation, while full-parameter tuning degrades knowledge and increases dataset-linked hallucinations; LoRA also outperforms the compared improvement methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Responses relying on pre-trained knowledge remain superior to models learning new knowledge through instruction tuning on the studied open-source datasets.

    Source: §1 · p. 2

    The analysis focuses on open-domain, language-only instruction tuning and does not study DPO, RLHF, or retrieval-augmented generation.

    Source: §9 · p. 9

Abstract

from arXiv · show

Instruction Tuning (IT), the process of training large language models (LLMs) using instruction-response pairs, has emerged as the predominant method for transforming base pre-trained LLMs into open-domain conversational agents. While IT has achieved notable success and widespread adoption, its limitations and shortcomings remain underexplored. In this paper, through rigorous experiments and an in-depth analysis of the changes LLMs undergo through IT, we reveal various limitations of IT. In particular, we show that (1) IT fails to enhance knowledge or skills in LLMs. LoRA fine-tuning is limited to learning response initiation and style tokens, and full-parameter fine-tuning leads to knowledge degradation. (2) Copying response patterns from IT datasets derived from knowledgeable sources leads to a decline in response quality. (3) Full-parameter fine-tuning increases hallucination by inaccurately borrowing tokens from conceptually similar instances in the IT dataset for generating responses. (4) Popular methods to improve IT do not lead to performance improvements over a simple LoRA fine-tuned model. Our findings reveal that responses generated solely from pre-trained knowledge consistently outperform responses by models that learn any form of new knowledge from IT on open-source datasets. We hope the insights and challenges revealed in this paper inspire future work in related directions.

1. Introduction

The paper examines why instruction tuning’s open-domain benefits and transformations remain insufficiently understood, using experiments and multiple evaluation methods to study its limitations. It reports that LoRA fine-tuning mainly teaches response initiation, while full-parameter tuning can degrade knowledge and response quality.

  • Open-domain instruction-following limitations remain underexplored, amid gaps in comprehensive evaluation, understanding of model changes through tuning, and visibility into chat models’ training data.Source: §1 · pp. 1–2
  • The study investigates open-domain conversational agents in single-turn interactions using varied open-source datasets, models, and training paradigms, with human, GPT-4, and token-distribution evaluations.Source: §1 · p. 2
  • LoRA fine-tuning mainly teaches response initiation and draws most of its response from pre-trained knowledge, while full-parameter fine-tuning degrades knowledge and response quality.Source: §1 · p. 2
  • Pattern-copying through LoRA mainly learns stylistic tokens, while full-parameter tuning adapts more deeply to training data and usually harms factual correctness.Pattern-copying can sometimes support detailed and comprehensive answers.Source: §1 · p. 2
  • Full-parameter fine-tuning increases hallucinations by borrowing inaccurate tokens from conceptually similar IT-dataset instances, an effect more prevalent than with LoRA.Source: §1 · p. 2
  • LoRA outperforms the compared improvement methods, which improve over full fine-tuning but do not advance knowledge beyond pre-trained models.Source: §1 · p. 2

2. Experimental Setting

The experiments use several base LLMs, open-source instruction-tuning datasets, and both LoRA and full-parameter tuning. Evaluation combines open-ended generation tests, human assessment, GPT-4 scoring, and token-distribution analysis.

  • The study uses LLaMa-2 7B, 13B, and 70B, Mistral 7B, and Phi-1.5 1.3B base models; LLaMa-2 70B appears in only some experiments due to compute constraints.Source: §2 · p. 2
  • Fine-tuning data include synthetic Alpaca and MedInstruct, human-written LIMA and Dolly, and the mixed-source Tulu-V2-Mix.Source: §2 · p. 2
  • Evaluation primarily uses the first 800 Just-Eval-Instruct examples for open-ended generation, with MedInstruct-test used for MedInstruct-tuned models.The last 200 Just-Eval-Instruct examples are removed because safety alignment is outside the paper’s scope.Source: §2 · p. 2
  • LoRA approximates weight matrices with low-rank matrices to reduce tuned parameters, whereas standard full-parameter tuning adjusts all or most model weights.Source: §2 · p. 2
  • Figure 1 reports lower token-distribution shifts for LoRA than full-parameter tuning, and says LoRA’s shifts are unaffected by IT-dataset scale.Source: Figure 1 · p. 3
  • GPT-4 Turbo scores responses from 1 to 5 on helpfulness, clarity, factuality, depth, and engagement, alongside expert human evaluation.Source: §2 · p. 3
  • The paper distinguishes instruction tuning (IT), aimed towards aligning a model towards specific response characteristics, from general fine-tuning, which may improve capabilities or knowledge and may require datasets with different characteristics.Source: §2 · p. 3

3. IT is (currently) Not a Knowledge Enhancer

At current open-source scale, IT does not reliably enhance knowledge: LFT largely preserves pre-trained knowledge, while SFT shifts responses toward IT-learned content. Across the reported evaluations, responses relying on pre-trained knowledge outperform those based on newly learned information.

Finding 1. LFT Responses align closely with the original

3. IT is (currently) Not a Knowledge Enhancer

4. Pattern Copying (often) Hurts Performance

The section finds that copying response patterns from instruction-tuning data often harms factual quality, especially when models imitate lengthy styles without enough relevant knowledge. It also reports that simplifying training responses reduces hallucinations, though the human comparisons show both benefits and trade-offs.

4. Pattern Copying (often) Hurts Performance

  • LFT shifts mainly style and response-initiation tokens, whereas SFT shifts many token types; 81.2% of words starting with shifted tokens and 66.7% starting with marginal tokens were found in the IT dataset.The authors suggest overfitting may explain SFT's increased borrowing; they report that borrowed tokens are often inaccurate.Source: §4 · p. 6
  • Style imitation can prompt lengthy answers that hallucinate when pre-trained knowledge is insufficient, while factual correctness is harmed by inaccurate training-data tokens.Responses may begin factually and then add unsupported content to prolong the answer.Source: §4 · p. 6
  • Style imitation can improve responses when the model knows enough, but can also alter open-ended answers or prompt unsupported extensions.The reported cases include hallucinations that prolong an answer or add facts when subject knowledge is inadequate.Source: §4 · p. 6
  • In the human study, LIMA 1K responses were sometimes more detailed and preferred, while other comparisons included differing information or hallucinations absent from the LIMA-Simple response.The figure distinguishes five comparison categories, including cases where both responses were accurate and cases where both were inaccurate.Source: Figure 6 · p. 6

Proposed Solution: Simplifying responses in the IT

  • GPT-4 was used to simplify LIMA 1K responses, and the resulting model showed reduced hallucinations and higher factuality and helpfulness, though it could lack depth.The section proposes concise, accurate training responses as a way to reduce hallucinations associated with pattern copying.Source: Proposed Solution: Simplifying… · pp. 6–7
  • The authors conclude that response depth should be matched to a model's existing factual knowledge, since pattern copying can lead to borrowing training-set tokens.Their takeaway links comprehensiveness and depth in IT data to the model's available knowledge.Source: Proposed Solution: Simplifying… · p. 7

5. Causal Analysis of Hallucinations

The causal analysis examines whether hallucinated phrases are borrowed randomly or from conceptually related IT examples. The authors report that SFT hallucinations often trace to such examples, while noting that identifying exact causal sources and quantifying concept similarity are difficult.

5. Causal Analysis of Hallucinations

  • SFT can produce hallucinations even in concise responses and after IT when the base model had the correct knowledge; the authors link these errors to erroneous causal links with training examples.The reported examples distinguish these cases from hallucinations caused by extending responses or adding facts.Source: §5 · p. 7
  • About 72% of phrases marked as hallucinations appeared in the IT dataset, and about 89% of tokens in those phrases were shifted or marginal.The analysis treats hallucinations as effects and attributes of the source IT instance as potential causes.Source: §5 · p. 7
  • Evaluators found that borrowed tokens often came from IT examples describing analogous concepts, and similar hallucinations were also reported for coding and summarization.The examples include responses drawing from unrelated instructions about basketball tournaments, atoms, languages, or the Northern Lights.Source: §5 · pp. 7–8
  • Expert human evaluators found an average of 87% of 1,000 randomly chosen hallucinated phrases in SFT-model responses causally related to the IT dataset, versus 13.9% for 500 LFT phrases.Source: §5 · p. 8
  • Tracing the exact source of hallucinations is difficult with manual or automated methods, and the contribution of concept similarity is hard to quantify.The reported mean semantic similarity was 0.418, while only 61.6% of hallucination keywords appeared in the identified source instance.Source: §5 · p. 8

6. Methods to Improve IT are Ineffective

The comparison finds that simple LFT outperforms the tested SFT enhancement methods, including dataset filtering, more complex instructions, and embedding noise. This holds across averaged metrics and task types in the reported setting.

6. Methods to Improve IT are Ineffective

  • Across averaged individual metrics and task types, LFT outperformed SFT models using AlpaGasus, WizardLM, or NEFTune in the reported comparison.The compared methods were applied in settings suggested by their original papers; AlpaGasus and WizardLM use improved Alpaca datasets, while NEFTune also uses Alpaca.Source: §6 · p. 8
  • AlpaGasus, WizardLM, and NEFTune each outperformed SFT on Alpaca, but none surpassed the LFT model in the reported comparisons.The authors state that the exact benefits of these methods need deeper investigation.Source: §6 · p. 8
  • The authors hypothesize that increased hallucination from tokens borrowed from IT data may account for the performance drop, and report only slight LFT improvement with these methods in Table 13.The paper calls for further investigation before these techniques can enhance response quality.Source: §6 · p. 8

7. Related Work

Prior work examines style imitation, alignment, and simplified instruction data, but the authors distinguish their study by investigating the causes of IT limitations and the effectiveness of LFT. They relate their findings to the superficial alignment hypothesis, which separates knowledge acquisition in pre-training from interaction-format shaping during alignment.

7. Related Work

  • Concurrent studies report style imitation rather than knowledge transfer from proprietary-model datasets and find that alignment teaches style; one proposes in-context learning as an alternative to IT.The authors connect these findings to LFT responses drawing on pre-trained knowledge without SFT-like knowledge degradation.Source: §7 · p. 9
  • The paper relates its findings to the superficial alignment hypothesis that models gain knowledge during pre-training while alignment shapes interaction format.It frames LFT's use of pre-trained knowledge as support for this account.Source: §7 · p. 9
  • Unlike cited work on imitation and simplified task data, this study investigates causes of IT limitations, pattern copying, hallucination, and LFT's effectiveness.The cited simplified-task studies report performance comparable to training on original instructions.Source: §7 · p. 9

8. Conclusion

The paper identifies limits of instruction tuning: LFT does not scale, while SFT and pattern-copying increase hallucinations, and an LFT model outperforms methods proposed in the literature.

  • LFT does not scale, SFT and pattern-copying increase hallucinations, and an LFT model outperforms methods proposed in the literature.Source: §8 · p. 9

9. Limitations and Future Work

The analysis is limited to open-domain instruction following and general-purpose IT settings, leaving several domains, methods, and data types outside its scope.

  • The analysis covers open-domain instruction following; fine-tuning for specific domains or tasks might enable models to gain new skills and knowledge.Source: §9 · p. 9
  • The study covers unimodal language-only IT and does not examine DPO, RLHF, or retrieval-augmented generation.Source: §9 · p. 9
  • Findings cover general-purpose IT datasets and may not apply to task-tailored datasets, such as those designed to improve math reasoning.Source: §9 · p. 9

10. Reproducibility

The paper reports its training and analysis implementation and notes that exact replication depends on a particular GPT-4 version and prompt.

  • All evaluated LLMs were trained using LLaMA-Factory, and the token-distribution analysis code was borrowed from Lin et al. (2023).Source: §10 · p. 9
  • Exact replication of the reported numbers requires the evaluation prompt with gpt-4-1106-preview, whose availability is subject to OpenAI.Source: §10 · p. 9
  • Using another GPT-4 version is expected to preserve most trends and core findings, though not necessarily the exact reported numbers.Source: §10 · p. 9

Impact Statement

The impact statement connects the paper’s findings to accuracy, trust, and future research, while the accompanying appendices provide prompts, examples, datasets, analyses, and human-study details.

Impact Statement

  • The authors connect IT limitations to developing more robust and reliable conversational agents for education, customer service, and accessibility.Source: Impact Statement · p. 9

A. Prompts

B. Qualitative Examples

  • Examples describe hallucinated tokens borrowed from conceptually similar IT instances, including in non-information-seeking tasks; accurate open-ended answers can also draw on IT data.Source: B. Qualitative Examples · p. 13

C.1. Training

  • The training appendix describes datasets including Databricks-Dolly, LIMA, and MedInstruct, spanning general-purpose and medical instruction-response data.Source: §C.1 · p. 24

D.1. Hallucination Analysis

  • The hallucination analysis compares semantic similarity between instruction-response pairs and searches for shared one- to three-gram keywords.Source: §D.1 · pp. 24–25

D.2. Details of human study

  • Human studies used four Ph.D.-level professionals, with pilot evaluation, two batches per study, and author checks for consistency.Source: §D.2 · p. 25

D.3. Detailed Results

D.3 reports detailed quantitative results averaged across evaluation instances, separating results by evaluation aspect and task type. It also visualizes token-frequency distributions for SFT- and LFT-fine-tuned models.

D.3. Detailed Results

  • Tables 13–15 report results averaged across all evaluation instances for separate aspects of the multi-aspect evaluation strategy.The supplied passages identify the tables' scope but do not provide their numerical results.Source: §D.3 · p. 25, Table 13 · p. 26, Table 14 · p. 27
  • Tables 16–17 report results averaged across aspects, divided by task type in just-eval-instruct 1k.The listed task types are coding, info-seek, math, procedure, reasoning, role-play, and writing.Source: §D.3 · pp. 25, 29, 30, Table 16 · p. 29, Table 17 · p. 30
  • Figure 12 compares frequency distributions of shifted and marginal tokens for LLaMa-2 7B trained on LIMA 1K, evaluating SFT and LFT on just-eval-instruct 1k.The figure is presented as a comparison of knowledge learned during fine-tuning; no distribution values or outcome are supplied here.Source: Figure 12 · p. 31

Related papers

Loading 2402.05119v5…