PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

Elita Lobo, Chirag Agarwal, Himabindu Lakkaraju

arXiv:2411.15382v2cs.CL

TL;DR

Fine-tuning’s impact on LLM reasoning has received little investigation despite its known side effects on other capabilities. This paper evaluates CoT performance and faithfulness after fine-tuning, finding that performance often declines, especially in smaller models, while faithfulness also declines in some settings.

  • Problem

    Although fine-tuning can cause forgetting and privacy and safety risks, its effects on LLM reasoning capabilities have been little studied.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The study evaluates CoT performance and faithfulness after supervised fine-tuning across reasoning and non-reasoning tasks, comparing pretrained and fine-tuned models.

    Source: §1 · p. 2

  • Results

    Fine-tuning generally reduces CoT performance, especially in smaller models, and fine-tuning smaller models on some non-reasoning or minimal-reasoning datasets tends to reduce CoT faithfulness.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fine-tuning’s effects on reasoning vary by model and dataset; smaller LLMs can show reduced accuracy on complex tasks and less faithful CoT reasoning.

    Source: §1 · p. 2, §6 · p. 9

    The authors call for evaluation across a wider range of reasoning and non-reasoning tasks and a broader variety of open-source models.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities that find applications across diverse domains. Despite their impressive performance, recent studies have highlighted the potential for significant enhancements in LLMs' task-specific performance through fine-tuning strategies like Reinforcement Learning with Human Feedback (RLHF), supervised fine-tuning (SFT), and Quantized Low-Rank Adapters (Q-LoRA) method. However, previous works have shown that while fine-tuning offers significant performance gains, it also leads to challenges such as catastrophic forgetting and privacy and safety risks. To this end, there has been little to no work in \textit{understanding the impact of fine-tuning on the reasoning capabilities of LLMs}. Our research investigates the effect of fine-tuning on the reasoning abilities of LLMs, addressing critical questions regarding the impact of task-specific fine-tuning on overall reasoning capabilities, the influence of fine-tuning on Chain-of-Thought (CoT) reasoning performance, and the implications for the faithfulness of CoT reasonings. By exploring these dimensions, our study shows the impact of fine-tuning on LLM reasoning capabilities, where the faithfulness of CoT reasoning, on average across four datasets, decreases, highlighting potential shifts in internal mechanisms of the LLMs resulting from fine-tuning processes.

1 Introduction

Fine-tuning adapts LLMs to specialized tasks, but its known side effects include degraded performance outside the target domain and safety and privacy risks. This paper examines the less-studied impact on reasoning, including CoT performance and faithfulness.

1 Introduction

  • Fine-tuning can degrade performance on tasks outside the target domain, weaken safety filters, and increase privacy-breach risk.The introduction identifies the effect of fine-tuning on reasoning capabilities as a comparatively unexplored question.Source: §1 · p. 1
  • The study asks whether fine-tuning changes CoT performance, CoT faithfulness, and general reasoning on specialized tasks.It reports that fine-tuning generally reduces CoT performance, with larger effects in smaller models, and that some fine-tuning datasets further reduce faithfulness in smaller models.Source: §1 · p. 2

1.1 Related Work

  • Prior work has examined fine-tuning’s effects on forgetting, privacy leakage, memorization, safety mechanisms, bias, and general knowledge, while CoT effects remained little studied.The authors identify CoT reasoning under fine-tuning as their focus.Source: §1.1 · p. 2

2 Preliminaries

The paper treats CoT as intermediate reasoning elicited before a final answer and evaluates both answer accuracy and whether those steps influence that answer.

2 Preliminaries

  • The evaluation considers both the accuracy of the final answer after CoT and the faithfulness of the reasoning steps.Faithfulness is assessed at the instance level using adapted metrics.Source: §2 · p. 2
  • The paper adapts three faithfulness tests to assess instances rather than only dataset-level results.The tests are Early Termination, Paraphrasing, and Filler Substitution.Source: §2 · p. 2
  • CoT prompting elicits intermediate reasoning steps before a final answer, either through added instructions or structured prompts.The section defines faithfulness as the degree to which reasoning steps influence the final answer rather than being post-hoc or unrelated.Source: §2 · pp. 2–3

3 Methods

The study uses supervised fine-tuning with QLoRA and compares pretrained and fine-tuned models on CoT accuracy and faithfulness. Its faithfulness tests modify reasoning chains and examine whether the final answer changes.

3 Methods

  • The study combines supervised fine-tuning with QLoRA, which trains low-rank adapters through a frozen 4-bit quantized model rather than updating all parameters.The experiments fine-tune on reasoning and non-reasoning question-answering tasks, then compare task performance before and after fine-tuning.Source: §3 · p. 3

3.1 Accuracy of Chain-of-Thought Reasoning

  • For CoT accuracy, models generate reasoning in a predefined format and then answer based on it; the final answers are compared between pretrained and fine-tuned models.The tested fine-tuning datasets include mathematics, common-sense reasoning, and medical tasks.Source: §3.1 · p. 3

3.2 Faithfulness of Chain-of-Thought Reasoning

  • Faithfulness is tested through early termination, paraphrasing, and filler substitution, checking whether modifying reasoning changes the final answer.Early termination progressively cuts the chain; paraphrasing changes wording in later steps; filler substitution replaces later steps with tokens such as “(…)”.Source: §3.2 · pp. 3–4

4 Experimental Setup

The experiments evaluate three language models across medical, commonsense, and math reasoning datasets, fine-tuning on one dataset and testing across all four. They assess answer accuracy and CoT faithfulness, including how results change when reasoning traces are perturbed.

Datasets

  • The study uses MedQA and MedMCQA for medical reasoning, CosmosQA for commonsense reasoning, and GSM8K for math, evaluating each fine-tuned model on all four datasets.A test dataset is IID when it belongs to the same category as the fine-tuning dataset, and OOD otherwise.Source: §4 · p. 4

Models and implementation

  • The evaluated models are 4-bit quantized Llama-3-8b-Instruct, GPT-3.5-0125, and GPT-4.The main Llama experiments use QLoRA with rank-16 adapters.Source: §4 · p. 4

Implementation details

  • Fine-tuning uses SFT with QLoRA for Llama models; GSM8K training includes reasoning steps, while the other datasets use question-answer pairs without them.The stated goal is to study vanilla fine-tuning, but reasoning steps were added for GSM8K because question-answer-only tuning did not improve its accuracy.Source: §4 · p. 4
  • Each dataset is split into training, validation, and test sets; models are fine-tuned for two epochs, with hyperparameters tuned on validation data.The paper uses CoT prompt templates and the OpenAI API for GPT fine-tuning, and TRL for QLoRA fine-tuning of Llama.Source: §4 · p. 4

Evaluation

  • Faithfulness tests compare answers from perturbed and original CoTs; early termination and filler substitution are evaluated at 25%, 50%, and 75% of CoT steps.CoT Pred Match is the fraction of examples whose perturbed-CoT answer matches the original-CoT answer; higher values in these tests generally indicate less faithful reasoning.Source: §4 · p. 4, Figure 3 · p. 5, Figure 4 · p. 6
  • Figure 2 compares zero-shot accuracy across fine-tuning and test datasets, while Figures 3 and 4 compare early-termination and filler-substitution CoT Pred Match.Figure 2 reports that fine-tuning appears to decrease CoT performance on average, more so for smaller models.Source: Figure 2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 6

5 Results

Fine-tuning reduced CoT accuracy, especially for smaller LLaMA models, while its effects on general accuracy and CoT faithfulness varied across models and fine-tuning datasets. Across QLoRA ranks, fine-tuning on MedQA and CosmosQA reduced CoT faithfulness on several evaluation datasets.

5 Results

  • GPT-4 CoT Pred Match remained relatively stable as reasoning steps were cut earlier, and this stability suggests many steps may have little influence on final answers.GPT-4's GSM8K reasoning was often only two steps, while its reasoning for other datasets typically exceeded four steps; the authors relate these results to prior work suggesting larger models produce fewer faithful CoTs.Source: §5 · p. 8
  • Fine-tuning had minimal impact on GPT-4 CoT faithfulness, but had a more pronounced effect on LLaMA-3-8B-Instruct, especially after MedQA or CosmosQA fine-tuning.For LLaMA, these datasets generally increased CoT Pred Match, indicating reduced faithfulness; the authors hypothesize that less complex reasoning can lead to overfitting.Source: §5 · p. 8
  • GPT-4 showed higher CoT Pred Match in filler substitution than GPT-3.5-turbo-0125 and LLaMA, while LLaMA faithfulness patterns varied between filler substitution and paraphrasing.For LLaMA fine-tuned on CosmosQA or MedQA, CoT Pred Match generally increased in filler substitution and decreased on average in paraphrasing; the paraphrasing test compares answers after substituting CoT steps with paraphrases at α = 25, 50, or 75.Source: Figure 5 · p. 7, §5 · p. 8
  • GPT models generally gained accuracy across datasets, whereas LLaMA fine-tuned on MedQA or CosmosQA had larger GSM8K performance drops.The authors suggest smaller LLMs are more susceptible to reduced generalization after fine-tuning.Source: §5 · p. 8
  • CoT reasoning accuracy decreased significantly across tasks and LLaMA models after fine-tuning, which the authors attribute to overfitting.All three fine-tuned models also had lower CoT accuracy than their pretrained counterparts on GSM8K; reductions were more pronounced for smaller models.Source: §5 · pp. 6, 8
  • Across QLoRA ranks 8, 16, and 32, MedQA and CosmosQA fine-tuning reduced CoT faithfulness on CosmosQA, MedQA, and MedMCQA.The authors report that these results validate findings from their previous experiments.Source: §5 · p. 9

6 Conclusion

The paper finds that fine-tuning can reduce accuracy on complex tasks, sometimes worsen CoT performance, and compromise CoT faithfulness. Its conclusions are bounded by immature faithfulness metrics, unexamined newer prompting variants and internal mechanisms, and limited evaluation breadth.

6 Conclusion

  • Fine-tuning smaller LLMs on non-reasoning and commonsense reasoning datasets reduces accuracy on complex tasks, particularly math-based tasks.Fine-tuning may sometimes lead to a larger decline in CoT reasoning performance, especially on math datasets.Source: §6 · p. 9
  • Fine-tuning can compromise CoT faithfulness, but the analysis relies on metrics still in early development.The authors say findings may need revisiting as those metrics evolve.Source: §6 · p. 9
  • The study does not examine newer CoT prompting variants or in-context demonstrations in fine-tuned models.The authors note these newer CoT methods often involve increased computational costs.Source: §6 · p. 9
  • The paper lacks deeper analysis of fine-tuning’s effects on internal mechanisms and broader evaluation across reasoning and non-reasoning tasks and open-source models.The authors identify internal changes and their downstream impact on reasoning as an area for future study.Source: §6 · p. 9

A.1 Additional Related Works

Prior CoT work explores prompting strategies, multiple-path and structured reasoning, and methods aimed at improving faithfulness. The cited work also identifies potential unfaithfulness in CoT and leaves the effect of fine-tuning on advanced CoT techniques for future study.

A.1 Additional Related Works

  • Appending “Let’s think step by step” is a zero-shot way to elicit reasoning, while in-context examples can improve CoT quality.Source: §A.1 · p. 12
  • Self-Consistency samples multiple reasoning paths, ToT branches at each step, and Least-to-Most solves sub-problems sequentially (Wang et al., 2023; Yao et al., 2023; Zhou et al., 2023).A deductive framework verifies intermediate steps before the final answer (Ling et al., 2023).Source: §A.1 · p. 12
  • Prior work identifies positional bias and in-context examples as potential sources of CoT unfaithfulness (Turpin et al., 2024).Lyu et al. (2023) proposed generating symbolic reasoning chains with an LLM, then using a deterministic solver to derive the answer; the effect of fine-tuning on advanced CoT techniques remains future work.Source: §A.1 · p. 12

A.2 Additional Results

The additional results figures compare answer matching against full-CoT outputs under filler substitution, paraphrasing, and early termination. The supplied captions describe the tested models, datasets, and axes but do not report comparative outcomes.

A.2 Additional Results

  • Figure 7 compares answer-match percentages after filler substitution across α values, QLoRA ranks, and four datasets.The x-axis is the percentage of CoT steps before substitution; the y-axis is the rate of matching the full-CoT final answer.Source: Figure 7 · p. 12
  • Tables 1 and 2 give CoT examples from pretrained and CosmosQA-fine-tuned LLaMA-3-8B-Instruct models on MedQA and GSM8K.Source: Table 1 · p. 13, Table 2 · p. 14
  • Figure 8 compares full-CoT answer-match percentages when portions of the reasoning are paraphrased across α values, ranks, and datasets.The x-axis is the percentage of CoT steps substituted with paraphrases, and the y-axis is the percentage matching the full-CoT answer.Source: Figure 8 · p. 15
  • Figure 9 compares full-CoT answer-match percentages after early termination across α values, QLoRA ranks, and datasets.The x-axis is the percentage of CoT steps before termination; the y-axis is the percentage matching the full-CoT final answer.Source: Figure 9 · p. 15
  • Tables 3–5 show example prompts for early termination, filler substitution, and paraphrase faithfulness tests, respectively.Table 6 shows an example prompt for paraphrasing partial CoT.Source: Table 3 · p. 16, Table 4 · p. 16, Table 5 · p. 17

A.3.1 Code and Computational Costs

The experiments used 1,100 GPU hours on V100s.

A.3.1 Code and Computational Costs

Related papers

Loading 2411.15382v2…