Source-linked AI summary
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models
Leonardo Ranaldi, Andrè Freitas
TL;DR
Demonstration-based tuning can generalize weakly when it trains on limited examples of reasoning paths. The paper transfers reasoning from LLM teachers to SLM students, then applies preference optimization for self-refinement. The authors report improved alignment and performance over instruction-tuning across in-domain and out-domain tasks.
Problem
Training SLMs on limited LLM-generated reasoning demonstrations can generalize weakly, although multiple valid Chain-of-Thought paths may solve the same question.
Source: §1 · p. 1Method
Results
Self-refined students outperform instruction-tuned students in in-domain and out-domain tasks, with improved out-domain performance even with fewer demonstrations.
Source: §4 · p. 5Takeaways & Limitations
The reported findings support improved teacher–student alignment of multi-step reasoning across in-family and out-family settings.
Source: §6 · p. 8The paper notes that CoT steps do not guarantee that a correct prediction came from the reasoning process they appear to represent, which may make end users overconfident.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
The alignments of reasoning abilities between smaller and larger Language Models are largely conducted via Supervised Fine-Tuning (SFT) using demonstrations generated from robust Large Language Models (LLMs). Although these approaches deliver more performant models, they do not show sufficiently strong generalization ability as the training only relies on the provided demonstrations. In this paper, we propose the Self-refine Instruction-tuning method that elicits Smaller Language Models to self-refine their abilities. Our approach is based on a two-stage process, where reasoning abilities are first transferred between LLMs and Small Language Models (SLMs) via Instruction-tuning on demonstrations provided by LLMs, and then the instructed models Self-refine their abilities through preference optimization strategies. In particular, the second phase operates refinement heuristics based on the Direct Preference Optimization algorithm, where the SLMs are elicited to deliver a series of reasoning paths by automatically sampling the generated responses and providing rewards using ground truths from the LLMs. Results obtained on commonsense and math reasoning tasks show that this approach significantly outperforms Instruction-tuning in both in-domain and out-domain scenarios, aligning the reasoning abilities of Smaller and Larger Language Models.
1 Introduction
Demonstration-based fine-tuning can generalize weakly when it relies on limited examples of reasoning paths, despite multiple valid paths for a question. The paper proposes transferring reasoning to smaller models with teacher demonstrations, then refining them with preference optimization.
1 Introduction
- Demonstration-based fine-tuning may generalize weakly from limited examples, even though multiple valid Chain-of-Thought annotations can solve the same question.The authors argue this motivates a more general CoT-based fine-tuning approach.Source: §1 · p. 1
- Self-refine Instruction-tuning first transfers step-wise reasoning to SLMs through LLM demonstrations, then reinforces the instructed models with preference optimization.The approach is framed as a teacher–student alignment method for CoT reasoning.Source: §1 · p. 1
- The method uses DPO heuristics to let students sample reasoning paths and learn from self-generated preferences, without a separately trained reward model.The experiments include LLM teachers from the same or different model families.Source: §1 · p. 2
- The paper evaluates generalization using in-family and out-family teachers across in-domain and out-domain tasks.Its experiments cover mathematical problems and commonsense question answering.Source: §1 · p. 2
2 Method
Self-refine Instruction-tuning transfers step-wise reasoning from LLMs to SLMs through teacher demonstrations, then uses preference optimization to refine the instructed students. The method combines instruction-conditioned learning with student-generated reasoning preferences.
2 Method
- The method first transfers step-wise CoT reasoning from LLMs to SLMs through Instruction-tuning on teacher-generated demonstrations, then self-refines the instructed students with preference optimization.The two phases initialize CoT alignment and subsequently refine the students’ internal CoT model.Source: §2 · p. 2
2.1 Instruction-tuning Phase
- Teacher demonstrations pair an instruction and question with an expected answer and CoT, to transfer foundational problem-solving abilities and instruction-following to the student.The student is fine-tuned on teacher-produced tuples of instruction, question, and answer/CoT.Source: §2.1 · p. 3
2.2 Self-refinement Phase
- The preference objective aims to shift the student’s default response style toward CoT answers when elicited to provide reasoning.The method assumes CoT responses are preferable when prompted for CoT, while ordinary responses are preferred for ordinary prompts.Source: §2.2 · p. 3
- In the refinement phase, students repeatedly sample answers, and DPO-based self-generated annotations make student-produced CoT responses preferred.The proposed DPOCoT configuration treats self-generated answers that deliver a CoT response as preferred.Source: §2.2 · p. 3
3 Experimental Setup
The experiments evaluate reasoning across commonsense, interaction, and mathematics benchmarks, with additional adaptability tests. The setup describes teacher–student training and a string-matching evaluation pipeline.
3.1 Tasks & Datasets
- Evaluation covers commonsense QA (CSQA, OBQA), physical and social interaction (PIQA, SIQA), arithmetic and math word problems (MultiArith, GSM8K), plus MATH and MMLU adaptability benchmarks.The paper uses the additional MATH and MMLU benchmarks to evaluate adaptability.Source: §3.1 · p. 4
- For SIQA, PIQA, CSQA, and OBQA, 4,000 examples with balanced target classes form training data and validation sets serve as test data; GSM8K and MultiArith use their full Hugging Face datasets.The paper notes that not all benchmarks prescribe a test split.Source: §3.1 · p. 4
3.2 Self-refine Instruction-tuning Pipeline
- Zero-shot LLM teachers generate answers used in demonstrations to instruct student models; after teacher-student CoT transference, an optimization technique is employed to improve alignment and self-refine generation quality.Teachers include Llama-2-70, Mixtral7x8, and GPT-3.5; students include Llama2-7, Llama2-13, and Mistral-7.Source: §3.2 · p. 4
3.2.1 Models Setup
- Instruction-tuning uses QLoRA for four epochs at learning rate 2e-5 and weight decay 1e-4; DPO uses learning rate 1e-6, β=0.1, batch size 128, and at most 1,000 steps.The selected checkpoint is the one with the lowest validation loss; experiments ran on four Nvidia RTX A6000 GPUs.Source: §3.2.1 · p. 4
- Figure 2 compares benchmark accuracies before tuning, after Instruction-tuning, and after Self-refine Instruction-tuning, using demonstrations from in-family LLM teachers.The figure legend uses method(Teacher->Student) notation.Source: Figure 2 · p. 5
3.3 Evaluation
- Evaluation uses multiple-choice probing with direct string matching between generated outputs and target choices, reporting the percentage of correct answers.The authors choose this method for a comparable, scalable pipeline that also accommodates GPT-family models without readily accessible probability values.Source: §3.3 · p. 5
4 Results & Discussion
Across the reported settings, Self-refine Instruction-tuning improves on Instruction-tuning and strengthens alignment between student and teacher models. The section examines initial transfer, refinement, generalization across task domains, and performance with fewer demonstrations.
4 Results & Discussion
- Self-refined students outperform Instruction-tuned students in in-domain and out-of-domain tasks, with out-family demonstrations also yielding more robust students.Figure 3 reports accuracies across baseline CoT, Instruction-tuned, and Self-refine IT stages for out-family teachers.Source: §4 · p. 5, Figure 3 · p. 6
4.1 The Instruction-tuning alignment
- Instruction-tuned students outperform baselines, but their performance remains below teachers, with average gaps varying by family and task.Reported gaps are 6.3 points on QA and 8.2 on math word problems for in-family alignment, versus 8.5 and 8.7 for out-family alignment.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
4.2 The Self-refine Impact
- Self-refinement brings in-family students to teacher-comparable performance and narrows out-family teacher-student gaps.The reported out-family comparison includes GPT-3.5 as a teacher; the passage also notes improved quality of student generations.Source: §4.2 · p. 6
4.3 In-Domain and Out-Domain
- Self-refined students outperform baselines and Instruction-tuned models on in-domain and out-of-domain tasks, including cross-setting optimization.In cross-setting evaluations, students optimized on a task other than their warm-up task outperform other models and obtain performance similar to in-domain models.Source: §4.3 · pp. 6–7, Table 1 · p. 7
4.4 Low-resource Optimization
- Self-refined students consistently outperform non-self-refined students across the tested low-resource demonstration splits.The combination of Instruction-tuning and DPO achieves the best results across the proposed in-family and out-family splits.Source: §4.4 · p. 7
5 Related Work
Related work covers methods for multi-step reasoning, reinforcement-learning refinement, and transferring reasoning from larger to smaller models. This paper positions its approach as combining demonstration-based instruction-tuning with DPO-based self-refinement and evaluating alignment across teacher-student families.
5.1 Multi-step Reasoning
- Prior CoT work studies prompting and specialized interventions, while teacher-student approaches transfer larger-model reasoning to smaller models through distillation or task-specific tuning.The cited examples include Python-program prompts and verification of intermediate reasoning steps.Source: §5.1 · pp. 7–8
5.2 Reinforcement Learning (RL)
- Prior reasoning-refinement work uses reward models or reinforcement learning; this paper applies DPO with automated rewards in a teacher-student setting.The proposed refinement operates on models downstream of Instruction-tuning and targets CoT-style, step-wise reasoning.Source: §5.2 · p. 8
5.3 Self-refined Instruction-tuning
- The paper’s stated contributions include demonstrations from different LLM teachers, in-family and out-family analysis, and adapting DPO to address teacher-student performance gaps.These contributions are framed as extensions to work transferring multi-step answers through supervised fine-tuning.Source: §5.3 · p. 8
6 Conclusion
The conclusion presents Self-refine Instruction-tuning as a two-stage approach for aligning multi-step reasoning between LLM teachers and SLM students. It describes the reported outcome as improved reasoning properties and self-refinement that completes alignment.
6 Conclusion
- Self-refine Instruction-tuning combines LLM-generated CoT demonstrations for instruction-tuning with a subsequent DPO self-refinement phase.Source: §6 · p. 8
- The approach contrasts in-family and out-family alignment across teacher and student models.Source: §6 · p. 8
- The reported conclusion is that instruction-tuning improves smaller models’ multi-step reasoning properties and self-refinement promotes alignment completion.Source: §6 · p. 8
Limitations
The study identifies limits in language coverage and in understanding how teacher-model training data shape comparisons. It also cautions that plausible chain-of-thought explanations may not reflect the process behind a correct answer.
Limitations
- The analysis is limited to English-language prompting and tasks, leaving understanding across languages unestablished.The authors state that methods across different languages still need to be studied.Source: Limitations · p. 9
- Because teacher models’ supporting training sets are not always fully known, pre-training differences cannot be fully contextualized.The authors say their analysis is consequently constrained to observing outputs in natural language.Source: Limitations · p. 9
- A seemingly consistent chain of reasoning may accompany a correct prediction without representing the process that produced it, risking user overconfidence.The authors describe this as a risk of learning from and with demonstrations.Source: Limitations · p. 9
Ethical Statement
The methods use teacher-generated demonstrations to instruct students, then preference optimization to refine their reasoning. The ethical statement cautions that further safety and out-of-distribution generalization work is needed before applying the approach in critical domains.
Ethical Statement
- Students still need greater robustness for critical domains, alongside further safety and out-of-distribution generalization mechanisms.The authors say these mechanisms need to be developed in tandem with applying the described methods.Source: Ethical Statement · p. 9
A Instruction-tuning
- Instruction-tuning follows annotation and knowledge-transfer steps, using teacher-generated instruction, input-question, and output triples to train students.Instructions guide task-related generation; instruction and input may be concatenated depending on task and prompt configuration.Source: A Instruction-tuning · p. 12
B Self-refine Instruction-tuning
- The preference setup compares student answers generated with and without a chain-of-thought prompt against target answers.The paper defines target answers for the input and demonstration answers for prompted inputs; DPO variants use these generated responses to structure preferences.Source: B Self-refine Instruction-tuning · p. 12
- DPOCoT treats student answers with correct chain-of-thought as preferred over answers without chain-of-thought.The paper focuses on DPOCoT in the main work and reports DPOanswer results separately in Table 4.Source: B Self-refine Instruction-tuning · p. 12
C.1 Data Splitting
The appendix describes evaluations across reduced demonstration splits, teacher and student configurations, and prompt formats. It also identifies the teacher-accuracy comparisons and the test-set accuracy figure used to report the experimental setup.
C.1 Data Splitting
- Demonstration data were progressively reduced across splits, and models were evaluated using demonstrations for instruction-tuning, self-refinement, and both phases.The passage describes random sampling from the available demonstrations and gives approximate example counts, though its split labels and counts are inconsistent.Source: §C.1 · p. 13
C.2 Parameters
- Teacher demonstrations were generated on training data, while teachers, students, and baselines were evaluated on test splits using question probing and string matching.Teacher prompts were customized by benchmark; student and baseline prompts followed the teacher format.Source: §C.2 · p. 13
- Figure 4 reports test-set benchmark accuracies for Self-refine Instruction-tuning across different data splits.The caption refers readers to Appendix C.1 for details of the splits.Source: Figure 4 · p. 13
D Prompting Approaches
- Tables 2 and 3 show example input prompts for multiple-choice and mathematical question-answering benchmarks, with Table 3 presenting the zero-shot chain-of-thought approach used in this work.Source: Table 2 · p. 14, Table 3 · p. 14
E Models
- The listed models include specific versions, and each uses the default configurations provided in its repository.The models are listed in Table 4.Source: Table 4 · p. 14
F Accuracy of LLMs on different Benchhmark
- Teacher accuracy is reported for training and test data under both chain-of-thought and baseline prompts.Table 5 covers Llama-2-70, GPT-3.5, and Mixtral7x8; the supplied passage gives no individual accuracy values.Source: Table 5 · p. 15
G Description of proposed Benchmark
The appendix describes benchmark resources and evaluation settings, including in-domain and out-of-domain tests. It also outlines a GPT-4-based assessment of generated answer quality and a comparison table of prior methods.
G Description of proposed Benchmark
- Tables 7 and 8 provide benchmark examples and dataset characteristics, while Table 9 lists benchmark versions.The supplied captions do not give the individual examples, statistics, or versions.Source: Table 7 · p. 16, Table 8 · p. 16, Table 9 · p. 16
- Tables 10 and 11 describe in-domain, out-of-domain, and cross-setting evaluations for Llama-2-13 and Mistral-7 instruction-tuned models.The captions distinguish QA-to-QA from QA-to-math-word-problem evaluation and identify baselines as non-instructed models.Source: Table 10 · p. 17, Table 11 · p. 17
H Quality of Generations
- Generation quality was assessed by prompting GPT-4 to judge responses from teachers and students on 300 test questions, with 50 questions per task.Tables 13 and 14 report average quality scores for teacher and student answers, respectively; the supplied passages give no scores.Source: H Quality of Generations · p. 18, Table 13 · p. 18, Table 14 · p. 18
- Table 15 summarizes prior methods by teacher and student models and marks whether they use SFT, instruction-tuning, or reinforcement learning.Its caption notes that the prior works listed do not use DPO.Source: Table 15 · p. 19
I Additional Evaluations
Figure 5 reports additional benchmark evaluations of the Self-refine Instruction-tuning approach, which the authors describe as scalable to the multi-task benchmarks MATH and MMLU.
I Additional Evaluations
- The authors report that their approach is scalable on the multi-task benchmarks MATH and MMLU.Figure 5 presents accuracies for these additional benchmarks.Source: Figure 5 · p. 19
- The additional evaluation uses the same pipeline and experimental setup as the Figure 2 and Figure 3 experiments.Self-refine Instruction-tuning used 25% of the training set and was omitted during evaluation.Source: Figure 5 · p. 19
- Figure 5 labels each method using the Teacher->Student notation.Source: Figure 5 · p. 19