Source-linked AI summary
Large Language Models Can Self-Improve
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han
TL;DR
LLM improvement typically depends on extensive supervised data, raising whether models can improve reasoning from unlabeled questions alone. The paper generates high-confidence Chain-of-Thought solutions through sampling and self-consistency, then fine-tunes on them. PaLM-540B improves across reasoning benchmarks and reaches state-of-the-art-level results without ground-truth labels.
Problem
Improving LLM performance beyond few-shot baselines typically requires extensive high-quality supervised datasets, while this paper studies self-improvement without supervised data.
Method
LMSI samples multiple Chain-of-Thought reasoning paths, selects high-confidence predictions by majority voting, and fine-tunes the LLM using mixed-format self-generated solutions.
Results
PaLM-540B improves accuracy by 1.1% to 7.7% across six datasets, achieving new state-of-the-art results on ARC, OpenBookQA, and ANLI without ground-truth labels.
Takeaways & Limitations
An LLM can improve reasoning and generalize to out-of-domain tasks by training on its own generated labels from input questions only.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.
1 INTRODUCTION
The paper investigates whether an LLM can improve its reasoning without supervised data. LMSI generates and filters self-produced reasoning solutions, then fine-tunes the model on them, improving in-domain and out-of-domain performance.
- The study asks whether an LLM can self-improve its reasoning using only input questions, without supervised output sequences.
- LMSI samples multiple Chain-of-Thought predictions, selects high-confidence answers by majority voting, and fine-tunes the model on those predictions.The method also mixes training formats to reduce overfitting to particular prompts or answer styles.
- PaLM-540B improved from 74.4% to 82.1% on GSM8K, 78.2% to 83.0% on DROP, 90.0% to 94.4% on OpenBookQA, and 63.4% to 67.9% on ANLI-A3.The reported gains include both training-task performance and out-of-domain improvements.
- The approach achieves competitive in-domain multi-task performance and out-of-domain generalization without supervised ground-truth answers.The paper reports state-of-the-art-level results on ARC, OpenBookQA, and ANLI datasets.
- The authors identify training with Chain-of-Thought reasoning formats as critical for successful self-improvement.Removing those formats still permits improvement, but substantially reduces the performance gain.
2 RELATED WORK
The related work situates LMSI among explanation-based learning, Chain-of-Thought reasoning, explanation refinement, self-training, and knowledge distillation. LMSI differs by using self-consistency-filtered reasoning solutions from unlabeled data for fine-tuning.
- Learning from explanations: Prior work uses human-annotated rationales to augment supervised learning, whereas this paper studies explanation-based learning without rationale-augmented training data.The related work notes that human-annotated rationales can be expensive.
- Few-shot explanations: Chain-of-Thought prompting generates intermediate reasoning steps, while self-consistency samples diverse paths and selects the most consistent answer by majority voting.
- Refining explanations: Other explanation-refinement methods assess or calibrate the reliability and satisfiability of generated reasoning paths.
- Self-training models: Conventional self-training assigns pseudo-labels to unlabeled examples, while LMSI uses Chain-of-Thought prompting and self-consistency to obtain high-confidence solutions.These solutions augment the fine-tuning process on a large set of unlabeled data.
- Distillation and dark knowledge: Knowledge distillation commonly transfers soft prediction targets, while LMSI directly uses imperfect Chain-of-Thought reasoning for self-improvement.
3 METHOD
The method generates multiple Chain-of-Thought reasoning paths for unlabeled questions, selects high-confidence answers by majority voting, and fine-tunes the model on the retained solutions. Mixed training formats and self-generated prompts are used to reduce prompt or answer-style overfitting and expand available training data.
- Generating and filtering multiple reasoning paths: The model samples multiple Chain-of-Thought reasoning paths and answers for each unlabeled training question, then selects the most consistent answer by majority voting.The retained paths are those that produce the majority-voted answer.
- Generating and filtering multiple reasoning paths: 56.5% → 74.4% on GSM8K test accuracy with self-consistency indicates potential for further improvement by training on self-selected reasoning paths.The method uses the gap between greedy and diverse decoding as motivation for self-training.
- Generating and filtering multiple reasoning paths: Higher confidence corresponds to more accurate majority-voted answers, while incorrect answers are typically supported by fewer reasoning paths.Confidence is defined as the fraction of sampled paths leading to the majority answer.
- Training with mixed formats: Each retained reasoning path is augmented into four prompt-and-answer formats to reduce overfitting to specific prompts or answer styles.The formats include Chain-of-Thought and standard prompting variants.
- Reducing human effort: When questions or human-written Chain-of-Thought examples are limited, the method investigates generating additional questions and prompts to increase self-training data.Question generation avoids requiring ground-truth answers, while prompt generation uses model-generated reasoning paths as examples.
4 EXPERIMENTAL SETUP
The experiments evaluate the method across arithmetic reasoning, commonsense reasoning, and natural language inference. They use GSM8K, DROP, OpenBookQA, ARC, and challenging ANLI subsets with a 540-billion-parameter autoregressive Transformer.
- Tasks and datasets: The evaluation covers arithmetic reasoning, commonsense reasoning, and natural language inference tasks.These task categories are represented by the listed benchmark datasets.
- Arithmetic reasoning: GSM8K and DROP are used for arithmetic reasoning, with DROP partitioned into football-related and non-football-related subsets for training.DROP requires numerical reasoning, while GSM8K is a math problem set.
- Commonsense reasoning: OpenBookQA and ARC Challenge are used to evaluate commonsense reasoning, and both contain multiple-choice questions.Only the ARC Challenge subset is used.
- Natural language inference: ANLI-A2 and ANLI-A3 evaluate natural language inference using entailment, neutral, and contradiction relations.These are the more challenging ANLI subsets compared with ANLI-A1.
- Models and training settings: The experiments use an autoregressive Transformer-based language model with 540 billion parameters and generate 32 reasoning paths per training question.Each path is augmented into four formats, producing up to 128×|Dtrain| final training samples.
5 RESULTS
LMSI improves PaLM-540B reasoning across in-domain and out-of-domain tasks using self-generated training data, with Chain-of-Thought formats and sampling choices materially affecting performance.
- Main results: LMSI outperforms previous state-of-the-art results on ARC-c, OpenBookQA, ANLI-A2, and ANLI-A3 without using ground-truth labels.On GSM8K and DROP, LMSI is close to methods using diverse prompting or supervised training while using fewer output paths or no labels.
- Main results: LMSI improves accuracy on all six out-of-domain tasks after training on questions from six in-domain datasets.The evaluated OOD tasks include arithmetic, commonsense, and natural-language-inference benchmarks.
- Ablations: Removing Chain-of-Thought reasoning paths preserves some self-improvement but substantially reduces the performance gain.The ablation compares training with four formats against training with direct answers only.
- Self-generated data: Self-training on generated questions still improves GSM8K reasoning, although real training-set questions produce better results.This evaluates the limited-data setting with ten real questions used to generate additional training questions.
- Self-generated prompts: Self-generated Step-by-Step examples substantially outperform the Step-by-Step baseline and nearly match human-written few-shot Chain-of-Thought prompts.At 40 paths, the reported accuracies are 74.2% versus 70.1% for the baseline and 74.4% for human-written prompts.
- Sampling studies: A sampling temperature of T = 1.2 performs best on both GSM8K and DROP after LMSI.The study evaluates temperatures from 0.7 to 1.5 and uses T = 1.2 for LMSI self-consistency across datasets.
- Sampling studies: After self-improvement, five self-consistency paths can exceed the performance of 32 paths from the unimproved model.Using more than 15 paths after LMSI yields only marginal additional improvements.
6 CONCLUSIONS
The paper concludes that an LLM can improve reasoning by training on its own generated labels from input questions alone. This self-improvement extends to generated questions and few-shot Chain-of-Thought prompts.
- Conclusion: An LLM improves reasoning-dataset performance by training on its own generated labels given input questions only.Using a 540-billion-parameter model, accuracy improves by 1.1% to 7.7% across six datasets without ground-truth labels.
- Conclusion: The method also supports self-improvement from self-generated questions and few-shot Chain-of-Thought prompts.The paper identifies combining generated data with existing supervised data as future work.
A.1 RESULTS ON UL2 MODEL
LMSI improves reasoning accuracy on most evaluated UL2 tasks, but its gains are smaller than those obtained with PaLM-540B.
- UL2 results: LMSI improves reasoning accuracy on most UL2 datasets under fixed fine-tuning settings.The setup generates 40 reasoning paths per training question and fine-tunes for 10,000 steps.
- UL2 results: LMSI improvements are smaller on UL2 than on PaLM-540B.The authors attribute this difference to larger models capturing more high-order semantics in implicit rationales.
A.2 CHAIN-OF-THOUGHT PROMPTS FOR EACH DATASET
The appendix lists dataset-specific few-shot Chain-of-Thought prompts and examples used for prompting and self-generated training samples.
- Prompt inventory: The appendix provides few-shot Chain-of-Thought prompts for OpenBookQA and several other reasoning datasets.Listed datasets include GSM8K, SVAMP, DROP, NLI tasks, ARC-challenge, AQUA, StrategyQA, and RTE.
- Prompt examples: The examples also illustrate direct lexical or semantic reasoning used to select yes, no, or uncertain answers.Additional examples involve synonymy, contradiction, and unsupported details.
- Prompt examples: The included examples pair premises or questions with concise rationale-augmented answers for natural-language inference tasks.Examples cover entailment, contradiction, and cases where the answer cannot be determined from the premise.