Large Language Models Can Self-Improve

• The study asks whether an LLM can self-improve its reasoning using only input questions, without supervised output sequences.Source: §1 · p. 1
• LMSI samples multiple Chain-of-Thought predictions, selects high-confidence answers by majority voting, and fine-tunes the model on those predictions.The method also mixes training formats to reduce overfitting to particular prompts or answer styles.Source: §1 · p. 1, Figure 1 · p. 2
• PaLM-540B improved from 74.4% to 82.1% on GSM8K, 78.2% to 83.0% on DROP, 90.0% to 94.4% on OpenBookQA, and 63.4% to 67.9% on ANLI-A3.The reported gains include both training-task performance and out-of-domain improvements.Source: §1 · p. 2
• The approach achieves competitive in-domain multi-task performance and out-of-domain generalization without supervised ground-truth answers.The paper reports state-of-the-art-level results on ARC, OpenBookQA, and ANLI datasets.Source: §1 · p. 2
• The authors identify training with Chain-of-Thought reasoning formats as critical for successful self-improvement.Removing those formats still permits improvement, but substantially reduces the performance gain.Source: §1 · p. 2

Learning from explanations

• Prior work uses human-annotated rationales to augment supervised learning, whereas this paper studies explanation-based learning without rationale-augmented training data.The related work notes that human-annotated rationales can be expensive.Source: §2 · pp. 2–3

Few-shot explanations

• Chain-of-Thought prompting generates intermediate reasoning steps, while self-consistency samples diverse paths and selects the most consistent answer by majority voting.Source: §2 · p. 3

Refining explanations

• Other explanation-refinement methods assess or calibrate the reliability and satisfiability of generated reasoning paths.Source: §2 · p. 3

Self-training models

• Conventional self-training assigns pseudo-labels to unlabeled examples, while LMSI uses Chain-of-Thought prompting and self-consistency to obtain high-confidence solutions.These solutions augment the fine-tuning process on a large set of unlabeled data.Source: §2 · p. 3

Distillation and dark knowledge

• Knowledge distillation commonly transfers soft prediction targets, while LMSI directly uses imperfect Chain-of-Thought reasoning for self-improvement.Source: §2 · p. 3

Generating and filtering multiple reasoning paths

• The model samples multiple Chain-of-Thought reasoning paths and answers for each unlabeled training question, then selects the most consistent answer by majority voting.The retained paths are those that produce the majority-voted answer.Source: §3 · p. 3, §3.1 · p. 4
• 56.5% → 74.4% on GSM8K test accuracy with self-consistency indicates potential for further improvement by training on self-selected reasoning paths.The method uses the gap between greedy and diverse decoding as motivation for self-training.Source: §3.1 · p. 4
• Higher confidence corresponds to more accurate majority-voted answers, while incorrect answers are typically supported by fewer reasoning paths.Confidence is defined as the fraction of sampled paths leading to the majority answer.Source: §3.1 · p. 4, Figure 2 · p. 4

Training with mixed formats

• Each retained reasoning path is augmented into four prompt-and-answer formats to reduce overfitting to specific prompts or answer styles.The formats include Chain-of-Thought and standard prompting variants.Source: §3.2 · p. 5, Table 2 · p. 5

Reducing human effort

• When questions or human-written Chain-of-Thought examples are limited, the method investigates generating additional questions and prompts to increase self-training data.Question generation avoids requiring ground-truth answers, while prompt generation uses model-generated reasoning paths as examples.Source: §3.3 · pp. 5–6

Tasks and datasets

• The evaluation covers arithmetic reasoning, commonsense reasoning, and natural language inference tasks.These task categories are represented by the listed benchmark datasets.Source: §4 · p. 6

Arithmetic reasoning

• GSM8K and DROP are used for arithmetic reasoning, with DROP partitioned into football-related and non-football-related subsets for training.DROP requires numerical reasoning, while GSM8K is a math problem set.Source: §4 · p. 6

Commonsense reasoning

• OpenBookQA and ARC Challenge are used to evaluate commonsense reasoning, and both contain multiple-choice questions.Only the ARC Challenge subset is used.Source: §4 · p. 6

Natural language inference

• ANLI-A2 and ANLI-A3 evaluate natural language inference using entailment, neutral, and contradiction relations.These are the more challenging ANLI subsets compared with ANLI-A1.Source: §4 · p. 6

Models and training settings

• The experiments use an autoregressive Transformer-based language model with 540 billion parameters and generate 32 reasoning paths per training question.Each path is augmented into four formats, producing up to 128×|Dtrain| final training samples.Source: §4 · p. 6

Main results

• LMSI outperforms previous state-of-the-art results on ARC-c, OpenBookQA, ANLI-A2, and ANLI-A3 without using ground-truth labels.On GSM8K and DROP, LMSI is close to methods using diverse prompting or supervised training while using fewer output paths or no labels.Source: §5.1 · p. 7
• LMSI improves accuracy on all six out-of-domain tasks after training on questions from six in-domain datasets.The evaluated OOD tasks include arithmetic, commonsense, and natural-language-inference benchmarks.Source: §5.1 · p. 7, Table 4 · p. 7

Ablations

• Removing Chain-of-Thought reasoning paths preserves some self-improvement but substantially reduces the performance gain.The ablation compares training with four formats against training with direct answers only.Source: §5.1 · p. 7, Table 5 · p. 8

Self-generated data

• Self-training on generated questions still improves GSM8K reasoning, although real training-set questions produce better results.This evaluates the limited-data setting with ten real questions used to generate additional training questions.Source: §5.2 · p. 8, Table 6 · p. 8

Self-generated prompts

• Self-generated Step-by-Step examples substantially outperform the Step-by-Step baseline and nearly match human-written few-shot Chain-of-Thought prompts.At 40 paths, the reported accuracies are 74.2% versus 70.1% for the baseline and 74.4% for human-written prompts.Source: §5.2 · pp. 8–9, Figure 3 · p. 8

Sampling studies

• A sampling temperature of T = 1.2 performs best on both GSM8K and DROP after LMSI.The study evaluates temperatures from 0.7 to 1.5 and uses T = 1.2 for LMSI self-consistency across datasets.Source: §5.4 · p. 9, Figure 4 · p. 9
• After self-improvement, five self-consistency paths can exceed the performance of 32 paths from the unimproved model.Using more than 15 paths after LMSI yields only marginal additional improvements.Source: §5.4 · pp. 9–10

Conclusion

• An LLM improves reasoning-dataset performance by training on its own generated labels given input questions only.Using a 540-billion-parameter model, accuracy improves by 1.1% to 7.7% across six datasets without ground-truth labels.Source: §6 · p. 10
• The method also supports self-improvement from self-generated questions and few-shot Chain-of-Thought prompts.The paper identifies combining generated data with existing supervised data as future work.Source: §6 · p. 10

UL2 results

• LMSI improves reasoning accuracy on most UL2 datasets under fixed fine-tuning settings.The setup generates 40 reasoning paths per training question and fine-tunes for 10,000 steps.Source: §A.1 · p. 14, Table 8 · p. 14
• LMSI improvements are smaller on UL2 than on PaLM-540B.The authors attribute this difference to larger models capturing more high-order semantics in implicit rationales.Source: §A.1 · p. 14

Prompt inventory

• The appendix provides few-shot Chain-of-Thought prompts for OpenBookQA and several other reasoning datasets.Listed datasets include GSM8K, SVAMP, DROP, NLI tasks, ARC-challenge, AQUA, StrategyQA, and RTE.Source: §A.2 · p. 14, Table 9 · p. 14, Table 10 · p. 15

Prompt examples

• The examples also illustrate direct lexical or semantic reasoning used to select yes, no, or uncertain answers.Additional examples involve synonymy, contradiction, and unsupported details.Source: §A.2 · p. 17
• The included examples pair premises or questions with concise rationale-augmented answers for natural-language inference tasks.Examples cover entailment, contradiction, and cases where the answer cannot be determined from the premise.Source: §A.2 · p. 17