Source-linked AI summary
STaR: Bootstrapping Reasoning With Reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman
TL;DR
Rationale generation can improve complex reasoning, but existing approaches require costly rationale datasets or sacrifice accuracy with few-shot prompting. STaR bootstraps rationales by iteratively generating and fine-tuning on correct solutions, including rationalized failures. It improves performance across reasoning tasks and reaches 72.5% on CommonsenseQA versus 73.0% for a 30× larger fine-tuned model.
Problem
Existing rationale-generation methods require expensive rationale datasets or few-shot prompting that generally underperforms larger-dataset direct-answer fine-tuning.
Method
STaR iteratively generates rationales, rationalizes incorrectly answered problems using their correct answers, and fine-tunes on the resulting correct solutions.
Results
STaR significantly improves generalization on symbolic and natural-language reasoning and achieves 72.5% on CommonsenseQA versus 73.0% for a 30× larger model.
Takeaways & Limitations
STaR allows a pre-trained language model to iteratively use its language-modeling capacity to improve its own rationale generation.
Takeaways & Limitations
STaR requires above-chance few-shot performance initially, and high-chance settings can produce poor rationales that confound the approach.
Abstract
from arXiv · showhide
Generating step-by-step "chain-of-thought" rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by using only few-shot inference. We propose a technique to iteratively leverage a small number of rationale examples and a large dataset without rationales, to bootstrap the ability to perform successively more complex reasoning. This technique, the "Self-Taught Reasoner" (STaR), relies on a simple loop: generate rationales to answer many questions, prompted with a few rationale examples; if the generated answers are wrong, try again to generate a rationale given the correct answer; fine-tune on all the rationales that ultimately yielded correct answers; repeat. We show that STaR significantly improves performance on multiple datasets compared to a model fine-tuned to directly predict final answers, and performs comparably to fine-tuning a 30$\times$ larger state-of-the-art language model on CommensenseQA. Thus, STaR lets a model improve itself by learning from its own generated reasoning.
1 Introduction
STaR addresses the limitations of manually or automatically constructing rationale datasets and the reduced accuracy of few-shot rationale prompting. It bootstraps rationale generation through iterative self-training and improves reasoning performance across tasks.
- Explicit rationales can improve language-model performance across mathematical, commonsense, code, social-bias, and natural-language-inference tasks.
- Manual rationale datasets are expensive, while template-based methods require known solutions or effective hard-coded heuristics.
- Few-shot rationale prompting improves over direct prompting but generally underperforms larger-dataset models fine-tuned to predict answers directly.
- STaR iteratively prompts a model to generate rationales, fine-tunes on correct solutions, and repeats the process with the improved model.The loop leverages the model’s pre-existing reasoning ability and creates a synergistic cycle between training data and rationale generation.
- Rationalization supplies correct answers to failed problems so the model can generate backward-justifying rationales for training.These additional rationales expose the model to difficult problems that would otherwise be absent from its fine-tuning data.
- +35.9% over few-shot prompting and +12.5% over direct-answer fine-tuning were observed on CommonsenseQA; STaR reached 72.5% versus 73.0% for a 30× larger model.
2 Background and Related Work
Related work frames STaR within in-context learning, rationale-based training, iterated learning, and natural-language explanations. These lines of research respectively study prompt-based pattern learning, explicit reasoning supervision, iterative solution reuse, and post-hoc justification.
- In-context Learning: In-context learning treats few-shot learning as language modeling over examples placed in the prompt.Researchers have analyzed it through Bayesian inference and mechanistic accounts involving induction heads.
- Rationales: Rationale-based training improves final-answer generation but traditionally requires many thousands of human-annotated reasoning examples.
- Iterated Learning: Iterated learning algorithms reuse successful solutions or methods to discover additional solutions, including Expert Iteration’s apprentice–expert loop.
- Natural Language Explanations: Natural-language explanation work emphasizes explainable decision making and generally does not find that post-hoc explanations improve model performance.
3 Method
STaR starts with a pretrained language model, a large problem-and-answer dataset, and a small prompt set containing rationale examples. It repeatedly generates rationales, filters for correct answers, adds rationalized explanations for failures, and fine-tunes on the combined data.
- Rationale generation bootstrapping: STaR begins with a pretrained LLM, a dataset of problems and answers, and a much smaller prompt set containing examples with intermediate rationales.The prompt set is concatenated to each dataset example to encourage rationale generation followed by an answer.
- Rationale generation bootstrapping: The model generates a rationale and answer for each problem, retaining only rationales whose generated answers match the ground-truth answers.The filtered rationales form the fine-tuning dataset for the next iteration.
- Rationale generation bootstrapping: STaR repeats rationale generation and fine-tuning with the newly improved model, treating correct-answer filtering as an approximation to an RL-style policy-gradient objective.The method greedily decodes rationale-answer samples and takes multiple gradient steps on the same batch.
- Rationalization: Rationalization addresses failed problems by providing the correct answer as a hint, generating a backward rationale, then adding it to the training data without the hint.The method uses this process for problems the model fails to solve through ordinary rationale generation.
- Rationalization: Rationalization exposes the model to difficult problems absent from its ordinary fine-tuning data and increases the dataset size.The paper describes this as challenging the model to think outside the box about unsuccessful problems.
4 Experiments
STaR is evaluated on arithmetic, commonsense reasoning, and grade-school math using GPT-J and several direct-answer and few-shot rationale baselines. Across these tasks, iterative rationale bootstrapping improves performance, while rationalization accelerates arithmetic learning and helps expose failed problems to training.
- Experimental Setup: STaR is evaluated on arithmetic, CommonsenseQA, and GSM8K using GPT-J and comparisons with direct-answer and few-shot rationale baselines.The arithmetic task sums two n-digit integers; CommonsenseQA tests multiple-choice commonsense reasoning; GSM8K contains grade-school math problems.
- Arithmetic: 89.5% overall arithmetic accuracy follows 16 STaR iterations, versus 76.3% for a 10,000-example no-rationale baseline.Few-shot rationale accuracy was below 1% for 2-digit addition and close to zero for longer numbers.
- Arithmetic: 32% 2-digit-addition accuracy follows one rationalization iteration, rising from less than 1%.Without rationalization, improvements are stagewise across digit lengths; with it, the model can learn multiple lengths at once.
- Arithmetic: Rationalization improves STaR's arithmetic learning speed by generating rationales for failed problems using their correct answers.These additional rationales provide training data for problems the model could not solve directly.
- CommonsenseQA: 72.5% CommonsenseQA accuracy with rationalization approaches 73.0% from a fine-tuned model 30× larger and exceeds direct-answer and few-shot baselines.STaR without rationalization already outperformed GPT-J fine-tuned to predict final answers directly.
- GSM8K: STaR substantially improves GSM8K over few-shot rationale and direct-answer training, while model-generated calculation-step counts match human counts 53%–57% of the time.The GSM8K results use 36 iterations without rationalization and 10 additional iterations with it.
5 Discussion and Challenges
The discussion examines rationalization, sampling temperature, and few-shot prompting as design choices that affect STaR’s learning signal, efficiency, style consistency, and performance. It also identifies open questions about hinting techniques and when rationalization improves learning.
- The Impact of Rationalization: Rationalization can improve learning by helping the model reverse-engineer solutions from known answers, while its relationship to reinforcement-learning objectives remains an open question.The paper contrasts rationale generation from p(r | x) with rationalization and calls for further study of when and why rationalization helps.
- The Impact of Rationalization: The method for adding hints does not follow directly from every question-answer pair, making hinting nontrivial in some contexts.The paper identifies the generality and impact of different hinting techniques as future work.
- Temperature: Higher-temperature sampling consistently produced worse models than reasoning alone because incorrect reasoning can accompany correct answers and undermine generalization.In arithmetic, higher-temperature scratchpads may become meaningless and cause training to stagnate.
- Temperature: Generating 10 sample outputs is approximately 10 times slower than generating one, making higher-temperature sampling less computationally efficient than rationalization.
- Few-shot Prompting: Few-shot prompting during sampling dramatically reduces rationale drift from the initial examples, supporting more consistent prompt-style adherence.
- Few-shot Prompting: Removing few-shot prompts after the initial iteration makes rationalization gradually worse during longer training, so extended training may require hints.
- Few-shot Prompting: The choice to retain few-shot prompts trades faster training against consistent prompt-style adherence and should be treated as a dataset- and model-dependent hyperparameter.The paper notes that larger models or other datasets may change the performance impact.
6 Conclusion
STaR iteratively improves rationale generation by training on correct and rationalized solutions, and the paper reports stronger generalization across symbolic and natural-language reasoning. Its applicability is limited when initial few-shot reasoning is near chance or many poor rationales arise.
- 6 Conclusion: STaR iteratively trains models to generate rationales by combining initially correct solutions with rationalized solutions for problems answered incorrectly.
- 6 Conclusion: STaR requires above-chance few-shot performance in its first iteration, so the initial model must possess some reasoning capability.GPT-2 could not bootstrap from few-shot reasoning even in arithmetic.
- 6 Conclusion: High-chance settings such as binary decisions can produce many poor rationales, leaving how to filter bad reasoning as an open problem.
- 6 Conclusion: Using examples without reasoning to bootstrap reasoning is presented as a general approach that may support more sophisticated techniques across domains.
A CommonsenseQA Error Patterns
The paper identifies reasoning failures in which CommonsenseQA answers are correct, but the rationales used to reach them are problematic.
- Correct final answers can still be accompanied by flawed reasoning in commonsense reasoning.
A.1 Question Implies Answer
Some rationales effectively restate the question’s property and answer without explaining why the selected answer satisfies it. These rationales may therefore be opaque and unexplanatory.
- The model may repeat the question property and assert that the chosen answer satisfies it without explaining why.
- For example, it may say that learning science increases knowledge and then select “increased knowledge.”
- Such rationales can be useful to the model while remaining opaque and unexplanatory to readers.
A.2 Begging the Question
The paper describes stronger forms of question-begging in which the model’s reasoning either implies the answer in the question or offers little justification beyond restating the answer.
- In a stronger but less common failure case, the model implies its chosen answer within the question.
- A wire catching the singer’s foot is used to restate that he would stumble, then select “stumble” as the answer.
- In a rare variant, the model considers justification unnecessary and simply gives an answer.
- For example, it selects “meeting” without providing a reason that it is a place to gain knowledge.
A.4 World State Assertions
The paper reports failures where the model makes unsupported or unhelpful assertions about the world, including assumptions about named entities and statements that do not demonstrate the answer.
- The model may assume knowledge about a subject or person named as a variable, producing comical reasoning examples.
- It may assert that a bottle is broken after a hand hits it, without explaining the relevant consequence.
- It may infer exhilaration from heights for James while treating a general association as sufficient support.
- Some reasoning statements are technically true but do not help demonstrate the claim.
- A response can identify a tree because squirrels climb trees while failing to connect that fact to where a dog would bark.
- The model may also use a generic claim that running is light-footed to justify “lightly,” without addressing the question’s contrast.
A.6 Hint Short-cutting
Hint-shortcutting experiments found that rationalization can make generated answers follow the supplied hint rather than provide reliable reasoning. Human evaluation compared explanations while treating the answer letter as authoritative, even when answer text and letter conflicted.
- Hint Short-cutting: Hint-based training sometimes caused the model to copy the hinted answer rather than independently reason.The observed outputs could contradict their own explanations, indicating shortcutting toward the hinted answer.
- Evaluation Assumption: For one example, the rationale argues for an airport but ends with train station (e), illustrating answer-text and answer-letter inconsistency.The evaluation protocol explicitly prioritized the answer letter in such cases.
- Human Evaluation: Twenty crowdworkers ranked explanations for 50 questions that were correctly answered by both few-shot prompting and STaR without rationalization.The examples were presented in random order, with explanations from the few-shot model, STaR-trained model, and a human.
- Human Evaluation: The evaluation instructed workers to rank three explanations from best to worst according to how well they justified the predetermined correct answer.Workers were told to treat the answer letter as correct even when another answer seemed more reasonable or conflicted with the preceding answer text.
- Rationalization Quality: The sampled rationalizations often used a generic property-matching template, resembling template matching more than reasoning when the model had initially answered incorrectly.The recurring structure states that the answer must have certain properties, asserts that the correct answer has them, and concludes that it is correct.
F Case Study
The case study examines how STaR-generated rationales change across iterations, including newly solved problems and improved rationales for problems already solved. For CommonsenseQA, rationale quality requires qualitative judgment because multiple choices may be technically correct.
- Examples: The examples include ordinary commonsense questions such as identifying Seattle, a warm place, or a frame as the best answer.The rationales connect each answer to a property of the question’s situation.
- Case Study: The case study includes problems that the model previously could not solve but later answered, alongside already-correct answers whose rationales improved.These categories are illustrated as separate case-study patterns.
- Rationale Improvement: Some rationales become more explicit after training, adding a supporting explanation before repeating the selected answer.For example, the university rationale adds that universities are places where people learn.
- Evaluation: CommonsenseQA rationale quality is harder to assess than arithmetic because evaluation is necessarily qualitative and several answers may be technically correct.The case study therefore focuses on coherence and structural similarity to the few-shot rationales.
G Bias and Faithfulness
STaR can amplify dataset-useful biases, while the faithfulness of its rationales remains difficult to establish. The paper reports limited encouraging observations about gender-neutral explanations but calls for more comprehensive study.
- Bias: STaR may amplify biases that are useful for solving a dataset, especially when rationalization elicits answers the model would not naturally produce.The interaction between dataset bias and pre-existing model bias remains unclear before real-world deployment.
- Bias: For questions where gender is irrelevant, the model appears to disregard gender in its explanations.The paper gives examples involving storing sunglasses and checking baggage, while stating that broader study is needed.
- Faithfulness: Faithfulness is a central challenge because a rationale may justify a selected answer without accurately representing the model’s internal reasoning.A model could implicitly choose an answer first and then generate a rationale, including an apparently unbiased rationale for a biased choice.
- Faithfulness: The paper reports that rationale generation improves answer quality, but evaluating whether any individual rationale is faithful remains difficult.The authors describe this difficulty as a broader challenge in understanding large language models, not only a STaR-specific issue.
J STaR GSM8K Solutions
STaR-generated GSM8K solutions sometimes use fewer calculation steps than the ground truth. These shorter solutions may reflect accidental correctness from faulty reasoning, but some provide genuinely simpler solutions.
- GSM8K Solutions: STaR solutions with substantially fewer calculation steps often correspond to accidentally correct answers despite reasoning mistakes.The paper distinguishes these cases from genuinely simpler solutions.
- Example Problem: The case-study problem asks how many bottles reach the first house when half of each drink category is delivered there.The problem distinguishes bottles containing only cider, only beer, or a mixture.
- GSM8K Solutions: Figure 8 presents a training-set problem where STaR derives a significantly simpler solution than the ground truth.The figure is offered as an example of a shorter solution that is not merely explained by an identified reasoning mistake.