STaR: Bootstrapping Reasoning With Reasoning

• Explicit rationales can improve LLM reasoning, but rationale datasets are costly or require known solutions or heuristics, while few-shot rationale prompting generally trails fine-tuning on larger datasets.Source: §1 · p. 1
• STaR iteratively generates rationales with a few-shot prompt, fine-tunes on those leading to correct answers, and repeats with the improved model, without needing to check new rationales’ correctness.Source: §1 · p. 2
• For problems it fails, STaR uses rationalization to generate a rationale given the correct answer, then includes that rationale in training; this often improves accuracy and accelerates bootstrapping.Source: §1 · p. 2
• The authors present STaR as a scalable method for pretrained LLMs to improve themselves iteratively while learning to solve increasingly difficult problems.Source: §1 · p. 2
• On CommonsenseQA [10], STaR improves over few-shot prompting by +35.9% and direct-answer fine-tuning by +12.5%, and scores 72.5% versus 73.0% for a fine-tuned model 30× larger.Source: §1 · p. 2

In-context Learning

• In-context learning uses prompted examples to induce patterns, while STaR emphasizes the model’s output and its ability to reason before concluding.Prior work examines Bayesian accounts [13], induction heads [14], and prompt configuration, including optimized soft prompts [15].Source: §2 · p. 3

Rationales

• Rationale research ranges from annotated training data [3] and scratchpads [5] to few-shot chain-of-thought prompts [6] and curriculum-based formal math solving [16].The curriculum approach [16] requires Lean translations, directly evaluable proofs, numerous sampled solutions, a separate value function, and GPT-f initialization [18]; the paper notes these conditions do not all apply in many domains. Other work analyzes rationale benefits through latent-variable models [19] and intermediate task supervision [20].Source: §2 · p. 3

Iterated Learning

• STaR resembles expert iteration [21] by filtering generated examples against target answers, but uses a fixed expert and no separate value function.ExIt [21] loops apprentice self-play, expert-feedback imitation, and expert replacement; related iterated-learning work applies the approach to formal reasoning [16] and compositional visual question answering [22].Source: §2 · p. 3

Natural Language Explanations

• Natural-language explanation work focuses on justification rather than reasoning and generally does not find that post-hoc explanations improve performance [23] [24].Its motivation is largely explainable decision making, paralleling [3].Source: §2 · p. 3

3.1 Rationale Generation Bootstrapping (STaR Without Rationalization)

• With a small rationale prompt set (P ≪ D, e.g. P = 10), STaR generates rationales and answers, keeps correct-answer examples, and fine-tunes the original model until performance plateaus.Each round trains from the original pretrained model rather than continually fine-tuning the previous model, to avoid overfitting.Source: §3.1 · p. 3
• STaR approximates an RL-style policy-gradient objective by greedily decoding to reduce variance, accepting potentially biased rationale exploration, and taking multiple gradient steps on the same batch.The indicator reward discards gradients for rationales that do not produce the correct answer; the approximations enable implementation with standard LLM training machinery.Source: §3.1 · p. 4

3.2 Rationalization

• Because ordinary bootstrapping gets no training signal from failed examples, rationalization uses the answer as a hint to generate a rationale, an approach inspired by [3].Rationalization is applied to problems the model failed to solve; the hint is omitted from the rationale's training prompt, and the resulting examples are combined with the previously generated dataset after filtering.Source: §3.2 · p. 4
• Rationalization adds difficult problems that would otherwise be absent from fine-tuning and also increases the dataset size.The method frames these examples as prompting the model to reason about problems on which it was unsuccessful.Source: §3.2 · p. 4

4.2 Datasets

• The evaluation covered n-digit addition, five-choice CommonsenseQA, and GSM8K word problems requiring two to eight calculation steps.Arithmetic prompts used scratchpads that track digit sums and carries; CommonsenseQA draws on ConceptNet, and GSM8K combines arithmetic and commonsense reasoning.Source: §4 · p. 5, §4.2 · pp. 5–6, Figure 3

4.1 Experimental Protocol

• Experiments used publicly available 6B-parameter GPT-J; training began with 40 steps and increased steps by 20% per outer loop, with a 100-step learning-rate warmup.The authors found slower initial training beneficial and left a thorough hyperparameter search for future work due to computational constraints.Source: §4.1 · p. 5

4.3 Symbolic Reasoning: Results on Arithmetic

• 89.5% overall accuracy after 16 STaR iterations exceeded the 76.3% accuracy of a baseline trained without rationales.After one rationalization iteration, 2-digit addition improved from less than 1% to 32%; rationalization also enabled learning multiple digit lengths at once.Source: Figure 4 · p. 6, §4.3 · p. 6

4.4 Natural Language Reasoning: Commonsense Question Answering

• 72.5% on CommonsenseQA with rationalization approached GPT-3’s 73%, and STaR surpassed the few-shot baselines, including 137B LaMDA.STaR without rationalization also outperformed GPT-J fine-tuned to predict final answers, despite training on less data.Source: Table 1 · p. 7, §4.4 · p. 7
• STaR-generated CommonsenseQA rationales were generally coherent, and crowdworkers were 30% more likely to rank them above few-shot rationales (p = .039).Participants were 74% more likely to prefer STaR rationales over human-generated rationales (p < .001), which the authors say does not demonstrate human-level rationale generation.Source: §4.4 · p. 7

4.5 Mathematical Reasoning in Language: Grade School Math

• STaR substantially improved GSM8K performance over few-shot and direct-answer baselines, while rationalization did not substantially improve performance on this task.Training was capped at the 30th iteration (7,912 steps) to limit runtime; results followed 36 iterations without rationalization and 10 additional iterations with it.Source: Table 2 · p. 8, §4.5 · p. 8

4.4 Natural Language Reasoning: Commonsense Question Answering

• CommonsenseQA rationale failures included relevant but non-argumentative statements, circular claims that the question implied its answer, and mistaken assumptions about individuals.Some errors, especially early in training, reflected specific-person assumptions rather than general statements.Source: §4.4 · p. 8
• Including few-shot prompts during fine-tuning raised CommonsenseQA performance from 60.9% to 68.8% without rationalization and from 69.9% to 72.5% with it.The authors generally recommend including prompts during at least some training, while noting caveats discussed in Section 5.Source: §4.4 · p. 8

The Impact of Rationalization

• Rationalization may help reverse-engineer solutions by conditioning rationale generation on the answer, accessing a potentially better search space than sampling rationales from p(r | x).The authors frame this as a possible off-policy estimate and call for research connecting rationalization to reinforcement-learning objectives and identifying when it improves learning.Source: §5 · p. 8
• Without rationalization, low-temperature samples favor the model’s most confident answers and may provide weaker gradient signals initially; hint construction can also be nontrivial.Because each fine-tuning iteration restarts from the pretrained model, the gradient effect is difficult to measure directly; the effects and generality of hinting techniques remain open questions.Source: §5 · p. 9

Temperature

• Higher-temperature sampling (e.g., 0.5 or 0.7) consistently produced worse models than reasoning alone, as bad scratchpads promoted incorrect answers and hindered generalization.Generating 10 outputs is approximately 10 times slower than generating one; majority-vote scratchpads could provide a ground truth for low-temperature rationales on question-only datasets.Source: §5 · p. 9

Few-shot Prompting

• Few-shot prompts appear to reduce rationale drift and shorten sampling sequences, but can shift rationale style away from the original prompt and their quality may constrain generalization.The point at which prompts are disabled is a tunable hyperparameter.Source: §5 · p. 9
• After prompts are removed, rationalization tends to worsen with longer training, so hints may need to remain; prompt use depends on whether style adherence or faster training is the priority.The authors recommend treating prompt use as a hyperparameter because other datasets or larger models may show performance effects.Source: §5 · p. 9

6 Conclusion

• STaR significantly improves generalization performance on both symbolic reasoning and natural-language reasoning.The conclusion describes these as results of the iterative technique.Source: §6 · p. 9
• The STaR loop few-shot prompts a model to generate step-by-step rationales for many problems, then asks it to rationalize correct answers for problems it initially gets wrong.Source: §6 · p. 9
• STaR fine-tunes on initially correct solutions and rationalized correct solutions, then repeats the process.Source: §6 · p. 9
• STaR’s first iteration requires few-shot performance above chance, so the initial model must already have some reasoning capability.GPT-2 could not bootstrap from few-shot reasoning even in arithmetic.Source: §6 · p. 9
• Tasks with high chance performance, such as binary decisions, can produce many poor rationales that confound STaR; filtering bad reasoning remains open.Source: §6 · p. 9
• The authors propose reasoning bootstrapped from examples without reasoning as a general approach and a basis for more sophisticated techniques across domains.Source: §6 · p. 9

A CommonsenseQA Error Patterns

• Across experiments, the authors found commonsense-reasoning failure cases despite correct final answers, focusing their concern on the reasoning process.They describe these cases as a variety of interesting failures.Source: A CommonsenseQA Error Patterns · p. 13

A.1 Question Implies Answer

• The model sometimes asserts that its chosen answer satisfies the question’s property but fails to explain why.This was identified as a key failure case.Source: §A.1 · p. 13
• For example, a rationale says learning science increases knowledge, then selects “increased knowledge” without explaining why that answer fits the question property.Source: §A.1 · p. 13
• The authors characterized these rationales as opaque and unexplanatory.Source: §A.1 · p. 13

A.2 Begging the Question

• A.2 identifies answer-implying questions as a stronger but less common version of the preceding failure case.The accompanying example asks what the singer had after his foot got caught on a wire, then repeats “stumble” as both the rationale and answer.Source: §A.2 · p. 13
• In the example, the rationale assumes the singer would stumble and concludes that the answer is “stumble.”The question’s description of the singer’s foot getting caught on a wire already points toward that answer.Source: §A.2 · p. 13

A.3 Exercise to the Reader

• A.3 describes as rare the failure to provide a justification for an answer.Its example gives “meeting” as the answer to where one can gain knowledge, without explaining why.Source: §A.3 · p. 13
• In the A.3 example, the model answers “meeting” but provides no supporting reasoning for that choice.The question asks where one can gain knowledge other than school or from text.Source: §A.3 · p. 13

A.4 World State Assertions

• The model sometimes assumes knowledge about a subject or person named as a variable, producing comical reasoning; general-class explanations are presented as more compelling than claims about a particular person.For example, “a person would typically feel exhilaration from heights” is described as more compelling than “James would feel exhilaration from heights.”Source: §A.4 · p. 14
• In one example, the rationale says the bottle is broken and selects “breaking” as the answer to what might happen to the bottle.The example illustrates a direct bottle-related assertion in response to the question.Source: §A.4 · p. 14
• In another example, the rationale attributes exhilaration from heights to James, a named individual, as the answer to what emotion he might feel.The passage’s preceding discussion contrasts such a specific-person claim with a general-class explanation.Source: §A.4 · p. 14

A.5 Red Herrings

• Some reasoning errors involve statements that are technically true but do not help demonstrate the claim.Source: §A.5 · p. 14
• In one example, the rationale notes that squirrels are not small dogs, then selects “tree” as the place where a small dog might bark.The rationale’s true distinction does not establish the selected answer.Source: §A.5 · p. 14
• In another example, the rationale calls running a fast, light-footed activity and selects “lightly” for how John’s feet touched the ground.The question asks for a way different from how John ran, while the rationale links its answer to running.Source: §A.5 · p. 14

A.6 Hint Short-cutting

• On hinted-answer training examples, the model appeared to learn that the final answer matched the hint, producing rationales whose reasoning and selected answer could conflict.One example argues that an airport fits the question but concludes with “train station (e).”Source: §A.6 · p. 15

B Modified CQA Prompts

• The appendix supplies modified CommonsenseQA prompts based closely on [6], with examples showing questions, answer choices, and step-by-step rationales.Examples include identifying a blotter as an ink absorber and a television as equipment requiring cable.Source: B Modified CQA Prompts · p. 16

C Human-evaluated Test Prompts

• For 50 questions answered correctly by both few-shot and STaR-trained models without rationalization, twenty crowdworkers preferred the STaR-generated answers.Workers ranked three explanations for a predetermined correct answer by how well they justified it; answer endings were standardized to focus judgments on rationale differences.Source: C Human-evaluated Test Prompts · p. 18
• For the poker prompt, multiple generations select “losing money” but give different explanations, including one that restates the answer without supporting it.The outputs first connect poor play to losing money, then offer other rationales for the same choice.Source: C Human-evaluated Test Prompts · p. 20
• Examples span questions about poker, greetings, pets, household items, and everyday situations, with generated rationales selecting an answer choice.For instance, the pantry and humidifier prompts each show answer-and-rationale generations.Source: C Human-evaluated Test Prompts · pp. 20–21
• For the ships-in-a-storm prompt, all shown generations answer “bottom out,” using explanations that do not establish that choice from the question.One cites the ships being in the middle of the ocean; another cites the sun as an energy source.Source: C Human-evaluated Test Prompts · p. 21
• The human-evaluated test prompts include questions about library behavior, dangers in the outdoors, file-sharing connections, and what can be injured.Source: C Human-evaluated Test Prompts · pp. 22–23

D Example Rationalizations on CQA

• Pretrained-model rationalizations were generally similar in quality to its rationale generation, but often resembled template matching rather than reasoning; rephrasing questions simply was also often effective.The paper notes that these are rationales the model struggled to learn from, since stronger rationales might already have been incorporated into its rationale generation.Source: D Example Rationalizations on CQA · p. 24

E STaR Without Rationalization (Rationale Generation Bootstrapping)

• Rationale-generation bootstrapping iteratively prompts the current model, retains generated rationales only when predicted answers match ground truth, and fine-tunes the original model on those examples.The procedure repeats this outer loop; the paper presents it as Algorithm 1 with the blue components removed.Source: E STaR Without Rationalization… · p. 26

F Case Study

• For a question about where people attend to gain knowledge, the model’s later rationale explains why “university” is the answer.The initial answer named university without explanation; the later rationale adds that universities are places where people go to learn.Source: F Case Study · p. 26
• For a question about a real estate agent, the later rationale connects misrepresenting a house’s quality to lying about it.The initial rationale gives the answer without that explanation.Source: F Case Study · p. 26
• The case study highlights both previously unsolved questions that the model solves and initially correct rationales that improve over iterations.For CommonsenseQA, selecting the “best” answer matters even when multiple choices may be technically correct.Source: F Case Study · p. 26, Figure 7

G Bias and Faithfulness

• STaR can amplify biases that help solve a dataset, while rationalization may elicit biased answers the model would not otherwise produce; how these interact with pre-existing model biases is unclear.The paper flags this as a concern for real-world deployment, particularly for STaR-trained models.Source: G Bias and Faithfulness · p. 27
• For questions where gender is irrelevant, the model appears to disregard gender in its explanations, but the authors call for a more comprehensive study.The paper illustrates this with questions about where a woman stores sunglasses and where she is heading with checked baggage.Source: G Bias and Faithfulness · p. 27
• Faithfulness is difficult to assess because rationales may not reflect internal processing: a model could select an answer first and then generate a rationale to justify it.The paper notes that this could yield unbiased rationales alongside biased answer selection.Source: G Bias and Faithfulness · p. 27
• Generating a rationale before an answer non-trivially improves answer quality over direct prediction, but the faithfulness of any particular rationale remains difficult to evaluate [6].The paper says these difficulties are not unique to STaR, but reflect the broader challenge of understanding large language models and their rationales.Source: G Bias and Faithfulness · p. 27

H Hyperparameters

• Training generally uses batches of 8 sequences of length 1024, and a learning rate of 10^-6 was consistently best in the Adam search from 10^-7 to 10^-4.The setup packs shorter examples, uses no weight decay, and trains and samples on a single TPU-v3 node; the model is GPT-J, a 28-layer decoder-only transformer.Source: H Hyperparameters · p. 27

I GSM8K Few-shot Prompt

• The GSM8K few-shot prompts are based on examples in [9] and provide worked arithmetic word-problem solutions, including clip sales, savings, and reading calculations.The supplied examples show step-by-step calculations ending in a final answer.Source: I GSM8K Few-shot Prompt · p. 28

J STaR GSM8K Solutions

• Substantially fewer calculation steps typically indicate that the model answered correctly despite mistakes in its reasoning.The passage notes that some shorter solutions are genuinely simpler instead.Source: J STaR GSM8K Solutions · p. 30
• In some cases, STaR produces simpler solutions than the ground truth.Figure 8 shows a training-set problem where its solution is significantly simpler.Source: J STaR GSM8K Solutions · p. 30, Figure 8 · p. 30
• Figure 8 presents a training-set example in which STaR derives a significantly simpler solution than the ground truth.Source: Figure 8 · p. 30