Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

Motivation

• Step-Back Prompting addresses difficult multi-step reasoning by grounding reasoning in high-level abstractions such as concepts and principles.The approach is motivated by the challenge of following correct intermediate reasoning paths when tasks contain many details.Source: §1 · p. 2

Evaluation

• The method is evaluated across STEM, knowledge-intensive question answering, and multi-hop commonsense reasoning tasks.The experiments include domain-specific Physics and Chemistry reasoning, factual question answering, and multi-hop reasoning.Source: §1 · p. 2

Results

• 7% and 11% gains on MMLU Physics and Chemistry, 27% on TimeQA, and 7% on MuSiQue are reported for PaLM-2L with Step-Back Prompting.These improvements are reported relative to PaLM-2L performance, with Step-Back improving performance across the listed tasks.Source: §1 · p. 3

Analysis

• Up to 36% improvement over Chain-of-Thought and take-a-deep-breath prompting is reported in additional analyses.The paper also reports that Step-Back fixes up to approximately 40% of base-model errors while introducing at most approximately 12% new errors.Source: §1 · p. 3

Definition

• A step-back question is a derived question that addresses the original question at a higher level of abstraction.For example, asking about education history can support answering which school someone attended during a specific period.Source: §2 · p. 3

Abstraction

• Abstraction prompts the LLM to ask a generic question about a higher-level concept or principle and retrieve relevant facts.The step-back question is unique to each task so that the retrieved facts are relevant to the original problem.Source: §2 · p. 3

Reasoning

• Reasoning uses facts about the high-level concept or principle to derive the solution to the original question.This grounding is intended to avoid intermediate reasoning errors caused by directly processing many task-specific details.Source: §2 · p. 3

Scope

• The paper studies Step-Back Prompting on challenging STEM, Knowledge QA, and Multi-Hop Reasoning tasks involving complex reasoning.Source: §2 · p. 3

Tasks

• The evaluation spans STEM, Knowledge QA, and Multi-Hop Reasoning tasks.STEM includes MMLU Physics and Chemistry and GSM8K; Knowledge QA includes TimeQA and SituatedQA.Source: §3.1 · pp. 3–4

Models

• The evaluated model families are instruction-tuned PaLM-2L, GPT-4, and Llama2-70B.Source: §3.2 · p. 4

Evaluation

• Model outputs are evaluated with PaLM-2L prompted to identify equivalence between target answers and model predictions.This evaluation is used because conventional accuracy and F1 metrics can have limitations for long-form LLM answers.Source: §3.3 · p. 4

Baselines

• The prompting setups include direct PaLM-2L baselines, one-shot demonstrations, and zero-shot or one-shot Chain-of-Thought variants.Source: §3.4 · p. 4

Inference

• STEM tasks use no retrieval augmentation, while inferences are performed with greedy decoding.The paper contrasts this with fact-seeking datasets, where retrieval augmentation is used.Source: §3.4 · p. 4

4.1 STEP-BACK PROMPTING

• STEP-BACK PROMPTING demonstrates abstraction skills through domain concepts and first principles such as Newton’s first law, the Doppler effect, and Gibbs free energy.The implicit step-back question asks which physics or chemistry principles and concepts are involved in solving the task.Source: §4.1 · pp. 4–5

4.2 RESULTS

• +7% and +11% compared to PaLM-2L are reported for MMLU Physics and Chemistry, respectively.The comparison is against PaLM-2L baseline performance, with results evaluated across PaLM-2L, GPT-4, and Llama2-70B.Source: §4.2 · p. 5

4.3 ABLATION AND ANALYSIS

• STEP-BACK PROMPTING is robust to the number of few-shot exemplars used as demonstrations.Adding examples beyond one does not improve performance, so the paper uses a single exemplar outside ablation studies.Source: §4.3 · pp. 5–6, Figure 3 · p. 5
• 20.5% of PaLM-2L baseline errors are corrected while 11.9% new errors are introduced on MMLU high-school Physics.The analysis categorizes remaining errors into Principle, Factual, Math, Context Loss, and Reasoning errors.Source: §4.3 · p. 6
• Reasoning is the dominating class among STEP-BACK PROMPTING errors on MMLU high-school Physics.The error taxonomy defines reasoning errors as mistakes in intermediate reasoning steps before the final answer.Source: Figure 4 · p. 6, §4.3 · p. 6

5.1 SETUP

• STEP-BACK PROMPTING is evaluated on TimeQA and SituatedQA with retrieval augmentation combined with in-context demonstrations of abstraction.The step-back question asks for a higher-level concept related to the original knowledge-intensive query.Source: §5.1 · p. 7, §5 · p. 7

5.2 RESULTS

• 68.7% accuracy on TimeQA is achieved by Step-Back + RAG, compared with 57.4% for regular RAG and 41.5% for PaLM-2L.CoT and TDB prompting do not improve the baseline on this task.Source: §5.2 · p. 7
• 62.3% Hard accuracy is achieved by Step-Back + RAG on TimeQA, compared with 46.8% for RAG and 42.6% for GPT-4.The passage attributes the improvement to retrieving facts about high-level concepts to ground final reasoning.Source: §5.2 · p. 7
• Step-Back + RAG improves SituatedQA accuracy from 54.3% to 61%, compared with GPT-4’s 63.2%.CoT and TDB do not help significantly on SituatedQA.Source: §5.2 · p. 7

5.3 ABLATION AND ANALYSIS

• More than half of the remaining TimeQA errors are reasoning errors, while 45% result from failing to retrieve the right information.StepBack itself rarely fails, despite abstraction making retrieval easier.Source: §5.3 · p. 8

• Step-Back Prompting is evaluated on the multi-hop reasoning benchmarks MuSiQue and StrategyQA.Source: §6 · p. 8
• 35.5% and 38.5% are the baseline PaLM-2L and GPT-4 performances on MuSiQue, respectively.The passage describes MuSiQue as a hard multihop reasoning benchmark.Source: §6 · p. 8
• 82.8% and 78.3% are the baseline PaLM-2L and GPT-4 performances on StrategyQA, respectively.The passage attributes the stronger baselines partly to StrategyQA being a binary classification task.Source: §6 · p. 8
• Step-Back Prompting produces the best performance of all methods tested on these multi-hop reasoning tasks.The supplied passage introduces the exact MuSiQue result immediately afterward but does not include the complete continuation.Source: §6 · p. 8

• Abstraction removes irrelevant details and distills high-level concepts and principles to guide problem solving.Source: §7 · p. 8
• Step-Back Prompting separates complex tasks into Abstraction and Reasoning, then grounds reasoning in high-level concepts and principles.The paper applies this framing to knowledge-intensive QA, multi-hop reasoning, and science questions.Source: §7 · p. 9
• Reasoning remains the dominant failure mode even after Step-Back Prompting substantially reduces task complexity.Source: §7 · p. 9
• Abstraction is neither necessary nor possible in all scenarios, including simple factual questions and questions that directly invoke first principles.Source: §7 · p. 9

• Step-Back Prompting belongs to the simple, generic family of few-shot prompting methods alongside Chain-of-Thought and scratchpad prompting.Source: §8.1 · p. 9
• Step-Back Prompting makes the question more abstract and high-level, whereas decomposition breaks the original task into simpler subquestions.Source: §8.2 · p. 9
• Abstract questions can map many specific questions to one generic concept, unlike decomposition’s typically one-to-one subquestion breakdown.The paper illustrates this contrast using employment-history questions about Steve Jobs.Source: §8.2 · p. 9

• Step-Back Prompting is introduced as a simple, generic method for eliciting deep reasoning through abstraction in LLMs.Source: §9 · p. 9
• Experiments across fact-seeking, commonsense reasoning, and domain-specific reasoning benchmarks show significant performance improvements.Source: §9 · p. 9
• Step-Back Prompting is the most competitive tested prompting method on GSM8K, including Take a Deep Breath and Decomposed Prompting.The passage also reports competitive performance relative to zero-shot CoT and 1-shot standard prompting.Source: §A.1 · p. 13
• GSM8K’s simple principles make retrieving principles before reasoning not absolutely necessary, according to the paper’s hypothesis.Source: §A.1 · p. 13

• 39.9% of incorrect baseline predictions are fixed by Step-Back + RAG, while 5.6% become errors.Source: §A.2 · p. 13, Figure 6 · p. 13
• 21.6% of errors from RAG are fixed by Step-Back + RAG.Source: §A.2 · p. 13, Figure 6 · p. 13
• 6.3% of RAG predictions receive errors introduced by Step-Back Prompting.Source: §A.2 · p. 13, Figure 6 · p. 13
• Table 5 reports the evaluation-data splits and example counts for TimeQA, StrategyQA, MMLU, and GSM8K.Source: Table 5 · p. 14, B DATASET DETAILS · p. 14

Evaluation

• PaLM-2L serves as the scoring model, with temperature T = 1 producing agreement with human ratings of 97%, 98%, 99%, and 99% across four trials.Source: §C.2 · p. 15

Step-Back prompting

• For MMLU Physics and Chemistry, the model first generates underlying principles and then uses them with few-shot demonstrations to produce the final answer.Source: §D.1 · pp. 15–16, Table 7 · p. 15, Table 8 · p. 16
• For TimeQA, SituatedQA, MuSiQue, and StrategyQA, step-back questions guide retrieval alongside the original question before final-answer generation.Source: §D.2 · p. 16, §D.3 · p. 16, Table 12 · p. 19

Baselines

• The appendix includes standard, zero-shot CoT, few-shot CoT, and deep-breath prompting baselines with task-specific demonstration exemplars.Source: §D.4 · p. 16, §D.5 · pp. 16, 21, §D.6 · p. 21

Error analysis

• Five error categories are illustrated: factual, mathematical, principle, context-loss, and reasoning errors.Source: §E.1 · p. 21, Table 23 · p. 27, Table 24 · p. 28

Examples

• Successful Step-Back Prompting examples are provided for MMLU Physics, MMLU Chemistry, TimeQA, SituatedQA, and StrategyQA.Source: §E.2 · p. 21, Table 28 · p. 32, Table 29 · p. 33