Source-linked AI summary
Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, Denny Zhou
TL;DR
Complex multi-step reasoning remains challenging for large language models, especially when relevant facts and principles are buried in many details. The paper introduces Step-Back Prompting, which first derives a higher-level concept or principle and then uses it to reason about the original question, yielding substantial gains across challenging reasoning tasks. The approach improves PaLM-2L by 7% and 11% on MMLU Physics and Chemistry, 27% on TimeQA, and 7% on MuSiQue.
Problem
Complex multi-step reasoning remains challenging for state-of-the-art LLMs, even though existing methods produce intermediate reasoning steps.
Method
Step-Back Prompting first derives a higher-level concept or principle from a detailed question, then grounds reasoning about the original question in that abstraction.
Results
Step-Back Prompting improves PaLM-2L by 7% on MMLU Physics, 11% on MMLU Chemistry, 27% on TimeQA, and 7% on MuSiQue.
Takeaways & Limitations
Abstraction is shown to be an easy skill to elicit with in-context learning, while reasoning remains the dominant failure mode after task complexity is reduced.
Takeaways & Limitations
Abstraction is neither necessary nor possible in all scenarios, including questions whose answers are readily available or directly state first principles.
Abstract
from arXiv · showhide
We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide reasoning, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of Step-Back Prompting with PaLM-2L, GPT-4 and Llama2-70B models, and observe substantial performance gains on various challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting improves PaLM-2L performance on MMLU (Physics and Chemistry) by 7% and 11% respectively, TimeQA by 27%, and MuSiQue by 7%.
1 INTRODUCTION
Complex multi-step reasoning remains challenging for state-of-the-art LLMs, motivating Step-Back Prompting, which uses abstraction to guide reasoning across STEM, Knowledge QA, and Multi-Hop Reasoning tasks. Experiments report substantial improvements, including gains of up to 27% and additional analyses of errors and baselines.
- Motivation: Step-Back Prompting addresses difficult multi-step reasoning by grounding reasoning in high-level abstractions such as concepts and principles.The approach is motivated by the challenge of following correct intermediate reasoning paths when tasks contain many details.
- Evaluation: The method is evaluated across STEM, knowledge-intensive question answering, and multi-hop commonsense reasoning tasks.The experiments include domain-specific Physics and Chemistry reasoning, factual question answering, and multi-hop reasoning.
- Results: 7% and 11% gains on MMLU Physics and Chemistry, 27% on TimeQA, and 7% on MuSiQue are reported for PaLM-2L with Step-Back Prompting.These improvements are reported relative to PaLM-2L performance, with Step-Back improving performance across the listed tasks.
- Analysis: Up to 36% improvement over Chain-of-Thought and take-a-deep-breath prompting is reported in additional analyses.The paper also reports that Step-Back fixes up to approximately 40% of base-model errors while introducing at most approximately 12% new errors.
2 STEP-BACK PROMPTING
Step-Back Prompting converts a detailed question into a higher-level step-back question, retrieves or derives relevant concepts and principles, and then reasons about the original question using them. This separates abstraction from reasoning to reduce intermediate reasoning errors.
- Definition: A step-back question is a derived question that addresses the original question at a higher level of abstraction.For example, asking about education history can support answering which school someone attended during a specific period.
- Abstraction: Abstraction prompts the LLM to ask a generic question about a higher-level concept or principle and retrieve relevant facts.The step-back question is unique to each task so that the retrieved facts are relevant to the original problem.
- Reasoning: Reasoning uses facts about the high-level concept or principle to derive the solution to the original question.This grounding is intended to avoid intermediate reasoning errors caused by directly processing many task-specific details.
- Scope: The paper studies Step-Back Prompting on challenging STEM, Knowledge QA, and Multi-Hop Reasoning tasks involving complex reasoning.
3 EXPERIMENTAL SETUP
The experiments cover diverse reasoning and knowledge tasks using PaLM-2L, GPT-4, and Llama2-70B, with model generations evaluated through PaLM-2L-based answer equivalence. The setup uses retrieval augmentation for fact-seeking datasets but not STEM tasks.
- Tasks: The evaluation spans STEM, Knowledge QA, and Multi-Hop Reasoning tasks.STEM includes MMLU Physics and Chemistry and GSM8K; Knowledge QA includes TimeQA and SituatedQA.
- Models: The evaluated model families are instruction-tuned PaLM-2L, GPT-4, and Llama2-70B.
- Evaluation: Model outputs are evaluated with PaLM-2L prompted to identify equivalence between target answers and model predictions.This evaluation is used because conventional accuracy and F1 metrics can have limitations for long-form LLM answers.
- Baselines: The prompting setups include direct PaLM-2L baselines, one-shot demonstrations, and zero-shot or one-shot Chain-of-Thought variants.
- Inference: STEM tasks use no retrieval augmentation, while inferences are performed with greedy decoding.The paper contrasts this with fact-seeking datasets, where retrieval augmentation is used.
4 STEM
The STEM evaluation tests whether abstraction through domain concepts and first principles improves specialized reasoning. STEP-BACK PROMPTING substantially improves performance across model families and remains robust with few-shot variation, while remaining errors are dominated by reasoning failures.
- 4.1 STEP-BACK PROMPTING: STEP-BACK PROMPTING demonstrates abstraction skills through domain concepts and first principles such as Newton’s first law, the Doppler effect, and Gibbs free energy.The implicit step-back question asks which physics or chemistry principles and concepts are involved in solving the task.
- 4.2 RESULTS: +7% and +11% compared to PaLM-2L are reported for MMLU Physics and Chemistry, respectively.The comparison is against PaLM-2L baseline performance, with results evaluated across PaLM-2L, GPT-4, and Llama2-70B.
- 4.3 ABLATION AND ANALYSIS: STEP-BACK PROMPTING is robust to the number of few-shot exemplars used as demonstrations.Adding examples beyond one does not improve performance, so the paper uses a single exemplar outside ablation studies.
- 4.3 ABLATION AND ANALYSIS: 20.5% of PaLM-2L baseline errors are corrected while 11.9% new errors are introduced on MMLU high-school Physics.The analysis categorizes remaining errors into Principle, Factual, Math, Context Loss, and Reasoning errors.
- 4.3 ABLATION AND ANALYSIS: Reasoning is the dominating class among STEP-BACK PROMPTING errors on MMLU high-school Physics.The error taxonomy defines reasoning errors as mistakes in intermediate reasoning steps before the final answer.
5 KNOWLEDGE QA
The Knowledge QA experiments combine STEP-BACK PROMPTING with retrieval augmentation for fact-intensive questions. Step-back abstraction improves retrieval and accuracy on TimeQA and SituatedQA, while remaining TimeQA errors largely arise from reasoning and retrieval.
- 5.1 SETUP: STEP-BACK PROMPTING is evaluated on TimeQA and SituatedQA with retrieval augmentation combined with in-context demonstrations of abstraction.The step-back question asks for a higher-level concept related to the original knowledge-intensive query.
- 5.2 RESULTS: 68.7% accuracy on TimeQA is achieved by Step-Back + RAG, compared with 57.4% for regular RAG and 41.5% for PaLM-2L.CoT and TDB prompting do not improve the baseline on this task.
- 5.2 RESULTS: 62.3% Hard accuracy is achieved by Step-Back + RAG on TimeQA, compared with 46.8% for RAG and 42.6% for GPT-4.The passage attributes the improvement to retrieving facts about high-level concepts to ground final reasoning.
- 5.2 RESULTS: Step-Back + RAG improves SituatedQA accuracy from 54.3% to 61%, compared with GPT-4’s 63.2%.CoT and TDB do not help significantly on SituatedQA.
- 5.3 ABLATION AND ANALYSIS: More than half of the remaining TimeQA errors are reasoning errors, while 45% result from failing to retrieve the right information.StepBack itself rarely fails, despite abstraction making retrieval easier.
6 MULTI-HOP REASONING
The paper evaluates Step-Back Prompting on MuSiQue and StrategyQA, finding the strongest performance among tested methods despite differing baseline difficulty.
- Step-Back Prompting is evaluated on the multi-hop reasoning benchmarks MuSiQue and StrategyQA.
- 35.5% and 38.5% are the baseline PaLM-2L and GPT-4 performances on MuSiQue, respectively.The passage describes MuSiQue as a hard multihop reasoning benchmark.
- 82.8% and 78.3% are the baseline PaLM-2L and GPT-4 performances on StrategyQA, respectively.The passage attributes the stronger baselines partly to StrategyQA being a binary classification task.
- Step-Back Prompting produces the best performance of all methods tested on these multi-hop reasoning tasks.The supplied passage introduces the exact MuSiQue result immediately afterward but does not include the complete continuation.
7 DISCUSSION
The discussion presents abstraction as a way to remove irrelevant detail and guide reasoning with high-level concepts or principles. It reports broad gains but notes that reasoning remains a dominant failure mode and abstraction is not universally applicable.
- Abstraction removes irrelevant details and distills high-level concepts and principles to guide problem solving.
- Step-Back Prompting separates complex tasks into Abstraction and Reasoning, then grounds reasoning in high-level concepts and principles.The paper applies this framing to knowledge-intensive QA, multi-hop reasoning, and science questions.
- Reasoning remains the dominant failure mode even after Step-Back Prompting substantially reduces task complexity.
- Abstraction is neither necessary nor possible in all scenarios, including simple factual questions and questions that directly invoke first principles.
8 RELATED WORK
The paper distinguishes Step-Back Prompting from prior prompting and decomposition methods by making questions more abstract and high-level rather than breaking them into low-level subquestions.
- Step-Back Prompting belongs to the simple, generic family of few-shot prompting methods alongside Chain-of-Thought and scratchpad prompting.
- Step-Back Prompting makes the question more abstract and high-level, whereas decomposition breaks the original task into simpler subquestions.
- Abstract questions can map many specific questions to one generic concept, unlike decomposition’s typically one-to-one subquestion breakdown.The paper illustrates this contrast using employment-history questions about Steve Jobs.
9 CONCLUSION
The paper introduces Step-Back Prompting as a generic abstraction-based method and reports improvements across several reasoning benchmark categories. On GSM8K, it remains competitive even though retrieving simple principles is not always necessary.
- Step-Back Prompting is introduced as a simple, generic method for eliciting deep reasoning through abstraction in LLMs.
- Experiments across fact-seeking, commonsense reasoning, and domain-specific reasoning benchmarks show significant performance improvements.
- Step-Back Prompting is the most competitive tested prompting method on GSM8K, including Take a Deep Breath and Decomposed Prompting.The passage also reports competitive performance relative to zero-shot CoT and 1-shot standard prompting.
- GSM8K’s simple principles make retrieving principles before reasoning not absolutely necessary, according to the paper’s hypothesis.
A.2 TIMEQA ERROR ANALYSIS
On TimeQA, Step-Back + RAG corrects a substantial share of baseline and RAG errors while introducing comparatively fewer new errors.
- 39.9% of incorrect baseline predictions are fixed by Step-Back + RAG, while 5.6% become errors.
- 21.6% of errors from RAG are fixed by Step-Back + RAG.
- 6.3% of RAG predictions receive errors introduced by Step-Back Prompting.
- Table 5 reports the evaluation-data splits and example counts for TimeQA, StrategyQA, MMLU, and GSM8K.
C.1 FEW-SHOT EXAMPLES FOR EVALUATION WITH PALM-2L
The evaluation appendix documents PaLM-2L scoring and the prompting templates, demonstrations, successful examples, and error categories used across the evaluated tasks.
- Evaluation: PaLM-2L serves as the scoring model, with temperature T = 1 producing agreement with human ratings of 97%, 98%, 99%, and 99% across four trials.
- Step-Back prompting: For MMLU Physics and Chemistry, the model first generates underlying principles and then uses them with few-shot demonstrations to produce the final answer.
- Step-Back prompting: For TimeQA, SituatedQA, MuSiQue, and StrategyQA, step-back questions guide retrieval alongside the original question before final-answer generation.
- Baselines: The appendix includes standard, zero-shot CoT, few-shot CoT, and deep-breath prompting baselines with task-specific demonstration exemplars.
- Error analysis: Five error categories are illustrated: factual, mathematical, principle, context-loss, and reasoning errors.
- Examples: Successful Step-Back Prompting examples are provided for MMLU Physics, MMLU Chemistry, TimeQA, SituatedQA, and StrategyQA.