Natural Language Reasoning, A Survey

1 INTRODUCTION

• The authors identify an unclear definition of reasoning as a source of confusion about whether tasks such as commonsense reasoning, natural language inference, and multi-hop question answering involve reasoning.They note that task names alone do not reliably indicate whether reasoning is involved.Source: §1 · p. 2
• The survey proposes a definition of NLP reasoning, discusses which tasks require it, and offers a taxonomy of reasoning.It also considers how deductive and defeasible reasoning differ and how those differences may affect NLP solutions.Source: §1 · pp. 2–3
• The authors highlight backward reasoning as a multi-step paradigm they consider underexplored and worth further exploration alongside forward reasoning.They describe backward reasoning as potentially more efficient because it searches a smaller space.Source: §1 · pp. 2–3
• The review focuses on single-modality, unstructured natural-language text and excludes neuro-symbolic techniques and mathematical reasoning.It reviews more than two hundred papers found mainly through searches of top conferences from 2019 to 2022.Source: §1 · p. 3
• The survey covers PLM-based natural-language reasoning across diverse benchmarks and provides a taxonomy of methodologies.Its review mainly covers classical logical reasoning, natural language inference, multi-hop question answering, and commonsense reasoning.Source: §1 · pp. 2–3
• The survey introduces defeasible reasoning as a future direction and compares it with deductive reasoning, including their possible effects on NLP solutions.It also reviews current methods for defeasible reasoning.Source: §1 · p. 3

2 WHAT IS NATURAL LANGUAGE REASONING

• The section aims to address the lack of a distinct definition of natural language reasoning in NLP, which the authors say affects understanding, analysis, and communication.It draws on philosophy and logic to adapt relevant reasoning theory to NLP.Source: §2 · p. 3
• The authors propose a definition of natural language reasoning in NLP and introduce categories whose differences can affect NLP solutions.Source: §2 · p. 3
• The section also discusses the potentials, challenges, and requirements for achieving natural language reasoning.Source: §2 · p. 3

2.1.1 Definition from philosophy and logic.

• The authors draw on philosophical accounts of reasoning, adopting theoretical and practical reasoning as a basis rather than restricting the definition to formal logic.The section presents reasoning as involving complex tasks, deriving knowledge beyond immediate perception, inferring assertions, or inferring actions from goals and knowledge.Source: §2.1.1 · pp. 4–5

2.1.2 Definition in NLP we suggest.

• NLP reasoning integrates multiple explicit or implicit knowledge sources to derive new assertions, events, or practical actions about the world.The proposed definition distinguishes reasoning from knowledge-intensive tasks that use multiple knowledge sources but do not derive new conclusions.Source: §2.1.2 · pp. 5–6
• Reasoning is effective when an answer is not already in context, model memory, or knowledge bases but can be reached by integrating available information.The section also describes reasoning as deriving new assertions, events, or actions without direct recourse to those stored or provided sources.Source: §2.1.2 · p. 6

2.1.3 Key Concepts.

• A proposition is a statement’s semantic content, while an inference is one step that produces an intermediate conclusion from premises.Reasoning can draw multiple premises from a sentence with multiple meanings, but its conclusion must have semantics different from all premises.Source: §2.1.3 · pp. 6–7

2.2 Categories of Inference

• The authors combine the monotonicity distinction with the familiar deduction, induction, and abduction taxonomy to organize inference as deductive or defeasible.Deduction is monotonic; induction and abduction are subclasses of defeasible inference.Source: §2.2 · p. 8, §2.2.1 · p. 8

2.2.1 Deduction, Induction, and Abduction.

• Deduction derives valid conclusions, whereas induction generalizes probable rules and abduction infers probable explanations for observed phenomena.The section notes that research on induction and abduction is much less explored than work on deduction, although deduction covers only a small part of everyday human reasoning.Source: §2.2.1 · p. 8

2.2.2 Deductive Inference and Defeasible Inference.

• Defeasible conclusions may be weakened or retracted by new knowledge, so NLP systems need opposing as well as supporting evidence and more complex reasoning paths.The authors also identify interpretability as more important for defeasible reasoning because people with different background knowledge may reach different or opposite conclusions.Source: §2.2.2 · pp. 8–9

• Natural language offers a human-computer interaction interface and permits defeasible reasoning that formal language does not.Source: §2.3 · p. 9
• Ambiguity and structural variety make natural-language propositions difficult to understand, while inference data is hard to obtain at scale and reasoning steps vary by instance.The section also emphasizes the importance of generalizing to unseen inference steps.Source: §2.3 · p. 9
• NLP reasoning requires knowledge acquisition, knowledge understanding, and inference, but this article focuses on inference because the first two topics are broad enough for another survey.Understanding includes capturing logical semantics such as negation, conjunction, and disjunction.Source: §2.3 · p. 10

3.1 Introduction to PLMs

• PLMs have shown potential for reasoning with explicit and implicit knowledge, including deductive and defeasible inference and prompted multi-step reasoning.The survey’s introductory discussion presents PLMs as transformer-based models used in pretraining, fine-tuning, in-context learning, and zero-shot prompting paradigms.Source: §3.1 · p. 10, §3.2 · p. 11

3.2 Empirical Development

• Chain-of-Thought prompting can unlock multi-step reasoning paths, with performance on many Big-Bench Hard tasks surpassing the average human rater.The survey also reports zero-shot prompting that elicits intermediate steps and learning from self-generated reasoning paths.Source: §3.2 · p. 11
• Question decomposition, a backward-reasoning method, is effective in multi-hop question answering for both medium-size PLMs and LLMs.Source: §3.2 · p. 11
• PLMs can produce faithful reasoning paths and make predictions based on them, according to the cited demonstrations.Source: §3.2 · p. 11
• PLMs can learn multi-step reasoning from supervised data or few-shot demonstrations, and their language understanding, generalization, and use of implicit knowledge make them promising for natural-language reasoning.Source: §3.2 · p. 11

4 METHODOLOGIES OF NLR

• End-to-end reasoning outputs only the answer, while forward and backward approaches can produce intermediate conclusions that link premises to conclusions.The survey focuses on reasoning paths rather than natural-language explanations alone.Source: §4 · p. 11
• Reasoning paths can improve interpretability by exposing supporting knowledge, and have been shown beneficial to multi-step reasoning performance.A strict path can make the supporting knowledge at each step explicit.Source: §4 · p. 12
• Forward reasoning derives new knowledge from existing knowledge; backward reasoning starts from a problem and breaks it into sub-problems.For a specified problem, backward reasoning has a smaller search space, while forward reasoning can uncover new knowledge without a preassigned problem.Source: §4 · p. 12

4.1 End-to-End Reasoning

• End-to-end reasoning is a black-box approach that outputs final answers without explanations or intermediate conclusions, using specialized models, vanilla PLMs, or decoder-only LLMs.The survey also reviews specialized pretraining methods.Source: §4.1 · p. 12

4.1.1 Training specialized models.

• Task-specific architectural biases, including graph neural networks, support evidence aggregation, while handcrafted assumptions may limit generalization to other tasks.ReasonFormer instead assigns modules to predefined fundamental reasoning capabilities.Source: §4.1.1 · p. 12

4.1.2 Finetuning vanilla medium-size PLMs.

• Finetuned bidirectional and causal PLMs can learn deductive reasoning, but data sparsity, spurious correlations, and weak defeasible reasoning remain challenges.Synthetic training data can fail to generalize across linguistic variations, while high-quality annotated data is difficult to scale.Source: §4.1.2 · pp. 13–14

4.1.3 Few-shot decoder-only LLMs.

• Reasoning performance improves with model size, but how much reflects stronger reasoning is unclear; scaling is slower on deductive reasoning than on other BigBench tasks.The survey also reports that vanilla LLMs struggle with multi-step reasoning.Source: §4.1.3 · p. 14
• Reasoning-related continual pretraining improves multi-hop and logical reasoning, especially in low-resource settings, but the reviewed methods use encoder-only PLMs.The approaches introduce type-specific or type-agnostic reasoning biases through data and training strategies.Source: §4.1.4 · p. 14

4.1.4 Specialized pretraining.

• Program-based pretraining with synthesized program–execution pairs significantly improves multi-hop and logical reasoning across medium- and large-size, encoder-only, and encoder-decoder PLMs.The surveyed approach uses programs such as SQL.Source: §4.1.4 · p. 14

4.2.1 Trustworthiness.

• Reasoning paths can expose knowledge and inference relations, but generated paths may contain hallucinations or invalid inferences; modular decoupling is used to promote faithfulness.Stepwise supervision can ease data sparsity but also introduces error propagation.Source: §4.2.1 · p. 15
• Evaluating reasoning paths is difficult because a problem may have multiple valid paths, making exhaustive annotation impractical and automatic validity assessment challenging.These challenges are especially acute for long-step problems with combinatorial explosion.Source: §4.2.1 · p. 15

4.2.2 Performance improvement.

• Generating a reasoning path before an answer, as in chain-of-thought prompting, significantly improves performance on commonsense reasoning, including deductive and defeasible tasks.The survey also reports gains for classical logical and multi-step reasoning.Source: §4.2.2 · p. 16
• Proof-graph construction improves PLMs’ zero-shot generalization to unseen reasoning steps and domains on theorem proving, with stepwise construction also generalizing across datasets.The survey suggests this may be because proof graphs encourage reasoning rather than shortcuts.Source: §4.2.2 · p. 16
• Forward search grows combinatorially with reasoning steps, while LLMs can still struggle to plan deep reasoning and the topic remains underexplored.Using reasoning paths to improve defeasible-reasoning performance is also underexplored.Source: §4.2.2 · p. 16

4.3 Backward Reasoning

• Backward reasoning breaks a problem into sub-problems until reaching answers and is more efficient than forward reasoning because it searches a smaller space.The survey distinguishes backward chaining, a proof-finding strategy, from question decomposition, a more general strategy.Source: §4.3 · p. 16

4.3.1 Backward Chaining.

• Backward chaining starts from a goal and uses abduction to propose premises as sub-goals until existing knowledge can prove or disprove them.It can predict only missing premises or generate all premises from scratch; the latter can address cases where required premises are absent from a knowledge base.Source: §4.3.1 · pp. 16–17
• Question decomposition breaks multi-hop questions into simpler sub-questions, which can be answered individually and combined into a final answer.This introduces abductive decomposition while removing the need for multi-step knowledge selection or retrieval.Source: §4.3.2 · p. 17

4.3.2 Question Decomposition.

• Implicit multi-hop questions are difficult to decompose because their needed sub-questions may not match the original wording; annotated data is scarce and labor-intensive to collect.In-context learning offers one response to data sparsity by requiring only a small set of demonstrations.Source: §4.3.2 · p. 17
• Decomposition frameworks include module-based, decompose-then-recompose, and generate-then-answer approaches, with different trade-offs in modeling dependencies and parallelism.Generate-then-answer handles sequential dependencies and tree structures but is less efficient than decompose-then-recompose when tree sub-questions could be solved in parallel.Source: §4.3.2 · p. 18
• Very long-hop questions remain difficult because decomposition supervision, demonstrations, and planning face combinatorial complexity; the survey suggests hierarchical decomposition and knowledge-aware planning.Hierarchical decomposition breaks a question into simpler multi-hop questions before reaching single-hop questions.Source: §4.3.2 · p. 18

4.4 Summary

• Early specialized models improved reasoning and built forward paths, but task- or dataset-specific constraints hurt generalization.Vanilla PLMs became more popular after transformers were found to perform soft deduction following in-domain fine-tuning.Source: §4.4 · p. 18
• Data sparsity and spurious correlations make it difficult for medium-size PLMs to learn general logical structures across reasoning types.It remains unclear whether specialized pretraining with inductive biases is worthwhile as model and pretraining-data sizes increase.Source: §4.4 · p. 18
• Generating a reasoning path before the answer can significantly improve multi-step reasoning when PLMs are sufficiently large.This emergent ability has boosted research on forward reasoning.Source: §4.4 · p. 18
• Backward reasoning targets a specific goal or solution and has a smaller search space than forward reasoning.Question decomposition is a typical backward approach and can improve multi-hop question answering for both medium-size PLMs and LLMs; defeasible reasoning remains under-explored and challenging for PLMs.Source: §4.4 · p. 19

5 NLR BENCHMARKS

• The survey covers four major benchmark areas—classical logic, NLI, multi-hop question answering, and commonsense reasoning—alongside complex reasoning and other tasks.It also discusses how far popular benchmarks actually involve reasoning.Source: §5 · p. 19

5.1 Classical Logical Reasoning

• Classical deductive tasks include single-step inference, multi-step theorem proving, and interpretable reasoning-path generation.Except for FOLIO, existing explicit deductive-reasoning datasets are synthesized; on synthetic theorem-proving datasets, proof generation can improve generalization to unseen proof depths and out-of-domain data, while FOLIO shows severe performance degradation on more diverse natural language.Source: §5.1.1 · pp. 19–20

5.1.2 Defeasible reasoning.

• Defeasible reasoning includes abduction and induction, but it remains under-explored compared with classical deduction.Experiments indicate substantial room for improvement.Source: §5.1.2 · p. 20

5.2 Natural Language Inference

• NLI includes paraphrase, compound-semantics, and implicit-premise entailments, so its examples can involve either understanding or reasoning.Human-authored hypotheses have significant biases, and models can sometimes predict labels without premises; some NLI benchmarks test only understanding, while converted datasets may be unrelated to reasoning.Source: §5.2 · pp. 21–22

5.3 Multi-Hop Question Answering

• Multi-hop question answering requires reasoning across evidence in multiple contexts, with contexts either supplied as distractors or retrieved from external sources.Dataset construction faces label-intensive annotation and combinatorial growth with more hops; approaches address artifacts through adversarial or carefully collected data, and reasoning commonly uses end-to-end answering or backward question decomposition.Source: §5.3 · pp. 22–23

5.4 Commonsense Reasoning

• Commonsense reasoning requires implicit knowledge that may be obvious to people but difficult for machines to retrieve, and not every dataset bearing that label requires reasoning.The survey groups problems by “what,” “what if / why,” and “how”; causal and temporal questions may reason forward from causes or backward from effects, while graph-based methods aggregate knowledge bases and vanilla PLMs act as implicit knowledge bases themselves.Source: §5.4 · p. 23, §5.4.1 · pp. 23–24, §5.4.2 · p. 25

5.5 Complex Reasoning

• Complex-reasoning benchmarks include realistic examinations and tests that may require domain knowledge and multiple reasoning skills.On Big-Bench Hard, GPT-3 with chain-of-thought prompting can outperform human performance on a majority of the challenging tasks.Source: §5.5 · p. 25

5.6 Others

• Natural-language reasoning also appears in tasks such as dialogue and reading comprehension, especially when complex problems require combining knowledge to reach conclusions.Source: §5.6 · p. 25

6.1 Open questions

• The survey leaves open why chain-of-thought prompting helps, why it works for LLMs but not medium-size PLMs, and whether larger models reason better.It also asks how reasoning abilities emerge as model size increases and whether more data or algorithms remain beneficial.Source: §6.1 · p. 26

6.2 Limitations

• Defeasible reasoning remains under-explored, and automatic evaluation of generated reasoning paths without ground truth remains challenging.The cited study reports that ChatGPT finds abduction and induction more challenging than deduction, especially induction.Source: §6.2 · p. 26
• PLMs may produce invalid deductive conclusions because transformers predict conclusions probabilistically rather than ensuring necessary truth.The survey notes that ChatGPT still fails to achieve perfect performance on a simple one-step deductive inference task.Source: §6.2 · p. 26
• LLMs can perform worse on abstract or counterfactual deduction than on realistic cases, and content biases may affect answers and downstream performance.The survey also notes potential societal harms from social biases, including gender bias reported in GPT-4.Source: §6.2 · p. 27

6.3 Future

• Future work should generalize reasoning to longer steps, defeasible reasoning, more languages and modalities, and improve interpretability and faithfulness.The survey notes that performance degrades when test examples require more steps than training data or few-shot exemplars, and that longer-step reasoning remains challenging for ChatGPT.Source: §6.3 · p. 27
• Technical directions include eliciting reasoning with prompts, self-improvement from generated paths, backward reasoning, planning, and self-correction.Backward reasoning may benefit both medium-size PLMs and LLMs, and its smaller search space may yield greater benefits as reasoning depth increases.Source: §6.3 · pp. 27–28