Source-linked AI summary
Augmented Language Models: a Survey
Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, Thomas Scialom
TL;DR
Traditional language models face hallucinations, reasoning mistakes, and limited context, motivating augmentations that extend their capabilities. The survey synthesizes models that combine reasoning with external tools under the standard missing-token objective and examines how these capabilities are implemented. It concludes that Augmented Language Models offer a framework for broader capabilities and may address limitations of traditional language models, while fully self-supervised augmentation remains open.
Problem
Traditional language models suffer from hallucinations, reasoning mistakes, limited context, and scaling demands, motivating a survey of augmentations that address these limitations.
Method
The survey categorizes language models augmented with reasoning and external tools, including whether these capabilities are implemented heuristically or learned through supervision or reinforcement.
Results
Augmented Language Models extend context with relevant information and non-parametric external modules while retaining the missing-token prediction objective, distinguishing them from classical language modeling.
Takeaways & Limitations
The survey concludes that combining reasoning and tools can support more capable assistants and agents across a wider range of tasks than standard language models.
Takeaways & Limitations
Intermediate reasoning steps are not guaranteed to be correct, and the extent to which models use stated reasoning steps for final predictions remains poorly understood.
Abstract
from arXiv · showhide
This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. The former is defined as decomposing a potentially complex task into simpler subtasks while the latter consists in calling external modules such as a code interpreter. LMs can leverage these augmentations separately or in combination via heuristics, or learn to do so from demonstrations. While adhering to a standard missing tokens prediction objective, such augmented LMs can use various, possibly non-parametric external modules to expand their context processing ability, thus departing from the pure language modeling paradigm. We therefore refer to them as Augmented Language Models (ALMs). The missing token objective allows ALMs to learn to reason, use tools, and even act, while still performing standard natural language tasks and even outperforming most regular LMs on several benchmarks. In this work, after reviewing current advance in ALMs, we conclude that this new research direction has the potential to address common limitations of traditional LMs such as interpretability, consistency, and scalability issues.
1 Introduction: motivation for the survey and definitions
The survey frames Augmented Language Models as language models enhanced with reasoning and external tools to address limitations of standard models. It defines these augmentations, explains their relationship, and organizes prior work by reasoning, tool use, and how these capabilities are implemented.
- Motivation: LLMs have achieved strong performance across language tasks but still produce plausible falsehoods and mistakes, while many capabilities emerge only at large scale.These limitations hinder broader deployment and motivate augmenting language models beyond pure statistical language modeling.
- Motivation: Standard LMs rely on a single parametric model and limited context, requiring massive scale to store knowledge absent from the immediate context.External information retrieval can improve context relevance and match some capabilities of larger models with fewer parameters.
- Definitions: In ALMs, reasoning decomposes complex tasks into simpler subtasks, while tools are external modules whose outputs enter the model’s context or affect the world.Reasoning may involve recursion, iteration, or step-by-step computation; tools include document retrievers and robotic arms.
- Motivation: Reasoning and tools complement one another: reasoning combines tools across subtasks, while tools help validate individual steps and improve missing-token prediction through context augmentation.The survey also treats tool calls as actions, including calls without external effects, and considers tools that gather information or manipulate virtual and physical environments.
- Survey organization: The survey organizes prior work across reasoning, tool interaction and action, and whether these capabilities arise from heuristics or are learned through supervision or reinforcement.It uses these three axes as its main taxonomy while acknowledging that other organizational axes are possible.
2 Reasoning
The survey presents reasoning as decomposing complex tasks into simpler subproblems and reviews prompting, recursive decomposition, working-memory training, and fine-tuning approaches for eliciting it. These methods improve reasoning capabilities but remain limited by scale, computation, and uncertainty about intermediate-step validity and faithfulness.
- Definition: Reasoning in language models is defined as decomposing potentially complex tasks into simpler subtasks, often through iterative or recursive structures.The survey distinguishes this operational definition from the broader ability to make inferences using evidence and logic.
- Eliciting reasoning with prompting: Few-shot chain-of-thought prompting supplies intermediate reasoning steps, while zero-shot chain-of-thought appends “Let’s think step by step” without task examples.Few-shot CoT generally requires very large models, whereas zero-shot-CoT performs well on reasoning tasks such as GSM8K but less well than few-shot-CoT.
- Recursive prompting: Recursive prompting explicitly decomposes problems into subproblems that are solved independently or sequentially, with later stages sometimes conditioned on earlier answers.Recent methods differ in whether decomposition is learned through supervision, generated in context, interwoven with answering, or guided by rules and slot filling.
- Comparison and limitations of abstract reasoning: Prompting approaches require suitable prompts and examples, can be computationally expensive for long prompts, and are constrained by model context size.These drawbacks motivate training language models to use working memory when solving tasks requiring multiple steps.
- Explicitly teaching language models to reason: Training-based approaches use scratchpads, detailed reasoning datasets, rationale bootstrapping, or instruction fine-tuning to improve reasoning beyond prompting alone.Reported instruction-fine-tuning results include improvements on BBH and MMLU, with some small fine-tuned models outperforming larger unfine-tuned models on instruction-following tasks.
- Comparison and limitations of abstract reasoning: Reasoning methods can produce invalid intermediate steps, and it remains poorly understood whether models actually use stated reasoning traces to support final predictions.External tools such as search engines and calculators are used in some work to validate intermediate steps and reduce avoidable errors.
3 Using Tools and Act
Language models can use external modules for information retrieval, multimodal processing, reasoning, and iterative text refinement. These augmentations also support more active behavior by letting models query tools and condition responses on their results.
- Tool use: External modules such as Python interpreters and search engines can offload exact computation and information retrieval from language models.Tools may be called by rules or special tokens, with their outputs included in the model’s context.
- Calling another model: Iterative model calling refines outputs through repeated generation, enabling Re3 to automatically produce stories exceeding two thousand words.Re3 generates a plan, setting, and characters from a premise, then repeatedly uses the current story state and plan to generate new passages.
- Calling another model: PEER repeatedly plans and edits text, decomposing article development into smaller subtasks until no further updates are needed.PEER was initialized from LM-Adapted T5 and trained on Wikipedia edits to learn both editing and planning.
- Leveraging other modalities: Multimodal interfaces connect language models with encoders trained on other modalities, supporting applications such as robot perception, video question-answering, and assistive dialogue.Socratic Models compose pretrained modality-specific models zero-shot and can interface with external APIs and databases.
- Retrieval and memory: Memory units and external retrieval augment language models with information beyond their weights, helping address non-factual or outdated outputs while reducing parameter requirements.Retrievers may use sparse bag-of-words or dense neural representations, and retrieved documents can be appended to the model context.
- Retrieval and memory: Retrieval-augmented models show strong knowledge-intensive performance, with RETRO integrating retrieval into pretrained models and Atlas achieving strong few-shot learning despite being orders of magnitude smaller than many large models.Table 2 compares retrieval token counts, granularity, retriever training, and retrieval integration across database-retrieval models.
- Reasoning with tools: Chain-of-thought and retrieval can be interleaved: retrieved evidence supports explanations and predictions, while reasoning steps guide retrieval for multi-step question answering.These approaches can operate without additional training or fine-tuning, and some systems query retrievers only when prompted to do so.
- Acting: Models become more active agents when they generate tool queries, and ReAct combines reasoning with search results to improve performance across language and decision-making tasks.ReAct conditions its response on search results and performs well on question answering, fact verification, and web and home navigation.
4 Learning to reason, use tools, and act
The survey reviews supervised, bootstrapped, and reinforcement-learning approaches for teaching language models to reason, use tools, and act. These approaches exploit demonstrations, feedback, or interaction, but face constraints including costly supervision, context limits, and unstable RL training.
- Supervised approaches: Few-shot prompting teaches reasoning and tool use from human-written in-context demonstrations without updating model parameters.Its effectiveness depends on example format, selection, and ordering, while supervision is limited by the context window.
- Supervised approaches: Fine-tuning teaches models to use search engines, web browsers, calculators, translation systems, and reasoning procedures through supervised parameter updates.
- Bootstrapping: Bootstrapping filters model-generated reasoning or actions using final-answer correctness, providing indirect supervision without requiring full human demonstrations.
- Reinforcement learning: Reinforcement learning is suited to tool use because many tools are nondifferentiable and tool-based tasks often form sequential decision problems with defined rewards.
- Reinforcement learning: RL-based systems have learned information seeking, web browsing, and shopping behaviors, with WebGPT answers preferred to human-generated answers on two QA datasets.Yao et al. found that RL fine-tuning after human-demonstration training improved shopping performance, although the model still lagged human experts.
- Limitations and future directions: Current RL methods remain unstable and can make training difficult and slow, while supervised approaches require large, costly collections of expert demonstrations.Bootstrapping and offline RL are presented as possible ways to combine more stable training with feedback-based improvement under limited data.
5 Discussion
The discussion frames ALMs as language models that extend token prediction with reasoning, external tools, and potentially action, while weighing their benefits, tradeoffs, and open limitations. It highlights non-parametric augmentation as a route toward broader capabilities and potentially smaller, more current systems, but notes unresolved questions about autonomy, truthfulness, interpretability, and scalable learning.
- Moving away from language modeling: ALMs extend language modeling by incorporating intermediate reasoning, external tools, and potentially non-parametric modules into prediction.The survey distinguishes this augmented setting from classical language modeling because external modules can expand the model’s context and capabilities.
- Generalizing the non-parametric framework: Tool augmentation generalizes non-parametric memory beyond retrieval, allowing models to delegate abilities such as calculus and access recent information externally.Avoiding rarely accessed knowledge in model weights may support smaller models and reduce the need for frequent updates.
- A path towards autonomous machine intelligence?: The survey questions whether token-level ALMs can achieve global consistency and human-like System 2 reasoning across long sequences of subgoals.It identifies inconsistency beyond maximum sequence length as a limitation for complex tasks such as writing a research paper.
- A path towards autonomous machine intelligence?: ALM-based agents can use language models as planners or communicators, but specialized planning models remain an alternative to centralized LM control.The survey contrasts Minecraft planning with Diplomacy, where an ad-hoc planner handles game dynamics and the LM serves as a communication module.
- Augmented Language Models benefits: ALMs may improve truthfulness, uncertainty estimation, interpretability, and task coverage, but each benefit remains qualified by evidence or implementation challenges.The survey notes limited evidence for retrieval-based truthfulness, potentially misleading reasoning traces, and the need to determine when models should query tools.
- Ethical concerns: Active tools raise ethical risks because incorrect outputs may appear more trustworthy and authoritative, especially when LM reasoning resembles human reasoning.The discussion distinguishes tools that gather information from tools that affect the virtual or physical world.
6 Conclusion
The conclusion presents ALMs as models augmented with reasoning and tools, often through additional relevant information in context. It identifies scalable self-supervised augmentation and the integration of reasoning with tool use as open challenges, while describing ALMs as a promising direction for complex human-machine interaction.
- Conclusion: The survey organizes ALM research around reasoning, tool interaction and action, and whether augmentations are heuristic or learned.Its taxonomy uses three axes to structure the surveyed works.
- Conclusion: Many ALMs use non-parametric modules to augment context, marking a departure from classical language modeling.The survey uses this distinction to motivate the term Augmented Language Models.
- Conclusion: Fully self-supervised learning of meaningful augmentations remains an open research question because many existing approaches rely on human annotation.The conclusion cites hand-crafted prompting and reinforcement learning from human feedback as examples of non-scalable supervision.
- Conclusion: Future work should study the integration and interaction of reasoning and tools, since relatively few works combine both abilities.The conclusion links this gap to the broader goal of systems capable of complex and useful human-machine interaction.