Source-linked AI summary
A Survey of Reinforcement Learning Informed by Natural Language
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktäschel
TL;DR
RL remains sample inefficient and often generalizes poorly, while natural language can express goals, domain knowledge, and world structure. This survey organizes research integrating language into RL and concludes that progress requires learning from richer text, stronger NLP tools, and more realistic environments.
Problem
Traditional RL agents remain sample inefficient and generalize poorly, motivating language integration for real-world sequential decision making.
Method
The paper surveys language-conditional and language-assisted RL, covering instruction following, textual environments, domain-knowledge transfer, and language-structured policies.
Results
The survey finds that research is concentrated in language-conditional and task-dependent settings, with limited work using descriptive text and compositional language-based policies.
Takeaways & Limitations
Advancing language-informed RL requires naturally occurring corpora, NLP tools such as pretrained language models, diverse real-world environments, and standardized evaluations.
Takeaways & Limitations
RL methods remain sample inefficient and often generalize poorly beyond tasks resembling their training tasks.
Abstract
from arXiv · showhide
To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand. Recent advances in representation learning for language make it possible to build models that acquire world knowledge from text corpora and integrate this knowledge into downstream decision making problems. We thus argue that the time is right to investigate a tight integration of natural language understanding into RL in particular. We survey the state of the field, including work on instruction following, text games, and learning from textual domain knowledge. Finally, we call for the development of new environments as well as further investigation into the potential uses of recent Natural Language Processing (NLP) techniques for such tasks.
1 Introduction
Natural language can give RL agents abstractions, world knowledge, and task guidance that traditional approaches often lack. The survey organizes this integration by language’s role, task dependence, and use across instruction following, textual knowledge transfer, and policy structure.
- Traditional RL and imitation learning agents typically struggle to learn efficiently from rich environments and lack language-enabled abstraction and communication capabilities.
- Language may improve generalization and sample efficiency when environment interactions are limited or human priors are easy to express linguistically.
- Generic corpora can provide transferable syntactic and semantic representations, while task-specific wikis and manuals may convey decision-relevant knowledge.
- The survey distinguishes language-conditional RL, where language is part of the task, from language-assisted RL, where it facilitates learning.
- The authors review prior work and advocate new environments plus pretrained language models and parsers for informing agents about world structure.
2 Background
RL formulates sequential decision making as optimizing expected discounted return, while imitation learning uses trajectories when rewards are unavailable. Despite deep-learning advances, RL remains sample inefficient and generalizes poorly, motivating language-based transfer of world and task knowledge.
- RL models sequential decision making as an MDP and seeks a policy maximizing expected discounted cumulative reward.
- Imitation learning learns policy approximations from optimal or sub-optimal trajectories when rewards are not observed.
- Deep learning has extended RL to high-dimensional games and complex planning, but these methods often require millions or billions of interactions.
- RL methods often generalize poorly to tasks only slightly different from training tasks, limiting their use in real-world settings.
- Natural language can communicate goals, policy constraints, reward information, and transition knowledge for transfer into sequential decision making.
3 Current Use of Natural Language in RL
The surveyed literature integrates language into RL either as a required part of the task or as assistance for transferring knowledge and structuring policies. These settings span instructions, reward induction, text environments, descriptive corpora, and language-informed representations.
- The survey separates language-conditional RL, where language is necessary, from language-assisted RL, where text facilitates learning.
- Language-conditional RL: Language-conditional work includes instruction following, reward induction from language, and environments with text in observations or actions.
- Instruction following: Instruction-following agents interpret high-level goals or policies and execute low-level actions to reach specified outcomes.
- Language in the Observation and Action Space: Natural-language interaction challenges RL because it requires commonsense and context while producing combinatorially expanding observation and action spaces.
- Text environments: Text games support structure learning, knowledge extraction, and transfer, while constrained verb-object actions can motivate decomposed Q-functions.
- Language-assisted RL: Language-assisted RL transfers domain knowledge through informative features, state or entity annotations, subtasks, retrieved text, and grounded policy representations.
4 Trends for Natural Language in RL
The literature is concentrated in language-conditional, task-dependent, and synthetic settings, with relatively little work on descriptive text or compositional policy structure. The authors therefore call for naturally occurring corpora, realistic environments, and standardized evaluations.
- Language-conditional studies outnumber language-assisted studies, and task-dependent text is more common than task-independent text.
- Only a handful of papers study unstructured descriptive text, and few explore language-based internal plans or compositional representations.
- Natural language remains less standard than synthetic language in instruction-following research.
- The authors advocate learning from naturally occurring text, more diverse real-world environments, and standardized environments and evaluations.
- Pretrained language models and explicit language-based knowledge querying are identified as promising directions for transferring knowledge to RL agents.
- Existing benchmarks often use small, artificial language corpora and do not adequately test adaptation to new goals, rewards, or dynamics.
5 Conclusion
The survey identifies tabula rasa RL as sample-inefficient and difficult to transfer, and calls for language-based knowledge transfer alongside more challenging, realistic environments.
- Tabula rasa RL gathers policy information only through direct interaction and reward, resulting in low sample efficiency and poor transfer across environments.
- Language can transfer knowledge from corpora to RL tasks and between tasks, potentially supporting more diverse and real-world applications.
- The survey advocates learning from unstructured or descriptive language with greater use of pre-trained language models.
- It also calls for more challenging environments that reflect real-world semantics and diversity.