Source-linked AI summary
The Rise of Verbal Reinforcement Learning
Kshitij Tayal, Arun Sharma, Genta Indra Winata, Anirban Das, Sambit Sahu
TL;DR
Language-agent tasks are difficult to specify and supervise when goals and contexts are ambiguous, motivating a unified account of natural-language feedback. The paper surveys Verbal Reinforcement Learning through a three-pillar taxonomy based on when language acts and what it modifies, finding broad improvements across settings while noting scope limitations in its organizing lens.
Problem
Agent development often requires formalized environments, interaction spaces, and rewards, while many tasks and goals remain ambiguous, context-dependent, and difficult to encode directly.
Method
The paper presents the first comprehensive VRL survey, organizing representative methods by when language enters the agent lifecycle into grounding, deliberative feedback, and learning signals.
Results
Verbal feedback produces substantial improvements across diverse settings, including 91% pass@1 on HumanEval through verbal self-reflection and a 1.3B model outperforming the 175B GPT-3 baseline after verbal preference training.
Takeaways & Limitations
The three pillars frame verbal feedback as a common medium for defining, updating, and improving agents across their development lifecycle.
Takeaways & Limitations
The survey focuses on representative rather than exhaustive papers, and its taxonomy may oversimplify methods spanning multiple language roles while excluding work where language is auxiliary.
Abstract
from arXiv · showhide
Natural language is emerging as a primary feedback channel for improving language agents, capable of conveying intent, preferences, and causal structure in forms interpretable by both humans and modern language models. We call this paradigm Verbal Reinforcement Learning (VRL) and offer the first unified account of it. We organize the field around a single axis, \textit{when} verbal feedback takes effect in an agent's lifecycle and \textit{what} it modifies, yielding three pillars: (1) \textbf{Language as Grounding Signal}, where language defines the task itself by specifying goals, states, and reward structures; (2) \textbf{Language as Deliberative Feedback}, where natural language guides reasoning at test time without the need to update model parameters; (3) \textbf{Language as Learning Signal}, where language-based feedback shapes model parameters through training. Within each pillar, we synthesize representative work, distinguish key subcategories of approaches, and outline the distinct role language plays in shaping agent behavior. Together, this taxonomy shows how verbal reinforcement is reshaping agent development, while also defining the challenges and opportunities for building more capable and aligned agents.
1 Introduction
The paper introduces Verbal Reinforcement Learning as a unified framework for using natural-language feedback to guide agent behavior. It surveys representative methods and organizes them around when language enters the agent lifecycle and what it modifies.
- Motivation: Classical reinforcement learning depends on heavily engineered environments, interaction spaces, and reward functions, especially for ambiguous or context-dependent tasks.The paper identifies this engineering burden as a long-standing bottleneck.
- Definition: VRL uses natural-language feedback from agents, humans, tools, or models to improve behavior and guide future decision-making.Feedback may remain textual or be converted into a training signal for parameter updates.
- Evidence: 91% pass@1 on HumanEval was achieved through verbal self-reflection alone, while a 1.3B-parameter model trained on verbal preferences outperformed the 175B GPT-3 baseline.The paper presents these findings as evidence that verbal feedback improves performance across diverse settings.
- Contribution: The survey unifies methods that operate during inference, modify training data, or redefine the task itself under a taxonomy based on language’s lifecycle timing and functional role.This framing distinguishes VRL from prior surveys focused on narrower language-conditioned or self-correction settings.
- Scope: The survey synthesizes representative methods, identifies cross-cutting challenges, and outlines future directions for robust, capable, and aligned verbal-RL agents.Its scope spans coding, scientific discovery, robotics, mathematical reasoning, clinical decision-making, and education.
2 Taxonomy
The taxonomy classifies VRL by when language acts and what it changes: task specification, single-episode reasoning, or persistent policy parameters. The three pillars can operate together across one agent lifecycle.
- Organizing principle: The taxonomy organizes VRL methods along one axis: when natural language enters the agent lifecycle and consequently what it modifies.The framework is summarized as three pillars.
- Language as Grounding Signal: Grounding signal acts before policy execution by specifying goals, states, actions, and rewards that define the task’s MDP.A natural-language task description can determine what the agent perceives, how it acts, and what counts as success.
- Language as Deliberative Feedback: Deliberative feedback acts at inference time by refining outputs or reasoning through critique, memory, debate, or search without updating model parameters.It redirects a single generation episode while leaving the model unchanged for future tasks.
- Language as Learning Signal: Learning signal acts during training by converting verbal feedback into gradient updates that persistently reshape the policy.Preference judgments can alter model weights and affect subsequent interactions.
- Lifecycle interaction: The pillars differ in persistence: grounding defines the task space, deliberative feedback improves one episode, and learning signal changes the future policy.The paper treats them as complementary rather than competing roles.
- Working Example: In a coding agent, issue descriptions ground the task, test traces support inference-time deliberation, and later trajectories become preference-optimization data for training.These stages consume verbal feedback at three distinct timescales.
3 Pillar 1: Language as Grounding Signal
Language as Grounding Signal uses verbal feedback to define the agent’s task by specifying goals, states, actions, and rewards. The central challenge is mapping language precisely enough to executable MDP components.
- Overview: Grounding maps language to four MDP elements: goals, states, actions, and rewards.Goal grounding specifies objectives; state grounding represents observations as text; action grounding resolves instructions to executable behavior; reward code generation compiles language into executable reward functions.
- Goal Grounding: Goal grounding parses instructions into executable objectives, including objects, relations, target conditions, and verifiable subgoals.Methods also filter plans for physical executability and extend grounding to multi-agent coordination and hierarchical decomposition.
- State Grounding: State grounding represents observations in language, but its effectiveness depends on preserving sufficient information for policy operation.Agents trained on verbal state descriptions can transfer to embodied 3D settings, indicating language can provide a modality-invariant state representation.
- Action Grounding: Action grounding connects verbal instructions to skills, tool calls, or motor commands and uses execution outcomes as language for replanning.The main challenge is granularity alignment because instructions may denote extended skills, discrete subgoals, or motor commands.
- Reward Code Generation: Reward code generation grounds success by compiling verbal descriptions into executable reward functions.Eureka iteratively refines reward code from training summaries, while later work addresses dense shaping, dynamic trajectory feedback, offline preference optimization, and video demonstrations.
4 Pillar 2: Language as Deliberative Feedback
Language as Deliberative Feedback acts at inference time to refine reasoning or outputs without updating model parameters. Its mechanisms range from self-critique and external verification to debate, memory, and search over candidate paths.
- Overview: Deliberative feedback refines a single episode without updating model parameters, using critique from models, tools, peers, past episodes, or search paths.This pillar closely aligns with test-time compute scaling and distinguishes mechanisms by the source of critique.
- Self-Critique: Self-critique asks a model to evaluate and revise its own output, but shared blind spots can limit correction.Separately trained critics can enhance reasoning, while internal feedback cannot identify knowledge gaps when errors arise from missing information.
- External Critique: External grounding anchors verbal feedback in deterministic outputs such as execution traces, unit-test verdicts, search results, and API responses.Verified signals support corrected outputs, and end-to-end reinforcement learning can improve models’ use of such feedback.
- Multi-Agent Debate: Multi-agent debate distributes critique across parallel model instances, often using distinct personas to increase perspective.Communication topology matters: full broadcast incurs token overhead, while debate is ineffective when participants share biases and produce redundant consensus.
- Experiential Memory: Experiential memory stores reusable lessons across episodes as episodic logs, semantic rules, or procedural skills.Reflexion stores raw feedback, ExpeL extracts insights from successes and failures, and Voyager accumulates executable skills.
- Search-Guided Deliberation: Search-guided deliberation explores multiple candidate paths and uses verbal feedback to expand, prune, or abandon them.Applying feedback at each thought level enables earlier identification of unproductive paths, but multiple LLM calls make these methods more expensive than single-pass generation.
5 Pillar 3: Language as Learning Signal
Language as Learning Signal persistently reshapes model policies through training, organized as a spectrum from preserving full critiques to compressing judgments into scalars. The methods differ in how much linguistic information remains in the training signal.
- Overview: Training-time methods form a compression spectrum from full verbal critiques to scalar judgments used for policy updates.The four categories are feedback-conditioned modeling, self-improvement, process supervision, and preference shaping.
- Feedback-Conditioned Modeling: Feedback-conditioned modeling trains on input, critique, and revised output, optionally retaining the critique as conditioning context.ALT showed that retaining feedback enables models to learn explicit mappings from critique to correction, whereas earlier work supervised only refined outputs.
- Self-Improvement: Self-improvement generates multiple trajectories, filters them with verbal feedback, and uses retained outputs as supervised fine-tuning data.Filtering can use final-answer correctness, safety-principle adherence, or majority voting across diverse samples.
- Process Supervision: Process supervision converts verbal evaluations of intermediate reasoning steps into scalar scores for training a process reward model.Step-level supervision substantially outperforms outcome-level supervision because localized feedback enables more precise credit assignment.
- Preference Shaping: Preference shaping maximally compresses verbal feedback by reducing comparisons between responses to a single scalar guiding policy updates.InstructGPT trains a reward model on pairwise rankings, while DPO further simplifies preference-based optimization.
6 Insights, Discussion and Future Prospects
The discussion identifies feedback quality, tool interfaces, adversarial robustness, and missing formal guarantees as central cross-cutting issues for Verbal RL. It highlights dedicated critics, interpretable tools, provenance-aware defenses, and theoretical frameworks as directions for addressing them.
- Feedback-model quality: Dedicated critics improve feedback quality over reusing the same LLM for generation and critique.Shepherd provides more actionable feedback, CriticGPT detects code bugs more reliably, and CritiqueLLM scales critique data through multi-path prompting.
- Feedback-model quality: Feedback-tuned models should target error localization, actionability, and calibration while preserving verbal information in training signals.Span-level feedback gradients, didactic interactions, and generative process reward models are identified as relevant developments.
- Tool-interface design: Tool interfaces need interpretable, structured outputs because ambiguous or terse feedback can cause agent failures.LLM-optimized interfaces enhance performance, ambiguous outputs caused many failures across 750 API calls, and executable code actions provide structured feedback.
- Adversarial robustness: Adversarial feedback can manipulate VRL policies beyond standard prompt injection, with adaptive attacks bypassing tested defenses and tool-level injection reaching 96.7% success on GPT-4o.The paper calls for feedback provenance mechanisms and adversarial feedback benchmarks.
- Formal foundations: VRL lacks formal guarantees about when verbal feedback improves sample efficiency or policy quality.The paper proposes PAC-learning, POMDP, and rate-distortion perspectives to formalize critic error, partial observation, and signal retention.
7 Conclusion
The survey unifies Verbal Reinforcement Learning through three pillars organized by when language enters an agent’s lifecycle. It argues that verbal feedback is becoming central to defining, updating, and improving agents, while future systems must verify feedback provenance, quality, and robustness.
- Conclusion: The survey organizes VRL into grounding signal, deliberative feedback, and learning signal according to when language enters the agent lifecycle.This taxonomy provides the paper’s unified account of the field.
- Conclusion: Verbal feedback is becoming a primary medium for defining, updating, and improving agent behavior.The conclusion frames this as a common pattern revealed by the taxonomy.
- Future prospects: Future agents may consume verbal feedback in unified loops spanning grounding, deliberation, and learning within one trajectory.The paper anticipates increasingly blurred boundaries between the pillars.
- Future prospects: Feedback provenance, quality benchmarks, and adversarial robustness are identified as infrastructure requirements as verification becomes more important.The paper predicts a shift in the bottleneck from generating feedback to verifying it.
Limitations
The survey offers broad coverage through representative papers, but its taxonomy is an organizing lens rather than an exhaustive or strict partition of the literature.
- Scope: The survey focuses on representative papers rather than attempting an exhaustive enumeration.Its coverage is broad but intentionally selective.
- Taxonomy boundary: The three-pillar taxonomy may oversimplify methods in which language simultaneously grounds, guides deliberation, and shapes learning.The paper therefore presents the taxonomy as an organizing lens rather than a strict partition.
- Scope: The survey does not fully cover adjacent work where language plays a more auxiliary role.Its primary focus is methods where verbal feedback is explicit and central to the agent’s process.