PaperlayerView exact revision on arXiv

Source-linked AI summary

EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning

Xiaoqian Liu, Ke Wang, Yongbin Li, Yuchuan Wu, Wentao Ma, Aobo Kong, Fei Huang, Jianbin Jiao, Junge Zhang

arXiv:2502.12486v6cs.CL

TL;DR

LLMs remain challenged by dynamic real-world scenarios that require strategic reasoning, while existing approaches face limitations in adaptation, transfer, and scalability. EPO trains a separate strategic reasoning LLM with multi-turn RL, process rewards, and iterative self-play to guide arbitrary LLM agents. Experiments report stronger long-term goal alignment across social and physical domains, with state-of-the-art results on social dialogue and web navigation.

  • Problem

    Current LLMs struggle to integrate long-term goal alignment, uncertainty management, and adaptation in dynamic settings, while existing strategic-reasoning methods face limits in adaptation, transfer, and scalability.

    Source: §1 · p. 1

  • Method

    EPO trains a separate strategic reasoning LLM through multi-turn RL with process rewards and iterative self-play to provide strategies for arbitrary LLM agents.

    Source: §1 · p. 2

  • Results

    Experiments across social and physical domains report improved long-term goal alignment and state-of-the-art performance on social dialogue and web navigation tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EPO's analyses identify collaborative reasoning mechanisms and novel strategies in interactions between its reasoning model and LLM agents.

    Source: §5 · p. 9

    Evaluations involve at most two agents, training is limited to 8B/7B models, and an off-the-shelf LLM serves as the process reward model.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown impressive reasoning capabilities in well-defined problems with clear solutions, such as mathematics and coding. However, they still struggle with complex real-world scenarios like business negotiations, which require strategic reasoning-an ability to navigate dynamic environments and align long-term goals amidst uncertainty. Existing methods for strategic reasoning face challenges in adaptability, scalability, and transferring strategies to new contexts. To address these issues, we propose explicit policy optimization (EPO) for strategic reasoning, featuring an LLM that provides strategies in open-ended action space and can be plugged into arbitrary LLM agents to motivate goal-directed behavior. To improve adaptability and policy transferability, we train the strategic reasoning model via multi-turn reinforcement learning (RL),utilizing process rewards and iterative self-play. Experiments across social and physical domains demonstrate EPO's ability of long-term goal alignment through enhanced strategic reasoning, achieving state-of-the-art performance on social dialogue and web navigation tasks. Our findings reveal various collaborative reasoning mechanisms emergent in EPO and its effectiveness in generating novel strategies, underscoring its potential for strategic reasoning in real-world applications. Code and data are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/EPO.

1 Introduction

Strategic reasoning requires long-term goal alignment, adaptation, and uncertainty management in dynamic settings, capabilities current LLMs struggle to integrate. EPO pairs LLM agents with a separate strategic reasoning model trained through multi-turn RL, and experiments report stronger goal alignment across social and physical domains.

1 Introduction

  • Strategic reasoning requires aligning long-term goals, managing uncertainty, and adapting to changing conditions, yet current LLMs struggle to integrate these capabilities in interactive contexts.Static tasks with fixed rules and deterministic outcomes do not capture the complexity of real-world scenarios such as business negotiations.Source: §1 · p. 1
  • Prior approaches use iterative prompting, imitation or reinforcement learning, and inference-time search, but face limits in real-time adaptation, transfer to unseen domains, or open-ended-action scalability.EPO is positioned as addressing these challenges with a separately trained strategist that can assist agents across settings.Source: §1 · pp. 1–2
  • EPO outperforms baselines in long-term goal alignment across diverse domains, with state-of-the-art performance on social dialogue and web navigation tasks.The analysis also reports emergent collaborative reasoning mechanisms and creative strategies.Source: §1 · pp. 2–3
  • EPO uses a dedicated LLM to provide real-time strategies that motivate arbitrary LLM agents toward goal-directed behavior in dynamic interactive environments.The strategic model can be plugged into agents while leaving the interacting agents unchanged, and strategies are formulated in open-ended language space.Source: §1 · p. 2
  • EPO trains its strategic reasoning model with multi-turn RL, process rewards, and iterative self-play to improve adaptability and policy transfer without preliminary supervised fine-tuning.The paper describes REINFORCE as its policy-gradient objective and a process reward model for assessing strategies.Source: §1 · p. 2

2 Related Work

Prior LLM strategic-reasoning work includes prompting, fine-tuning, modular tools, and multi-agent methods, but transfer, scalability, and action-space constraints remain. This paper frames strategic reasoning as a multi-turn RL challenge and supports open-ended natural-language strategies.

Strategic reasoning in LLMs

  • Prior strategic-reasoning approaches include prompting for opponent analysis, theory of mind, or demonstrations; IL/RL fine-tuning; and modular memory, tools, or multi-agent systems.Examples include dialogue action planners trained with RL and game-theoretic algorithms.Source: §2 · p. 3
  • These approaches are limited by inherited reasoning abilities, unfaithful explanations, weak cross-domain generalization, or reliance on finite predefined action sets.Some modular planners also lack interpretability, while search and prompting approaches face scalability or adaptation challenges.Source: §2 · p. 3

Reinforcement learning for LLMs

  • Much prior LLM reinforcement learning targets static problem solving or one-step preference rewards, whereas this work treats strategic reasoning as an interactive multi-turn RL challenge.The paper states its feasibility result is independent of the specific RL algorithm used.Source: §2 · p. 3

3 Method

EPO separates strategic reasoning from an LLM agent’s behavior: a dedicated model proposes strategies that guide the agent toward long-term goals. It trains that model through multi-turn reinforcement learning, using process rewards and iterative self-play to refine strategies from environmental feedback.

3.1 Overview

  • EPO’s strategic model uses the goal, interaction history, prior strategies, and latest observation to propose open-ended strategies that guide an LLM agent’s goal-directed behavior.The figure distinguishes the model’s RL training process from its use to motivate agent behavior.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Learning to Reason for Optimal Strategies

  • The strategic policy is formulated as a POMDP because the reasoning model has access only to partial information about the environment.The objective is to maximize expected return over the trajectory.Source: §3.2 · p. 4
  • An REINFORCE objective updates strategy-token probabilities using advantages based on discounted future rewards, with maximum absolute normalization serving as a variance-reduction baseline.The advantage reflects a strategy’s contribution to long-term goal alignment relative to other strategies in the trajectory.Source: §3.2 · p. 4
  • A process reward marks a strategy as critical to goal progress, and a process reward model identifies critical strategy indexes using trajectory history and final goal-completion rate.Critical strategies receive reward 1; other strategies receive 0.Source: §3.2 · p. 4
  • Iterative self-play trains two EPO instances as different partners, using recorded trajectories and process rewards to retrain the strategy model until its policy stabilizes.The LLM agents’ weights remain unchanged during training to support generalization across domains without overfitting to particular behavior patterns.Source: §3.2 · pp. 4–5
  • EPO’s open-ended strategies and plug-in design let the reasoning model train across domains and transfer its policy to novel scenarios without additional training of the agent.The agent retains its general-domain capabilities.Source: §3.2 · p. 5

4 Experiment

EPO is evaluated across social and physical tasks, with results reported for its overall performance, iterative self-play, collaborative reasoning, ablations, and case studies. The experiments report state-of-the-art results on SOTOPIA and WebShop, alongside findings about when collaboration helps and where EPO may struggle.

4.1 Experimental Settings

  • Evaluation spans SOTOPIA, WebShop, and ALFWorld; SOTOPIA training scenarios do not overlap with test scenarios.SOTOPIA uses goal completion and overall score; WebShop and ALFWorld use average reward.Source: §4.1 · p. 5

4.2 Results and Analysis

  • EPO outperforms baselines and exceeds state-of-the-art results on SOTOPIA and WebShop.The reported comparisons are in Tables 1 and 2.Source: §4.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6
  • Test goal completion improves with iterative self-play, with gains diminishing after round five; training-data goal completion rises until policy convergence.Figure 3 shows test scenarios on the left and training data by iteration on the right.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Bidirectional EPO-RL achieves the highest average goal in both symmetric self-play and asymmetric interactions, and participants tend to identify mutual-benefit opportunities.Figure 4 compares ReAct, mixed EPO-RL configurations, and EPO-RL on both sides.Source: §4.2 · p. 7, Figure 4 · p. 8
  • In competitive dialogues, RL-trained participants can score below weaker partners, potentially because exposed strategies are exploited; asymmetric strong–weak play can also lower joint benefits.The authors suggest strategies embedded in dialogue history may be exploited; such interactions may fail to reach Pareto optimality.Source: §4.2 · pp. 7–8
  • Removing RL or process rewards degrades performance, while adding SFT before RL brings no significant gain and omitting SFT halves training time.The ablations are reported in Table 4.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Case Studies

  • In a negotiation case study, EPO-RL generates flexible strategies such as urgency creation and value-based persuasion, unlike static ReAct strategies and patterned SFT strategies.The examples are from SOTOPIA and are attributed to explicit policy optimization and RL-based strategy reasoning.Source: §4.3 · p. 9
  • EPO may struggle against adversarial, deceptive, or aggressive behavior; in a conflicting-goal case, a rigid opponent is associated with reduced flexibility and a participant failing its goal.The authors present this as a case study of a potential failure mode.Source: §4.3 · p. 9

5 Conclusion

EPO uses a strategic reasoning model trained through multi-turn RL to assist LLM agents in dynamic environments. The conclusion reports policy transfer across scenarios and superior long-term goal alignment, while noting emergent collaborative reasoning and novel strategies.

5 Conclusion

  • EPO uses a strategic reasoning model trained through pure RL to support arbitrary LLM agents in goal-directed behavior in dynamic environments.The reasoning model is trained with process rewards and iterative self-play in a lightweight multi-turn RL pipeline.Source: §5 · p. 9
  • The pipeline supports policy transfer across scenarios while leaving the interacting LLM agent unchanged to maintain cross-domain generalization.The authors report computational costs similar to SFT methods.Source: §5 · p. 9
  • EPO reports superior long-term goal alignment, diverse collaborative reasoning mechanisms, and novel strategies produced by its reasoning model.The conclusion frames these results as advancing strategic reasoning for complex real-world scenarios.Source: §5 · p. 9

Limitations

The evaluation is limited to settings with at most two agents and 7B/8B models, and uses an off-the-shelf process reward model and final goal-completion scores. The authors identify larger models, more domains, more reliable process supervision, and strategy-quality metrics as directions for future work.

Evaluation scope

  • The tested social and physical environments involve at most two agents, leaving more complex multi-agent settings such as Diplomacy and Hanabi for future study.The authors also note that computational constraints limited training to 8B/7B models and did not allow scaling multi-turn RL to larger models.Source: Limitations · p. 9

Reward modeling

  • Training uses an off-the-shelf LLM as the process reward model; the authors suggest learning a more reliable process-supervised reward model.Source: Limitations · p. 9

Evaluation metrics

  • Performance is evaluated using the final goal-completion score, while strategy quality and diversity are left for future metrics.Source: Limitations · p. 9

Ethics Statement

The ethics statement warns that strategic reasoning may favor success metrics over value-driven decision-making without appropriate safeguards. It also points readers to a table comparing strategic reasoning with static problem-solving.

Ethics Statement

  • Without safeguards, strategic reasoning models might prioritize success metrics over nuanced, value-driven decision-making.Source: Ethics Statement · p. 9
  • In competitive or adversarial interactions, an unsupervised model may pursue favorable outcomes without considering ethical implications or long-term human-centric values.The authors warn this narrow focus could reinforce undesirable behaviors or overlook empathy and integrity, and call for mechanisms balancing strategic efficacy with ethical considerations.Source: Ethics Statement · p. 10

A Strategic Reasoning vs. Static Problem-Solving

B Environments and Datasets

The appendix describes social and interactive physical environments used in EPO experiments and the data sources for training. These include SOTOPIA and SOTOPIA-π social tasks, WebShop shopping tasks, and ALFWorld household tasks.

SOTOPIA

  • SOTOPIA simulates open-ended, goal-oriented social interactions, while SOTOPIA-π uses newly constructed scenarios, profiles, and goals that do not overlap with SOTOPIA.SOTOPIA tasks combine a scenario, two role profiles, and private social goals; episodes are assessed across seven dimensions whose scores are averaged.Source: B Environments and Datasets · p. 14

SOTOPIA training data

  • GPT-4-Turbo self-chat in SOTOPIA-π supplies reasoning, strategy, and response data, while iterative self-play uses the RL-trained reasoning model with GPT-4-Turbo.The model collects strategy data and GPT-4-Turbo collects dialogue histories before each iteration.Source: B Environments and Datasets · p. 14

WebShop

  • WebShop tasks ask agents to purchase products matching user specifications, with final rewards based on programmatic matching of product attributes, options, and price.Source: B Environments and Datasets · p. 14

WebShop training data

  • WebShop training uses GPT-4 exploration trajectories with rewards above 0.7 and GPT-4-generated rationales for each action as strategies.Source: B Environments and Datasets · p. 14

ALFWorld

  • ALFWorld presents text-based household environments and high-level instructions, with training data combining successful trajectories and failed GPT-4 trajectories containing rationales and action information.Source: B Environments and Datasets · p. 14

C Additional Implementation Details

Implementation details cover the training protocol, evaluation setup, and baseline methods. Experiments use a fixed gradient-update budget and a single reported run, alongside comparisons involving ReAct, PPDPP, and DAT.

Training protocol

  • Training uses a fixed budget of gradient updates without hyperparameter changes, and results are reported from a single run because of computational costs.Final checkpoints are selected for each run, with a small held-out validation set available; experiments use six NVIDIA A100 80G GPUs.Source: C Additional Implementation Details · p. 14

Additional Implementation Details

  • Table 6 is titled as a comparison of challenges in strategic reasoning versus static problem-solving.Source: Table 6 · p. 15
  • Table 7 covers SOTOPIA self-chat performance and computational costs, with GPT-4o serving as the EPO dialogue agent.It compares SFT and RL training of a single Llama3-8B-Instruct model that outputs strategy and behavior at each interaction turn.Source: Table 7 · p. 15

Baseline Implementations

  • The baselines include ReAct, a supervised-initialized RL dialogue planner called PPDPP, and DAT, a continuous-action planner optimized with TD3-BC.The PPDPP planner supports one LLM agent during communication; DAT is trained on SOTOPIA-π and evaluated on all SOTOPIA scenarios.Source: C Additional Implementation Details · p. 15

D Computational Costs

EPO-RL and EPO-RL with self-play have similar training times, while adding SFT first doubles total training time. The appendix also specifies how prompts guide social dialogue and substitute SOTOPIA template fields.

D Computational Costs

  • EPO-RL and EPO-RL+Self-play have similar training times, while EPO-RL+SFT takes twice as long as EPO-RL.The self-play methods use the same amount of training data, although each self-play iteration uses only a subset of SOTOPIA-π scenarios.Source: D Computational Costs · p. 15
  • The social prompt asks the agent to reason about its goal and dialogue history, generate a concise dialogue policy, select an action type, and produce an argument.The prompt emphasizes natural conversation, relationships, information gathering, and protecting secrets while pursuing the agent’s goal.Source: D Computational Costs · pp. 16–17
  • The SOTOPIA-π prompt template substitutes participant names, turn number, dialogue history, action list, and output-format instructions for placeholders.Source: Table 8 · p. 17

E Prompts

The appendix provides environment-specific prompts for guiding agents and selecting process-reward evidence. The prompts describe the information presented to the model and the outputs it should return.

E.1 Evaluation Prompts

  • The WebShop evaluation prompt guides an agent from its shopping goal, latest action, and resulting observation toward a useful next action.It asks the guide to assess progress and anticipate future actions or obstacles.Source: §E.1 · p. 18
  • The ALFWorld evaluation prompt frames the model as a guide to household tasks using the goal, environment description, actions, and observations.An observation of “Nothing happens” indicates that the preceding action was invalid.Source: §E.1 · p. 18
  • The SOTOPIA evaluation prompt’s placeholders can be replaced with the names of the two dialogue participants.Source: Table 9 · p. 19

E.2 PRM Prompts

  • For SOTOPIA process-reward prompts, the model selects important strategy rounds from dialogue history and returns indexes with its reasoning.The score ranges from 0 to 10, with higher values indicating progress toward the agent’s goal; the table caption notes that strategies are private to their corresponding agents.Source: §E.2 · pp. 19–20, Table 10 · p. 21
  • The WebShop process-reward prompt selects thoughts considered critical to achieving the shopping goal and returns selected indexes with reasoning.The input includes the shopping goal, thoughts, and the agent’s action-observation trajectory.Source: §E.2 · p. 20
  • The ALFWorld process-reward prompt selects thoughts judged critical to task completion, using agent thoughts, actions, and environment observations.The score is binary: 0.0 for an incomplete task and 1.0 for a completed task.Source: §E.2 · pp. 20–21

F Case Studies

The case studies illustrate strategic reasoning in bargaining and inmate dialogue, alongside WebShop and ALFWorld action examples. A SOTOPIA example also shows a failure against uncompromising behavior.

F Case Studies

  • In the ReAct negotiation, Lily opens at $4600 and Hendrick proposes $4800 with the smog certificate included.The scenario sets Lily’s target at $4600 and Hendrick’s at $3260.Source: F Case Studies · pp. 21–22
  • EPO-SFT’s negotiation ends with Lily accepting $4750, while EPO-RL’s excerpt shows a $4300 offer, a $4700 counter, and a later $4600 counter.The examples describe EPO-SFT and EPO-RL as strategic reasoning models plugged into both dialogue agents.Source: F Case Studies · pp. 22–23, Table 11
  • The WebShop and ALFWorld examples pair EPO-Llama3-8B RL-generated strategy thoughts with GPT-4o-produced actions, using short CoT rationales as strategies.Source: Table 12 · p. 23, Table 13 · p. 24
  • In the inmate dialogue, Lena maintains her innocence despite appeals, and Isabella later suggests using Lena’s architectural skills to help others.Isabella’s initial appeal presents confession as a chance for a life sentence rather than execution.Source: F Case Studies · pp. 24–25
  • A SOTOPIA failure example describes Person1 failing to achieve the goal against the other person’s uncompromising behavior.Source: Table 14

Related papers

Loading 2502.12486v6…