PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, Tat-Seng Chua

arXiv:2311.00262v2cs.CLcs.AI

TL;DR

Proactive dialogue planning is important for LLM agents, but prompting is limited by frozen-model planning ability and iterative refinement is difficult to transfer to new cases. PPDPP trains a tunable policy-planning plug-in with annotated data and goal-oriented AI feedback from simulated interactions. It outperforms existing approaches across negotiation, emotional support, and tutoring dialogues.

  • Problem

    LLM dialogue agents often prioritize user intentions, while proactive tasks require planning toward goals; existing prompting is bounded by frozen-model capability and iterative refinement is difficult to transfer to new cases.

    Source: §1 · pp. 1–2

  • Method

    PPDPP trains a tunable dialogue-policy plug-in with human-annotated data and reinforcement learning from goal-oriented AI feedback collected through LLM self-play.

    Source: §6 · p. 9

  • Results

    PPDPP consistently outperforms existing approaches across negotiation, emotional-support, and tutoring dialogues, improving goal-completion effectiveness and efficiency.

    Source: §1 · p. 2, §5.1 · pp. 6–7

  • Takeaways & Limitations

    The learned plug-in can generalize to different cases and be substituted across applications without affecting the LLM’s response capabilities.

    Source: §6 · p. 9

Abstract

from arXiv · show

Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the dialogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. However, these approaches are either bounded by the policy planning capability of the frozen LLMs or hard to be transferred to new cases. In this work, we introduce a new dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems with a tunable language model plug-in as a plug-and-play dialogue policy planner, named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data as well as reinforcement learning from goal-oriented AI feedback with dynamic interaction data collected by the LLM-based self-play simulation. In this manner, the LLM-powered dialogue agent can not only be generalized to different cases after the training, but also be applicable to different applications by just substituting the learned plug-in. In addition, we propose to evaluate the policy planning capability of dialogue systems under the interactive setting. Experimental results demonstrate that PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications, including negotiation, emotional support, and tutoring dialogues.

1 INTRODUCTION

LLM-powered agents struggle with proactive dialogue because they tend to follow users’ intentions, while success requires planning actions toward goals over dynamic interactions. PPDPP adds a tunable policy-planning plug-in and interactive evaluation to address limits in existing planning and assessment approaches.

1 INTRODUCTION

  • Proactive dialogue planning means choosing actions during dynamic user interactions to achieve specific goals, unlike passively accommodating user intentions.The paper gives negotiation, emotional support, and tutoring as examples of proactive dialogue problems.Source: §1 · p. 1
  • Existing prompting is bounded by frozen LLM planning ability, while iterative refinement requires new simulations for each case and standard turn-level evaluation misses multi-turn goal achievement.These limitations constrain transfer to new cases and leave policy-planning effectiveness and efficiency unmeasured.Source: §1 · p. 2
  • PPDPP uses a tunable language-model plug-in to predict dialogue strategies, trained with human-annotated data and reinforcement learning from goal-oriented feedback in LLM self-play.The learned plug-in is intended to transfer across cases and applications without changing the dialogue LLM’s response-generation capabilities.Source: §1 · p. 2
  • PPDPP outperforms existing LLM-based dialogue systems across negotiation, emotional-support, and tutoring tasks, effectively and efficiently guiding conversations toward designated goals.The proposed interactive evaluation measures goal success and average turns using LLM-based user simulators and reward models.Source: §1 · p. 2

2 RELATED WORKS

Proactive dialogues lack a fixed agenda and often have subjective outcomes, requiring complex reasoning and domain knowledge. PPDPP differs from earlier approaches by making policy planning learnable and using goal-oriented AI feedback as scalar rewards for long-term interaction.

2 RELATED WORKS

  • Unlike slot-based task-oriented dialogue, proactive dialogue has no predefined agenda and involves complex reasoning, domain knowledge, and subjective outcomes.Examples of outcomes include learning gain in tutoring and emotional relaxation in counselling.Source: §2 · p. 3
  • Prompted self-thinking plans the next turn without long-term goals, while iterative refinement requires simulations for each case and leaves LLM parameters frozen.PPDPP instead fine-tunes a learnable plug-in for policy planning without altering other LLM functions.Source: §2 · p. 3
  • Existing learnable plug-ins can use supervised fine-tuning or reinforcement learning, whereas fixed external plug-ins cannot learn from feedback.Examples of plug-ins include APIs, vision models, and smaller language models.Source: §2 · p. 3
  • PPDPP turns verbal AI feedback into scalar rewards for long-term goals in dynamic multi-turn dialogue, rather than single-turn response preferences.Prior feedback-based approaches commonly use natural-language feedback to refine prompts.Source: §2 · p. 4

3 METHOD

PPDPP is a plug-in policy planner trained first on annotated dialogue and then on simulated interactions with goal-oriented AI rewards. Its policy operates over dialogue histories and strategies, and the tuned planner guides responses at inference without the reward model.

MDP Environment

  • The dialogue is modeled as an MDP in which dialogue history informs strategy selection, users respond, and interaction ends at goal completion or the turn limit.The policy objective is to maximize expected cumulative rewards over observed dialogue episodes.Source: §3 · p. 4

Plug-and-Play Dialogue Policy Planner

  • A tunable language model predicts dialogue actions and can be initialized by supervised fine-tuning to match human-labeled actions from available dialogue corpora.This corpus-based initialization is intended to accelerate convergence of interactive online training, although it may produce a sub-optimal policy.Source: §3 · p. 4

Self-play Interaction

  • Two prompted LLMs role-play assistant and user with distinct goals; PPDPP maps its predicted action to a natural-language instruction for the assistant.The simulated interaction continues until the goal is completed or the maximum turn is reached.Source: §3 · p. 4

LLM as Reward Model

  • A third LLM evaluates goal completion and policy outcomes with verbal feedback that is mapped to scalar rewards; sampled feedback outputs are averaged into a scalar value.Nonterminal interactions receive a small negative reward, such as -0.1, to encourage efficient goal completion.Source: §3 · p. 5

Reinforcement Learning

  • At terminal turns, the policy agent is optimized with vanilla policy gradient using goal-oriented rewards, and the tuned planner supplies action prompts at inference without the reward LLM.This allows application to new situations without repeated case-specific simulation iterations.Source: §3 · p. 5

4 EXPERIMENTAL SETUPS

Experiments cover negotiation, emotional-support, and tutoring datasets, with annotated training dialogues used for supervised initialization and case backgrounds used for reinforcement learning. Evaluation measures goal completion interactively; implementation compares the planner against dialogue-model and LLM-planning baselines.

4.1 DATASETS

  • The experiments use CraisglistBargain for negotiation, ESConv for emotional support, and CIMA for tutoring; annotated training dialogues initialize the planner and case backgrounds support reinforcement learning.CraisglistBargain uses 11 negotiation strategies, ESConv has 1,300 cases and eight support strategies, and CIMA covers five pedagogical strategies.Source: §4.1 · pp. 5–6

4.2 EVALUATION PROTOCOLS

  • Interactive evaluation measures average turns to goal and success rate within eight turns; negotiation also uses Sale-to-List Ratio, with an LLM user simulator and reward model.Average turns measures efficiency, while success rate measures effectiveness; higher Sale-to-List Ratio means greater buyer benefit.Source: §4.2 · p. 6

4.3 IMPLEMENTATION DETAILS AND BASELINES

  • RoBERTa-large is the default planner, ChatGPT supplies role-playing and reward models, and comparisons include DialoGPT and five LLM-based planning methods.The LLM baselines include Standard, Proactive, ProCoT, Ask-an-Expert, and ICL-AIF; Vicuna-13B and LLaMA-2-13B are also evaluated.Source: §4.3 · p. 6

5 EXPERIMENTS

PPDPP is evaluated across negotiation, emotional support, and tutoring, with analyses of performance over conversation turns, different LLM backbones, and human judgments. Results report gains over baselines, alongside task-specific trade-offs and an exception in comforting.

5.1 OVERALL EVALUATION

  • PPDPP consistently outperforms baselines across three proactive dialogue tasks, achieving goals in fewer turns and using fewer API tokens than AnE and ICL-AIF.Supervised fine-tuning contributes little to final performance compared with reinforcement learning from dynamic interactions.Source: §5.1 · pp. 6–7, Table 3 · p. 7
  • In negotiation, reinforcement learning raises Sale-to-List Ratio from 0.2280 to 0.3376, while success rate falls from 0.6649 to 0.6117.The paper attributes the success-rate decline to reduced seller willingness to deal when maximizing seller benefit.Source: §5.1 · p. 7
  • On Emotional Support Dialogues (ESConv), learning from dynamic interactions improves PPDPP’s success rate from 0.7308 to 0.8462 and outperforms all other baselines by a noticeable margin.PPDPP without RL performs worse than ChatGPT with Standard prompting, which achieves a 76.92% success rate.Source: §5.1 · p. 7
  • In tutoring, corpus-based learning substantially outperforms baselines, and reinforcement learning further raises performance from 0.7965 to 0.8407.The paper links corpus-based performance to the shared structure of CIMA exercises and similarity between training and test cases.Source: §5.1 · p. 7

5.2 IN-DEPTH ANALYSIS

  • Across datasets and nearly every conversation turn, PPDPP outperforms baselines; AnE is relatively strong early, while tutoring baselines fall below Standard after turn three.Figure 2 compares turnwise success-rate differences against Standard; Figure 3 reports turnwise Sale-to-List Ratio differences for negotiation.Source: Figure 2 · p. 8, Figure 3 · p. 8, §5.2 · p. 8
  • Reinforcement learning generally improves each tested backbone, but no LLM leads every task: open-source models gain more in negotiation, while ChatGPT excels at Italian translation tutoring.Vicuna is competitive with ChatGPT in emotional support; in negotiation, Vicuna and LLaMA2-Chat achieve higher Sale-to-List Ratio but lower deal success rate than ChatGPT.Source: §5.2 · p. 8, Figure 4 · p. 9

5.3 HUMAN EVALUATION

  • Human evaluation favors PPDPP on almost every response perspective and overall, with AnE winning the comforting perspective in emotional support.The evaluation uses paired comparisons on sampled ESConv and CraisglistBargain dialogues.Source: Table 4 · p. 9, §5.3 · p. 9

6 CONCLUSIONS

The paper presents PPDPP as a plug-in policy planner trained with supervised fine-tuning and reinforcement learning, and reports results across three proactive dialogue problems. It also identifies tunable plug-ins and collaboration among agents as potential directions for conversational AI.

6 CONCLUSIONS

  • PPDPP combines supervised fine-tuning on human-annotated corpora with reinforcement learning from goal-oriented AI feedback.The conclusion describes generalization across cases and application versatility through substituting the learned plug-in without affecting LLM response ability.Source: §6 · p. 9
  • The conclusion reports superior results on negotiation, emotional support, and tutoring dialogues, and presents the framework as an interactive evaluation protocol for dialogue-level effectiveness and efficiency.The protocol measures dialogue effectiveness and efficiency in multi-turn conversations.Source: §6 · p. 9
  • The authors suggest tunable plug-ins could address specific LLM shortcomings across applications, while multiple agents could be combined for more complex dialogue challenges.They also note that different LLMs may have strengths in different problems due to their training processes, and that specialized-model training is resource-intensive.Source: §6 · p. 9

A RELIABILITY ANALYSIS OF LLMS AS REWARD MODELS AND USER SIMULATORS

A.1 ANALYSIS OF LLMS AS REWARD MODEL

The analysis compares Vicuna-13B, LLaMA2-Chat-13B, and ChatGPT as reward models across three dialogue problems. ChatGPT is reported as suitable for all three, while the other two models struggle with Italian translation evaluation.

A.1 ANALYSIS OF LLMS AS REWARD MODEL

  • Vicuna-13B and LLaMA2-Chat-13B fail to evaluate students’ Italian translation correctness on CIMA, which the authors relate to their lack of large-scale Italian training.Source: §A.1 · p. 14
  • The reward model selects the situation that best matches the current user state, and its predictions are assessed against human-annotated labels using F1.Source: §A.1 · p. 14
  • ChatGPT is qualified to serve as the reward model for CraisglishBargain, ESConv, and CIMA.The analysis reports that all three LLMs perform well for CraisglishBargain and ESConv, but only ChatGPT is qualified across all three problems.Source: Figure 5 · p. 14, §A.1 · p. 14

A.2 ANALYSIS OF LLMS AS USER SIMULATOR

The user simulator is evaluated for naturalness and usefulness in single- and multi-turn free-form conversations. The authors report that ChatGPT-based simulation outperforms DialoGPT and is competitive with human-annotated dialogues.

A.2 ANALYSIS OF LLMS AS USER SIMULATOR

  • Simulated users are expected to enact assigned roles in specific contexts, such as a patient with emotional issues arising from real-world scenarios.Source: §A.2 · p. 14
  • Simulator utterances are assessed for naturalness and usefulness in single- and multi-turn free-form conversations.Naturalness concerns fluency and human-likeness; usefulness concerns consistency with role descriptions.Source: §A.2 · p. 15
  • Two annotators rated sample pairs as Win, Tie, or Lose, with Cohen’s Kappa of 0.72.Source: §A.2 · p. 15, Table 5 · p. 15
  • The ChatGPT-based simulator outperforms DialoGPT, particularly in multi-turn response naturalness, and is competitive with human-annotated dialogues.The authors interpret these results as validating ChatGPT for use as a user simulator.Source: §A.2 · p. 15

B TRAINING DETAILS

Training has supervised fine-tuning and reinforcement-learning phases. The experiments use validation performance for SFT checkpoint selection and randomly sampled training cases for online RL.

B TRAINING DETAILS

C HUMAN EVALUATION INSTRUCTIONS

The instructions assess emotional-support responses for identifying the problem, comforting the user, and offering suggestions, and negotiation responses for persuasion, coherence, and naturalness.

C HUMAN EVALUATION INSTRUCTIONS

  • Emotional-support responses are assessed on identifying the problem, comforting the user, and offering helpful suggestions.These are the three perspectives specified for ESConv.Source: C HUMAN EVALUATION INSTRUCTIONS · pp. 15–16
  • Table 7 is identified as an ablation study of the sampling strategy.Source: Table 7 · p. 16
  • Negotiation responses are assessed for persuasiveness, coherence with the conversation, and human-like naturalness.These are the three perspectives specified for CraigslistBargain.Source: C HUMAN EVALUATION INSTRUCTIONS · p. 16

D ABLATION STUDY OF THE SAMPLING STRATEGY

The sampling-strategy ablation reports improved state prediction and policy-planning performance with sampling. The appendix also specifies response and user-simulation prompts, strategy mappings, reward models, and baseline implementations.

D ABLATION STUDY OF THE SAMPLING STRATEGY

  • Sampling substantially improves state-prediction F1 and provides fine-grained continuous rewards that contribute to better policy-planning performance.Sampling reduces variance in LLM-generated state predictions; continuous averaged rewards make policy outcomes more distinguishable during reinforcement learning.Source: D ABLATION STUDY OF THE SAMPLING… · p. 16

E.1 RESPONSE GENERATION

  • Response prompts assign the assistant buyer, therapist, or teacher roles and use each task’s negotiation target, patient situation, or translation exercise as background.The dialogue strategy is supplied as an [action] instruction.Source: §E.1 · pp. 16–17

E.2 USER SIMULATOR

  • User-simulator prompts omit strategy instructions, keeping simulated users responsive to dialogue history while assigning seller, patient, or student roles.Emotional-support prompts also specify emotion and problem types; tutoring prompts instruct the student to forget the translation.Source: §E.2 · pp. 17–18

E.3 STRATEGY PROMPTING

  • Strategy prompts map dataset-annotated strategies into natural language, with 11 negotiation, 8 emotional-support, and 5 tutoring strategies.Source: §E.3 · pp. 18–19

E.4 REWARD MODEL

  • Negotiation rewards first score deal completion, then use the deal price to measure assistant-side benefit.The deal threshold is 1.0; the Sale-to-List Ratio is used to measure benefit.Source: §E.4 · pp. 19–20
  • Emotional-support rewards map sampled patient-state feedback to scalar scores; tutoring rewards distinguish four translation-progress levels.The emotional-support completion threshold is 0.5, while the tutoring threshold is 1.0.Source: §E.4 · pp. 20–21

E.5 BASELINES

  • The baselines range from strategy-free self-play and turn-level strategy prompting to expert-LLM advice and iterative dialogue-level feedback.ProCoT adds chain-of-thought analysis to Proactive; Ask-an-Expert uses another LLM as strategic expert, while ICL-AIF iterates dialogue-level feedback.Source: §E.5 · pp. 21–22

F EXAMPLE CONVERSATIONS

The example conversations illustrate how systems handle a shared negotiation case and emotional-support situations, with different prompting approaches producing distinct interaction patterns.

F EXAMPLE CONVERSATIONS

  • In negotiation examples, Standard reaches no deal, Ask-an-Expert reaches a deal with a large buyer compromise, ProCoT reaches a better deal, and ICL-AIF ends without agreement.PPDPP also reaches a better deal and tries to maximize buyer benefit when the seller is willing to compromise.Source: F EXAMPLE CONVERSATIONS · p. 26
  • Standard and Ask-an-Expert sustain empathy or reassurance across turns, which becomes less useful as the patient’s emotional intensity falls.Source: F EXAMPLE CONVERSATIONS · p. 26

PROCOT

ICL-AIF

  • ICL-AIF’s emotional-support suggestions typically cover three flexible stages of helping the patient.The paper relates these stages to Hill (2009).Source: F EXAMPLE CONVERSATIONS · p. 26

PROCOT

BACKGROUND INFORMATION

ASK-AN-EXPERT

  • In the job-loss email scenario, one conversation example has 7 turns, while the second example, where the patient fears supporting their family, has 4 turns.Source: ASK-AN-EXPERT · p. 31, PROCOT · p. 32
Loading 2311.00262v2…