Source-linked AI summary
Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration
Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, Tat-Seng Chua
TL;DR
LLM-based conversational systems show strong context understanding and response generation but remain limited in proactive dialogue, motivating evaluation of their clarification, target-guided, and non-collaborative abilities. The paper evaluates these settings and proposes ProCoT, which adds descriptive reasoning and goal planning before action selection. ProCoT improves clarification and smooths target-directed topic shifts, while strategic decision-making in non-collaborative dialogue remains challenging.
Problem
LLM-based conversational systems may guess answers to ambiguous queries or fail to refuse problematic requests, raising whether they can manage proactive dialogue problems.
Method
The paper comprehensively evaluates clarification, target-guided, and non-collaborative dialogues and proposes ProCoT to trigger reasoning and planning toward conversational goals.
Results
ProCoT largely improves clarification and makes target-directed topic planning smoother, but LLMs still struggle with domain-specific clarification and strategic decisions in non-collaborative dialogue.
Takeaways & Limitations
LLMs have potential for proactive dialogues, but effective domain-specific clarification and strategy learning remain unresolved challenges.
Takeaways & Limitations
Evaluation results are likely sensitive to prompt choice, and the designed prompts may not be optimal for the applications studied.
Abstract
from arXiv · showhide
Conversational systems based on Large Language Models (LLMs), such as ChatGPT, show exceptional proficiency in context understanding and response generation. However, despite their impressive capabilities, they still possess limitations, such as providing randomly-guessed answers to ambiguous queries or failing to refuse users' requests, both of which are considered aspects of a conversational agent's proactivity. This raises the question of whether LLM-based conversational systems are equipped to handle proactive dialogue problems. In this work, we conduct a comprehensive analysis of LLM-based conversational systems, specifically focusing on three aspects of proactive dialogue systems: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thought prompting scheme, which augments LLMs with the goal planning capability over descriptive reasoning chains. Empirical findings are discussed to promote future studies on LLM-based proactive dialogue systems.
1 Introduction
The paper examines whether LLM-based conversational systems can proactively manage clarification, target-guided, and non-collaborative dialogues. It introduces ProCoT to support goal-oriented reasoning and planning, finding persistent weaknesses alongside improvements in clarification and topic shifting.
- Scope: LLM-based conversational systems are evaluated on clarification, target-guided, and non-collaborative proactive dialogues.These settings require asking clarification questions, leading conversations toward designated targets, or strategically reaching agreement despite competing goals.
- Approach: ProCoT prompts LLMs to generate descriptive intermediate reasoning and planning before selecting the next dialogue action.The scheme is designed to give systems planning capability over conversational goals.
- Evaluation: The study compares ChatGPT and Vicuna with fine-tuned state-of-the-art dialogue models under three prompting schemes.The experiments cover standard, proactive, and proactive chain-of-thought prompting.
- Findings: LLMs barely ask clarification questions for ambiguous queries, although ProCoT largely improves this behavior.Performance remains unsatisfactory in domain-specific applications.
- Findings: LLMs can shift topics toward designated targets but tend to make aggressive transitions; ProCoT supports smoother topic planning.The evaluation identifies topic shifting as a capability that improves with explicit goal planning.
- Findings: LLMs tend to compromise with opponents rather than make strategic decisions in non-collaborative dialogues.The paper identifies strategy optimization as the key challenge in this setting.
2 Related Works
Prior work develops proactive dialogue systems for target guidance, information seeking, and competing-goal task completion. Related research also includes fine-tuned dialogue models and prompting methods across several dialogue applications.
- Proactive Dialogues: Target-guided systems lead conversations toward designated topics or knowledge entities using keyword transitions or knowledge-graph reasoning.These methods proactively plan the topic thread toward a target.
- Proactive Dialogues: Information-seeking dialogue research includes systems designed to address clarification and question-answering needs.The related-work discussion places clarification within proactive information-seeking dialogue.
- Proactive Dialogues: Non-collaborative dialogue systems pursue agreements favorable to themselves when system and user goals compete.Applications include product-price negotiation and donation persuasion.
- Large Language Models for Dialogues: Earlier dialogue systems commonly fine-tune pretrained language models on public dialogue data, while newer systems use supervised fine-tuning of open-source LLMs.The cited examples include DialoGPT, Meena, BlenderBot, LaMDA, and LLaMA-based systems.
- Prompting in Dialogue Systems: Prompting methods have been applied to task-oriented, knowledge-grounded, and open-domain dialogue applications.The related work describes prompting as a way to induce knowledge from LLMs.
3 Prompting LLMs to be Proactive
The paper contrasts standard prompting, proactive prompting, and ProCoT for making LLMs more proactive. ProCoT adds descriptive reasoning and planning about the next action before response generation.
- Standard Prompting: Standard prompting gives the task background and conversation history to an LLM and asks it to generate a response.The task background may be a grounded document or a target description.
- Proactive Prompting: Proactive prompting supplies possible dialogue acts so the LLM selects an action before generating its response.Examples include asking versus answering, negotiation strategies, and alternative conversation topics.
- Proactive Chain-of-Thought Prompting: ProCoT adds a thought description for dynamic reasoning and planning toward the conversational goal.The system analyzes the next action before deciding and generating the response.
- Proactive Chain-of-Thought Prompting: In clarification dialogues, ProCoT can use ambiguity analysis of the current user question to guide the next action.In non-collaborative dialogues, the reasoning concerns the negotiation progress and goal completion.
4 Evaluation
Across clarification, target-guided, and non-collaborative dialogues, LLMs show uneven proactivity: ProCoT improves goal-directed behavior, but important gaps remain in domain-specific clarification, smooth topic transitions, and strategic negotiation.
- Clarification: LLM-based systems rarely ask clarification questions under standard prompting, whereas ProCoT substantially improves ChatGPT’s clarification capability.Vicuna remains limited even when clarification is available, while ChatGPT improves on both F1 and Help. scores on Abg-CoQA.
- Clarification: ProCoT improves clarification behavior, but domain-specific PACIFIC performance remains far behind fine-tuned methods.ChatGPT with ProCoT approaches fine-tuned performance on general-domain clarification need prediction, while PACIFIC remains substantially weaker.
- Target-guided dialogues: LLMs can steer conversations toward designated targets, but their topic transitions are often aggressive and lack global coherence.ChatGPT achieves nearly perfect target success rates and strong contextual coherence, yet human evaluation reports low global coherence and potential harm to user engagement.
- Target-guided dialogues: ProCoT enables smoother target-guided transitions, although it can reduce success rate and remains insufficient for smooth transitions in ChatGPT.Under ProCoT, Vicuna achieves smoother and more engaging conversations, while ChatGPT still struggles with smooth topic transition.
- Non-collaborative dialogues: LLMs struggle to learn negotiation strategies and dialogue acts, producing responses that lag behind fine-tuned methods.Reference strategies improve human-rated persuasion and deal outcomes, while prompted systems often compromise rather than strategically maximizing their own benefit.
- Non-collaborative dialogues: Conditioning responses on reference strategies improves negotiation quality, while ProCoT mainly improves coherence without resolving strategic weaknesses.ChatGPT generates more coherent responses with ProCoT, but persuasive responses and better Sale-to-List Ratio are associated with reference strategies.
5 Conclusion
The paper evaluates LLM-based dialogue systems across clarification, target-guided, and non-collaborative proactive dialogues, and proposes ProCoT to trigger reasoning and planning. ProCoT improves clarification and smooths target transitions, while strategic planning remains challenging in non-collaborative dialogues.
- The study provides a comprehensive evaluation of LLM-based dialogue systems on clarification, target-guided, and non-collaborative dialogues.
- ProCoT enhances LLMs’ originally poor clarification performance but remains limited in domain-specific applications.
- ProCoT enables smoother planning of topic shifts, whereas LLM-based systems otherwise shift aggressively toward designated targets.
- Strategy learning and planning remain key challenges for LLMs handling non-collaborative dialogues despite their controllable response-generation capability.
Limitation
The paper identifies prompt sensitivity, financial and computational costs, and weaknesses in planning and decision making as important limitations. These constraints affect evaluation robustness, model selection, and real-world applicability.
- Sensitivity of Prompts: Evaluation results may be sensitive to prompt choices, and the designed prompts may not be optimal for the problems studied.
- Sensitivity of Prompts: Prompt sensitivity and prompt optimality remain open research problems for dialogue systems.
- Financial and Computational Cost of LLMs: ChatGPT experiments cost about $120 through the OpenAI API, while local open-source-model experiments are computationally expensive.
- ProCoT is a preliminary attempt because LLM planning and decision-making remain weaker than fine-tuned methods.
- Simply prompting LLMs to be proactive may not handle decision making under dynamic real-world environments.
A Details of Datasets
The evaluation uses five datasets spanning clarification, target-guided, and non-collaborative dialogue settings. Their constructions encode ambiguity, financial-domain reasoning, target transitions, topic reachability, and negotiation strategies.
- The experiments use five datasets: Abg-CoQA, PACIFIC, OTTers, TGConv, and CraigslistBargain.
- Clarification Dialogues: Abg-CoQA truncates CoQA conversations and selects ambiguous questions, while PACIFIC rewrites financial-domain TAT-QA questions to introduce clarification turns.
- Target-guided Dialogues: OTTers requires proactive transition utterances toward designated targets, with topics represented as topical keyword sets.
- Target-guided Dialogues: TGConv, built from ConvAI2, includes easy-to-reach and hard-to-reach target settings with topics represented as topical keyword sets.
- Non-collaborative Dialogues: CraigslistBargain models buyer–seller price negotiation and assigns 10 dialogue acts and 21 negotiation strategies to utterances.
B Error Analysis Details for Clarification Dialogues
The appendix documents clarification-error categories, prompt construction, case-study materials, and ProCoT demonstration choices. It also describes evaluations across multiple proactive dialogue datasets and compares prompting schemes through generated-response case studies.
- Error Categories: Clarification failures are categorized as wrong aspect, under-specified clarification, over-specified clarification, or generation error.
- Prompt Construction: Prompts combine task instructions with samples; few-shot prompts additionally include demonstrations and a test sample.
- Prompt Construction: Task instructions define each proactive dialogue problem and require responses in the output format used for evaluation.
- Case Studies: The appendix presents examples from Abg-CoQA, OTTers, and CraigslistBargain, including target topics, dialogue histories, and negotiation behavior.
- ProCoT Demonstrations: Three ProCoT annotations are evaluated, and the study adopts the best annotation for each dataset because annotation variance is smaller than one-shot-sample variance.
- Case Studies: Case studies compare generated responses across clarification, target-guided, and non-collaborative dialogue problems.
D.1 Clarification Dialogues
The clarification case contrasts random guessing with proactive clarification and ProCoT’s analysis-driven approach to identifying ambiguity. ProCoT uses reasoning about the ambiguity to generate a more effective clarifying question.
- Standard prompting answers an ambiguous question by randomly guessing, which may not match the user’s intended outcome.
- Proactive prompting asks a clarification question, but the question may focus on an aspect unrelated to the conversation.
- ProCoT first analyzes the underlying reasons for ambiguity before generating a clarifying question.The analysis serves as the foundation for clarifying and disambiguating the user’s intended meaning.
D.2 Target-guided Dialogues
In target-guided dialogue, standard prompting can shift topics aggressively or fail to reach difficult targets, while ProCoT plans smoother transitions and target-directed conversations. The case studies show this pattern for both ChatGPT and Vicuna.
- Standard prompting produces an aggressive topic transition that is fluent but lacks natural logical or topical coherence.
- Proactive prompting predicts less aggressive next topics but blends current and target topics without naturally connecting them.
- ProCoT generates a smoother transition by planning the topic-shifting process between current and target topics.
- Within 8 turns, ProCoT enables Vicuna to smoothly drive conversation toward the hard-to-reach target “Gaullist.”Standard and proactive prompting struggle to lead the conversation toward this target.
D.3 Non-collaborative Dialogues
The non-collaborative dialogue cases show that ProCoT uses negotiation-status analysis and planning to avoid premature or contradictory bargaining decisions. It therefore produces responses better aligned with the conversation’s negotiation progress.
- At turn 3, standard prompting initiates negotiation before the buyer expresses bargaining intent, potentially disadvantaging the seller.
- Proactive and ProCoT prompting predict the next dialogue act and strategy, with ProCoT identifying that negotiation has not yet started.
- At turn 9, standard and proactive prompting propose $45 after the seller had lowered the price to $40, creating a contradictory counteroffer.
- ProCoT summarizes negotiation progress and selects a better next negotiation goal through reasoning and planning.
E Analysis of Strategy Learning (Cont.)
The strategy analysis contrasts reference negotiation behavior with prompting-based behavior. Reference responses express sentiment and negotiate positively or negatively, whereas standard and proactive prompting favor hedging and politeness.
- Reference responses use positive or negative sentiment and negotiate in positive or negative ways.
- Standard and proactive prompting prefer hedge words and polite expressions such as “please” and gratitude.This pattern indicates that ChatGPT essentially adopts a nice role in negotiation.
- Clarification dialogues: Table 10 presents examples of prompting LLMs for clarification dialogues.
- Target-guided dialogues: Table 11 presents examples of prompting LLMs for target-guided dialogues.
- Non-collaborative dialogues: Table 12 presents examples of prompting LLMs for non-collaborative dialogues.