Source-linked AI summary
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Lior Baruch, Moshe Butman, Kfir Bar, Doron Friedman
TL;DR
Specialized goal-oriented dialogue, particularly Motivational Interviewing, is difficult because domain data are scarce and conversational objectives are nuanced. The paper proposes PTO, which generates look-ahead preference data with virtual patients and an oracle, then iteratively trains agents with DPO. PTO-trained models outperform the baseline on MI session satisfaction, working alliance, and final score, while deeper look-ahead produces higher-scoring and more stable interactions.
Problem
Specialized goal-oriented dialogue lacks abundant domain-specific data and involves nuanced interactions, while suitable reward functions are difficult to define in domains such as Motivational Interviewing.
Method
PTO simulates conversational trajectories with a Preference Tree with Look-Ahead, uses virtual patients and an oracle evaluator to generate preferences, and iteratively trains the agent with DPO.
Results
PTO-trained models consistently outperform the baseline across Session Satisfaction, Working Alliance, and Final Score, with deeper look-ahead achieving higher scores and lower variance.
Takeaways & Limitations
Look-ahead preference optimization supports more effective, empathetic, streamlined, and stable goal-oriented dialogues in the evaluated MI setting.
Takeaways & Limitations
Automated evaluation may exhibit positional or preference bias, potentially encouraging optimization for superficial stylistic or content-related attributes rather than conversational quality.
Abstract
from arXiv · showhide
Developing dialogue systems capable of engaging in multi-turn, goal-oriented conversations remains a significant challenge, especially in specialized domains with limited data. This research proposes a novel framework called Preference Tree Optimization (PTO), designed to iteratively improve agent models in such dialogue systems, by generating preference data using a method called Preference Tree with Look-Ahead. Focusing on Motivational Interviewing (MI) -- a counseling technique aimed at facilitating behavioral change -- we leverage virtual patients and an oracle evaluator to simulate conversations and generate rich preference datasets. By combining this method with Direct Preference Optimization (DPO), we aim to enhance the agent's decision-making capabilities over iterative training cycles. The proposed framework addresses data scarcity and advances the development of more nuanced and effective dialogue systems in goal-oriented domains. Experimental evaluations demonstrate that the PTO framework enhances dialogue agents' performance in goal-oriented conversations within the domain of Motivational Interviewing (MI). Models trained with PTO consistently outperformed the baseline in key metrics such as session satisfaction and working alliance. Additionally, incorporating look-ahead simulations led to improved long-term planning and more effective conversational strategies, with deeper look-ahead configurations yielding the most stable and high-scoring results.
1 INTRODUCTION
This work introduces Preference Tree Optimization (PTO) to address data scarcity and nuanced decision-making in specialized, goal-oriented dialogue, evaluated in Motivational Interviewing (MI). PTO combines look-ahead preference-data generation with iterative DPO training, while remaining an offline training paradigm.
- Motivation: Specialized goal-oriented dialogue is difficult because interactions are complex and domain-specific data are scarce, particularly in Motivational Interviewing.MI requires collaborative, client-centered, and nuanced dialogue that facilitates behavioral change.
- Approach: PTO generates preference data by systematically simulating conversational paths and evaluating them with an oracle.The Preference Tree with Look-Ahead explores alternative trajectories before recording preferences.
- Approach: PTO iteratively trains the current agent model with the generated preference data using Direct Preference Optimization.Each training cycle produces an updated agent model for the next iteration.
- Evaluation: The framework is evaluated in MI using virtual patients and evaluators to simulate specialized conversational interactions.This setup is intended to address data scarcity and the need for nuanced interactions.
- Scope: PTO is designed exclusively for offline training, with computational cost incurred during development rather than real-time deployment.The paper states that inference is fast and efficient once the model is trained.
2 BACKGROUND AND RELATED WORK
The background frames specialized goal-oriented dialogue as a data-scarce problem with difficult-to-specify objectives and surveys preference-based, synthetic-data, and search-based approaches. The paper positions PTO as combining tree exploration with oracle scoring for Motivational Interviewing, a human-centric domain where preference optimization remains largely unexplored.
- Challenges: Specialized goal-oriented systems face limited domain data, nuanced multi-turn interactions, and difficulty defining suitable rewards in psychological domains.The background contrasts these challenges with structured domains such as mathematics and gaming.
- Preference Optimization: DPO directly optimizes language models from preference data without a separate reward model or reinforcement-learning procedure.It maps language-model policies directly to reward functions through a cross-entropy loss.
- Related Approaches: Synthetic-data and iterative-improvement methods include score-based selection, self-evaluation, and search-based tree-structured approaches.These categories differ in how preference data are generated and which domains they target.
- PTO Positioning: PTO uses a dedicated user model to simulate full conversational trajectories and an oracle evaluator to score them for preference-based refinement.This bridges search-based exploration and score-based comparison in goal-oriented dialogue.
- Motivational Interviewing: Motivational Interviewing requires empathy, adaptability, and interpretation of subtle conversational cues, making it a challenging AI dialogue setting.Prior work found virtual patients feasible for training and evaluating therapeutic dialogues but limited in capturing full human interaction depth.
- Motivational Interviewing: Unlike prior MI work using existing datasets, PTO iteratively generates training data from simulated conversations and refines the therapist model with DPO.The distinction is between dataset-based fine-tuning and simulated preference-data generation.
3 METHOD
The method generates preference tuples by exploring multiple agent responses, simulating future dialogue, and scoring branches with an oracle. PTO filters these preferences and repeatedly applies DPO, using the preferred response to continue the simulated conversation and update the agent.
- Preference Tree with Look-Ahead: At each agent decision point, the Preference Tree with Look-Ahead generates multiple candidate responses and branches the conversation.Each branch appends one candidate response to the conversation history.
- Preference Tree with Look-Ahead: Each branch simulates K future steps by alternating between the agent and virtual patient to anticipate longer-term implications.The simulated paths represent possible future conversational trajectories.
- Preference Tree with Look-Ahead: An oracle evaluates each branch using criteria such as MI adherence, empathy, and goal progression, assigning scores to the candidate trajectories.The scores support comparison of alternative responses at the current turn.
- Preference Tree with Look-Ahead: The highest- and lowest-scoring responses are recorded as preferred and least-preferred elements of a preference tuple.The tuple contains the conversation, losing response, and winning response.
- Preference Tree with Look-Ahead: The conversation continues with the preferred response until a termination condition is met, after which the process repeats across decision points.Termination may occur at maximum conversation length or after goal achievement.
- Preference Tree Optimization: PTO iterates preference-data generation, filtering, and DPO training to produce a sequence of increasingly optimized agent models.Samples are retained when the winning score exceeds the losing score by at least threshold τ, set to 0.1 in the experiments.
- Preference Tree Optimization: The framework balances exploration of conversational paths with exploitation of improved responses through successive training cycles.The paper describes this balance as producing incremental performance enhancements.
4 EXPERIMENTAL SETUP
The experiments use Llama-2-7B as the therapist agent, GPT-3.5 for fixed virtual-patient and oracle roles, and two look-ahead depths over iterative DPO training. Performance is evaluated through simulated MI conversations scored for adherence and conversational quality.
- Models and Tools: Llama-2-7B serves as the base therapist-agent model, while GPT-3.5 simulates virtual patients and evaluates conversations in separate fixed roles.The GPT-3.5 user and oracle roles use distinct prompts and are not updated during training.
- Models and Tools: The virtual-patient setup defines 96 profiles varying in demographics, problems, history, and cooperation level.The profiles cover smoking and obesity problems and diverse counseling attitudes.
- Models and Tools: The oracle evaluator scores MI adherence and conversational quality using two questionnaires whose average determines the final score.The questionnaires are based on guidelines from prior MI research.
- Training Conditions: The experiment compares look-ahead depths 0 and 5 across 7 iterative training cycles per depth.The look-ahead variable tests whether anticipating future conversational turns affects performance.
- Evaluation: After each iteration, each trained model is evaluated in 96 separate virtual-patient conversations.The conversations are scored by the oracle evaluator using the two questionnaires.
5 RESULTS
Across the evaluated metrics, PTO-trained models outperform the Llama-2-7B baseline, with depth-5 look-ahead generally scoring higher and L5 M7 showing the strongest stability and conversation-length reduction. Statistical tests confirm significant baseline gains, while the advantage of deeper look-ahead over depth-0 is clearest for Working Alliance.
- Experimental setup: Seven iterative training cycles compared look-ahead depths 0 and 5 against the Llama-2-7B baseline.The evaluation focused on PTO configurations with no look-ahead and depth-5 look-ahead.
- Evaluation metrics: Session Satisfaction (Q1) measures satisfaction, relevance, motivation facilitation, learning outcomes, and everyday applicability.Q1 aggregates scores from Questionnaire 1.
- Evaluation metrics: Working Alliance (Q2) measures interpersonal skills, empathy, communication effectiveness, and collaborative relationship-building.Q2 aggregates scores from Questionnaire 2.
- Evaluation metrics: Final Score averages Session Satisfaction and Working Alliance to provide an overall performance indicator.The metric combines Q1 and Q2 rather than introducing a separate evaluation dimension.
- Performance comparison: Every PTO-trained model outperforms the baseline across all evaluated metrics, and depth-5 models score higher than depth-0 models.The reported comparison covers Session Satisfaction, Working Alliance, and Final Score.
- Statistical analysis: L5 M7 achieves the highest Final Score, with its improvement over L0 M4 statistically significant only for Q2.Both L0 M4 and L5 M7 significantly outperform the baseline across Q1, Q2, and Final Score; model choice also significantly influences conversation length.
- Stability and efficiency: L5 M7 has the lowest variance across Q1, Q2, and Final Score and reduces average dialogue turns from 43.7 to 34.4 versus the baseline.PTO-trained models generally reduce conversation length, with L5 M7 showing the largest reported reduction.
6 DISCUSSION
PTO consistently improves dialogue performance over the baseline, while look-ahead supports more effective and stable interactions. Automated evaluation remains a concern, and deeper look-ahead requires further investigation.
- PTO consistently improves dialogue performance compared with the baseline, with depth-0 and depth-5 models gaining on Q1, Q2, and Final Score.The models were trained from Llama-2-7B using only data generated by Preference Tree with Look-Ahead.
- The best-performing depth-5 model achieved the highest scores and lowest variance, indicating more stable and reliable interactions.
- Look-ahead may help agents anticipate future turns, supporting more effective, empathetic, and streamlined dialogues.
- Automated evaluation can exhibit positional or preference bias, potentially encouraging optimization for superficial attributes rather than genuine conversational quality.The paper describes this risk as reward hacking, although oracle scoring remained largely consistent across the dialogue.
- Future work will examine whether and why deeper look-ahead improves long-term planning in this soft domain.Planned comparisons include online alignment and self-rewarding language-model approaches.
A.1 PREFERENCE TREE WITH LOOK-AHEAD ALGORITHM
The Preference Tree with Look-Ahead algorithm constructs preference data by generating candidate agent responses and simulating conversational branches to a specified depth.
- The algorithm requires agent, user, and oracle models, plus maximum conversation length, look-ahead depth, and candidate-response count.
- It initializes an empty preference dataset and conversation history with the starting context before iterating until the maximum length is reached.
- At each agent decision phase, the agent generates N candidate responses for evaluation.
- Each candidate creates a cloned conversation branch with the response appended before look-ahead simulation begins.
A.2 EVALUATION QUESTIONNAIRES FOR THERAPIST PERFORMANCE
Table 3 presents the questions posed to the language model to evaluate therapist performance.
- Table 3 lists the questions used by the language model to evaluate the therapist’s performance.
- The table functions as the questionnaire reference for the therapist-performance evaluation procedure.
- The supplied caption identifies the table’s content but does not state a specific comparison or outcome.
A.3 TUKEY HSD POST-HOC ANALYSIS
Table 4 reports Tukey HSD post-hoc results for pairwise comparisons between models.
- Table 4 presents Tukey HSD post-hoc test results for pairwise model comparisons.
- The table is the reported reference for the post-hoc analysis of differences between model choices.
- The supplied caption identifies the table’s comparison structure but does not state specific pairwise values or outcomes.