Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment

1 Introduction

• Personalized alignment must infer and adapt to user attributes from dialogue, including when little or no prior user information is available.The challenge includes constructing and refining coherent user profiles across extended conversations.Source: §1 · p. 2
• Existing prompting relies on static profile injection, while offline optimization needs large labeled datasets and adapts poorly across users in cold-start settings.Prompting also faces context-window limits on integrating long-term user knowledge.Source: §1 · p. 2
• RLPA frames personalized alignment as a multi-turn MDP and trains through simulated-user interaction, with per-turn profile and response rewards supporting profile inference and personalized replies.The framework is intended for cold-start and dynamically evolving user scenarios.Source: §1 · p. 2
• Qwen-RLPA outperforms prompt-based and offline optimization baselines in personalization quality, surpasses Claude-3.5 and DeepSeek-V3, and performs on par with GPT-4o.Further analysis reports coherent personalized responses over long dialogues, preference-conflict resolution, and greater inference efficiency than recent reasoning LLMs.Source: §1 · p. 3
• The paper conceptualizes personalization as evolving preference modeling and fine-tunes Qwen-2.5-3B-Instruct with RLPA to produce Qwen-RLPA.The stated contribution includes adapting to cold-start and real-time scenarios.Source: §1 · p. 3

2 Preliminaries

• The preliminaries introduce personalized alignment as a multi-turn MDP, then present reinforcement learning as the policy-optimization paradigm.The framework overview depicts a simulated user, inferred profile and response, two reward functions, and policy optimization.Source: §2 · p. 3, Figure 2 · p. 4

2.1 Personalized Alignment as a Multi-Turn Markov Decision Process

• The MDP state is dialogue history, the action is the generated response, and the simulated user supplies the next utterance; per-turn rewards measure profile accuracy and response alignment.The formulation captures dynamic profile inference and refinement across turns.Source: §2.1 · p. 3

2.2 Reinforcement Learning for Policy Optimization

• Reinforcement learning optimizes a dialogue policy for expected cumulative reward by updating it from simulated-user interaction and long-term outcomes.The two goals are accurate preference tracking and coherent personalized responses; delayed and cumulative feedback supports both.Source: §2.2 · pp. 3–4

3 Method: Reinforcement Learning for Personalized Alignment (RLPA)

• RLPA trains language models to dynamically infer and respond to user-specific preferences through interaction, using rewards for profile inference and personalized response quality.The framework has three components: a simulated user, a Profile Reward, and a Response Reward.Source: §3 · p. 4

3.1 Simulated User Design

• The simulated user reveals profile information gradually while maintaining consistent preferences and conversational style across turns.Human evaluation informed the choice of GPT-4o-mini as the simulator for subsequent experiments.Source: §3.1 · pp. 4–5

3.2 Profile Reward Function Design

• The Profile Reward compares predicted and ground-truth slot-value profiles, rewarding overlapping matches and penalizing omissions and incorrect predictions.The model updates its profile estimate from dialogue history because user attributes are not directly observable.Source: §3.2 · p. 5

3.3 Response Reward Function Design

• The Response Reward assesses personalization fidelity across preference expression, style consistency, goal alignment, and persona coherence, while requiring five binary quality criteria.The criteria are naturalness, relevance, logical consistency, engagement, and informativeness; GPT-4o-mini is used for efficiency during RL training.Source: §3.3 · p. 5

3.4 Training with Proximal Policy Optimization (PPO)

• PPO trains the model with a turn-level combined reward that supervises both user modeling and personalized response generation.The turn-level signal is intended to encourage continued profile inference, refinement, and use throughout interaction.Source: §3.4 · pp. 5–6

4.1 Experimental Setup

• Evaluation uses ALOE multi-turn dialogues with annotated profiles, testing both within-schema personalization and generalization to unseen attribute types and values.The Vanilla setting holds the profile schema constant; Extended changes both attribute types and values.Source: §4.1 · p. 6
• The reported evaluation metrics are average alignment score, N-IR, and N-R2; experiments run on eight NVIDIA A100 80GB GPUs.The implementation uses OpenRLHF and vLLM.Source: §4.1 · p. 6
• The study compares prompt-based methods, including Reminder, Self-Critic, CoT, and RAG, with offline SFT and DPO optimization.Experiments use Qwen-2.5-3B-Instruct and GPT-4o-mini for the user simulator and reward model.Source: §4.1 · p. 6

4.2 Overall Results

• RLPA improves Vanilla alignment over same-backbone SFT by +29.06 and DPO by +28.11, has best N-R2 (0.855) and N-IR (0.090), and leads all three Extended metrics.In the Extended setting, it outperforms SFT and DPO by over +28 points.Source: §4.2 · p. 7
• On Extended ALOE, RLPA's alignment rises steadily across all 10 turns, while SFT and DPO peak around turn 5 before declining significantly.The passage attributes the trend to continual refinement of inferred profiles and their use in response generation.Source: §4.2 · p. 7, Figure 3 · p. 7
• Table 2 examines the impact of reward components, while Figure 4 covers preference conflict and long-term profile modeling.Source: Table 2 · p. 7, Figure 4 · p. 8

5.1 Ablation Study

• Profile Reward alone yields an average alignment score of 45.54, while Response Reward alone yields AVG 66.19 but scores 58.92 at k=1 and is unstable later.The ablation attributes profile construction to Profile Reward and effective use of profiles in generation to Response Reward.Source: §5.1 · p. 8

5.2 Adaptation to Preference Conflict

• At turn 6, RLPA scores 70.27 after a preference shift and recovers to 82.43 at turn 9, while DPO drops from 71.62 to 25.68.The shift is introduced at turn 6 to simulate mid-dialogue preference changes.Source: §5.2 · p. 8

5.3 Stable Profile Modeling in Long-Term Interaction

• Profile scores rise from 9.26 at k=1 to 52.54 at k=50, then remain near 50 through k=70, approaching the theoretical maximum after turn 50.The AVG line averages profile scores at eight checkpoints; the theoretical maximum reflects attributes revealed under ALOE annotation rules.Source: §5.3 · pp. 8–9

5.4 Comparison with Reasoning LLMs

• Qwen-RLPA consistently achieves higher response scores while using fewer reasoning tokens than reasoning-centric models.The comparison concerns tokens used to infer and maintain user profiles.Source: §5.4 · p. 9, Figure 5 · p. 9

Prompt-based methods

• Prompt-based methods use static templates and offer superficial personalization, with context-length constraints and no long-term memory or adaptive behavior.The examples include profile-augmented and retrieval-augmented prompting.Source: §6 · p. 9

Offline optimization

• Offline SFT and DPO require extensive labeled data and their static nature limits cold-start suitability, user generalization, and real-time adaptation.These approaches aim to train models to produce profile-consistent responses.Source: §6 · p. 9

Summary

• Both prompting-based and offline methods struggle to provide dynamic, long-term, adaptive personalization, motivating interactive learning frameworks.Source: §6 · p. 9

• Dynamic personalization under cold-start conditions and evolving user preferences is the central challenge.Source: §7 · p. 9
• RLPA models alignment as a multi-turn MDP and uses ongoing interaction with dual profile-level and response-level rewards to infer, retain, and use user profiles.Source: §7 · p. 10
• Qwen-RLPA achieves substantial empirical gains across personalization benchmarks, matches or exceeds leading proprietary systems, and supports long-term consistency.Source: §7 · p. 10
• Comparisons with reasoning LLMs highlight profile-based reasoning as a more efficient and contextually appropriate paradigm for personalized dialogue generation.Source: §7 · p. 10

• RLPA currently assumes a single-user interaction thread; multi-user, multi-session, and cross-domain personalization remain extensions for more realistic usage patterns.Source: §8 · p. 10
• The theoretical understanding of long-term alignment dynamics and convergence properties remains underexplored.The authors suggest more principled frameworks for continual user modeling and policy generalization.Source: §8 · p. 10

NeurIPS Paper Checklist

• The checklist states that the abstract and Section 1 describe the main claims, limitations are discussed in Appendix A, and the empirical study includes no theoretical results.The checklist also marks conformity with the NeurIPS Code of Ethics and says broader impacts and safeguards are in Appendix B.Source: NeurIPS Paper Checklist · pp. 14, 15, 17, 18
• The authors answer Yes on reproducibility, reporting implementation details in Section 4.1 and Appendix F, data and code in supplementary files, and statistical significance and compute information in Appendix F.They also state that training and test details are in Section 4.1 and Appendix F.Source: NeurIPS Paper Checklist · pp. 15–17
• The checklist answers Yes for human-subjects and crowdsourcing information in Appendix F, and NA for LLM use because it was limited to writing, editing, or formatting.It also says existing asset owners are credited and new assets are documented in Appendix F and supplementary files.Source: NeurIPS Paper Checklist · pp. 18–20

A Broader Impact

• The authors identify potential benefits in personalized education, mental health support, and assistive communication through adaptive interactions, including under cold-start conditions.They frame these applications as potential benefits of RLPA's dynamic inference of evolving preferences through multi-turn interactions.Source: A Broader Impact · p. 21
• Dynamic user modeling may infer sensitive attributes without consent and enable manipulation, surveillance, harmful bias, reduced neutrality, or filter bubbles.The authors advocate future privacy-preserving personalization with consent, profile inspection, and real-time preference correction.Source: A Broader Impact · p. 21

B Training with Proximal Policy Optimization (PPO)

• PPO optimizes the personalized dialogue policy using turn-level rewards and a clipped surrogate objective, with GAE-derived advantages and clipping that constrains updates to a trust region.The stated purpose of clipping is to prevent destabilizing policy changes.Source: B Training with Proximal Policy… · p. 21

C.1 Training Data

• RLPA training requires user-profile data to simulate behavior, not ground-truth responses; applying evaluation preprocessing to ALOE yielded 3,821 slot-based training samples.The training set contains 3,821 samples.Source: §C.1 · p. 21

C.2 Evaluation Dataset

• Vanilla ALOE uses 74 test instances, a fixed “Hello” opening, and GPT-4o-mini to structure profiles into ten frequently occurring fields.The fields were selected after manual screening and automatic deduplication.Source: §C.2 · pp. 21–22

C.3 Metrics

• AL(k), the average judge score across 74 instances at turn k, is the primary metric, supplemented by normalized improvement and fit measures.N-IR normalizes alignment scores to reduce the effect of high starting alignment; N-R2 measures the robustness and consistency of alignment progression.Source: §C.3 · p. 23

C.4 Eval Prompts

• The user-role-play prompt is revised for consistent persona and gradual disclosure beyond ten turns, while response evaluation expands from three dimensions to five.The revised user-role-play prompt is shown in English and Chinese, as are the response-reward prompts.Source: §C.4 · p. 23

D.1 Model Version

• The paper reports the versions of the language models evaluated with ALOE in Table 3.The table is labeled as a detailed listing of model versions.Source: §D.1 · p. 28, Table 3 · p. 28

D.2 Methods Description

• Baselines include direct answering, preference reminders, self-critique and revision, five-shot CoT, and top-five retrieval-augmented prompting.The retrieval baseline uses SimCSE to select relevant prior exchanges.Source: §D.2 · pp. 28–29

E.1 RLPA Prompts

• The actor-model system prompt is shown in English and Chinese, while the User Model prompt remains consistent between training and evaluation.Source: §E.1 · p. 29

E.2 HyperParameters

• SFT and DPO each use one training epoch, while RLPA allows up to 10 rounds and uses four samples per prompt.Source: §E.2 · pp. 29–30

F Detailed Experimental Results

• Tables 4 and 5 report per-turn alignment scores for Vanilla and Extended ALOE, while Figures 14 and 15 visualize the corresponding alignment curves across methods.The materials provide a turn-wise comparison, but these passages do not state which method performs best.Source: F Detailed Experimental Results · p. 31

G User Model Evaluation

• Three annotators each conduct five-turn dialogues with ten personas per model, yielding 30 cases per model rated on four dimensions using a five-point scale.The dimensions are coherence, stability, proactivity, and persona-fit.Source: G User Model Evaluation · p. 34
• DeepSeek-V3 achieves the highest average score (4.23) and inter-annotator agreement (κ = 0.81) among evaluated user models.Source: Table 6 · p. 34, G User Model Evaluation · p. 34

H.1 Profile Reward Model

• The Profile Reward Model test set rewrites profiles by paraphrasing one subset of items and replacing a disjoint subset with semantically different content, then predicts the overlap count.The test dataset was constructed to select a prompt and LLM configuration for this task.Source: §H.1 · p. 35
• Overlap-count predictions are assessed by exact accuracy, fuzzy accuracy within one, MSE, and RMSE; human labels use majority vote, with uncertain cases counted incorrect.The human evaluation covers 300 sampled profile-item pairs.Source: §H.1 · p. 35

H.2 Response Reward Model

• Human annotators re-evaluated 300 randomly sampled training instances using the reward model’s dimensions and five binary response-quality criteria.The sampled records included an inferred profile, dialogue context, generated response, and predicted scalar response reward.Source: §H.2 · pp. 35–36
• The final human reward label is binary and is computed as the product of naturalness, relevance, logical consistency, engagement, and informativeness judgments.The evaluation also compared model predictions with human labels using a confusion matrix and classification metrics.Source: §H.2 · pp. 36–37, Table 9 · p. 37
• 87% accuracy and Cohen’s Kappa of 0.74 indicate substantial agreement between the response reward model and human judgments.The authors interpret this agreement as validating the model’s reliability as a reinforcement signal during training.Source: §H.2 · p. 37, Table 10 · p. 37

I Reward Curve

• PRR outperforms RR and PR across training steps, and its reward curve exceeds the sum of the individual Profile and Response rewards.The authors describe this as a synergistic effect of integrating both reward components.Source: I Reward Curve · pp. 37–38, Figure 18 · p. 38