PaperlayerView exact revision on arXiv

Source-linked AI summary

COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation

Ruike Cao, Fugen Yao, Liang Dong, Jian Xu, Guanjun Jiang, Li Xiao

arXiv:2609.26853v1cs.LGcs.AI

TL;DR

LLMs aligned to normative preferences can produce homogenized responses, while existing personalization approaches face context or continual-adaptation constraints. COPE combines learnable user embeddings with preference capture, self-evaluation, and response optimization, using proxy rewards when explicit feedback is absent. Experiments report improved personalization under sparse feedback, complementarity with retrieval prompting, and robustness across several analyses.

  • Problem

    LLMs commonly aligned to normative preferences can produce homogenized responses that may underrepresent minority preferences, while existing personalization approaches face context-space or continual-adaptation constraints.

    Source: §1 · p. 1

  • Method

    COPE jointly updates learnable user embeddings and the assistant through preference capture, self-evaluation calibration, and personalized response optimization, using proxy rewards when explicit feedback is unavailable.

    Source: §1 · pp. 1–2, §3.2 · p. 3

  • Results

    COPE improves personalization under sparse feedback, complements retrieval-based prompting, and is reported to show meaningful preference patterns, aligned self-evaluation, preserved general capabilities, and robustness to preference shifts and alternative evaluators.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    COPE is presented as a practical step toward continually adaptive personalized assistants, with learned embeddings capturing meaningful preference patterns and retrieval prompting remaining complementary.

    Source: §6 · p. 8

    Evaluation uses benchmark-derived simulated PersonaLens users, covers 51 optimization interactions per user, and is limited to single-turn sessions; privacy-preserving real-user and multi-turn validation remain future work.

    Source: Limitations · pp. 8–9

Abstract

from arXiv · show

While Large Language Models (LLMs) have achieved remarkable results across various benchmarks, their alignment with normative values often results in homogenized responses that fail to address diverse user preferences. Existing training-free methods often occupy valuable context windows through prompt engineering, while training-based methods typically remain static post-training, failing to support the continual optimization required in real-world settings. To address these challenges, we propose COPE (Continual Optimization with Personalized embedding and self-Evaluation), a novel optimization framework tailored for real-world-motivated interaction settings with sparse user feedback. Our framework assigns learnable personalized embeddings to each user and synergistically integrates preference capture, self-evaluation calibration, and personalized response optimization within a single update step. A key innovation of our method is the use of self-evaluation to generate proxy rewards, enabling continuous model updates even when explicit user feedback is unavailable. Experiments show that COPE consistently outperforms strong training-free and training-based baselines under sparse feedback, and remains complementary to Retrieval-Augmented Prompting (RAP). Further analyses confirm COPE's reliable self-evaluation, meaningful preference patterns, stable general capabilities, and robustness under shifting preferences and alternative evaluators.

1. Introduction

The introduction frames personalization as a gap between group-level alignment and diverse individual preferences, while existing training-free and training-based approaches face context, scaling, or adaptation constraints. COPE combines learnable user embeddings with preference capture, self-evaluation, and response optimization to support continual personalization under sparse feedback.

1. Introduction

  • Existing LLM alignment often assumes shared normative preferences, producing homogenized responses that may underrepresent minority preferences.Training-free personalization uses context signals, while training-based approaches include optimization over preference dimensions, user-specific modules, and interaction modeling.Source: §1 · p. 1
  • Training-free methods consume context space and may suffer retrieval noise or summarization loss, while many training-based methods remain static and limit adaptation to evolving preferences.Some approaches also rely on predefined preference dimensions or discrete user types, and PEFT-style methods face data and scaling constraints.Source: §1 · p. 1
  • COPE assigns each user a learnable embedding and combines preference capture, self-evaluation calibration, and personalized response optimization in one update step.The optimization loop uses real feedback when available and proxy rewards otherwise.Source: §1 · pp. 1–2
  • COPE outperforms strong training-free and training-based baselines across feedback probabilities, complements retrieval-based prompting, and shows robust self-evaluation, meaningful preference patterns, preserved general capabilities, and resilience to preference shifts and alternative evaluators.The introduction summarizes these results across the paper’s experiments and analyses.Source: §1 · p. 2

2. Setup

The setup models a personalized assistant serving multiple users with distinct preferences over a sequence of single-turn sessions. Feedback is sparse, and the scenario supports continual optimization from interaction data as it becomes available.

2. Setup

  • The assistant interacts with N users, each with distinct preferences, over T single-turn sessions in which queries concern tasks from domains in X.The setup denotes the current model parameters as θ_t and each user’s query as q_t^i.Source: §2 · p. 2
  • Each user has a latent preference profile η_i hidden from the assistant, and explicit feedback is provided only probabilistically.The supplied passage introduces feedback probability p but does not provide its full definition.Source: §2 · p. 2
  • The setup aims at continuous personalized optimization by updating the assistant using interaction tuples when available, with preferences expected to be internalized and generalized across related tasks.The passage describes early responses as weakly personalized before the assistant has learned user preferences.Source: §2 · p. 2

3. Methodology

COPE continually updates user-specific embeddings and the assistant through an “Interact-Collect-Optimize” loop. Its unified update combines preference capture, calibrated self-evaluation, and personalized response optimization.

3. Methodology

  • Learnable user embeddings are iteratively updated alongside assistant parameters as interaction data is collected and optimized across sessions.The loop separates users with and without feedback and applies the three objectives in a unified update.Source: §3 · pp. 2–3

3.1. Supervised Fine-Tuning for Preference Capture

  • Textual user feedback trains each personalized embedding to capture implicit preferences while the model parameters remain frozen.The embedding is prepended to the query and response embeddings, and gradients are calculated exclusively for that embedding.Source: §3.1 · p. 3

3.2. Reinforcement Learning for Self-Evaluation Calibration

  • The assistant self-evaluates responses by predicting textual feedback and a numerical score, then uses calibrated scores as proxy rewards when explicit feedback is unavailable.Calibration uses real feedback; gradients refine both personalized embeddings and the assistant model.Source: §3.2 · pp. 3–4

3.3. Reinforcement Learning for Personalized Response Optimization

  • Response optimization combines task-completeness rewards with personalization signals from real or proxy feedback, making completeness a prerequisite for policy reward.The response objective updates model parameters while keeping the personalized embedding frozen.Source: §3.3 · p. 4

3.4. Synergistic Updating

  • Gradients from all three objectives are aggregated for simultaneous parameter updates, keeping both reinforcement-learning objectives on-policy.The reinforcement-learning gradients use PPO; simultaneous aggregation is intended to avoid off-policy issues from sequential optimization.Source: §3.4 · pp. 4–5

4. Experiments and Results

COPE is evaluated on PersonaLens using a controlled chronological protocol and compared with training-free, training-based, and retrieval-augmented methods. Results favor COPE, especially when paired with retrieval, while analyses examine training dynamics, self-evaluation, embeddings, and optimization objectives.

4.1. Experimental Setup

  • The protocol uses 256 PersonaLens users, each interested in 17 domains, with three traversals for continual updates and a fourth for evaluation on unseen tasks.The chronological split tests queries about previously exposed preferences; performance averages personalization multiplied by binary completeness across users, domains, and interactions.Source: §4.1 · p. 5

4.3. Main Results

  • At p=0.75, COPE+SR peaks at 5.36, while COPE scores 4.74 versus PAP’s 3.69; COPE+SR exceeds T-PAP+SR by 0.32–0.40 points.Table 1 reports personalization × binary completeness scores out of 10; COPE+SR achieves the best scores across feedback settings.Source: Table 1 · p. 5, §4.3 · p. 6

4.2. Comparison Methods

  • Comparisons include Base, prompting and retrieval methods, training-based T-PAP, and COPE+SR across feedback probabilities of 0.25, 0.50, and 0.75.T-PAP+SR controls for retrieval’s contribution when added to training-based personalization.Source: §4.2 · p. 5

4.4. Ablation and Analysis

  • Across 20 domains, COPE improves personalization over Base and maintains completeness near 1.0; completeness rises rapidly early, while personalization improves more gradually.Completeness stabilizes after roughly 20 sessions, and personalization later tapers as additional history provides diminishing marginal information.Source: §4.4 · p. 6, Figure 2 · p. 6
  • Self-evaluation tracks user feedback: population-level trajectory correlations are r=0.81 during training and r=0.71 ± 0.06 on held-out runs.At the interaction level, its pairwise rankings agree with users on 72.63% of pairs where both make a directional judgment.Source: §4.4 · p. 6
  • Embedding clusters group users with similar broad preferences while separating finer tastes, such as modern versus traditional interests among high-income travelers.The example interprets two adjacent clusters, G1 and G2, as differing in finer subpreferences despite shared travel preferences.Source: §4.4 · p. 6, Figure 3 · p. 6
  • COPE performance improves as embedding length l increases, with diminishing marginal returns beyond l=4; the final configuration uses l=10.At l=10, user-specific parameters add approximately 1.2 × 10^-5 of the Qwen3-1.7B model’s parameters per user.Source: §4.4 · pp. 6–7, Figure 3 · p. 6
  • Removing either LSFT or Jeval degrades performance, while parameter ablations favor updating both embeddings and model weights for Jeval but only model weights for Jpolicy.Omitting Jeval has the largest objective-ablation impact; updating model weights for LSFT degrades performance, and updating embeddings for Jpolicy causes a drop to 4.45.Source: Table 2 · p. 6, §4.4 · p. 7, Table 3 · p. 7

5. Discussion

COPE retains general capabilities over the evaluated training horizon, adapts to simulated preference shifts, and improves over Base across tested backbones. Its gains also persist under alternative evaluators and blinded human comparison.

General Capability Retention

  • After 51 interactions across three tasks and 17 domains, the optimized model maintains general capabilities relative to the original Qwen3-1.7B.The authors report no catastrophic forgetting over this evaluated training horizon.Source: §5 · p. 7

Adaptation to Preference Shifts

  • After a within-domain preference shift, personalization quickly recovers while completeness remains largely unaffected; under a new-domain shift, both measures drop before COPE rapidly adapts.The embeddings are carried over rather than re-initialized across both shifts.Source: §5 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

Scaling to Larger Models

  • COPE improves over Base across tested backbones; within the Qwen3 family, its score rises from 4.69 at 1.7B to 5.49 at 8B.The Llama-3.1-8B-Instruct result also indicates effectiveness beyond the Qwen family.Source: §5 · p. 8, Table 4 · p. 8

Robustness to Evaluator Choice

  • Across 204 aligned validation pairs, a blinded human annotator consistently prefers COPE over T-PAP, while DeepSeek-V3 preserves COPE’s best-performance ranking despite lower absolute scores.The two checks support effectiveness across judging styles and indicate that the gains are not driven by closed-loop bias.Source: §5 · p. 8

6. Conclusion

COPE combines learnable personalized embeddings with self-evaluation for continual personalization. The authors report improved personalization under sparse feedback alongside retained general capabilities and adaptation to preference shifts.

6. Conclusion

  • COPE is a unified framework for continual personalization optimization using learnable embeddings and self-evaluation.The conclusion also reports meaningful preference patterns and complementarity with retrieval-based prompting.Source: §6 · p. 8
  • COPE improves personalization under sparse feedback while its self-evaluation aligns with user feedback and its optimized model largely preserves general capabilities.The conclusion also reports recovery from preference shifts and robustness under alternative evaluators.Source: §6 · p. 8
  • The authors propose updating one user subset at a time while retrieval over interaction histories supports users awaiting parametric updates.They identify more complex multi-domain, multi-turn tasks and privacy-preserving real-user evaluations as future work.Source: §6 · p. 8

Limitations

COPE is evaluated with benchmark-derived simulated users rather than longitudinal real-user interactions, and the experiments cover single-turn sessions. The authors identify longer optimization horizons and privacy-preserving real-user and multi-turn evaluation as future tests.

Limitations

  • COPE’s evaluation uses benchmark-derived simulated users, not longitudinal real-user interaction data, with privacy-preserving real-user evaluation left to future work.The simulator uses pre-generated, manually inspected query-rubric pairs, making the setting a controlled approximation of sparse personalized interaction.Source: Limitations · p. 8, §H.2 · p. 16
  • COPE is evaluated over 51 optimization interactions per user, while longer horizons are identified as a further stress test.The authors also note that real-user evaluation would complement the controlled evidence.Source: Limitations · p. 8
  • The current evaluation is limited to single-turn sessions; empirical validation on privacy-preserving multi-turn data remains future work.COPE can be extended to multi-turn sessions with session-level feedback without changing its three-objective design.Source: Limitations · p. 9, D. Extension to Multi-Turn Sessions · p. 13

J. Instance-Level Self-Evaluation Reliability

Instance-level analysis finds that the calibrated proxy’s decisive rankings often agree with user rankings, while examples illustrate preference-sensitive scoring when current feedback is unavailable. The authors connect this behavior to jointly representing preferences and calibrating scores.

J. Instance-Level Self-Evaluation Reliability

  • 72.63% of the 285 user-ranked pairs where the proxy produced a ranking had proxy direction agree with the user.The proxy gave a directional judgment on 67.4% of 423 user-ranked pairs; 32.6% were proxy-silent after 710 of 1,133 pairs were discarded as user-tied.Source: J. Instance-Level Self-Evaluation… · p. 17

J.1. Why Self-Evaluation Calibration Works

  • Preference capture encodes textual feedback in user embeddings, while the LLM maps response quality under those preferences to a user-specific score.Teacher forcing grounds response properties in user judgments, and PPO jointly updates the embedding and LLM against numerical feedback.Source: §J.1 · p. 17
  • Performance rises from 3.36 to 4.57 with Jeval and to 4.69 after adding LSFT; updating both the LLM and embedding yields 4.69, versus 4.49 and 4.53 when updating only one.These values are reported in Table 2 for the objective additions and Table 3 for the Jeval update variants.Source: §J.1 · p. 18
  • Two held-out examples show the self-evaluator matching the mapped user grade for both preference-aligned and preference-conflicting responses without current textual feedback.The shopping response received the highest coarse grade, while a merchandise response conflicting with the user’s club preference received the lowest.Source: §J.1 · p. 18

K. General Capability Evaluation

After continual personalization, COPE’s general-capability benchmark changes were within one standard deviation, and the authors interpret performance as comparable to the original model.

K. General Capability Evaluation

  • After 51 optimization interactions, MMLU overall changed from 60.12 to 59.93; all reported benchmark changes were within one standard deviation.The authors interpret the results as comparable performance, not evidence of broad general-capability improvement or degradation, and report no catastrophic forgetting over this evaluated horizon.Source: K. General Capability Evaluation · p. 18

L. Preference Shift Adaptation Evaluation

COPE is evaluated by continuing training with carried-over user embeddings after simulated preference shifts. Personalization recovers after within-domain shifts, and both personalization and completeness improve after shifts to non-overlapping domains.

L. Preference Shift Adaptation Evaluation

  • The shift tests continue training with each user slot’s existing embedding rather than reinitializing it, evaluating whether the representation can be revised.Within-domain and new-domain settings use different user groups, with the latter drawn from non-overlapping domain sets.Source: L. Preference Shift Adaptation… · p. 18
  • After a within-domain preference shift, personalization decreases slightly then quickly recovers, while completeness remains largely unaffected and high.The test switches user slots after 51 interactions while continuing training with the same personalized embeddings.Source: L. Preference Shift Adaptation… · p. 18
  • After a shift to non-overlapping domains, both personalization and completeness decrease, then consistently increase as training continues.The first user group trains for 24 interactions before the second group uses the carried-over embedding for each user slot.Source: L. Preference Shift Adaptation… · pp. 18–19

M. Cross-Family User Model Evaluation

With DeepSeek-V3 replacing the Qwen-family user simulator, COPE retains the highest score and the overall method ordering, despite lower absolute scores.

M. Cross-Family User Model Evaluation

  • The evaluation replaces the Qwen-family user simulator with DeepSeek-V3 while keeping the assistant model, chronological traversal, and comparison methods unchanged.The setting uses p=0.5 and q=0.5; DeepSeek-V3 supplies interaction feedback and evaluates personalization and task completeness.Source: M. Cross-Family User Model Evaluation · p. 19
  • Under DeepSeek-V3, COPE obtains the highest score while the overall ordering is preserved; absolute scores are lower because its judgments are stricter.The preserved ordering is retrieval and profile-based prompting above the base assistant, with training-based personalization further improving performance.Source: Table 7 · p. 19, M. Cross-Family User Model Evaluation · p. 19
  • Table 7 reports personalization score multiplied by binary completeness score, out of 10, as mean±std.The table caption marks training-involving methods with a dagger.Source: Table 7 · p. 19

N. Independent Blinded Pairwise Evaluation

A blinded, rubric-free human evaluation prefers COPE to T-PAP across 204 sampled response pairs, with the annotator judging which response better matches the user.

N. Independent Blinded Pairwise Evaluation

  • The study randomly sampled 12 aligned COPE/T-PAP pairs from each of 17 held-out validation steps in the fourth traversal and randomized response order.The annotator saw user demographics, relevant domain affinities, the query, and both responses; simulator rubrics and scores were withheld.Source: N. Independent Blinded Pairwise… · p. 19
  • COPE wins 76.92% of decisive human comparisons, with 120 preferences for COPE, 36 for T-PAP, and 48 ties across 204 pairs.The blinded, rubric-free evaluation supports the authors’ conclusion that COPE adapts to human preferences rather than merely learning simulator scoring behavior.Source: N. Independent Blinded Pairwise… · pp. 19–20, Figure 9 · p. 20
  • Figure 9 summarizes blinded pairwise preferences over all 204 randomly sampled validation interactions.The reported percentage comparison is between COPE and T-PAP.Source: Figure 9 · p. 20

Related papers

Loading 2609.26853v1…