PaperlayerView exact revision on arXiv

Source-linked AI summary

Hypotheses-Guided Self Distillation for Continual Personalization

EunJeong Hwang, Kushan Mitra, Dan Zhang, Hannah Kim, Estevam Hruschka

arXiv:2609.00251v1cs.AI

TL;DR

Continual personalization must infer incomplete, noisy preferences from interaction signals, while raw-history and reward-based approaches have scalability or cost limitations. HYPREFLECT maintains explicit, uncertainty-aware, revisable hypotheses, reflectively refines them, and uses them in self-distillation. Across three settings, it improves over raw-history and incremental-update baselines, with reported gains up to 4.4%, 10.3%, and 5.0%.

  • Problem

    User preferences are rarely fully stated and instead emerge as heterogeneous, latent, noisy signals spread across interactions, challenging continual personalization.

  • Method

    HYPREFLECT infers explicit, uncertainty-aware preference hypotheses from diverse signals, reflectively refines them over time, and guides self-distillation with the refined user model.

  • Results

    Up to 4.4%, 10.3%, and 5.0% relative improvements are reported across explicit-feedback, multi-session, and implicit-behavior personalization settings, respectively.

  • Takeaways & Limitations

    Explicit, reusable, revisable preference hypotheses support reliable and scalable continual personalization across settings and context budgets.

  • Takeaways & Limitations

    Preference hypotheses may encode sensitive user information and therefore remain private user data requiring consent, security, and user control.

Abstract

from arXiv · show

As people increasingly interact with LLM assistants in daily life, continually adapting to individual preferences has become essential for effective long-term interactions. However, user preferences are rarely stated in full, and instead emerge through heterogeneous, latent, and noisy signals, with existing methods relying on raw interaction histories or costly reward-based optimization to manage personalization. We introduce HypReflect, a reliable, scalable framework for continual personalization that infers explicit, uncertainty-aware preference hypotheses from diverse user signals, reflectively refines them as new evidence accumulates, and incorporates the resulting user model through hypotheses-guided self-distillation. Experiments across three personalization settings: online personalization, multi-session interactions, and implicit behavioral signals, show that HypReflect outperforms a range of baselines, including raw-history and incremental-update methods. We further demonstrate strong generalization to unseen users and cross-domain settings, along with stability across context budgets, reusable hypotheses, and more focused personalization. These results suggest a step towards reliable and scalable continual personalization through explicit, revisable user preference hypotheses.

1 INTRODUCTION

Continual personalization is difficult because user preferences emerge as sparse, noisy, context-dependent signals across interactions, while existing approaches retain costly histories or rely on reward-based optimization. HYPREFLECT addresses this by maintaining explicit preference hypotheses and using reflective, hypotheses-guided self-distillation, improving across three personalization settings.

  • Motivation: User preferences emerge incrementally from explicit feedback, implicit behavior, and profile information, making stable preference identification difficult.Signals may be sparse, noisy, context dependent, and distributed across long multi-turn or multi-session histories.
  • Prior limitations: Existing personalization methods condition on profiles, memories, or raw histories, while reinforcement-learning approaches require costly reward models.Raw-history approaches also become increasingly expensive to retain and retrieve over time.
  • HYPREFLECT: HYPREFLECT maintains explicit, uncertainty-aware, revisable preference hypotheses from diverse user signals and uses them to guide personalized response generation.The framework combines signal distillation, reflective consolidation, and hypotheses-guided self-distillation.
  • Results: Up to 4.4%, 10.3%, and 5.0% relative improvements are achieved across explicit-feedback, multi-session, and implicit-behavior settings, respectively.The evaluation also reports generalization to unseen users, shifted user characteristics, and cross-domain personalization.

2 PROBLEM FORMULATION

The problem is to continually infer and use user preferences from accumulating interactions when personalized ground-truth responses and complete preference statements are unavailable. Self-distillation uses the subsequent user turn as privileged information, while HYPREFLECT introduces explicit, reusable, revisable hypotheses between interaction history and response generation.

  • Problem setting: Continual personalization requires inferring, maintaining, and using increasingly informative interaction histories to personalize future responses.The assistant repeatedly observes conversation history and current user turns containing feedback, requests, or both.
  • Self-distillation: Self-distillation treats the subsequent user turn as privileged information, avoiding an external teacher or reward model when personalized responses lack ground truth.The teacher receives richer information than the student and provides feedback-informed supervision.
  • Pipeline: The refined user state conditions both student and teacher, while the teacher additionally observes the subsequent user turn during self-distillation.This shared state connects maintained user understanding to personalized response generation.
  • Preference representation: HYPREFLECT inserts an explicit preference-hypothesis layer between raw interactions and response generation.The hypotheses are intended to make inferred preferences reusable and revisable as new evidence arrives.

3 HYPREFLECT: CONTINUAL USER MODELING THROUGH REFLECTIVE REFINEMENT

HYPREFLECT builds a bounded, revisable user model by generating local preference hypotheses from interaction chunks, reflectively consolidating them across time, and conditioning self-distillation on the refined state. This design preserves long-term evidence while limiting context growth.

  • Inferring local hypotheses: HYPREFLECT first infers reusable preference hypotheses from individual interaction chunks rather than processing the full history at once.Chunks contain the longest sequence of complete interactions within token budget B, and inference avoids unsupported generalizations.
  • Hypothesis representation: Each hypothesis is a natural-language preference statement paired with a confidence score from 1 to 5.Multiple candidate hypotheses preserve alternative beliefs, while confidence distinguishes stronger from more uncertain preferences.
  • Summary augmentation: The summary-augmented variant jointly maintains summaries and hypotheses so conversational context can complement reusable preference abstractions.Summaries ground hypotheses in observed interactions while hypotheses capture explicit or inferred user signals.
  • Why refinement is needed: Incremental updating can accumulate errors or overwrite earlier valid preferences because each update operates on an already compressed representation.The paper empirically associates this behavior with unstable long-term personalization.
  • Reflective refinement: Reflective refinement jointly compares hypotheses across chunks to consolidate consistent evidence, resolve conflicts, merge redundancies, and revise unsupported beliefs.A bounded reflection window keeps this process tractable while retaining a refined representation of earlier preferences.
  • Hypothesis-guided self-distillation: During self-distillation, student and teacher share the refined hypothesis state, while the teacher uses the subsequent user turn to provide feedback-informed supervision.The objective is computed token-wise over the generated response.

4 EXPERIMENTAL SETUP

The evaluation tests continual personalization from explicit feedback, multi-session profile information, and implicit behavioral choices against raw-history, summary, hypothesis, and reflective-prompting baselines. Experiments use multiple datasets, users, backbone models, and an LLM judge.

  • Evaluation settings: Three settings evaluate personalization from explicit feedback, multi-session conversational profiles, and implicit behavioral choices.These settings test whether models infer reusable preferences from diverse interaction signals and apply them to future responses.
  • HelpSteer2: The HelpSteer2 online setting trains over 250 interactions and evaluates every 50 interactions on held-out prompts using an LLM judge.A user simulator supplies feedback on one or two relevant writing-style preference dimensions after each response.
  • HiCupid: The HiCupid setting trains on QA interactions from 300 users and evaluates personalization for seen and unseen users.Evaluation uses held-out prompts and pairwise win rates from an LLM judge.
  • Flight Recommendation: The Flight Recommendation setting uses trajectories from 624 users with up to 25 interactions per user to infer preferences from partial behavioral evidence.User choices can encode non-obvious trade-offs rather than fixed assumptions.
  • Baselines: Baselines include vanilla generation, raw-history self-distillation, incremental summaries, incremental hypotheses, reflective prompting, and summary-augmented HYPREFLECT.The summary-augmented variant examines whether summaries provide complementary information during hypothesis refinement.
  • Implementation: Experiments use Qwen3.5-4B, Qwen3.5-9B, and Gemma4-4B as backbone language models.The primary evaluation judge is deepseek-v4-flash, with consistency checked against other judges in an appendix.

5 RESULTS

Across three continual-personalization benchmarks, HYPREFLECT generally outperforms raw-history and incremental-update baselines. Analyses further associate reflective refinement with stronger generalization, reusable hypotheses, focused personalization, and robustness to context and reflection-window choices.

  • Main results: HYPREFLECT outperforms SD in eight of nine settings, while HYPREFLECT+SUM improves over HYPREFLECT in all nine.The combined memory state benefits from complementary evidence in summaries and inferred hypotheses.
  • Main results: SD-INCSUM and SD-INCHYP underperform SD in six and three settings, respectively, whereas reflective refinement maintains more reliable hypotheses as histories grow.Incremental updates can propagate earlier errors or let recent observations overwrite valid prior information.
  • Main results: Reflective refinement plus self-distillation outperforms inference-time reflective prompting by an average of 25.2 accuracy points on Flight Recommendation.Reflective prompting alone improves the base model, but self-distillation more effectively incorporates preferences into response behavior.
  • Generalization results: HYPREFLECT+SUM improves over SD in all nine generalization settings, while HYPREFLECT improves over SD in eight of nine.The settings include unseen users, varying feature counts, and cross-domain transfer; SD-INCSUM shows large drops on Flight transfer tasks.
  • Further analysis: Transferred HYPREFLECT hypotheses yield average gains of 11.5 accuracy points on Flight Recommendation and 10.0 winrate points on HiCupid.Final hypotheses are provided to corresponding non-finetuned base models, indicating that the hypotheses contain reusable user information.

6 RELATED WORK

Prior work personalizes conversational LLMs through prompting, parameter adaptation, alignment, and evolving user representations. Feedback-driven methods further adapt behavior by distilling privileged information, using external feedback, or refining user representations.

  • Personalization in Conversational LLMs: Personalized LLMs adapt responses as users’ preferences, goals, and behaviors evolve through prompting, parameter adaptation, and alignment-based techniques.
  • User Modeling: User representations in prior work include personas, textual summaries, RL-based profile updates, profiles, behavioral patterns, and predefined attributes.
  • User Modeling: Recent user-modeling approaches treat personalization as evolving inference by maintaining beliefs over goals, refining hypotheses, or inferring communication styles.
  • Feedback-Driven Learning and Adaptation: Feedback-driven methods distill privileged information, combine self-distillation with policy optimization and external feedback, leverage subsequent responses, or refine user representations through interaction.

7 CONCLUSION

The conclusion presents HYPREFLECT as a reliable and scalable approach that maintains explicit, uncertainty-aware, revisable preference hypotheses and uses them for self-distillation. Across three settings, it reports consistent improvements and emphasizes explicit, revisable user models for continual personalization.

  • HYPREFLECT infers explicit, uncertainty-aware preference hypotheses from diverse signals, reflectively refines them, and incorporates them through hypotheses-guided self-distillation.
  • Reflective refinement maintains stable yet revisable user hypotheses over long interaction histories as new evidence accumulates.
  • Across three personalization settings, HYPREFLECT consistently improves approaches based on raw histories or incremental updates and remains stable across context budgets.
  • The results highlight explicit, revisable user hypotheses as important for reliable and scalable continual personalization.

AI USE STATEMENT

The authors used ChatGPT for language editing and proofreading during manuscript preparation, not for the research’s substantive scientific work.

  • ChatGPT was used to identify grammatical errors, improve sentence structure, enhance clarity, and refine wording and phrasing.
  • ChatGPT was not used to generate or substantively modify the research ideas, methodology, experiments, analysis, results, or conclusions.
  • The statement attributes all scientific content, interpretations, technical decisions, code, and experiments to the authors.

ETHICS STATEMENT

The ethics statement identifies privacy, inaccurate inference, and sycophancy risks in continual personalization. It calls for user control, preference correction, transparency, uncertainty, and corrective or alternative perspectives.

  • Privacy: Explicit preference hypotheses may encode sensitive information and should be treated as private user data despite not directly exposing raw histories.
  • Privacy: Deployments should obtain consent, handle inferred data securely, and support user control over storage, modification, and deletion.
  • Inference Risks: Inaccurate, outdated, or overgeneralized preferences can reinforce assumptions or reduce users’ exploration of alternative behaviors.
  • Inference Risks: Practical systems should provide preference correction and transparency so inferred preferences remain aligned with users’ intended goals.
  • Sycophancy: Personalization can amplify sycophancy by turning preferences or prior feedback into signals to agree with beliefs rather than merely adapt responses.
  • Sycophancy: Systems should distinguish preference adaptation from endorsing claims, preserve uncertainty, and retain corrective information or alternative perspectives.

REPRODUCIBILITY STATEMENT

The paper documents reproducibility resources while noting boundaries in preference modeling, evaluation realism, computation, and representation.

  • The authors provide detailed training, evaluation, judging, and prompt descriptions in Appendices C and F, and plan to release code upon acceptance.
  • The experiments cover explicit feedback, implicit behavioral signals, and conversational user profiles, while richer and more dynamic user behavior remains future work.The authors specifically mention indirect corrections, contextual cues, changing preferences, and task-dependent preferences.
  • The evaluation covers three personalization settings but may not fully reflect the complexity of long-term human interactions.The authors propose longitudinal user studies and real-world deployments as future complements.
  • Experiments are limited to models up to 9B parameters because continual personalization remains computationally demanding.Bounded refinement avoids reward-model training and repeated policy optimization, but does not eliminate computational demands.
  • HYPREFLECT assumes preferences can be represented as natural-language hypotheses with confidence scores, which may miss difficult-to-verbalize or latent contextual factors.The authors suggest richer representations that combine language with other forms.
  • Offline processing partitions histories into token-budgeted chunks, while online processing accumulates complete interactions until adding another would exceed budget B.Local hypotheses are inferred from each completed chunk.

C.1 DATASET DETAILS

The evaluation uses three datasets representing distinct evidence types and personalization settings, with dataset-specific procedures, judges, and qualitative analyses.

  • HelpSteer2: HelpSteer2 uses eight synthetic user profiles with three writing-style preferences and sparse simulator feedback on one or two preference dimensions.A separate model is trained for each user profile.
  • HiCupid: HiCupid contains 9K personalized QA examples from 300 users and evaluates personalization for both seen and unseen users using profile data as privileged information.
  • Flight Recommendation: Flight Recommendation contains trajectories from 624 users with up to 25 interactions, where selections reveal latent preferences and non-obvious trade-offs.The task evaluates inference from implicit behavioral signals rather than fixed assumptions.
  • Evaluation: HelpSteer and HiCupid use DeepSeek-V4-Flash for LLM judging, whereas Flight uses exact matching against ground-truth choices.Additional judges produce consistent evaluation outcomes and the same method ranking.
  • Qualitative analyses: Reflective refinement improves hypothesis consolidation by avoiding broad, overlapping abstractions and preserving more distinctive guidance for personalization.Qualitative comparisons show reflection re-evaluating conflicting evidence across the recent interaction window.
  • Qualitative analyses: Explicit weighted hypotheses help HYPREFLECT emphasize relevant preferences in HiCupid and prioritize shorter duration in Flight when prices are equal.In the cited Flight example, HYPREFLECT selects Flight 2 while SD selects Flight 3.
  • Additional analyses: Figure 10 compares performance across reflection-window sizes on a HelpSteer2 subset at every 50 interactions.
  • Dataset-specific methodology: Flight hypotheses represent directional preferences over departure time, duration, stops, and price; dataset-specific instructions adapt extraction to each evidence type.The overall prompt structure is shared across datasets.
Loading 2609.00251v1…