Source-linked AI summary
Aligning Language Models from User Interactions
Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor, Idan Shenfeld, Giorgia Ramponi, Andreas Krause
TL;DR
Language models produce abundant user interactions containing implicit feedback, but learning from these unlabeled conversations remains difficult. The paper introduces SDPO, which conditions the model on follow-ups and distills the resulting hindsight distribution into the original policy. Training on WildChat improves alignment and instruction following without degrading other capabilities, while also supporting personalization and continual adaptation; safety requires additional guardrails because the method does not distinguish benign from adversarial signals.
Problem
Effective methods are lacking for learning directly from multi-turn user interactions, whose feedback is implicit rather than provided as labels, preferences, demonstrations, or rewards.
Method
SDPO conditions the model on each user follow-up, compares its hindsight token distribution with the original policy, and distills the resulting signal back into the policy.
Results
Training on raw WildChat conversations improves alignment and instruction following across evaluated benchmarks without degrading other capabilities, while supporting personalization and continual adaptation.
Takeaways & Limitations
Naturally occurring user interactions can serve as a data modality for alignment, personalization, and continual adaptation during deployment.
Takeaways & Limitations
User follow-ups may reward unsafe or manipulative behavior, and SDPO does not by itself distinguish benign from adversarial learning signals.
Abstract
from arXiv · showhide
Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user's preferences. Importantly, language models are already able to make use of this information in context. After observing a user's follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user's follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model's behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation.
1 Introduction
User interactions contain implicit signals about errors, instruction failures, and preferences, but effective methods for learning from them are lacking. The paper uses hindsight self-distillation to extract these signals and reports improvements in alignment, instruction following, personalization, and continual adaptation.
- User follow-ups can reveal that a response was incorrect, violated an instruction, or mismatched user preferences.
- Language models often revise errors, formatting, tone, and style effectively after observing a user’s follow-up.
- SDPO compares the original policy with a hindsight policy conditioned on the follow-up, then distills the resulting token-level signal into the original policy.
- Training on WildChat conversations improves alignment and instruction following across standard benchmarks without degrading other capabilities.
- The same mechanism supports personalization and continual adaptation through continued interaction without explicit supervision.
2 Problem Formulation
The paper formulates direct learning from multi-turn conversations as training without external labels, annotations, or extracted rewards. It asks whether this can improve general alignment while enabling continual personalization from interactions alone.
- A conversation is represented as alternating assistant responses and user messages, with each interaction defined by history x, response y, and follow-up o.
- A multi-turn conversation yields overlapping interaction tuples because each later history contains the preceding interaction.
- Existing approaches may add semantic categorization, preference annotation, or post-hoc rewards, but reliable signal construction from raw interactions remains unclear.
- Direct Learning from User Interactions asks whether models can be trained from multi-turn interactions in a simple, principled, and scalable manner.
- The formulation also asks whether interaction-only training can improve general alignment and enable continual personalization without explicit supervision.
3 Directly Learning from User Interactions via Self-Distillation
The method treats follow-up messages as hindsight information, comparing conditioned and original token distributions to produce a fine-grained self-distillation signal. The paper also connects this signal to instruction correction and latent-reward optimization under idealized assumptions.
- Naturally occurring follow-ups provide implicit signals about whether an assistant response was adequate, even without explicit feedback.
- Conditioning on a follow-up produces a hindsight distribution that often better aligns responses with the task than the original policy.
- Comparing original and hindsight token probabilities identifies tokens to penalize or reinforce based on the user’s subsequent message.
- Table 1 specifies the hindsight-policy chat template and recovers the base-policy template by removing the hindsight context.
- Under idealized assumptions, the self-distillation advantage can be interpreted as implicitly maximizing the interacting user’s latent reward function.
4 Experimental Results
SDPO improves alignment and instruction-following from real-world user conversations while generally preserving other capabilities, and it supports rapid personalization and adaptation from interaction alone. Its learning signals are informative when follow-ups provide feedback and are naturally suppressed when follow-ups are irrelevant.
- General alignment: SDPO improves alignment and instruction-following on real-world conversations without regressing other evaluated capabilities.The evaluation covers alignment, instruction-following, math, coding, creative writing, and knowledge tasks.
- General alignment: Qwen3-4B gains 8.2% on AlpacaEval 2.0 and 1.3% on IFEval but loses 1.2% on ArenaHard-v2 math and coding.The results indicate a trade-off for this model, whereas other models show more consistent but often modest improvements.
- Data quality: SDPO remains effective on fully uncurated conversations, improving alignment and instruction-following while reducing ArenaHard-v2 math and coding performance by 0.6%.Filtering feedback-rich conversations strengthens the signal, but uncurated data does not cause widespread degradation.
- Comparison with SFT: Standard supervised fine-tuning on WildFeedback assistant completions substantially degrades performance across all benchmarks.The contrast reflects that logged completions can come from weaker or dissatisfied-response contexts, whereas SDPO uses hindsight-based signals.
- Continual personalization: SDPO rapidly personalizes responses, exceeding an 85% win rate after 50 interactions and 95% after 200 interactions.It can match or exceed an in-context oracle supplied with the full user profile, using only limited interaction data and policy updates.
- Continual personalization: After 250 interactions, SDPO quickly reverses learned behavior when user preferences flip and can retain compatible earlier preferences while adding new ones.The method therefore supports both unlearning outdated preferences and continual accumulation of compatible preferences.
- Interpretability and robustness: Relevant follow-ups create interpretable token-level advantages, whereas unrelated follow-ups produce near-zero signals and little or no learning update.This behavior links strong updates to corrections, revisions, and explicit preferences while suppressing updates for uninformative topic shifts.
5 Related Work
Prior work uses explicit feedback, reward modeling, curated preference data, or in-context revision to improve language models, while SDPO applies self-distillation directly to user interactions. Its distinguishing approach is to use hindsight-conditioned behavior rather than auxiliary reward extraction or manually curated feedback.
- Preference-Based Alignment: Preference-based alignment methods optimize rankings or rewards from curated datasets with explicit feedback for each generation.These approaches include supervised instruction tuning, RLHF, and direct preference optimization.
- Learning from Natural Language Feedback and through Retrospection: Natural-language feedback can be converted into token-level or state-wise rewards using external models, manually designed rubrics, or explicit reward construction.These methods introduce additional reward-modeling steps between user feedback and policy optimization.
- Learning from Natural Language Feedback and through Retrospection: Other approaches improve responses in context or train on feedback-refined and manually paired examples without necessarily modeling explicit rewards.Examples include in-context improvement, direct preference optimization on before-and-after responses, and supervised fine-tuning on refined generations.
- Self-Distillation: Self-distillation transfers a teacher’s output behavior into model weights, with prior work using fixed contexts or extra hints through SFT, DPO, or GRPO.The paper positions SDPO as on-policy self-distillation, contrasting with prior off-policy approaches trained on teacher-generated outputs.
6 Discussion
The paper interprets SDPO as hindsight self-distillation: user follow-ups condition the model into a better-aligned distribution whose differences from the original policy provide a local learning signal. Empirically, SDPO improves alignment, instruction following, personalization, and adaptation from raw interactions, while safety risks remain because the method does not distinguish benign from adversarial feedback.
- Discussion: SDPO treats a user’s next message as hindsight information and distills the resulting token-level behavior into the model without auxiliary mechanisms.The method leverages in-context learning to derive an interpretable local update from naturally occurring interactions.
- Empirical Findings: SDPO improves general alignment and instruction-following performance, supports continual personalization, and remains robust to noisy, uncurated, or irrelevant follow-ups.The paper reports these outcomes across experiments using raw real-world user conversations.
- Broader Implications: User interactions offer a distinct training modality because they arise during deployment and reflect how model outputs are used, evaluated, and acted upon.The paper argues that their scale and diversity create potential for learning systems that connect deployment with training.
- Safety and Ethical Considerations: SDPO does not distinguish benign from adversarial learning signals, so continual personalization can be exploited to steer models toward unsafe or manipulative behavior.The paper identifies guardrails, transparency, consent, and governance as necessary considerations for learning from user interactions.
- Latent Reward Perspective: Under a stylized user model, maximizing the sequence-level SDPO advantage is equivalent to maximizing latent user reward up to an additive normalization term.This interpretation depends on strong assumptions about user behavior and the hindsight distribution.
B Gradient Derivation
The derivation establishes that the one-sample gradient estimator is unbiased for the SDPO gradient by expressing token-level contributions under autoregressive sampling and matching their expectations.
- The one-sample approximation is an unbiased estimator of the SDPO gradient in Equation (3).
- The proof samples the response autoregressively from πθ and analyzes each token position conditional on its preceding tokens.
- At each position, the derivation defines a score-function contribution weighted by the advantage and its conditional expectation under the policy.
- The two estimators are compared through their expectations, with the analytic SDPO gradient identified as the expectation of one estimator.
- Using conditional expectation and the tower property, the proof shows that the two estimator expectations coincide under finite-expectation assumptions.
C.1 Hyperparameters
The appendix reports SDPO hyperparameters and standard benchmark evaluation settings, including judge models and task-specific evaluation protocols.
- SDPO hyperparameters are reported across all experiments in Table 5.
- Learning rates were swept over {1, 2, 3, 5} × 10^-6 for Qwen3-4B and the SFT checkpoint, with 2 × 10^-6 best for the SFT setup.
- For reported benchmarks, AlpacaEval 2.0 used Weighted Alpaca Eval GPT-4 Turbo and ArenaHard-v2 used GPT-4.1 as judges.
- IFEval results use prompt-level loose evaluation, while MMLU-Pro uses the recommended chain-of-thought 5-shot setting.
D.1 Additional Results from Section 4.1
Additional pre-training evaluations show that SDPO preserves Qwen3-8B performance rather than changing it across the reported benchmarks.
- No performance changes were observed for Qwen3-8B on the pre-training benchmarks in Table 6.
- Table 6 evaluates TruthfulQA, HellaSwag, and CommonsenseQA as additional standard pre-training benchmarks.
D.2 Additional Results from Section 4.2
Additional personalization experiments test SDPO on Qwen3-4B across user preferences for detail, tone, and expertise. The model adapts quickly from a handful of interactions and can sometimes exceed an in-context oracle.
- Across user profiles, SDPO quickly adapts Qwen3-4B using only a handful of user interactions.
- The evaluated preference dimensions are detailed versus concise, casual versus professional, and beginner versus expert.
- SDPO sometimes exceeds the in-context oracle prompted directly with the desired writing style.
- The win rate is computed against the base model and judged by Qwen3-8B.
- Each response pair is judged twice with flipped positions, and win rates are evaluated on 256 held-out prompts per experiment.
- The user simulations use profile prompts describing preferences such as avoiding emojis, reducing filler praise, and answering directly with less formatting.
- Additional profiles include concise, casual, beginner-friendly responses and detailed, professional, expert-level responses.
- User simulators generate brief preference-focused follow-ups, while evaluators choose A, B, or C based on style, tone, formatting, verbosity, and complexity.