The Era of Real-World Human Interaction: RL from User Conversations

1 Introduction

• Static expert annotations reflect context-free judgments rather than real users’ diverse, evolving goals and preferences.The paper contrasts annotation-budget-limited datasets with supervision arising from organic user interaction.Source: §1 · p. 1
• Organic conversations expose preferences through corrections, follow-up requests, approvals, and changing goals, offering supervision for personalized and adaptable behavior.The authors argue that modern language models improve the ability to extract these signals from interactions.Source: §1 · p. 2
• RLHI learns from in-the-wild conversations through user-guided revisions and persona-conditioned rewards, linking long-term user histories to turn-level preferences.Rewrites pair revised and original responses; persona-conditioned rewards rank candidate responses to generate preference pairs.Source: §1 · p. 2
• Both RLHI variants outperform strong baselines in personalization and instruction-following, while User-Based Rewards leads on AlpacaEval 2.0 and User-Guided Rewrites improves reasoning accuracy.User-Based Rewards reaches a 77.9% length-controlled win rate on AlpacaEval 2.0; User-Guided Rewrites raises average accuracy from 26.5 to 31.8 across four reasoning benchmarks.Source: §1 · p. 2
• Ablations find benefits from user guidance and interaction diversity, stronger performance from reinforcement learning than supervised finetuning, and a need for quality filtering.Filtering is described as essential for effectively leveraging noisy interaction data.Source: §1 · p. 2

2.1 The Era of Real-World Human Interaction

• Human interaction provides contextual, evolving, and diverse supervision, including corrections and clarifications as well as implicit cues such as disengagement or frustration.The authors describe interactions as tied to ongoing tasks, changing user needs, and personalized histories.Source: §2.1 · pp. 2–3
• RLHI combines long-term persona summaries with multi-turn context in two modes: follow-up-guided rewrites and persona-conditioned ranking of candidate responses.Rewrites form preference pairs from original and revised outputs; reward-ranked candidates form chosen-rejected pairs.Source: Figure 2 · p. 3

2.2 Analysis of Real-World Human Interaction

• In WildChat, 26.51% of user messages are feedback-bearing re-attempts, rising to 83.15% of utterances after the fifth turn; these re-attempts average 272 characters versus 725 for initial requests.The analysis classifies follow-ups into new requests, re-attempts with or without feedback, and positive feedback; the authors note that the statistics may change as model capabilities evolve.Source: §2.2 · pp. 3–4
• WildChat request contexts are more diverse than sampled annotated preference datasets, and user personas include distinct, domain-specific, and evolving preferences.WildChat’s average pairwise cosine distance is 0.865, compared with 0.751 for HH-RLHF and 0.848 for HelpSteer2; preference dimensions include expertise, informativeness, tone, and structure.Source: §2.2 · p. 4

2.3 RLHI with User-Guided Rewrites

• User-Guided Rewrites turn feedback-bearing follow-ups into preference pairs, while persona summaries connect long-term preferences to local corrections and guide generation at inference.Rewrites are filtered both for improvement over the original and for overall quality.Source: §2.3 · p. 5
• Persona-conditioned DPO optimizes relative preference for revised responses given both prompt and persona, with β controlling preference-learning sharpness.The objective uses a frozen reference model as a baseline.Source: §2.3 · p. 5

2.4 RLHI with User-Based Rewards

• User-Based Rewards samples candidate responses, scores them using a reward model conditioned on the user persona, and forms preference pairs from the highest- and lowest-scoring candidates.The method extends persona-grounded alignment to initial requests even when explicit follow-up feedback is unavailable, using offline or online DPO.Source: §2.4 · pp. 5–6

3.1 Training Data Generation

• The user-facing training data derives from WildChat, excludes Midjourney-related instructions, and reconstructs assistant responses with Llama-3.1-8B-Instruct; reasoning conversations simulate feedback on 10,000 erroneous solutions.The reasoning simulations use PRM800K-derived math examples, and simulated users indicate error locations without supplying corrections or personas.Source: §3.1 · p. 6

3.2 Training Details

• Both RLHI methods initialize from Llama-3.1-8B-Instruct and use persona-conditioned DPO, with different batch sizes and learning-rate settings.User-Guided Rewrites uses batch size 64 and sweeps 5 × 10^-7 and 1 × 10^-6; User-Based Rewards uses batch size 32, learning rate 1 × 10^-6, and KL penalty β = 0.01.Source: §3.2 · p. 6

3.3 Models and Baselines

• Baselines include rewrites generated without prior outputs or feedback, generic user-agnostic rewards, supervised finetuning on chosen responses, and rewrites trained without quality filtering.Source: §3.3 · p. 6

3.4 Evaluation Setting

• WildChat UserEval tests held-out turns for personalization, instruction-following, and holistic preference, using an o3-based judge and a human study; standard benchmarks use GPT-4 Turbo or sampled-solution accuracy.UserEval samples 100 users with at least 10 conversations, holds out each user’s final five dialogues, and includes 10 human evaluators; reasoning evaluation covers four benchmarks with 50 sampled solutions per problem.Source: §3.4 · p. 7

4.1 Main Results

• On WildChat UserEval, User-Guided Rewrites yields the largest personalization (+24.3) and overall (+22.4) gains, while User-Based Rewards leads instruction-following (+14.1).Source: §4.1 · p. 7, Table 2 · p. 8
• The methods were described as confirming their effectiveness under direct human judgment, with Llama-3.1-8B-Instruct named but no quantitative win-rate result stated in the cited passage.Source: §4.1 · p. 7
• User-Based Rewards reaches a 77.9% length-controlled win rate on AlpacaEval 2.0, outperforming all baselines, and matches User-Agnostic Rewards on ArenaHard.Source: §4.1 · p. 7, Table 3 · p. 8
• User-Guided Rewrites raises average reasoning accuracy from 26.5 to 31.8 across four benchmarks, including tasks beyond the math conversations used for training.Source: §4.1 · pp. 7–8, Table 4 · p. 8

4.2 Understanding Human Interaction and RLHI

• User-Guided Rewrites outperforms regeneration from scratch, with a 60.4% head-to-head win rate and +7.9 points in personalization.Source: §4.2 · pp. 8–9, Figure 3 · p. 9
• Persona-conditioned User-Based Rewards improves personalization (+8.3 points) and instruction-following over user-agnostic rewards while maintaining competitive standard-benchmark performance.Source: §4.2 · p. 9, Figure 3 · p. 9
• RL outperforms supervised finetuning across both RLHI variants and evaluations; the paper attributes the gap to RL’s use of preferred and dispreferred examples.Source: §4.2 · p. 9, Figure 3 · p. 9
• Reward-model filtering changes User-Guided Rewrites gains from +2.5 and +3.3 points without filtering to +23.4 and +17.7 points with filtering on user-based and standard evaluations.Source: §4.2 · p. 9, Figure 3 · p. 9

• RLHF work trains reward models on preferences and optimizes models with RL, while later methods use direct preference objectives on pairwise comparisons or implicit signals [Ziegler et al., 2019; Ouyang et al., 2022; Rafailov et al., 2023; Ethayarajh et al., 2024; Azar et al., 2024].Source: §5 · p. 10
• Post-deployment feedback has been harvested through classifiers, heuristics, and organic signals such as ratings or comments, with positive signals used for fine-tuning or other optimization [Hancock et al., 2019; Chen et al., 2024b; Don-Yehiya et al., 2024; Pang et al., 2023; Jaques et al., 2020; Xu et al., 2023].Source: §5 · p. 10
• Personalization methods use retrieval, prompting, representation learning, or user-conditioned reward signals; RLHI explicitly links long-term histories to turn-level preferences through RL [Zhang et al., 2024; Mysore et al., 2023; Salemi et al., 2024; Jiang et al., 2023; Tan et al., 2024; Chen et al., 2025; Jang et al., 2023; Yang et al., 2024; Li et al., 2024b; Chen et al., 2024a; Lee et al., 2024; Poddar et al., 2024].Source: §5 · p. 10

6 Conclusion

• RLHI learns directly from in-the-wild user conversations using long-term conversation history and organic natural-language feedback.The authors describe it as a simple and scalable framework.Source: §6 · p. 10
• The authors report clear user-level improvements over strong baselines, with organic feedback improving both non-reasoning and reasoning tasks.Source: §6 · p. 10
• Future opportunities include human-in-the-loop learning, richer and safer reward modeling, privacy-preserving personalization, and broader modality and task coverage.The authors also believe an online learning loop could bring major gains over the fixed-training-data setup used in their experiments.Source: §6 · p. 10

A Examples of User Conversation Data

• Initial requests account for 27.07% of user messages; follow-ups are classified as new requests (40.40%), re-attempts with feedback (26.51%), re-attempts without feedback (4.77%), or positive feedback (1.25%).Source: A Examples of User Conversation Data · p. 14

A.1 New requests

• New requests can follow a greeting, change the specific content while retaining a theme, or shift to an unrelated topic.Source: §A.1 · p. 14

A.3 Re-attempts without feedback

• Re-attempts without feedback repeat a request without actionable guidance, unlike repeats that add a signal such as “make them creative.”Source: §A.3 · p. 15

A.4 Positive feedback

• Positive feedback accounts for only 1.25% of user messages.Source: §A.4 · p. 16
• The passage attributes this rarity to users often ending conversations once satisfied.Source: §A.4 · p. 16
• The example exchange ends with the user saying “You’re welcome!” after the assistant answers a question about Hinduism and Buddhism.Source: §A.4 · p. 16

B Details of User Conversation Data Analysis

• The authors report that real-world human-interaction data are more diverse than existing preference datasets.Source: B Details of User Conversation Data… · p. 16
• The visualization uses prompt embeddings calculated in Section 2.2.Source: B Details of User Conversation Data… · p. 16
• Figure 5 visualizes context embeddings for HH-RLHF, HelpSteer2, and the human-interaction dataset used for RLHI.Source: Figure 5 · p. 16

C.1 Details of Training Conversation Filtering

• The pipeline excludes non-English prompts and Midjourney-related instructions, retaining users with at least three and no more than 100 conversations; users above 100 are often associated with low-quality program-generated instructions.Source: §C.1 · pp. 16–17
• Conversations longer than 10 turns are excluded, and an LLM filters for users who provide meaningful feedback.Source: §C.1 · p. 17

C.2 Details of Preference Pair Filtering

• Preference pairs require rejected-response length ≥1878, rejected-response reward ≥−1, and reward gap ≤1.Source: §C.2 · p. 17

C.3 Details of Synthesizing Math Conversations

• Because no open-source dataset captures complex-reasoning interactions, the authors sample 10,000 erroneous PRM800K solutions and simulate users identifying the first error without corrections.Source: §C.3 · p. 17

D Additional Results on WildChat UserEval

• 60.3% on initial turns, 52.6% on follow-up turns, and 54.9% overall: User-Guided Rewrites with Persona-Guided Inference leads WildChat UserEval.Source: D Additional Results on WildChat… · p. 18

E Prompts Used in RLHI

• RLHI prompts classify follow-ups into four categories and support persona inference, user-guided rewrites, and persona-guided inference linking histories to turn-level preferences.Source: E Prompts Used in RLHI · p. 18, Figure 7 · p. 18, Figure 8 · p. 19

F Prompts Used in WildChat UserEval

• WildChat UserEval uses separate personalization and instruction judges, plus a combined judge that compares instruction-following, answer quality, and persona alignment.Source: Figure 10 · p. 20, Figure 11 · p. 20, Figure 12 · p. 21