Source-linked AI summary
WildReward: Learning Reward Models from In-the-Wild Human Interactions
Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li
TL;DR
Reward models usually depend on large-scale human-annotated preference pairs, motivating the question of whether in-the-wild interactions can provide reward signals directly. The paper extracts reliable feedback from WildChat, trains WildReward by ordinal regression on 186k point-wise instances, and reports competitive or superior benchmark performance with improved calibration, consistency, and downstream DPO results.
Problem
Reward models typically rely on large-scale human-annotated preference pairs, raising whether in-the-wild interactions can provide reward signals directly.
Method
The paper extracts reliable feedback from WildChat and trains WildReward via ordinal regression on point-wise user feedback without preference pairs.
Results
WildReward achieves performance comparable or superior to conventional reward models, with improved calibration and cross-sample consistency, and improves several tasks through online DPO.
Takeaways & Limitations
The results highlight in-the-wild human interactions as a promising resource for training reward models and show benefits from user diversity.
Takeaways & Limitations
The dataset covers only English and Chinese WildChat conversations, and the paper does not sufficiently search reward-model configurations such as hyperparameters or backbone models.
Abstract
from arXiv · showhide
Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.
1 Introduction
The paper asks whether reward models can learn directly from in-the-wild human-LLM interactions, which contain rich but sparse and noisy feedback. It proposes extracting reliable feedback from WildChat and shows that WildReward performs competitively while offering calibration, consistency, and downstream-training benefits.
- In-the-wild conversations contain textual feedback that directly reflects response quality and human preferences.
- Approximately 82% of follow-up queries do not explicitly express feedback, while explicit positive feedback accounts for only 1%.
- A conservative automated pipeline classifies feedback into five satisfaction levels and refines labels to reduce noise.
- WildReward achieves performance comparable or superior to conventional reward models on RewardBench, RM-Bench, PPE, and JudgeBench without dedicated human-annotated preference pairs.
- WildReward benefits from user diversity, is well-calibrated, and provides cross-sample consistency through meaningful global scores.
- Online DPO guided by WildReward improves mathematical reasoning, instruction following, and creative writing.
2 Methodology
The methodology constructs reliable ordinal feedback labels from WildChat interactions and trains WildReward on point-wise feedback rather than preference pairs. It addresses sparse and noisy feedback through classification, refinement, and refusal validation.
- WildChat is used as the primary human-LLM interaction source for constructing the WILDFB dataset and training WildReward.
- In a 10,000-instance analysis, 82% of follow-up queries were neutral, 17% negative, and 1% positive.
- Manual inspection found that 86% of neutral queries were unrelated new requests, while negative and positive categories generally expressed corresponding feedback.
- The pipeline uses five satisfaction levels and treats relevant follow-up engagement as implicit positive feedback.
- Implicit feedback mining reclassifies semantically similar neutral instances within a two-turn window, adding approximately 12,310 samples and expanding positive feedback by 29%.
- Ordinal regression models the ordered feedback categories without assuming uniform intervals and produces probabilistic outputs for confidence filtering.
- During inference, WildReward computes a continuous score representing the expected value of predicted feedback, which supports confidence filtering.
3 Experiment
The experiments evaluate WildReward against conventional reward models, analyze data construction and user diversity, and examine calibration, cross-sample consistency, and DPO training. Results show competitive benchmark performance, benefits from diverse interactions, strong calibration, and gains in online DPO.
- Experimental setup: WildReward-4B and WildReward-8B are evaluated against representative reward models on standard reward-model benchmarks, including RewardBench, RM-Bench, PPE, and JudgeBench.The baselines are typically trained on large-scale preference pairs.
- Reward model benchmarking: WildReward achieves comparable or superior benchmark performance without human-annotated preference pairs, with 4B and 8B models surpassing some much larger 70B reward models.The results support training reward models directly from human feedback in in-the-wild interactions.
- Data strategy: Removing either implicit feedback mining or refusal validation significantly degrades performance, while excluding refusal validation reduces SRF performance by 60%.The refusal-validation strategy contributes only 572 instances but substantially affects results, especially on the safety subset.
- User diversity: For a fixed data size, models trained with more unique users consistently perform better, while performance also generally improves as training data size increases.The user-diversity analysis compares same-sized datasets whose numbers of unique users differ tenfold.
- Calibration: WildReward achieves an ECE of 2.76%, compared with 8.81% for ArmoRM-Llama3-8B-v0.1, indicating substantially better calibration.Calibration is evaluated using a 50% fitting split and a 50% evaluation split.
- Application in DPO training: Online DPO with WildReward yields significant gains over Llama3.1-8B-Instruct and ArmoRM, whereas offline DPO provides nearly no improvements.The strongest gains occur on Alpaca Eval 2.0 and Arena Hard, with additional gains in mathematical reasoning and instruction following.
4 Related Work
Prior reward-model research primarily relies on human-annotated preference pairs, while newer work mines implicit feedback from interactions without training a reward model. WildReward instead trains a reward model directly from interaction data without preference pairs.
- Conventional reward models are typically trained on large-scale preference pairs using the Bradley-Terry objective.
- Recent approaches use negative feedback from human-LLM interactions to construct data for DPO or SFT, but do not train reward models.
- WildReward trains directly on interaction data without using preference pairs and reports improved calibration.
5 Conclusion
The paper uses WildChat interactions to extract valid human feedback and constructs WILDFB, then trains WildReward directly on those data without preference pairs. Experiments report improved calibration and cross-sample consistency, supporting in-the-wild interactions as a promising reward-modeling resource.
- WildChat serves as the interaction source for an automated pipeline that extracts valid human feedback.
- 186k high-quality WILDFB instances support training WildReward via ordinal regression without preference pairs.
- Extensive experiments report improved calibration and cross-sample consistency for WildReward.
Limitations
The work has language, configuration-search, and policy-training scope limitations. WILDFB covers only English and Chinese conversations, configurations were not sufficiently optimized, and RL training was not attempted.
- WILDFB includes only English and Chinese WildChat conversations, limiting application to other languages.
- The authors did not sufficiently search hyperparameters or backbone models, so other configurations may perform better.
- The study uses online DPO rather than RL training because RL is resource-intensive and difficult to stabilize.
Ethical Considerations
The authors describe licensing and intended-use considerations for the datasets and model, while noting that real-world training data may contain biases. They recommend verification before using WildReward.
- The work uses WildChat and Infinity-Instruct under their stated licenses and plans to release WILDFB under Apache License 2.0.
- WildReward may contain biases from real-world data, and users are advised to verify it before use.
A WILDFB Construction Details
WildReward is constructed from WildChat real-world human–LLM interactions using heavy filtering and automated classification of implicit user feedback. The pipeline restricts the data to suitable text conversations and assigns feedback categories for dataset curation.
- WildChat-4.8M5 provides the real-world human–LLM interactions used as the data source.
- Heavy filtering retains English and Chinese conversations while removing multimodal, tool-dependent, trivial, identity, and context-dependent queries.
- Conversations exceeding 20 turns, queries with fewer than five words, and responses shorter than ten words are excluded.
- After filtering, 10,000 instances are sampled and gpt-oss-120b classifies user follow-up queries as Negative, Neutral, or Positive.
B Experimental Details
The experiments use H100 GPUs and standardized reward-model training and evaluation configurations. User-feedback and refusal behaviors are annotated with dedicated classification prompts, while performance is assessed across several reward-model benchmarks and consistency evaluation.
- Experiments use NVIDIA H100 GPUs, with reward-model training conducted using the TRL framework.
- Training uses batch size 512, learning rate 1 × 10−5, maximum sequence length 4,096, and one epoch.
- Reward-model evaluation uses RewardBench, RM-bench, JudgeBench, and official PPE evaluation code.
- The feedback annotation prompt infers satisfaction from the user’s latest message using only strong explicit evidence and labels ambiguous follow-ups Neutral.
- A separate five-class prompt distinguishes satisfaction-related feedback, while refusal validation checks whether the model explicitly declined to answer.
- The refusal-validation prompt is used to identify justified refusals, with gpt-oss-120b reported as effective for this task.