PaperlayerView exact revision on arXiv

Source-linked AI summary

DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning

Yifan Wang, Bolian Li, Junlin Wu, Zhaoxuan Tan, Zheli Liu, Ruqi Zhang, Ananth Grama, Qingkai Zeng

arXiv:2510.02341v2cs.CLcs.AI

TL;DR

Real-world interactions yield abundant implicit dissatisfaction, while explicit satisfaction is scarce and human preference annotations are costly to scale. DRIFT uses real dissatisfaction responses as negatives and samples positives from the evolving policy. It outperforms SPIN and IterDPO across the reported datasets and benchmarks, with gains over base models on WildBench and AlpacaEval2.

  • Problem

    Preference learning often depends on costly human annotations, while deployed systems generate abundant implicit dissatisfaction and scarce explicit satisfaction.

    Source: §1 · pp. 1–2

  • Method

    DRIFT anchors preference pairs on real dissatisfaction negatives and dynamically samples chosen responses from the current policy.

    Source: §1 · p. 3

  • Results

    Across WildFeedback and UltraFeedback, DRIFT outperforms SPIN and IterDPO; it gains up to +7.61% (14B) on WildBench Task Score over base models.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    DRIFT provides a scalable direction for real-world post-training, with an asymmetric pairing strategy that yields a stable, informative learning signal.

    Source: §6 · p. 10

    The method is scoped to prompts with observed dissatisfaction signals, denoted X_DSAT.

    Source: §3 · p. 4

Abstract

from arXiv · show

Real-world large language model deployments (e.g., conversational AI systems, code generation assistants) naturally generate abundant implicit user dissatisfaction (DSAT) signals, as users iterate toward better answers through refinements, corrections, and expressed preferences, while explicit satisfaction (SAT) feedback is scarce. Existing preference learning approaches are poorly aligned with this data profile, as they rely on costly human annotations or assume plentiful positive responses. In this paper, we introduce \textbf{DRIFT} (\textbf{D}issatisfaction-\textbf{R}efined \textbf{I}terative pre\textbf{F}erence \textbf{T}raining), which anchors training on real-world DSAT signals and samples positives dynamically from the evolving policy. Empirically, DRIFT models trained on real-world \textit{WildFeedback} datasets and synthetic \textit{UltraFeedback} datasets achieve up to +6.23\% (7B) / +7.61\% (14B) on WildBench Task Score and up to +8.95\% (7B) / +12.29\% (14B) on AlpacaEval2 win rate over base models, outperforming strong baseline methods such as iterative DPO and SPIN. At larger scales, the improvements are particularly pronounced: 14B models trained with DRIFT surpass GPT-4o-mini on WildBench. Further analysis shows that DRIFT also preserves exploratory capacity, yielding more diverse high-reward solutions rather than collapsing to narrow subsets. Theoretically, we demonstrate that this design preserves preference margins and avoids the gradient degeneration. These results show that DRIFT is an effective and scalable recipe for real-world post-training that leverages the most abundant and informative signal. The code and data are available at https://github.com/cacayaya/DRIFT.git.

1 INTRODUCTION

Real-world interactions offer abundant, information-rich implicit feedback, while explicit feedback is sparse and costly human preference annotations are difficult to scale. DRIFT uses real dissatisfaction as negative supervision and samples positives from the evolving policy, improving benchmark performance while preserving exploration and preference margins.

1 INTRODUCTION

  • Explicit feedback is provided by only 1–3% of users, while conversations contain refinements and corrections that can reveal richer preference information.WildFeedback labels dissatisfaction more often than satisfaction: 12% versus 5%.Source: §1 · p. 2, Figure 1 · p. 2
  • DRIFT anchors each training pair with a real dissatisfaction response and samples the chosen response from the current policy, enabling dynamic, policy-aligned adaptation.This contrasts with approaches that fix supervised responses as positives.Source: §1 · p. 3
  • +6.23% (7B) / +7.61% (14B) WildBench Task Score and +8.95% (7B) / +12.29% (14B) AlpacaEval2 win rate over base models; DRIFT surpasses SPIN and IterDPO.These are the reported maximum gains.Source: §1 · p. 3
  • DRIFT preserves a larger exploration space and produces more diverse responses with broader coverage of the global high-reward region.Source: §1 · p. 3
  • The theoretical analysis finds that DRIFT maintains a non-vanishing expected preference margin and prevents gradient collapse.Source: §1 · p. 3

2 RELATED WORK

Related work mines feedback from real-world interactions and develops iterative self-improvement methods. DRIFT instead pairs genuine dissatisfaction negatives with fresh positives from the evolving policy to preserve a non-vanishing preference margin.

2.1 LEARNING FROM REAL-WORLD USER FEEDBACK

  • Real-world feedback approaches draw on edits, demonstrations, and conversational signals; DRIFT requires no positive responses from stronger models, reward model, or human-provided golden truth.Source: §2.1 · p. 3

2.2 SELF-IMPROVEMENT AND ITERATIVE DIRECT PREFERENCE OPTIMIZATION

  • Iterative self-improvement methods can produce increasingly similar chosen and rejected responses, weakening preference signals; DRIFT anchors on genuine dissatisfaction negatives and samples fresh policy positives.The paper links this design to maintaining a non-vanishing preference margin and preventing gradient collapse.Source: §2.2 · pp. 3–4

3 DRIFT: DISSATISFACTION-REFINED ITERATIVE PRE-FERENCE TRAINING

DRIFT treats abundant, informative user dissatisfaction as negative supervision and generates positive responses dynamically from the evolving policy. Its iterative procedure filters real dissatisfaction cases, samples fresh positives for the same prompts, and updates the policy with DPO.

3 DRIFT: DISSATISFACTION-REFINED ITERATIVE PRE-FERENCE TRAINING

  • Genuine dissatisfaction provides negative supervision, while fresh positives sampled from the current policy help maintain the chosen–rejected margin and mitigate gradient collapse.Source: §3 · p. 4
  • DRIFT applies to prompts with observed dissatisfaction signals, denoted X_DSAT, within a prompt space X and response space Y.Source: §3 · p. 4
  • Each iteration filters real dissatisfaction cases, samples a fresh positive from the current policy for the same prompt, and updates the policy by minimizing DPO loss.Source: §3 · p. 4

4 EXPERIMENT

The experiments compare DRIFT with self-improvement baselines on real-world WildFeedback and synthetic UltraFeedback, then assess iterative stability and high-reward coverage.

4.1 SETUP

  • WildFeedback provides SAT/DSAT labels, UltraFeedback supplies GPT-4-scored completions, and training warm-starts on 491 DSAT→SAT pairs before iterative preference training.Evaluation uses WildBench and AlpacaEval2.Source: §4.1 · p. 5

4.2.1 RESULTS ON WILDFEEDBACK

  • WildFeedback training raises WildBench Task Score by 6.23% (+3.03) for 7B and 5.97% (+3.29) for 14B, and AlpacaEval2 win rate by 8.95% for 7B and 12.11% for 14B over base models.DRIFT also outperforms SPIN and IterDPO across metrics in both controlled and full data settings.Source: Table 3 · p. 6, §4.2.1 · p. 6
  • SPIN degrades across iterations, while DRIFT's controlled setting with around 4k samples matches or exceeds IterDPO's full-data performance.The authors report stronger gains at 14B and suggest DRIFT's real-world DSAT examples provide a superior training signal.Source: §4.2.1 · p. 6
  • Across five iterations on Qwen2.5-7B, DRIFT improves through iteration 4 (52.47) and plateaus at 51.22 at iteration 5, while SPIN and IterDPO peak at iteration 1 and decline.The authors describe the limited performance collapse at iteration 5 as evidence of long-horizon robustness.Source: Figure 2 · p. 7, §4.2.1 · p. 7

4.2.2 RESULTS ON ULTRAFEEDBACK

  • On UltraFeedback, DRIFT beats the base model and best SPIN/IterDPO results on WildBench Task Score and AlpacaEval2 win rate at both model sizes.Compared with the best SPIN/IterDPO results, DRIFT gains +2.14 (7B) and +4.49 (14B) on Task Score, and +6.51% (7B) and +6.10% (14B) on win rate.Source: Table 4 · p. 7, §4.2.2 · p. 7

4.3 EXPLORATORY CAPACITY ANALYSIS: DRIFT EXPLORES MORE DIVERSE HIGH REWARD SOLUTIONS

  • DRIFT achieves the largest share of the global high-reward region at both 7B and 14B, with a larger margin at 14B across 50 prompts.The authors interpret this coverage as greater diversity among high-reward solutions and better scalability.Source: Figure 3 · p. 8, §4.3 · p. 8
  • The coverage analysis embeds 128 responses per method and prompt, projects them with UMAP, and measures method-specific reward-weighted density inside the global high-reward region.The study also illustrates DRIFT spanning broader high-reward semantic regions and uniquely using markdown formatting in one example.Source: §4.3 · p. 8

5 THEORETICAL ANALYSIS

DRIFT’s fresh positives, anchored by genuine DSAT negatives, are presented as maintaining a usable training signal. The analysis gives conditions for non-vanishing expected gradient and utility improvement, and contrasts these with possible signal degeneration for SPIN.

THEORETICAL ANALYSIS

  • DRIFT’s theoretical analysis attributes its usable gradient signal to fresh positives anchored by genuine DSAT negatives, unlike updates fitted to a fixed SAT set.Source: §5 · p. 8

Non-vanishing expected training signal

  • A positive-mass set of pairs with sufficient implicit preference margin and a nonzero conditional gradient gap keeps the expected training signal away from zero.The lower bound applies when the event σ(−s) ≥ τ has probability at least p0 > 0.Source: §5 · p. 9

Expected improvement of actual utility

  • Under the stated improvement-event and smoothness conditions, a sufficiently small DPO step increases J by a linear-in-η margin, up to O(η^2).The stated margin is βτpimpλ.Source: §5 · p. 10

Why DRIFT outperforms SPIN?

  • At SPIN fixed points on a finite SAT catalogue, the pairwise DPO signal is controlled by log-density-ratio variance and can degenerate when that variance is small.The analysis contrasts this with DRIFT’s non-vanishing signal.Source: §5 · p. 10

6 DISCUSSION

DRIFT outperformed SPIN and IterDPO across WildBench categories on Gemma-3-12B-it, and safety evaluations found no increase in measured harms relative to baseline models. The paper concludes that its asymmetric pairing generalizes across model families while preserving baseline safety characteristics.

Generalization

  • On Gemma-3-12B-it, DRIFT outperformed SPIN and IterDPO across all WildBench categories, with similar stability and improvement patterns.The paper describes this as generalization beyond a single model family.Source: §6 · p. 10

Safety implications of training with DSAT signals

  • Across iterations, DRIFT did not increase jailbreak success rates, toxicity, or group-specific harms relative to baseline models.The evaluations used AdvBench and ToxiGen.Source: §6 · p. 10

DISCUSSION

  • The paper concludes that off-policy DSAT negatives paired with on-policy positives provide a stable, informative learning signal while preserving baseline safety characteristics.It presents preference optimization grounded in real dissatisfaction as a reliable and scalable direction for real-world post-training.Source: §6 · p. 10

7 CONCLUSION

DRIFT pairs authentic dissatisfaction signals with positives sampled from the evolving policy, and the paper reports benchmark gains, broader high-reward exploration, and a non-vanishing gradient lower bound. The section also describes the work’s ethics, reproducibility, disclosure, and theoretical assumptions.

7 CONCLUSION

  • DRIFT pairs authentic DSAT negatives with policy-sampled positives and outperforms SPIN and IterDPO on WildBench and AlpacaEval2, with stronger margins at larger base models.Its advantage also holds on the synthetic UltraFeedback dataset.Source: §7 · p. 11
  • DRIFT explores more diverse high-reward solutions rather than overfitting to a narrow region.The paper also reports a uniform, non-vanishing gradient lower bound, avoiding collapse from concentrating probability on a finite fixed chosen set as in SPIN.Source: §7 · p. 11
  • The work reports using anonymized WildFeedback conversations without collecting or using personally identifiable information, and states that experiments comply with dataset licenses and terms.It reports no foreseeable negative social or ethical impacts.Source: §7 · p. 12
  • The authors describe the main paper’s training setup, datasets, and evaluation metrics, with dataset construction, filtering, hyperparameters, iteration procedures, and training dynamics documented across the paper and appendix.They state that code, data-processing pipelines, training scripts, and analysis will be open-sourced upon acceptance.Source: §7 · p. 12

A DISCLOSURE OF LLM USE IN PAPER PREPARATION

  • The authors report using LLMs to assist with writing and polishing, with suggested content proofread and manually adjusted before inclusion.They take responsibility for the manuscript’s accuracy and factuality.Source: A DISCLOSURE OF LLM USE IN PAPER… · p. 15

B.1 ASSUMPTIONS

  • The assumptions excerpt specifies local smoothness of J and a finite second moment for the score difference.The local smoothness condition bounds L_J; the second-moment condition bounds E||d_θ||² by C_d.Source: §B.1 · p. 15

B.2 PROOF OF THEOREM 1 (EXPECTED IMPROVEMENT OF J))

The theorem proof uses stated assumptions and an improvement event to establish an expected-improvement bound for J. The accompanying SPIN discussion explains how convergence to a finite satisfaction catalogue can lead to quantitative degeneration of the DPO signal.

B.2 PROOF OF THEOREM 1 (EXPECTED IMPROVEMENT OF J))

  • Under assumptions (A1)–(A3) and the local advantage correlation condition, the theorem gives an expected-improvement result for J.The proof conditions on the improvement event, where σ(−s) is bounded below by τ.Source: §B.2 · p. 15

B.3 SPIN REAL-WORLD PERFORMANCE DISCUSSION

  • At a SPIN fixed point, the positive and negative responses are conditionally i.i.d. from the finite SAT catalogue, so the DPO signal can quantitatively degenerate.The discussion attributes the signal to variance in the log-density ratio on that finite set.Source: §B.3 · p. 15
  • The proposition formalizes quantitative degeneration at a SPIN fixed point under conditional i.i.d. sampling and a finite second moment assumption.The supplied proposition statement introduces h(y) as the log-density ratio between the policy and reference distributions.Source: §B.3 · p. 16

C.2 UNGUIDED ABLATION STUDY

The supplied appendix materials describe an unguided ablation, a Gemma-family WildBench evaluation, safety metrics, and warm-start data and training setup; they do not provide the corresponding outcome values.

C.2 UNGUIDED ABLATION STUDY

  • The unguided ablation removes the reference DSAT response and explicit improvement instruction, matching SPIN’s setting.Table 6 is identified as the Qwen2.5-7B full-WildFeedback ablation, but the supplied caption gives no result values.Source: Table 6 · p. 18

C.3 GENERALIZATION ACROSS MODEL FAMILIES

  • The Gemma-3-12B-it generalization experiment uses the same training recipe across methods and evaluates them on WildBench.Table 7 indicates bolding the highest score for each metric; its caption does not state the scores.Source: Table 7 · p. 18

C.4 SAFETY AND ETHICAL CONSIDERATIONS

  • The safety evaluation reports AdvBench jailbreak attack success rate and ToxiGen toxicity scores, with lower values indicating better safety.ToxiGen uses a 1–5 scale.Source: Table 8 · p. 19

D.1 WARM START TRAINING DETAILS

  • The warm-start seed set contains 491 WildFeedback DSAT→SAT pairs, where a dissatisfied user turn is followed by a revised response that satisfies the user.The pair encodes the DSAT response as failing expectations and the subsequent SAT response as preferred.Source: §D.1 · p. 20
  • Warm-start training initializes from pretrained instruction-tuned models and uses DPO on seed preference data to establish initial alignment.Experiments used eight H100 GPUs with the same hardware configuration across training phases.Source: §D.1 · p. 20

D.2 ITERATIVE TRAINING DETAILS

Iterative training refines alignment using newly generated preference data and successive model checkpoints. Each iteration uses one epoch, while training hyperparameters remain consistent across iterations.

D.2 ITERATIVE TRAINING DETAILS

  • Each iteration builds on the previous checkpoint and incorporates newly created preference data.The iterative process changes only the training data and base-model checkpoint while maintaining consistent hyperparameters.Source: §D.2 · p. 20
  • Each iteration trains for a single epoch to prevent overfitting on iteratively generated data.Table 10 is identified as containing iterative-training hyperparameters, but no values are supplied here.Source: §D.2 · p. 20, Table 10 · p. 20

D.3 TRAINING DYNAMICS

DRIFT’s two training iterations show stable loss convergence, with chosen rewards increasing and rejected rewards decreasing. Figures 6–8 present SPIN, IterDPO, and DRIFT response examples for a shared prompt.

D.3 TRAINING DYNAMICS

  • Across both DRIFT iterations, loss converges stably while chosen rewards rise and rejected rewards fall.Figure 5 shows the training dynamics for iterations 1 and 2 on Qwen2.5-14B-Instruct.Source: §D.3 · p. 20, Figure 5 · p. 21

E MODEL RESPONSES EXAMPLE

  • Figures 6–8 present response examples from SPIN, IterDPO, and DRIFT for the same carnivorous-plant research-paper prompt.The prompt requests comparisons of three plant types, including their adaptations, evolutionary history, ecological significance, and potential uses in medicine and biotechnology.Source: Figure 6 · p. 22, E MODEL RESPONSES EXAMPLE · p. 22, Figure 7 · p. 23
Loading 2510.02341v2…