PaperlayerView exact revision on arXiv

Source-linked AI summary

PerSPL: Personalized Semi-supervised Preference Learning

Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi

arXiv:2610.29219v1cs.LG

TL;DR

Per-user preference comparisons are costly and limited, while existing personalized reward models leave shared unlabeled responses unused. PerSPL centers pair scores and uses per-user calibration with iterative pseudo-labeling to turn those responses into personalized training signals. Across four benchmarks, it reports the highest mean accuracy among evaluated methods for both seen and unseen users.

  • Problem

    Per-user preference labels are scarce, and existing personalized reward models do not use shared unpaired responses as additional supervision.

    Source: §1 · p. 1

  • Method

    PerSPL uses pair-mean score centering, per-user calibration, confidence filtering, and iterative self-training to turn shared unlabeled responses into user-specific signals.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    PerSPL has the highest reported mean accuracy among seven representative baselines across four benchmarks for both seen and unseen users.

    Source: §1 · p. 3, §6 · p. 8

  • Takeaways & Limitations

    The results indicate that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.

    Source: §1 · p. 3

    Per-user calibration requires labeled anchors, so the evaluated setting is few-shot rather than zero-shot, and generator-shift evidence applies only to tested generators.

    Source: §3 · p. 3, §6 · p. 8

Abstract

from arXiv · show

Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of responses can be generated without user identities or preference labels, yet existing personalized reward models leave this data unused. Directly learning from these responses is difficult because pairwise objectives constrain relative scores within a prompt, whereas pseudo-labeling individual responses requires scores that are comparable across prompts for the same user. We propose \textbf{PerSPL} (Personalized Semi-supervised Preference Learning), a backbone-agnostic framework that uses pair-mean score centering to stabilize this cross-prompt score coordinate and iteratively turns unlabeled responses into user-specific training signals. Against seven representative baselines across four benchmarks---PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena---PerSPL achieves the highest reported mean accuracy among the evaluated methods for both seen and unseen users. The two latter benchmarks provide real-user preference comparisons in summarization and open-domain chatbot interaction, respectively. These results indicate that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.

1 Introduction

Per-user comparisons are scarce, while shared unpaired responses remain unused by existing personalized reward models. PerSPL centers pair scores to support cross-prompt comparability, then uses user-specific pseudo-labeling; it reports the highest mean accuracy among evaluated methods for seen and unseen users.

1 Introduction

  • Personalized models need scarce per-user comparisons, but existing approaches leave a much larger shared pool of unpaired responses unused.The responses can be generated or collected without additional user feedback.Source: §1 · p. 1
  • Pairwise training leaves cross-prompt score locations undetermined, although individual-response pseudo-labeling requires comparable scores across prompts for each user.A common shift to both scores for one prompt preserves the pairwise prediction but can change whether a response crosses a threshold.Source: §1 · p. 2
  • PerSPL centers the two scores in each labeled pair around zero, then fits per-user calibrators and iteratively trains on confident pseudo-labels from unpaired responses.Ambiguous responses are discarded.Source: §1 · p. 2, Figure 1 · p. 2
  • PerSPL supplements limited per-user comparisons with user-specific supervision from shared unpaired responses for seen users and new users with small support sets.The same response can receive different pseudo-labels for different users.Source: §1 · p. 2

Evaluation across complementary personalization settings

  • PerSPL has the highest reported mean seen- and unseen-user accuracy among seven representative baselines across four benchmarks and every primary setting.The benchmarks include real-user preferences from Reddit TLDR and Chatbot Arena.Source: §1 · p. 3

2 Related Work

Prior personalized reward models address sparse feedback through architectural sharing, while semi-supervised preference methods use unlabeled data for homogeneous, population-wide rewards. PerSPL differs by using unpaired responses to create personalized supervision.

Personalized reward modeling

  • Personalized reward-model approaches use input conditioning, user-specific parameter modules, or meta-learned initializations to address per-user feedback scarcity.These approaches share the assumption that the available signal is each user’s labeled preference pairs; PerSPL instead adds unpaired data at the data level.Source: §2 · p. 3

Semi-supervised preference learning under limited feedback

  • Semi-supervised preference methods use unlabeled data for one population-wide reward model or policy, without a mechanism for cross-user preference variation.PerSPL targets personalized reward modeling with unpaired data, which these homogeneous-reward methods do not address.Source: §2 · p. 3

3 Problem Setup

The setup models user-conditioned reward scores and compares responses for the same prompt using pairwise preferences. It assumes a shared unpaired response pool and few-shot feedback, while highlighting that pairwise training alone does not identify the cross-prompt score coordinate needed for single-response judgments.

3 Problem Setup

  • For a user, prompt, and response, the reward model assigns a scalar score, and the Bradley–Terry objective uses the score difference between preferred and rejected responses.The preference probability is σ(r+ − r−), with loss −log σ(r+ − r−).Source: §3 · p. 3
  • The setting gives each user labeled comparisons and shares a larger pool of unpaired prompt–response examples without preference labels or user identities.Previously unseen users provide a small support set, so the study is few-shot rather than zero-shot personalization.Source: §3 · p. 3
  • Adding any prompt–user offset to both response scores preserves the pairwise loss but changes cross-prompt comparisons and threshold-based pseudo-labeling.PairAcc measures same-prompt ranking, whereas GlobalAcc diagnoses cross-prompt ordering for a user.Source: §3 · p. 4
  • Centering each pair around zero preserves PairAcc and yields GlobalAcc = 1 in the example where the uncentered comparison fails and GlobalAcc is 3/4.The contrast motivates pair-mean score centering.Source: §3 · p. 4

4 PerSPL: Personalized Semi-supervised Preference Learning

PerSPL turns shared unpaired responses into user-specific training data through score centering, per-user calibration and pseudo-labeling, and iterative self-training. It retains confident responses as pseudo-labels while keeping labeled comparisons in the training objective.

4 PerSPL: Personalized Semi-supervised Preference Learning

  • PerSPL combines cross-prompt score centering, per-user calibration and pseudo-labeling, and iterative self-training to use the shared unpaired pool.Figure 1 shows how labeled and unpaired data interact in the process.Source: §4 · p. 4

4.1 Pair-mean score centering

  • Pair-mean score centering supplies a training signal for labeled-pair means, which the pairwise loss otherwise leaves unidentified.The penalty acts on pair centers rather than output-coordinate margins, and the same reward model produces both scores.Source: §4.1 · p. 4

4.2 Per-user calibration and pseudo-labeling

  • Per-user isotonic calibration maps reward scores to monotone empirical probabilities of appearing on the chosen side of that user’s comparisons.For unpaired responses, confident probabilities above τ or below 1 − τ receive labels; intermediate cases are rejected, and only retained responses contribute to the pointwise loss.Source: §4.2 · p. 4

4.3 Iterative self-training

  • Self-training first fits a labeled-only model, then combines labeled and pseudo-labeled losses in later iterations while updating the pseudo-label sets.The calibrators are refit using labeled rewards, and the reward model is randomly reinitialized before the next iteration.Source: §4.3 · p. 5

5 Experiments

PerSPL is evaluated against seven representative baselines across four benchmarks, with shared data splits and budgets for comparison. It leads the reported seen- and unseen-user means, while ablations and controlled studies probe centering, label budgets, noise, and generator shift.

5.1 Datasets

  • The four benchmarks span synthetic responses and simulated users, human-rated responses with simulated users, and real-user preferences on summaries and open-domain chatbot responses.Methods share assigned splits, labeled budgets, and seed identities; PerSPL additionally uses unlabeled responses without user identities or preference labels.Source: §5.1 · p. 5, §5.2 · p. 5

5.3 Main Results

  • 95.6 ± 0.1 overall on PersonalLLM versus LoRe’s 94.3 ± 0.7; PerSPL has the highest reported mean in every seen- and unseen-user column across the four benchmarks.Reddit TLDR and Chatbot Arena extend the pattern to real-user preferences in summarization and open-domain dialogue.Source: Table 1 · p. 6, §5.3 · p. 6

5.4 Ablations and Analysis

  • With regularization, adding unpaired training changes seen accuracy by +0.5 to +1.8 points; without it, the change is −6.5 to −2.9 points across four settings.The reported interaction ranges from +3.4 to +8.3 points; moderate centering also reduces pair-mean variation and cross-prompt loss while preserving held-out pairwise fit, and tested accuracy is stable across τ and α sweeps.Source: §5.4 · pp. 6–7
  • At a 0.4 label-flip ratio PerSPL’s accuracy gap over MRM reaches 5.07 points, rising to 6.69 points at 0.5; both methods become less accurate as noise increases.PerSPL remains ahead throughout the tested noise range.Source: §5.4 · p. 7, Figure 3 · p. 8
  • Across 50, 100, and 200 labeled pairs per seen user, PerSPL stays 0.81–1.08 points above MRM; for unseen users, its accuracy rises from 65.51% to 66.03% as support grows from 10 to 50 pairs.The unseen-user gain over MRM rises from 0.81 to 1.33 points with the seen-user budget fixed at 50.Source: §5.4 · p. 7, Figure 3 · p. 8

5.5 Controlled Response-distribution Shift

  • Across tested generator mixtures, PerSPL’s overall gain over regularized labeled-only training ranges from 0.66 to 3.21 points, with every paired interval above zero.The evidence is specific to the controlled generator shift studied, including mixtures composed entirely of tested held-out generators; the experiment uses corrupted labels and its absolute accuracies are not directly comparable with clean-label PersonalLLM results.Source: Table 3 · p. 8, §5.5 · p. 8

6 Conclusion

PerSPL converts a shared pool of unlabeled responses into user-specific supervision and reports the highest evaluated mean accuracy for seen and unseen users. Its current personalization setting remains few-shot, and the generator-shift evidence is limited to tested generators.

6 Conclusion

  • PerSPL combines pair-mean score centering, per-user isotonic calibration, confidence filtering, and iterative self-training to turn shared unlabeled responses into user-specific supervision.Across seven baselines and four benchmarks, it has the highest reported mean accuracy among evaluated methods for seen and unseen users.Source: §6 · p. 8
  • Per-user calibration requires a labeled anchor, so the studied setting is few-shot rather than zero-shot personalization.The controlled generator-shift benefit is supported for the tested generators.Source: §6 · p. 8

A.1 Dataset Construction Details

The appendix details benchmark construction, implementation, and training, then characterizes why pair-mean centering matters for decisions across prompts. Its bounds connect pair-center dispersion to cross-prompt and single-response risks.

A.1 Dataset Construction Details

  • The benchmarks combine simulated-user settings with real-user preference data: Reddit TLDR covers summary quality, while Chatbot Arena covers open-domain comparisons.PersonalLLM uses model-generated responses and simulated users; HelpSteer2 uses human ratings with simulated users weighted over attributes.Source: §A.1 · pp. 10–11

A.2 Implementation Details

  • PerSPL keeps 4,096-dimensional cached representations frozen and learns a basis with user weights; unseen-user adaptation freezes the basis and fits only that user’s logits.Main configurations use pair-mean regularization and per-user isotonic calibration, with matched seeds and last-epoch checkpoints.Source: §A.2 · p. 12, Table 5 · p. 12

A.3 PerSPL Training Procedure

  • Training alternates labeled reward updates with per-user isotonic calibration, confidence-filtered pseudo-label refreshes, and subsequent unlabeled-loss updates.The procedure initializes pseudo-label sets empty and reinitializes model parameters at each outer iteration.Source: §A.3 · p. 13

B.1 Pair-mean geometry and cross-prompt risk

  • Pair-mean centering controls a score component unconstrained by within-prompt comparisons: pair-center dispersion governs the gap to cross-prompt ranking risk and also links to single-response classification around a shared threshold.The pair-mean penalty selects zero as the common score anchor without directly penalizing the within-pair contrast.Source: §B.1 · p. 14

B.2 An illustrative Gaussian model

In an idealized Gaussian model, pair-mean centering can improve single-response accuracy by changing the scoring direction, with the gain requiring nonalignment. The appendix also traces centering’s initial effect through a fixed-budget Adam training path.

B.2 An illustrative Gaussian model

  • The Gaussian analysis models pair centers and contrasts as independent with positive-definite covariances, and uses their covariance geometry to study the optimized scoring direction.The penalty is a quadratic form on pair centers, so it targets sensitivity to directions shared within a pair.Source: §B.2 · pp. 14–15
  • Under nonalignment, small positive λ improves single-response accuracy at first order while paired BT excess risk grows only quadratically.The local tradeoff follows from the direction change along the regularized population path.Source: §B.2 · pp. 15–16, Figure 4 · p. 16
  • The direction-preserving control has zero accuracy gain, showing that regularization alone is insufficient; the gain requires the stated nonalignment condition.The plotted population path covers only the small-λ neighborhood addressed by the theorem.Source: §B.2 · p. 16, Figure 4 · p. 16

B.3 Turning on centering under a fixed training budget

  • If the Adam-path derivative of the evaluation target is positive, then the target decreases for all sufficiently small positive λ.This is a conditional result under the stated smoothness assumptions and finite training horizon.Source: §B.3 · p. 17
  • For a fixed training budget, the Adam analysis propagates midpoint-penalty gradients through curvature and moment updates to determine the initial change in an evaluation target.The sign depends on the data, representation, and training path.Source: §B.3 · pp. 16–17

B.4 Real-data checks

Real-data checks find that centering reduces held-out cross-prompt loss while largely preserving held-out pairwise fit at the paper’s coefficient. The appendix also reports full PersonalLLM and HelpSteer2 results across diversity settings.

B.4 Real-data checks

  • At the tested coefficient, held-out PairAcc changes by −0.02 ± 0.14 points on HelpSteer2 and 0.00 ± 0.36 points on Reddit.On HelpSteer2, precision among the highest-confidence 10% of candidate pseudo-labels rises by 2.16 ± 0.29 points.Source: §B.4 · p. 18
  • 0.159 on HelpSteer2 and 0.093 on Reddit are the mean held-out cross-prompt loss reductions at λr = 0.1; all 70 nonzero-λr seed comparisons are positive.Figure 5(a) plots reductions relative to the matched λr = 0 model.Source: Figure 5 · p. 18, §B.4 · p. 19
  • At λr = 0.01, held-out BT loss gains are 0.0064 on HelpSteer2 and 0.0021 on Reddit, while values at λr = 1 are −0.0664 and −0.0778.Figure 5(b) reports matched-model BT loss reductions; positive values indicate lower loss.Source: Figure 5 · p. 18, §B.4 · p. 19

C.1 Full Per-dataset Results

  • The appendix reports full PersonalLLM and HelpSteer2 pairwise accuracy results across their listed diversity levels, with means and standard deviations over five seeds.The supplied captions specify αdiv ∈{0.001, 0.01, 0.1} for PersonalLLM and αdiv ∈{0.01, 0.1} for HelpSteer2.Source: §C.1 · p. 20, Table 7 · p. 20, Table 8 · p. 20

C.2 Full Chatbot Arena Results

The Chatbot Arena evaluation reports PerSPL’s highest mean accuracy for seen and unseen users. Additional analyses examine paired uncertainty, the interaction between centering and unlabeled training, generator shift, and hyperparameter sensitivity.

C.2 Full Chatbot Arena Results

  • The evaluation retains 38 active users, assigning 19 per seed to seen and unseen groups with 25 and 12 labeled pairs, respectively.Comparisons are split into an 80% training pool and a held-out 20% test set; all methods share the assigned splits, budgets, and seeds.Source: §C.2 · p. 20
  • PerSPL has the highest reported mean accuracy for seen and unseen users, with paired gains over LoRe of 1.72, 2.66, and 2.19 points for seen, unseen, and overall accuracy.All three paired-bootstrap 95% confidence intervals exclude zero.Source: §C.2 · p. 21, Table 10 · p. 21

C.3 Additional Analyses

  • Paired-bootstrap intervals summarize variation across five matched seeds using 20,000 resamples of paired accuracy differences.The intervals are descriptive summaries over the five executed matched seeds.Source: §C.3 · p. 21, Table 11 · p. 21
  • Adding unpaired training with centering raises seen accuracy by 0.5–1.8 points across four settings, while adding it without centering lowers seen accuracy by 2.9–6.5 points.The four centered seen-user intervals exclude zero; regularizer-only changes range from −0.1 to +0.1 points.Source: §C.3 · p. 22
  • The controlled generator-shift study holds labeled data fixed and tests mixtures of six in-domain and two held-out generators across ten matched seeds.Table 12 reports seen- and unseen-user breakdowns; overall results and paired gains appear in the main table.Source: §C.3 · p. 22, Table 12 · p. 22

C.4 Hyperparameter Sensitivity Analysis

  • Across tested settings, threshold and labeled-loss weight change overall accuracy by at most 0.30 and 0.23 points, while the main learning rate is 2.75 points above η = 0.3.The sweep varies one parameter at a time on HelpSteer2, with the other settings and training schedule fixed.Source: §C.4 · p. 22, Figure 6 · p. 22

D Scope and Limitations

PerSPL’s scope depends on per-user labeled feedback and is bounded by the evaluated data, model scale, and tasks. Several deployment-relevant settings remain untested.

D Scope and Limitations

  • Per-user isotonic calibration requires labeled comparisons for every user, so zero-label cold start is outside the study’s scope.Seen-user histories train shared structure, while unseen-user support sets adapt it to a target user.Source: D Scope and Limitations · pp. 22–23
  • Mining user-generated content could reduce explicit annotation cost but raises privacy and consent issues and requires evaluation under a matched total information budget..Source: D Scope and Limitations · p. 23
  • Confidence filtering can overrepresent easy or well-calibrated response regions, and subgroup-specific pseudo-label precision and coverage have not been measured.The authors recommend auditing both measures for deployment.Source: D Scope and Limitations · p. 23
  • The generator-shift experiment does not establish robustness to arbitrary semantic, demographic, adversarial, or temporal shifts.Recalibration or abstention may be required when the unlabeled pool is far from labeled support.Source: D Scope and Limitations · p. 23
  • Experiments use frozen representations and a learned low-rank reward basis, without a capacity-matched full-fine-tuning control; user populations contain at most 200 users.Behavior at production scale remains uncharacterized.Source: D Scope and Limitations · p. 23
  • The benchmarks are English-language and cover instruction following, open-domain chatbot comparisons, and summarization, not cross-lingual transfer, multi-turn interaction, code, or long-horizon consequences..Source: D Scope and Limitations · p. 23

E Data and Software

The study uses four public preference-learning benchmarks and released baseline implementations. Dataset access and licensing conditions vary, including restrictions on Chatbot Arena data and LoRe.

E Data and Software

  • The four benchmarks are PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena Conversations, used under their respective licenses and access terms.Chatbot Arena is gated, and its access terms prohibit attempts to identify users.Source: E Data and Software · p. 23
  • The Reddit TLDR human-feedback release is Modified MIT, while its underlying filtered corpus is separately licensed CC-BY-4.0..Source: E Data and Software · p. 23
  • The study uses official released code for GPO, VPL, PAL, LoRe, and MRM; VPL’s license is unspecified in the passage.The listed licenses include MIT, Apache-2.0, and CC-BY-NC-4.0.Source: E Data and Software · p. 23
  • Reward-model embeddings come from Skywork-Reward-V2-Llama-3.1-8B, derived from Llama-3.1-8B-Instruct and inheriting its license terms..Source: E Data and Software · p. 23

Related papers

Loading 2610.29219v1…