PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models

Qiyao Ma, Junshan Zhang, Zhe Zhao

arXiv:2609.35695v1cs.LGcs.CL

TL;DR

Personalized generation must accommodate pluralistic user preferences, which monolithic alignment paradigms do not capture. The paper uses a lightweight ranker on the generator’s hidden states, trained with candidate pools and user-authored references. Across nine datasets, it outperforms generalist reward models on every dataset while using far fewer parameters.

  • Problem

    Standard alignment optimizes for a monolithic set of preferences, although users’ idiosyncratic choices matter for responses tailored to them.

    Source: §1 · p. 1

  • Method

    A small MLP scores candidates from the generator’s hidden states, trained on its candidate pools labeled against user-authored references.

    Source: §3.3 · p. 5

  • Results

    Across nine datasets, the ranker outperforms generalist 8B reward models on every dataset and keeps improving as the pool grows where they stall.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The oracle analysis reframes personalized alignment as selecting well-matched candidates already produced by the generator, rather than addressing a generator capability deficit.

    Source: §1 · pp. 1, 3

Abstract

from arXiv · show

Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empirical studies are first used to reveal the existence of a massive, untapped performance headroom for personalized generation through test-time alignment. We demonstrate that personalized generation is uniquely suited for test-time scaling methods like Best-of-N (BoN) because it can be viewed primarily as a candidate matching problem rather than a generator capability bottleneck. While reward models could in principle exploit this headroom, they are poorly calibrated for personalization, and their billion-parameter scale makes scoring large candidate pools prohibitively expensive. To overcome this limitation, we propose a parameter-efficient framework utilizing million-parameter scale multi-layer perceptron (MLP) ranking models. Our personalized ranking model directly reuses the internal embeddings of the base generator with minimal overhead. By scaling train-time data to provide fine-grained personalized preferences, this million-parameter ranking model accurately scores large candidate pools and can seamlessly guide generation to reduce the cost of materializing N candidates. Extensive experiments on nine datasets spanning three personalized generation settings show that our personalized ranking model effectively exploits the discovered headroom, outperforming billion-parameter generalist reward models on every dataset, with under 0.4% of their parameters and four orders of magnitude lower scoring latency.

1 Introduction

The introduction argues that personalized generation is limited less by what the generator can produce than by selecting candidates that match an individual user. It presents a lightweight ranker using the generator’s hidden states to exploit this headroom, with candidate selection and guided decoding as test-time approaches.

1 Introduction

  • The paper frames personalization as candidate matching: well-matched responses already appear in the generator’s pool, but generalist reward models often fail to select them.The oracle experiment contrasts continued improvement from metric-based selection with the reward model’s plateau.Source: §1 · pp. 1–3
  • Oracle Best-of-N selection keeps improving as the candidate pool grows, while generalist reward models capture only a small fraction of the available personalization headroom.The oracle selects using the evaluation metric itself; the comparison is averaged over the three datasets of each task.Source: §1 · pp. 1–3, Figure 1 · p. 2
  • Generalist reward models are both costly to apply to large pools and poorly matched to candidates whose main differences are in whose voice they reflect.The paper reports that scoring a pool with an 8B reward model costs about as much as generating it.Source: §1 · p. 2
  • The proposed ranker maps the generator’s final-layer hidden states to a score, using candidate pools labeled by ROUGE-L against user-authored references.These pools supply realistic hard negatives; the MLP has 1M to 30M parameters and requires no per-user parameters.Source: §1 · p. 2
  • Guided decoding steers uncertain next-token choices toward the ranker’s prediction, recovering part of the selection gain with one generation rather than replacing pool-based selection.Most of the headroom still requires scoring a candidate pool.Source: §1 · p. 2
  • Across nine datasets, the ranker outperforms generalist reward models on every dataset, recovers most finetuned-model gains at over two hundred times smaller size, and matches personalized models without per-user preference labels.The evaluation spans three personalized-generation settings.Source: §1 · pp. 2–3

2 Related Work

Related work addresses general alignment, personalized generation, and test-time alignment. The paper distinguishes its approach by using the generator’s own hidden states and existing user history for personalized selection.

Generalized Alignment and Reward Modeling

  • RLHF [20] [1] and DPO [26] align models to monolithic preferences; Bradley-Terry rewards average pluralistic tastes and are not calibrated across candidate pools.The paper notes that Bradley-Terry scores are identified only up to a prompt-dependent shift.Source: §2 · p. 3

Personalized Generation

  • Prior personalization methods use persona prompts or per-user finetuning, or adapt reward models using user annotations; deployed reward models encode each candidate separately.The paper’s ranker instead derives user signal from interaction history in the generation context and reuses generator hidden states.Source: §2 · p. 3

Test-Time Alignment

  • Existing test-time selection and steering methods inherit per-candidate encoding costs and target monolithic rewards, while interactive approaches elicit preferences through pairwise queries.The paper’s ranker needs no queries or labels beyond existing user history and can steer decoding using the generator’s hidden states.Source: §2 · p. 3

3 Methodology

The methodology casts personalized test-time alignment as selecting among generated responses and introduces a decoupled ranker built on generator representations. It develops pointwise preference training, uncertainty-gated decoding guidance, and a complexity analysis.

3.1 Problem Formulation: Personalized Test-time Alignment

  • Personalized test-time alignment samples N candidates and learns a user-specific scorer to select maximum-utility responses, avoiding per-user policy fine-tuning but retaining costly full reward-model passes.The candidate pool is generated by the base model, while standard reward-model scoring requires N full forward passes.Source: §3.1 · p. 4

3.2 Personalized Ranking Model

  • The ranker reuses generator final-layer states for query, profile, and response, mapping concatenated features to scalar utility with an MLP in fixed feature space.The profile representation uses the generator’s summary of user-profile text, with no separate user encoder or per-user learned parameters.Source: §3.2 · pp. 4–5

3.3 Pointwise Preference Optimization

  • Pointwise regression fits absolute candidate utilities, standardized within each pool, on generator-produced hard negatives scored against user-authored references rather than random negatives.The paper uses the downstream evaluation metric as the target utility; its experiments use ROUGE-L.Source: §3.3 · p. 5

3.4 Ranking Guided Generation

  • At confident steps decoding is unchanged; above the entropy threshold, ranker gradients are projected through the LLM head into logits, with steering strength modulated by entropy.The method blends the steering signal with the generator logits and emits the resulting argmax token.Source: §3.4 · p. 6

3.5 Time Complexity Analysis

  • Reward-model selection costs O(N · L · d_r^2), whereas recycled-embedding ranking costs O(N · d^2), independent of response length L.The latter selection step applies a shallow MLP to each candidate embedding after generation.Source: §3.5 · p. 6

4 Experiments

Across nine datasets, personalized ranking improves with larger candidate pools, while generalist reward models plateau or decline; its gains transfer to BLEU and remain useful across varying user-history amounts. A small ranker approaches a per-dataset-finetuned 8B reward model’s gains at much lower scoring cost.

4.1 Experimental Settings

  • The nine-dataset evaluation uses Qwen2.5-7B-Instruct and ROUGE-L against target-user-authored references, with BLEU reported as a held-out metric.Prompts use benchmark-specific retrieved history or a pre-written summary rather than each user’s full history.Source: §4.1 · pp. 6–7
  • Reward-model baselines receive the same generation prompt and user-profile context as the ranker, testing whether generalist preference models can exploit personalization when ranking pools.The comparison includes four generalist reward models; one is also finetuned per dataset on the same candidate pools and objective.Source: §4.1 · p. 7

4.2 Main Results

  • The oracle improves through N = 64 on every task, but the best generalist reward model captures at most 23% of the headroom and under 12% on eight datasets.The oracle uses the evaluation metric to select the best candidate; its rising scores indicate that the generator continues to produce better-matched candidates as the pool grows.Source: §4.2 · p. 8
  • As N grows, the personalized ranker improves on every task while generalist reward models stay flat or decline, and the ranker’s advantage widens.Gains saturate earlier on short-form generation than on long-form QA and explainable recommendation; the authors attribute the ranker’s fine-grained distinctions to training on generator-produced candidate pools.Source: Figure 3 · p. 7, §4.2 · pp. 8–9
  • Ranking-guided generation recovers Best-of-2 to Best-of-4 performance on short-form generation and long-form QA, and Best-of-1 to Best-of-2 on recommendation, with one generation.It exceeds Best-of-64 only on Tweet and falls below Best-of-1 on News; default decoding settings change ROUGE-L by at most about 0.005 per dataset.Source: §4.2 · p. 9

4.3 More Analysis

  • From 1M to 30M parameters, Best-of-64 changes by at most 0.006 ROUGE-L per dataset, while the finetuned 8B model leads on all nine.The 30M ranker recovers 45% to 94% of the finetuned model’s gain per dataset at four orders of magnitude lower scoring cost.Source: Figure 4 · p. 8, §4.3 · p. 9
  • Under BLEU, the ranker selects the highest-BLEU candidate on all nine datasets, although training used ROUGE-L labels.ROUGE-L and BLEU select the same best candidate in only 12–37% of pools, with within-pool rank correlations of 0.36–0.59; across-prompt correlation is 0.48–0.78.Source: §4.3 · pp. 9–10, Table 1 · p. 9
  • Performance generally rises with more user history on Tweet, Scholarly, and all three XRec datasets, while long-form QA is flatter and News declines for users with 100 or more pairs.The authors characterize sparse-history degradation as graceful, while noting adjacent-bucket ordering is within noise.Source: §4.3 · p. 10, Table 2 · p. 10

5 Conclusion

The paper frames personalized alignment as candidate selection and introduces a small MLP ranker that reuses generator hidden states. Across nine datasets, it outperforms generalist 8B reward models and approaches per-dataset finetuning at far lower scoring latency, while evaluation relies on lexical overlap and leaves human style evaluation for future work.

5 Conclusion

  • The ranker outperforms generalist 8B reward models on all nine datasets and approaches per-dataset finetuning at four orders of magnitude lower scoring latency.It also steers decoding to recover part of the selection gain from one generation, and is competitive with personalized reward models without per-user preference labels.Source: §5 · p. 10
  • The evaluation uses lexical overlap with user-authored references; a human study of style consistency remains future work.Source: §5 · p. 10

A Full Per-Dataset Results

A.1 Personalization Headroom Analysis

Oracle Best-of-N selection continues to improve through 64 candidates across all nine datasets, while generalist reward models capture little or none of that headroom.

A.1 Personalization Headroom Analysis

  • At N = 64, the strongest generalized reward model captures 4%, 2%, and 8% of oracle headroom on News, Scholarly, and Tweet; 23%, 6%, and 11% on Amazon, Yelp, and Google; and −5%, −5%, and 0% on Art, Lifestyle, and Society.The Oracle ceiling rises monotonically as N expands from 1 to 64, while the generalized model plateaus or declines. On Art, Lifestyle, and Society, its Best-of-64 pick is no better than a single sample (ROUGE-L).Source: §A.1 · p. 14

A.2 Baseline Comparisons Across Datasets

Across the nine datasets, generalist reward-model trajectories are mostly flat or declining, whereas the personalized ranker is the best N = 64 selector everywhere. Single-trajectory ranking-guided generation recovers only a small part of the pool-selection gain.

A.2 Baseline Comparisons Across Datasets

  • Generalist reward models end at or below their N = 1 score on all long-form QA datasets; short-form performance is flat or declining, with explainable recommendation the exception.On recommendation, Skywork-Reward-V2 and URM gain 0.019 and 0.025 ROUGE-L on Amazon, but the best reward model captures no more than 23% of oracle headroom.Source: §A.2 · p. 15
  • At N = 64, the personalized ranker leads the best reward model by 0.010 ROUGE-L on Tweet to 0.048 on Amazon and improves with N on every dataset.Its curve saturates around N = 16 to 32 on News, Scholarly, Tweet, and Society; ranking-guided generation exceeds Best-of-64 only on Tweet and falls below Best-of-1 on News.Source: §A.2 · p. 15, Figure 6 · p. 15

A.3 Scaling Analysis and Parameter Efficiency

The scaling analysis finds that increasing the personalized ranker from 1M to 30M parameters barely changes its scores, while the finetuned 8B reward model remains ahead on every dataset. The 30M ranker nevertheless recovers a substantial share of the 8B model’s gain at much lower scoring latency.

A.3 Scaling Analysis and Parameter Efficiency

  • Figure 7 compares Best-of-64 ROUGE-L for 1M, 3M, 10M, and 30M rankers against the finetuned 8B reward model on each dataset.Its plotted delta is the 8B score minus the 30M score.Source: §A.3 · p. 15
  • At Best-of-64, ranker scores move by at most 0.006 ROUGE-L on News and less than 0.002 on Scholarly, Lifestyle, Society, Amazon, Yelp, and Google across 1M–30M parameters.The four sizes have no consistent ordering across datasets, so the 2.8M model is used by default.Source: §A.3 · pp. 15–16
  • 45% (Tweet) to 94% (Amazon) of the finetuned 8B model’s gain is recovered by the 30M ranker at roughly 10^-4 of its scoring latency.The 8B model is ahead on every dataset; its score-minus-30M gaps range from +0.004 to +0.026 ROUGE-L.Source: Figure 7 · p. 16, §A.3 · p. 16

B.1 Ranking Objectives

The ranking-objective ablation finds no single objective dominates across the nine datasets, and the score spread is narrow. The authors therefore treat pointwise MSE as a well-founded default, while attributing the framework’s gains to embedding reuse and training-time candidate scaling rather than the loss.

B.1 Ranking Objectives

  • No objective dominates across the nine datasets, and the spread is at most 0.012 ROUGE-L per dataset and below 0.003 on the three explainable recommendation datasets.MSE is best on News, ListMLE on Tweet, Lifestyle, and Society, pairwise objectives on Scholarly, Art, Yelp, and Google, and ListNet on Amazon.Source: §B.1 · p. 16, Table 3 · p. 17
  • Pointwise MSE is a well-founded default because it uses cardinal-label magnitudes and scales linearly with pool size, while pairwise objectives also perform on par.The ablation holds the MLP architecture, training data, and budget fixed while changing only the objective.Source: §B.1 · pp. 16–17
  • The narrow objective spread supports the authors’ conclusion that framework gains stem from embedding recycling and train-time candidate scaling, not the loss.Source: §B.1 · p. 17

B.2 Personalized Reward Model Baselines

On XRec, the ranker matches VPL within noise without per-user preference labels or per-candidate encoder passes. The comparison uses held-out user interactions and baselines fitted or inferred from each user’s labeled comparisons.

B.2 Personalized Reward Model Baselines

  • For each evaluation user, baselines use up to eight held-out training interactions to form labeled comparisons from on-policy candidate pools, while the ranker receives only the content profile and zero preference labels.All methods select from the same Best-of-64 pools and are evaluated on each user’s held-out test interaction.Source: §B.2 · p. 17
  • On all three XRec datasets, the ranker matches VPL within noise without annotated per-user pairs or a per-candidate encoder pass.PAL, PReF, and LoRe score below the ranker on every dataset; GPO scores about 0.005 ROUGE-L higher per dataset but uses labeled examples and post-hoc scoring.Source: §B.2 · p. 18, Table 5 · p. 18
  • Giving trained methods another user’s fitted parameters or context set changes selection by at most 0.0008 ROUGE-L, and SynthesizeMe captures 1–2% of oracle headroom.The authors connect the small effect of user-conditioning machinery to candidates already being generated with the user’s profile in the prompt.Source: §B.2 · p. 18

C.1 Datasets

The evaluation covers nine datasets in short-form generation, long-form personalized question answering, and explainable recommendation. Inputs include user histories or profiles, and the reported reference and prompting protocols vary by setting.

C.1 Datasets

  • Short-form LaMP generates a user-styled headline, research-paper title, or rephrased tweet from a document and the user’s history, using three BM25-retrieved profile items.Evaluation sets contain 1,889 News, 2,498 Scholarly, and 1,495 Tweet prompts.Source: §C.1 · p. 18
  • Long-form LaMP-QA answers open-ended questions tailored to the asker’s background across Arts & Entertainment, Lifestyle & Personal Development, and Society & Culture.Prompts use six BM25-retrieved past questions; released per-user reference answers are split 90/10 for ranker training and evaluation.Source: §C.1 · p. 19
  • XRec generates explanations for user–item interactions from user and item profiles on Amazon, Yelp, and Google, with each user’s review-style explanation as reference.Each platform has 3,000 evaluation prompts, and generation uses textual summaries without XRec’s collaborative embedding tokens.Source: §C.1 · p. 19
  • ROUGE-L is reported against the references, with ROUGE-1 where noted and BLEU as a held-out metric in Section 4.3.Source: §C.1 · p. 19

C.2 Training Details

The appendix specifies how candidates are generated and labeled, how the ranker’s frozen-generator embeddings are obtained, and how the ranker and finetuned reward-model comparator are trained.

C.2 Training Details

  • Qwen2.5-7B-Instruct generates 256 samples per prompt at temperature 1.0; each candidate receives pool-standardized ROUGE-L labels for the pointwise objective.Best-of-N uses the first N candidates in stored order, with N = 64 throughout the appendix.Source: §C.2 · p. 19
  • The ranker uses final-layer last-token embeddings of the frozen generator for the query, serialized user profile, and candidate; no embedding is fine-tuned.The implementation computes embeddings in a separate pooling pass, though exposing generation states would remove that pass.Source: §C.2 · p. 19
  • The default ranker concatenates the three embeddings and applies three linear layers with width 256, GELU, and dropout 0.1, totaling 2.8M parameters.Appendix ablations use AdamW for 15 epochs, batch size 64, and a single seed.Source: §C.2 · p. 19
  • The comparator is Skywork-Reward-V2-Llama-3.1-8B, fully finetuned per dataset with the ranker’s pointwise objective on the same candidate pools.Training uses eight candidates per prompt and 1,500 prompts, except LaMP-QA, which uses all 690–966 available prompts; it runs one epoch, or two for LaMP-QA.Source: §C.2 · p. 19

C.3 Ranking Guided Generation Hyperparameters

The appendix tests ranking-guided generation hyperparameters, reports inference costs for ranking and reward-model scoring, and records checklist responses about scope and reproducibility.

C.3 Ranking Guided Generation Hyperparameters

  • The same decoding settings—τ = 1, Hmax = 8, α0 = 5, and a three-token warmup—are used across all nine datasets.The settings were chosen a priori; τ governs intervention frequency, while Hmax and α0 govern intervention strength.Source: §C.3 · p. 19
  • Across the hyperparameter grid, ROUGE-L spread is at most 0.0052 per dataset (mean 0.0037), while τ changes intervention frequency from 4–39% at default to at most 11% at 2.Hmax, α0, and warmup length each change the outcome by at most 0.005 ROUGE-L.Source: Table 7 · p. 20, §C.3 · p. 20

C.4 Inference Cost

  • 0.098 ms versus 2.4 s: scoring 64 candidates with the 2.8M ranker is four orders faster and uses about 22× less memory than 8B reward models.With a separate embedding pass, end-to-end selection costs 2.1 ms per candidate, still 18× cheaper; part of the latency gap reflects differing serving implementations.Source: Table 8 · p. 20, §C.4 · p. 20

NeurIPS Paper Checklist

  • Checklist responses say the paper’s claims reflect its scope, experimental details are disclosed, and the authors release their code repository.The responses also mark the paper’s reproducibility information as sufficient.Source: NeurIPS Paper Checklist · pp. 21, 22, 26

Related papers

Loading 2609.35695v1…