Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles

• Optimizing reward-model scores beyond a threshold can diminish human preferences, in part because reward models trained on limited data are imperfect proxies for human preferences.Source: §1 · p. 1
• KL regularization constrains policy deviation from the supervised fine-tuning model, but can overfit and reduce gold performance; alternatives may carry substantial cost.Source: §1 · p. 2
• UP-RLHF penalizes rewards using uncertainty from diverse reward LoRA ensembles, adding uncertainty regularization alongside KL to address overoptimization.The ensembles are diversified by maximizing the nuclear norm of concatenated LoRA matrices.Source: §1 · p. 2
• Experiments report that UP-RLHF eliminates overoptimization and improves performance in terms of gold reward.Source: §1 · p. 2

2.1. Reinforcement Learning from Human feedback

• Supervised fine-tuning trains a pretrained LLM on demonstrations, producing the initial policy πSFT.Source: §2.1 · p. 2
• Reward modeling trains on prompts paired with labeler-preferred and less-preferred answers, using a Bradley–Terry ranking loss.Source: §2.1 · p. 2
• In RL fine-tuning, a policy generates answers scored by the reward model and is optimized with a KL penalty regulating deviation from πSFT.Source: §2.1 · p. 2

2.2. Low-Rank Adaptations

• LoRA adapts fixed pretrained weights through low-rank down- and up-projection matrices, with fewer trainable parameters than full-model tuning.The passage states that LoRA often uses less than 1% of the original model size in trainable parameters.Source: §2.2 · pp. 2–3

3.1. Analysis of Regularizations in RLHF

• KL regularization can become weak or negative for low-quality OOD answers, especially when the reward model assigns them high rewards.The analysis frames this as a source of overoptimization.Source: §3.1 · p. 3
• The method approximates an intractable offline-RL objective's OOD penalty using reward-model uncertainty and formulates a constrained policy optimization problem.The constraint keeps the policy close to the dataset's behavior distribution so the surrogate remains a reasonable approximation.Source: §3.1 · p. 3

3.2. Training Diverse Reward LoRA Ensembles

• Maximizing the nuclear norm of concatenated LoRA matrices regularizes diversity, using matrix rank as the diversity measure and nuclear norm as its computationally efficient surrogate.The method addresses the hypothesized lack of diversity among LoRA members, whose outputs can be more homogeneous than those of traditional deep ensembles.Source: §3.2 · p. 4
• Reward uncertainty is estimated from the standard deviation across the trained diverse reward LoRA ensemble.The ensemble members are LoRAs rather than full reward models, making the approach more parameter-effective.Source: §3.2 · pp. 3–4

3.3. Overall Optimization Objectives

• UP-RLHF optimizes uncertainty-penalized rewards with RL while keeping KL regularization independent of the actor loss.The mean uncertainty of previously seen samples approximates the scale adjustment for ensemble uncertainty; KL and uncertainty penalties correspond to regularization from pipeline steps 1 and 2, respectively.Source: §3.3 · p. 4

4.1. Datasets and Training Setups

• Experiments cover TL;DR summarization and Anthropic Helpful question answering, using gold-reward relabeling for summarization and human preferences for question answering.Both datasets are split into 20% for Step 1, 40% for Step 2, and 40% for Step 3; policy and reward model sizes differ by task.Source: §4.1 · p. 5

4.2. Reward Model Evaluation

• LoRA ensembles improve reward-model accuracy and ECE on both test datasets, and nuclear norm maximization further improves overall performance across those metrics.Table 1 reports results for five-member ensemble methods.Source: §4.2 · p. 5, Table 1 · p. 6
• With diversity regularization, ensemble uncertainty increases gradually with policy KL divergence, indicating better OOD detection than the rapid early increase of ordinary LoRA ensembles.The ordinary ensemble’s uncertainty rises rapidly from KL divergence 0 to 50, making high-gold-reward samples harder to distinguish from overoptimized samples at roughly 50 to 100.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3. Effect of Uncertainty Penalty

• Although uncertainty regularization improves gold-reward performance, it can lower the reward-model score and may restrict exploration of both high- and low-quality OOD outputs.The authors relate this possible exploration restriction to the exploration–exploitation dilemma in RL.Source: §4.3 · p. 6
• Uncertainty penalties eliminate overoptimization even though training on the diverse ensemble’s mean reward still shows significant overoptimization.With penalties, uncertainty in generated samples remains within a reasonable range.Source: Figure 4 · p. 6, §4.3 · p. 6

4.4. Policy Model Evaluation

• UP-RLHF outperforms RLHF on gold performance in both tasks, with higher performance at lower KL divergence cost in summarization.Figure 5 compares the methods; the diverse reward LoRA ensemble fine-tunes 4.53% of OPT-350M parameters and 1.25% of Llama2-7B parameters.Source: Figure 5 · p. 6, §4.4 · p. 6

5.1. Reinforcement Learning from Human Feedback

• Reward-model-based RLHF remains relevant alongside DPO because cited work reports robustness to KL-related overfitting and advantages on out-of-preference samples.DPO bypasses reward modeling through classification on human-preference data, with implementation and training-stability benefits.Source: §5.1 · p. 7
• Most overoptimization work focuses on RL fine-tuning, whereas this paper introduces uncertainty quantification at reward modeling and carries it into uncertainty-aware fine-tuning.Related work also describes overoptimization as a challenge associated with imperfect reward models.Source: §5.1 · p. 7

5.2. Uncertainty Aware Reinforcement Learning

• Offline RL uses uncertainty conservatively, and this work applies that framing to imperfect reward models trained on limited offline preference data.It contrasts its parameter-effective diverse LoRA ensembles with larger reward-model ensembles and says KL and uncertainty regularization affect the objective independently.Source: §5.2 · p. 7

5.3. Uncertainty for LLMs

• Full LLM ensembles can be expensive and insufficiently diverse, motivating LoRA ensembles with diversity regularization for reward modeling.The paper distinguishes its diversity regularization from concurrent LoRA-ensemble work using different regularization techniques for each LoRA.Source: §5.3 · p. 7

6. Conclusion and Limitations

• The diverse reward LoRA ensemble provides reward uncertainty estimates, which UP-RLHF uses to address overoptimization in LLM alignment.The conclusion identifies uncertainty regularization as pivotal to addressing this challenge.Source: §6 · p. 7
• Nuclear-norm computation adds time overhead, and uncertainty regularization may be over-conservative for near-distribution, high-quality outputs.The authors suggest balancing KL and uncertainty regularization for specific samples as a future direction.Source: §6 · pp. 7–8