LIRE: listwise reward enhancement for preference alignment

• The training pipeline gathers responses, scores them before optimization, and optionally regenerates the candidate pool with the updated policy for iterative training.Source: Figure 1 · p. 1, §1 · p. 2
• PPO uses sparse pointwise rewards and online sampling, while pairwise methods depend on sample-pair quality and become computationally costly over large candidate pools.Source: §1 · pp. 1–2
• LIRE addresses harmful or human-misaligned LLM outputs by reformulating preference alignment as an RL-free listwise optimization problem over multiple responses.Source: §1 · pp. 1–2
• LIRE uses offline rewards to update response probabilities, providing richer preference information than binary chosen-versus-rejected labels.Source: §1 · p. 2
• Unlike traditional Plackett-Luce listwise methods, LIRE computes its exponential probability distribution once, making computation efficient as the candidate count increases.Source: §1 · p. 2
• LIRE is easy to implement with little parameter tuning, extends pairwise alignment to multiple responses, and consistently outperforms existing methods across dialogue and summarization benchmarks.Source: §1 · p. 2

• Policy-gradient methods maximize trajectory rewards under a parameterized policy, with trajectory probability P(τ, θ) and return R(τ).Source: §2 · p. 2
• In RLHF, a learned reward function guides policy optimization, but training data must be sampled online as πθ updates, with a KL penalty limiting policy drift.Source: §2 · p. 2
• LIRE uses multiple pre-allocated responses to reformulate the trajectory-probability distribution and transform the RL problem into an offline setting.Source: §2 · pp. 2–3
• This offline reformulation improves computational efficiency and training stability relative to online sampling.Source: §2 · pp. 2–3

3.1 The construction of the LIRE objective

• LIRE uses multiple responses per query, each paired with a reward-model score, to construct a listwise preference-alignment objective.The response set is used to reformulate the response-probability distribution before optimization begins.Source: §3.1 · p. 3
• The objective applies a temperature-controlled probability distribution over the response set and derives a listwise loss from that distribution.The temperature controls distribution smoothness, while the reformulated distribution approximates the underlying response trajectory distribution.Source: §3.1 · p. 3
• LIRE weights normalized model-prediction gradients by demeaned rewards, encouraging higher-reward responses and depressing lower-reward responses.Unlike the policy-gradient formulation, all responses for one query are treated as a single listwise or groupwise sample.Source: §3.1 · p. 3
• When the response count M becomes 2, LIRE degenerates into a pairwise loss that can be directly compared with DPO.LIRE uses explicit proxy rewards to determine response preferences, whereas DPO derives weights from implicit rewards involving the policy and reference model.Source: §3.1 · pp. 3–4
• Compared with PL-based listwise objectives, LIRE computes the exponential probability distribution only once rather than using a ranked-list likelihood with O(n) time complexity.This distinction separates LIRE from traditional listwise ranking approaches based on ranking permutations.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 The self-enhancement algorithm

• The self-enhancement algorithm alternates Evolve data generation with Iterate policy training, repeatedly sampling responses, scoring them with the reward model, and updating the policy.Its inputs include the reward model, samples per query, initial policy, and the numbers of Evolve and Iterate steps.Source: §3.2 · p. 4

7 end

• RLHAIF expands the candidate pool beyond pairwise human preferences by including LLM generations produced with diverse decoding strategies.The paper motivates this extension by citing results that RLHAIF can outperform human-feedback-only settings.Source: §7 · p. 4
• The candidate-pool construction combines response generation with reward-based policy optimization rather than restricting training to pairwise human-preference data.This description connects diverse LLM generations with the subsequent objective optimization.Source: §7 · pp. 4–5
• With E = 1, responses are sampled only once and no iterative sampling follows during training.The target policy is initialized as the pretrained language model before optimizing J(θ).Source: §7 · p. 5

4.3 Performance comparison when training with pairwise preference

• LIRE achieves the highest average win rates against human-written baselines on both HH Dialogue and Summarization using two reward models.Human evaluation also gives LIRE win rates above 50 against all compared baselines.Source: §4.3 · p. 5, Table 2 · p. 6, Table 3 · p. 6
• LIRE and PPO obtain comparable GPT-4 support rates for summarization, while LIRE and PPO maintain relatively comprehensive MT-Bench performance after distribution shift.The MT-Bench result reports an overall score of 347 for the transferred models.Source: Figure 2 · p. 6, Figure 3 · p. 6, §4.3 · p. 6

4.4 Effects of increasing sequence number

• Increasing sequence number improves RM win rates for both LIRE and Best-of-n, but Best-of-n declines more under RM*, whereas LIRE provides a more balanced compromise across metrics.With Llama2 base models, LIRE also shows an obvious performance gap over Best-of-n when learning from augmented Alpaca-7B samples.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5 How far is LIRE drifted away from the reference policy?

• LIRE achieves higher rewards than SFT and PPO at the same KL range, while additional sequences increase rewards without substantially increasing policy KL divergence.The reference model is used to augment responses, helping mitigate model regression and policy drifting.Source: §4.5 · p. 7, Figure 5 · p. 7

4.6 Do we need to add regularization loss?

• Adding an SFT loss helps prevent model collapse observed with pairwise training from an unfine-tuned Llama2-7B base model and can provide a limited reward boost with suitable α.The collapse produces repeated and meaningless words; the added loss is defined as L(θ) = J(θ) + αL_SFT(θ).Source: §4.6 · pp. 7–8, Table 4 · p. 8

4.7 Additional performance boost with special sampling technique and iterative training

• Combining RSO with LIRE yields considerable RM improvement across the compared methods, while LIRE retains the best score.RSO adds four rejection-sampled responses after eight Alpaca-7B responses are sampled per prompt.Source: §4.7 · p. 8, Table 5 · p. 8
• Iterative sampling and training generally increase LIRE rewards as Evolve and Iterate steps grow, while enhancement suppresses most extremely low-reward cases.Fresh policy-generated responses replace model responses while human preference data remain in the candidate pool.Source: §4.7 · p. 8, Figure 6 · p. 8, Table 6 · p. 8

4.8 Effects of temperature parameters T

• Temperature changes cause slight performance fluctuations, with larger T producing more uniformly weighted samples and smaller T concentrating probability on the best sample.Experiments suggest T ∈[1, 10] is a good starting range.Source: §4.8 · p. 9, Table 7 · p. 9

• RLHF uses preference feedback to optimize LLMs, but obtaining sufficient human feedback is expensive and impractical in general cases.Source: §5 · p. 9
• DPO reformulates constrained reward maximization as direct policy optimization, while SLiC-HF learns pairwise human feedback with ranking and regularization losses.Source: §5 · p. 9
• RRHF and PRO extend preference alignment beyond binary data through pairwise ranking or iterative Bradley-Terry comparisons among multiple responses.Source: §5 · p. 9
• ReST and RAFT iteratively improve alignment by augmenting or filtering datasets with reward-based samples before offline fine-tuning.Source: §5 · p. 9
• Despite these advances, further research is needed for settings with multiple available responses.Source: §5 · p. 9

• LIRE combines listwise policy-gradient optimization with a self-enhancement algorithm that progressively maximizes rewards from diverse candidate responses.Source: §6 · p. 9
• LIRE requires minimal parameter tuning and achieves commendable performance on dialogue and summarization tasks.Source: §6 · p. 9

• Using proxy reward models as both optimization targets and evaluation metrics risks Goodhart-style divergence from human preferences.The authors call for reward models that better reflect human preferences, potentially requiring additional reward modeling during iterative sampling and training.Source: §7 · p. 10
• LIRE relies on the current policy to sample completions, while the best way to construct highly qualified and diverse candidate pools remains open.Source: §7 · p. 10

• More capable LLMs bring increased risks, including untruthful answers, deception, biased opinions, and harmful content.Source: §8 · p. 10
• The paper frames controlling and steering model generations toward human intentions and values as essential for safer AI systems.Source: §8 · p. 10

• The section derives the optimization gradient by substituting an earlier equation into Equation (6) and proceeding toward Equation (7).Source: §A.1 · p. 12
• Here, y denotes a set of model completions used in the objective.Source: §A.1 · p. 12
• The resulting objective is differentiated with respect to model parameters θ for back-propagation.Source: §A.1 · p. 12
• The derivation treats T as a probability that can be incorporated into the expectation.Source: §A.1 · p. 12

A.2 Relation to the DPO derivative

• The DPO derivative is rewritten using the implicit reward defined by πθ and πref, then expressed in a pairwise form.Source: §A.2 · pp. 12–13

A.3 Model and benchmark dataset details

• Experiments use Alpaca and Llama2 7B base models with hh-static for dialogue and Summarize From Feedback for summarization.The benchmark statistics are reported in Table 8.Source: §A.3 · p. 13, Table 8 · p. 13

A.4 LIRE implementation code

• The implementation provides a minimal PyTorch loss that forms a temperature-scaled candidate distribution, weights it by reward scores, and negates the batch average.Source: §A.4 · p. 13

A.5 More experimental results with multiple responses

• LIRE achieves superior and consistent performance on dialogue and summarization tasks as the number of training responses increases.The comparison covers HH-6 with six responses and Summarization-3 with three responses.Source: §A.5 · p. 13, Table 9 · p. 14

A.6 Human evaluation details and evaluation prompts using GPT-4

• Human evaluation compares method outputs with human-written baselines and other methods using randomized student judgments and GPT-4 prompts.The student evaluation gathers 52 feedbacks, while GPT-4 rates responses or selects between summarizations.Source: §A.6 · pp. 13–14

A.7 Comparison of Model generations

• The appendix includes randomly selected model-generation examples and reports that larger candidate sets let LIRE provide more detail and receive higher rewards.Source: §A.7 · p. 14, Table 10 · p. 16

A.8 Regression problem of the alignment algorithms

• LIRE reduces negative reward flips from 38% to 27% when its self-enhancement algorithm is used, outperforming the comparison methods.The rates measure test responses whose reward drops after policy tuning relative to Alpaca-7B.Source: §A.8 · p. 14, Figure 8 · p. 18