PaperlayerView exact revision on arXiv

Source-linked AI summary

CREAM: Consistency Regularized Self-Rewarding Language Models

Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, Huaxiu Yao

arXiv:2410.12735v5cs.LGcs.CL

TL;DR

Self-rewarding language models can generate preference data without human annotation, but inaccurate and overconfident rankings may accumulate bias during iteration. CREAM formulates this process and regularizes training using reward consistency between iterations. The paper reports improved reward consistency and alignment performance, while experiments primarily use Llama-2 and Llama-3 models of about 7B parameters.

  • Problem

    Self-rewarding models lack guaranteed reward-label accuracy, allowing overconfident and accumulated bias to produce unreliable preference data.

  • Method

    CREAM formulates iterative preference fine-tuning and uses consistency between current and previous reward rankings as a regularization signal.

  • Results

    CREAM improves reward consistency and alignment performance across a series of natural-language benchmarks and beats baseline methods at the same iteration in Arena comparisons.

  • Takeaways & Limitations

    Cross-iteration reward consistency provides a practical signal for learning from more reliable preference data in self-rewarding alignment.

  • Takeaways & Limitations

    Experiments mainly use Llama-2 and Llama-3 models of about 7B parameters, while validation on other LLMs remains future work.

Abstract

from arXiv · show

Recent self-rewarding large language models (LLM) have successfully applied LLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the LLM via direct alignment technologies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for ensuring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g., 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead to unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning framework for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a Consistency Regularized sElf-rewarding lAnguage Model (CREAM) that leverages the consistency of rewards across different iterations to regularize the self-rewarding training, helping the model to learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superiority of CREAM in improving both reward consistency and alignment performance. The code is publicly available at https://github.com/Raibows/CREAM.

1 INTRODUCTION

Self-rewarding language models synthesize preference data by using one LLM as both policy and reward model, but unreliable rankings can accumulate bias and weaken alignment. CREAM addresses this by regularizing training with reward consistency across iterations.

  • Self-rewarding LLMs generate and rank responses with the same model, then use ranked responses as preference pairs for iterative DPO training.
  • Unreliable reward accuracy threatens preference-data quality and alignment performance, especially when iterative training accumulates rewarding bias.
  • Overconfident labeling forces the model to distinguish between similarly high-quality responses, producing noisy preferences that can harm preference tuning.
  • CREAM compares rankings from the current and previous reward models to estimate consistency and regularize preference training.
  • Empirical results on natural-language benchmarks report improved reward consistency and alignment performance with CREAM.

2 RELATED WORKS

Prior alignment methods rely heavily on human preference data, while self-rewarding language models synthesize preferences using the LLM itself. Related reward-hacking work improves reward reliability differently from CREAM’s consistency regularization.

  • RLHF and direct alignment methods such as DPO commonly depend on human-annotated preference pairs, whose collection is time-consuming and labor-intensive.
  • Self-rewarding language models use one LLM as policy and reward model to generate, score, rank, and train on preference pairs.
  • Reward-hacking research studies reward-model flaws and ensemble or conservative estimation, whereas CREAM regularizes preference comparisons in self-rewarding training.

3 METHODOLOGY

CREAM generalizes iterative preference fine-tuning, identifies overconfident preference labels as a source of bias, and uses cross-iteration ranking consistency to soften unreliable supervision. Its algorithm samples responses, compares current and previous rankings, and trains with adaptive regularization.

  • 3.1 GENERALIZED ITERATIVE PREFERENCE FINE-TUNING FRAMEWORK: The framework supports SFT, DPO-style preference learning, RL with feedback, and self-rewarding by varying the preference-labeling function.
  • 3.1 GENERALIZED ITERATIVE PREFERENCE FINE-TUNING FRAMEWORK: The generalized framework alternates preference labeling with model optimization, and its two-step procedure converges under the stated solvability and nonnegative-SFT-loss assumptions.
  • 3.2 CONSISTENCY REGULARIZED SELF-REWARDING: Hard preference labels can propagate bias because self-rewarding does not guarantee accurate judgments, particularly for similarly high-quality responses.
  • 3.3 PROPOSED ALGORITHM: Each iteration samples N responses, ranks them with current and previous models, computes Kendall’s Tau-based consistency, and updates the policy using consistency-weighted preference losses.
  • 3.2 CONSISTENCY REGULARIZED SELF-REWARDING: CREAM regularizes similar-quality comparisons toward a uniform preference distribution, yielding a soft-labeled DPO interpretation of the regularized objective.

4 EXPERIMENT

Experiments compare CREAM with self-rewarding baselines and oracle-style variants on downstream accuracy, alignment arena performance, ranking accuracy, and reward consistency. Across these evaluations, CREAM improves across iterations, maintains ranking consistency, and mitigates degradation associated with prompt rewarding.

  • 4.2 MAIN RESULTS: CREAM consistently improves across iterations and outperforms directly KL-regularized SRLM and manually tuned CREAM without ranking consistency regularization.The dynamically calculated consistency rate also avoids the hyperparameter-search cost of manually setting it.
  • 4.2 MAIN RESULTS: CREAM beats SRLM and Oracle baselines at the same iteration in GPT-4o-judged Arena comparisons, while its win and tie rates against Oracle increase with iteration.These results are reported in the Arena evaluation shown in Figure 3.
  • 4.3.1 ANALYSIS OF REWARDING: CREAM preserves ranking consistency across iterations, whereas SRLM exhibits consistency problems that introduce noise into generated preference data.Consistency is measured with Consistency Rate, Kendall correlation, Spearman correlation, and TopOrder, which checks whether the least- and most-preferred responses remain consistently ranked.
  • 4.3.1 ANALYSIS OF REWARDING: Prompt rewarding causes 7B SRLM performance to decrease from M1 to M2, while DPO rewarding is more suitable because it is intrinsically aligned with the model’s learning objective.Figure 4 evaluates pairwise ranking accuracy on RewardBench and curated preference data, and Figure 5 compares average performance across SRLM, P-SRLM, and CREAM.
  • 4.3.2 RELIABILITY OF SELF-CONSISTENCY: Using an oracle reward model as CREAM’s baseline reward model provides stronger regularization for Llama-2, while the last iteration’s model supplies a reasonably reliable consistency signal for Llama-3.The comparison evaluates Oracle and the previous model M0 as baseline reward models.
  • 4.3.1 ANALYSIS OF REWARDING: CREAM works with Kendall, Spearman, and TopOrder consistency measurements, indicating that the regularized training approach generalizes across consistency metrics.The comparison is reported in Table 4 for Llama-3.

5 CONCLUSION

The paper formulates a generalized iterative preference fine-tuning framework and proposes CREAM to regularize self-rewarding with cross-iteration reward consistency.

  • CREAM uses reward consistency across iterations as a regularization signal, emphasizing reliable preference data and reducing overconfident labeling.

A LIMITATIONS

The work has computational, scale, and design-scope limitations that motivate more efficient training and broader consistency-regularized settings.

  • Full fine-tuning across multiple iterations is time-consuming and computationally intensive, motivating future exploration of PEFT methods.
  • Experiments primarily focus on small 7B-level language models, which may limit applicability despite the method being designed for any model size.
  • More complex consistency-regularized settings remain open, including weighting models by importance and enabling their collaboration for improved reward accuracy.

B.1 PROOF OF THEOREM 3.1

The proof shows that the two-step optimization decreases the objective monotonically and therefore produces a convergent sequence under the stated assumptions.

  • The preference-labeling update minimizes LDPO for fixed responses, while the learning update minimizes L for the resulting labels.
  • The learning and preference-labeling steps yield a nonincreasing loss sequence across iterations.
  • Because the loss is upper bounded by 0, the sequence L(θt, zt) converges as the iteration count grows.

B.2 PROOF OF LEMMA 3.2

The lemma proof rewrites the regularization expectation by decomposing preference events and uses exchangeability of sampled responses to obtain a uniform binary distribution.

  • The regularization expectation is expanded into logarithmic terms for the two preference-label outcomes.
  • The expectation is decomposed according to the events z = 0 and z = 1, weighted by their iteration-specific probabilities.
  • Because both responses are sampled from the same policy, each binary preference event has probability 0.5, yielding the uniform distribution u(z).

B.3 PROOF THEOREM 3.3

The proof expands the generalized iterative preference fine-tuning objective into supervised fine-tuning, DPO, and regularization components, then rewrites the regularized terms as a mixture of normal and reversed DPO losses. It also derives the relationship between preference rankings across iterations through indicator expectations and symmetry.

  • Objective decomposition: The objective decomposes into supervised fine-tuning plus an expected combination of DPO and regularization losses over sampled preference pairs.The loss is written as L_SFT plus an expectation of L_DPO and λL_Reg.
  • Regularized loss form: Dividing by the normalization factor yields a mixture of normal DPO and reversed-preference DPO weighted by 1−C_λ and C_λ.For z(y,y′,x)=1, C_λ=λ/(1+2λ), producing the equivalent weighted form.
  • Optimization equivalence: The proof concludes that minimizing the normalized objective is equivalent to minimizing the original objective.The final step uses the fact that positive scalar normalization does not change the minimizer.
  • Ranking comparison: The ranking analysis characterizes agreement and disagreement using indicator products for current and previous model preferences.The derivation converts ranking differences into conditions on reward gaps and then combines the corresponding indicator terms.
  • Kendall’s Tau derivation: Symmetry of independently sampled responses lets the proof combine the indicator expectations and derive the claimed Kendall’s Tau relationship.The final expression follows after adding the intermediate equations and dividing by two.
  • Empirical implication: Across later iterations, CREAM converges at M4 while SRLM begins degrading at M2, and CREAM avoids the severe post-convergence drop observed for SRLM.The additional Llama-3 experiments cover iterations M4–M6 and support improved long-term stability.

C.2 DIFFERENCE BETWEEN CREAM LOSS AND WEIGHTED DPO LOSS

CREAM combines normal and reversed DPO rather than merely downweighting the normal DPO loss. This produces a label-smoothing-like regularizer, whose behavior differs from weighted DPO and is associated with more stable later-iteration training.

  • Loss distinction: CREAM’s loss combines normal and reversed DPO, whereas weighted DPO only scales the normal loss.The paper states that weighted DPO behaves like normal DPO with a smaller learning rate, while CREAM changes the objective itself.
  • Regularized objective: CREAM’s combined objective is C log P + (1−C) log(1−P), equivalent to binary cross-entropy and a label-smoothing regularizer.Here C is the consistency rate, while P is the preference probability induced by the reward gap.
  • Optimization behavior: The reversed-DPO component encourages less overconfident preference probabilities by opposing the normal preference signal.The derivation expresses normal DPO as log σ(x_θ) and reversed DPO as log σ(−x_θ).
  • Theoretical limitation: The paper notes that there is no theoretical evidence that both preferred and dispreferred response likelihoods decrease under the combined optimization.The subsequent discussion instead cites empirical results suggesting that the reward gap may increase while both likelihoods decrease.
  • Empirical comparison: CREAM outperforms weighted-DPO variants, while negative DPO weighting can cause catastrophic forgetting and failure to generate fluent sentences.The weighted-DPO comparison is reported on Llama-3, with “NA” marking this failure mode.
  • Iteration stability: CREAM converges at M4 while SRLM degrades from M2, and CREAM maintains performance through M5 and M6.The consistency-based regularization is reported to stabilize self-improvement and prevent long-term degeneration.

C.4 COMPARISON OF ENSEMBLE METHODS

CREAM is compared with ensemble-based reward methods that aggregate multiple models or checkpoints. The paper argues that CREAM achieves competitive alignment performance while requiring only the latest two checkpoints rather than an expanding collection of reward models.

  • Ensemble construction: Ensemble methods train multiple models with different learning rates to provide alternative reward rankings in the self-rewarding setting.The described ensemble uses three models with learning rates 7e-7, 1e-6, and 3e-6.
  • Performance comparison: CREAM has an advantage over Ensemble-Worst and Ensemble-Mean in alignment performance across iterations.Ensemble-Worst uses minimum rewards, whereas Ensemble-Mean uses average rewards for ranking.
  • Self-rewarding constraint: Using a single evolving model for reward distillation can yield the trivial minimizer θ_t=θ_{t−1}, preventing improvement.The paper uses this point to explain why ordinary distillation assumptions do not directly transfer to self-rewarding.
  • Scaling cost: As self-rewarding iterations progress, an ensemble needs an increasing number of earlier models and must retain them throughout training.For example, training π4 would ensemble three earlier models, and the storage requirement grows with the iteration count.
  • Efficiency: CREAM requires storing and evaluating only the latest two checkpoints, making it more efficient than standard ensemble methods.The comparison highlights reduced storage and computation relative to methods using a batch of reward models.
  • Regularization target: The paper distinguishes CREAM’s preference-pair regularization from reference-policy KL regularization, which is described as more conservative for iterative self-improvement.CREAM targets overconfident preference modeling, whereas the alternative constrains the policy toward a reference policy.

C.5 CREAM WITH DATA CONSISTENCY

CREAM can treat consistency as a model-level uncertainty signal or selectively regularize data according to consistency thresholds. The paper reports that average consistency is effective without extra threshold complexity, while applicability depends on initial alignment capability.

  • Uncertainty interpretation: CREAM’s consistency is intended to quantify uncertainty from stochastic training noise or incorrectly labeled preference data.The paper contrasts this model-level uncertainty with variants that treat consistency as a property of individual data samples.
  • Data consistency: Increasing the consistency threshold eventually produces converging performance gains, so fine-grained data selection adds complexity without clear further benefit.The threshold variant includes more samples for regularization as the threshold increases.
  • Consistency estimation: Average consistency reduces variance in dataset-uncertainty estimation and improves performance relative to dynamic consistency.The paper presents the average-consistency method as sufficiently effective without an added threshold hyperparameter.
  • Applicability boundary: CREAM requires some initial alignment capability because DPO-based rewards are not meaningful when the model is unaligned.The same initial post-training requirement is also attributed to the original SRLM setup.
  • Unaligned models: After limited seed SFT, both SRLM and CREAM improve an initially unaligned model, with CREAM often providing larger gains.The reported gains from the third iteration exceed those from the second, consistent with stronger alignment improving the method’s effectiveness.
  • Rewarding strategy: Prompt-rewarding is less effective than DPO rewarding for both SRLM and CREAM in the reported comparison.The section compares P-CREAM and Distilled DPO with DPO-based rewarding.
Loading 2410.12735v5…