Reasons to Reject? Aligning Language Models with Judgments

1 Introduction

• Unlike scalar rewards, which indicate response goodness sparsely, language judgments can identify good and bad aspects, explain evaluations, and suggest improvements.The introduction argues this richer feedback may be more advantageous for alignment.Source: §1 · p. 1
• CUT contrasts response-token probabilities under aligned and misaligned conditions, then applies unlikelihood training to penalize inappropriate content.The framework is introduced as enabling fine-grained detection and correction based on judgments.Source: §1 · p. 2
• With 1,317 offline judgments, CUT on LLaMA2-13b achieves a 61.06 winning rate and beats the best baseline by 50.84 points on AlpacaEval.The introduction also reports that online CUT on LLaMA2-chat-13b improves from 81.09 to 91.68 points on AlpacaEval.Source: §1 · p. 2
• The paper systematically investigates judgment-based alignment and evaluates CUT across offline and online, cold-start and warm-start, and generalist and specialist settings.It also compares judgments with rewards and concludes that judgments hold greater potential for alignment.Source: §1 · p. 2

2 Related Work

• Feedback-learning approaches divide into prompting, which leaves model parameters unchanged, and fine-tuning, which updates them.The distinction is whether the LLM's parameters are updated.Source: §2 · p. 2
• Prompting methods use judgments on prior responses to elicit revisions, but rely on in-context learning and require more computation than single-pass generation.They can draw judgments from diverse aspects and iterate refinement (Welleck et al., 2022; Akyurek et al., 2023; Nathani et al., 2023; Yang et al., 2022; Peng et al., 2023a; Madaan et al., 2023).Source: §2 · p. 2
• Fine-tuning has used scalar rewards with PPO, whose complexity and instability have motivated efforts to simplify or stabilize training (Schulman et al., 2017; Ziegler et al., 2019; Zheng et al., 2023).Other efforts include direct preference optimization (Rafailov et al., 2023).Source: §2 · pp. 2–3
• Hindsight translates scalar rewards into language instructions, while training on prompted improvements uses revised responses as data; both approaches still struggle to learn from mistakes (Zhang et al., 2023; Liu et al., 2023a; Scheurer et al., 2022, 2023; Yu et al., 2023).Source: §2 · p. 3

3.1 Problem Setting

• Judgment alignment uses instruction-response-judgment triplets to retain strengths and address response weaknesses, with offline and online settings distinguished by the source of responses.Offline alignment uses off-the-shelf, model-agnostic data; online alignment reflects on the target model’s own outputs through interaction with a judge.Source: §3.1 · p. 3

3.2 Potential Solutions

• Forward Prediction trains an LLM to generate a response from the instruction and then generate its judgment from the instruction-response sequence.Because response generation comes first, learning to generate judgments does not necessarily improve responses.Source: §3.2 · p. 3
• ILF refines an initial response using judgment, then trains on the improved response with MLE or pairs it with the original response for DPO.ILF-DPO treats the improved response as preferred to the original for the same instruction.Source: §3.2 · p. 3
• Hindsight conditions response generation on the instruction and judgment, extending a method that rewrites instructions based on scalar rewards.The reward-based version appends an instruction phrase when a response’s scalar reward falls below a threshold.Source: §3.2 · p. 3

4 Contrastive Unlikelihood Training

• CUT combines maximum-likelihood training for appropriate content with unlikelihood training for inappropriate content identified through contrasting conditions.The framework uses fine-grained detection and correction based on judgments.Source: §4 · p. 4

4.1 Incorporating Judgments for Alignment

• Align-P, Align-N, and Misalign examples distinguish responses aligned with the instruction alone from those aligned only with the instruction plus judgment.Misalign substitutes a fake positive judgment for Align-N’s authentic negative judgment, leaving the response aligned with neither input condition.Source: §4.1 · p. 4

4.2 Learning from Contrasting

• Align-N versus Misalign exposes tokens whose generation probability rises with an authentic negative judgment relative to a fake positive judgment.The method treats such tokens as inappropriate when their probability difference exceeds the threshold set by λ.Source: §4.2 · pp. 4–5
• CUT applies unlikelihood training to identified inappropriate tokens and uses standard maximum-likelihood loss for other tokens.A dynamic weight scales the unlikelihood penalty with the token’s probability under the negative judgment, penalizing more strongly tokens more correlated with negative judgments.Source: §4.2 · p. 5
• Align-P versus Align-N teaches different responses depending on whether a negative judgment accompanies the instruction, distinguishing satisfactory from unsatisfactory responses.Both responses align with the instruction-judgment pair, but only Align-P aligns with the instruction alone.Source: §4.2 · p. 5

4.3 Relation to Prior Solutions

• Compared with Forward Prediction and ILF, CUT directly uses judgments and can learn from misaligned data; unlike Hindsight, it combines likelihood and unlikelihood objectives.The paper characterizes Hindsight as risking increased likelihood of unsatisfactory responses.Source: §4.3 · p. 5

5 Experiments

• Experiments span general instruction following and summarization, offline and online alignment, and multiple base-model and baseline configurations.Source: §5 · pp. 5–6, §5.1 · p. 6

5.1 Offline Alignment

• For LLaMA2-13b, CUT raises AlpacaEval winning rate from 1.87 to 61.06, beats Hindsight by 50.84 points, improves TruthfulQA by 13.05 points, and beats Forward Prediction by 1.38 rougeLsum.The reported results cover general instruction following and summarization.Source: Table 2 · p. 6, Table 3 · p. 6, §5.1 · p. 6
• On warm-start AlpacaEval, ILF-MLE and ILF-DPO outperform Forward Prediction and Hindsight, unlike in cold-start, where the authors say ILF relies on high-quality improved responses.Source: §5.1 · p. 6
• Removing the Align-N/Misalign contrast substantially reduces TruthfulQA; omitting inappropriate-token detection fails to converge, and removing dynamic weighting also impairs general instruction following.Removing Align-P affects summarization more than removing Align-N.Source: §5.1 · pp. 6–7, Table 4 · p. 7
• CUT improves LLaMA2-chat at 7B, 13B, and 70B, and elevates performance across almost all evaluated tasks for several other instruction-tuned backbones.The evaluated backbones include Mistral-7binstruct-v1, gemma-7b-it, and llama3-8b-instruct2.Source: Table 5 · p. 7, §5.1 · p. 7

5.2 Online Alignment

• After four online iterations, AlpacaEval rises from 81.09 to 91.68, but improvement ceases at the fifth iteration.The first online iteration also outperforms offline alignment on TruthfulQA and AlpacaEval.Source: Table 6 · p. 8, §5.2 · p. 8

5.2.1 Training A Judgment Model

• An AI judge trained on 5,000 judgments improves TruthfulQA by 3.02 points and AlpacaEval by 4.17 points over LLaMA2-chat-13b; the 3,000-example judge has limited effectiveness.The authors identify judge quality as crucial to success.Source: Figure 3 · p. 8, §5.2.1 · p. 8

5.3 Judgment vs. Reward

• With 1,000 instructions, CUT beats DPO by 37.54 and 23.04 AlpacaEval points across two base models; at larger data it is generally better or comparable, except on ARC and HellaSwag.The authors note ranking-based evaluation may favor DPO.Source: Figure 4 · p. 9, §5.3 · p. 9

• The exploration finds that three adapted methods cannot fully capitalize on judgments, motivating a framework for direct and explicit learning from them.Source: §6 · p. 9
• CUT uses judgments for fine-grained inappropriate-content detection and correction, and evaluations span offline and online, specialist and generalist, and cold-start and warm-start settings.Source: §6 · p. 9
• The comparison of rewards and judgments suggests that aligning LLMs with judgments is a promising research area.Source: §6 · p. 9

Limitations

• AI judges can misclassify inappropriate tokens as appropriate, or vice versa, potentially increasing hallucination risk; periodic human annotation is proposed to reduce accumulated hallucinations.The authors also report that adding 1,317 Shepherd judgments degraded their AI judge and hypothesize that its reasoning and math tasks were too complex for a 13B model.Source: Limitations · pp. 9–10
• Experiments use only several thousand judgment examples because of budget constraints, leaving scaling with larger judgment datasets for future investigation.Source: Limitations · p. 10

A.1 Implementations

• Models are trained with LoRA and Platypus-recommended configurations, with λ, α, and γ selected from specified small candidate sets.λ is selected from {1.1, 1.2}; α from {0.25, 0.5, 1}; and γ from {0.25, 0.5, 1, 2}.Source: §A.1 · p. 13
• The Alpaca template is used for fine-tuning and inference, while the inference template does not require judgments.Source: §A.1 · p. 13

A.2 Prompt for Judgment Annotation

• Judgments beginning with “Perfect.” are labeled positive; all others are labeled negative, a requirement GPT-4 is reported to fulfill proficiently.Source: §A.2 · p. 13

A.3 Offline Alignment Tasks

• Offline alignment is evaluated on instruction following using 1,317 Shepherd examples and on summarization using 10,827 training examples, with ROUGE reported on 1,939 test examples.Instruction-following evaluation includes four ranking-based benchmarks and AlpacaEval; summarization uses judgment annotations from Saunders et al. (2022).Source: §A.3 · p. 13

A.4 Downsampling Align-P

• 713 of 1,000 LLaMA2-chat responses received positive judgments; the authors report that retaining a moderate share of Align-P examples is crucial and use a 0.25 ratio subsequently.They conjecture that too many Align-P examples may cause overfitting and disrupt alignment.Source: §A.4 · p. 13, Figure 7 · p. 14

A.5 Case Study: Online Alignment

• Online case studies show gradual response improvements, including correction of a hallucinated filename and a later correction of an introduced error; CUT also adds Twitter-like hashtags in one case.Most content was already satisfactory before training and remained unchanged across iterations.Source: §A.5 · pp. 13–14, Table 7 · p. 16

A.6 Case Study: CUT v.s. DPO

• CUT responses are described as more specific, helpful, and instruction-adherent than DPO’s, while DPO responses are more polite.Source: §A.6 · p. 14, Table 8 · p. 17