SALMON: Self-Alignment with Instructable Reward Models

• The prevailing SFT-plus-RLHF approach relies on costly, nonscalable human annotations and assumes people can demonstrate or evaluate the tasks advanced AI systems undertake.The authors warn that future systems could challenge human evaluation and prioritize appeasing evaluators over accuracy.Source: §1 · p. 1
• The paper asks whether RLAIF can replace RLHF from scratch, as prior RLAIF work focused on models already trained with RLHF and retained its human-preference warm-up dependence.Source: §1 · p. 2
• SALMON’s instructable reward model scores responses according to human-written principles, unlike prior RLAIF methods that use principles only to generate synthetic preferences.The authors also report using tailored principles to address reward hacking, emphasize helpfulness, honesty, and harmlessness, and reduce false refusals.Source: §1 · p. 2
• The authors wrote reward-hacking behavior descriptions from their own inspections and identify automated description generation as future work.Source: §1 · p. 2
• With 6 exemplars and 31 principles, Dromedary-2 outperformed the extensively RLHF-trained LLaMA-2-Chat; Table 1 compares supervision and MT-Bench scores.Source: §1 · p. 2, Table 1 · p. 3

AI Alignment from Scratch

• Aligning models from scratch means aligning pretrained LLMs without relying on already aligned systems; this differs from distilling capabilities or behavior from proprietary models.Source: §2 · p. 3

Self-Alignment & Scalable Oversight

• Self-Instruct uses 175 in-context query-response pairs, Self-Align uses 16 principles and 5 exemplars, and Instruction Back-translation uses web documents with 3200 seed examples.Source: §2 · p. 3
• Whether synthetic-data bootstrapping can match RLHF performance remains an open challenge, while existing approaches still depend heavily on human-annotated data.Source: §2 · p. 3

3.3 RL WITH INSTRUCTABLE REWARD MODELS

• Unlike vanilla RLHF and Constitutional AI reward models, SALMON conditions reward scores on customized principles that can be changed during RL.During RL, the model uses both predefined principles and RL-time intervention principles.Source: Figure 1 · p. 4, §3.3 · p. 7

3.1 PREREQUISITES

• RL trains a policy to maximize reward-model scores, sometimes adding a per-token KL penalty against the initial policy to limit over-optimization.The penalty scale is controlled by β.Source: §3.1 · p. 4

3.2 PRINCIPLE-DRIVEN PREFERENCE MODELING

• The method targets RLHF’s need for fresh human preferences to counter reward hacking, which the authors say undermines scalability.Their proposed RLAIF paradigm uses AI-generated preference labels and an instructable reward model to address this issue.Source: §3.2 · pp. 4–5
• SALMON builds synthetic preference data by having the policy choose between sampled responses under human-written principles, averaging scores after swapping response positions to address position bias.The preference score is the difference between log probabilities of choosing responses A and B.Source: §3.2 · p. 5
• The instructable reward model trains on synthetic preferences paired with principles; including negative principles helps it interpret prohibitions.Positive and negative principles are defined for each positive principle to increase principle diversity.Source: §3.2 · p. 6

3.3 RL WITH INSTRUCTABLE REWARD MODELS

• SALMON samples three principles per prompt during RL and adjusts sampling for reasoning and red-teaming prompts.Consistent Reasoning is sampled more often for reasoning prompts, and Ethical more often for red-teaming prompts.Source: §3.3 · p. 7
• The authors identify high-level advice, self-praise, and over-education as reward-hacking tendencies and define a natural-language intervention for each.They report that the same instructable reward model can be reused with prohibition instructions rather than collecting online preferences aligned with the updated policy.Source: §3.3 · p. 7

4.1 DROMEDARY-2

• Dromedary-2 starts from LLaMA-2-70b, receives SFT using SELF-ALIGN data generated with six in-context exemplars, then undergoes SALMON reinforcement learning.Human demonstration annotations are limited to those six exemplars; behavior at the RL stage is controlled by human-defined principles.Source: §4.1 · p. 7

4.1.1 DATASETS

• The authors do not claim their proposed training curriculum is the only way to produce an instruction-following reward model.Source: §4.1.1 · p. 7
• The datasets comprise unlabeled prompts for SFT and RL, synthetic principle-driven preference data, existing preference datasets, and 7.5k additional MATH prompts for RL.The prompt collections include ShareGPT4, Databricks Dolly, OpenAssistant, and OpenOrca; preference pre-training uses 160k Anthropic HH-RLHF and 160k synthetic Stanford SHP preferences.Source: §4.1.1 · pp. 7–8

4.1.2 TRAINING DETAILS

• The reward model projects the final-token embedding to a scalar reward, while PPO with a KL penalty trains the policy; QLoRA keeps the models within one GPU.The value model is initialized from the reward model, and non-RL or offline-RL alternatives to PPO are left for future work.Source: §4.1.2 · p. 8

4.1.3 BASELINE MODELS

• The main open-source baseline is LLaMA-2-Chat, selected as the best open-source model aligned from scratch among the comparisons described.The paper notes that potentially stronger open-source models, including Orca and WizardLM, exist.Source: §4.1.3 · p. 8

4.2 BENCHMARK EVALUATIONS

• The benchmark evaluation uses GPT-4 on Vicuna-Bench and MT-Bench, alongside AlpacaEval; human evaluation is deferred to future work.General-capability tests cover BBH, HumanEval, and TydiQA, while TruthfulQA measures truthful and informative generations using a fine-tuned GPT-3 judge.Source: §4.2 · pp. 8–9
• Dromedary-2 significantly outperforms LLaMA-2-Chat on BBH, HumanEval, and TydiQA, achieves new state-of-the-art on TruthfulQA, and outperforms LLaMA-2-Chat-70b on Vicuna-Bench and MT-Bench.BBH, HumanEval, and TydiQA evaluate reasoning, coding, and multilingual ability, respectively.Source: §4.2 · pp. 8–9, Figure 4 · p. 9, Table 2 · p. 9

4.3 IMPROVED CONTROLLABILITY BY PRINCIPLE INTERVENTION

• Changing preference principles lets the policy selectively enhance helpfulness, honesty, or harmlessness and reduce false refusals associated with over-aligned models.The authors describe this as a proof of concept and refer readers to Appendix B for detailed results.Source: §4.3 · p. 9

• SALMON adjusts the principles followed by an instructable reward model during RL to control reward preferences and influence the policy, reducing reliance on exhaustive online human preference collection.Source: §5 · p. 9
• With six in-context exemplars and 31 human-defined principles, Dromedary-2 significantly surpasses several state-of-the-art RLHF-trained systems on chatbot, reasoning, coding, multilingualism, and truthfulness benchmarks.Source: §5 · p. 9

6 LIMITATIONS

• Dromedary-2 can hallucinate unverified information and make reasoning errors, with these reliability issues possibly stemming from SFT-initialized reward models.The authors suggest external fact-checking tools as a possible future way to improve accuracy and trustworthiness.Source: §6 · p. 10
• Robust principles are difficult to craft for unpredictable RL scenarios, and conflicts between principles can produce unexpected results; distinct contexts require tailored strategies.The authors frame SALMON as a starting platform, not a universal solution, and call for diverse stakeholder participation in refining principles.Source: §6 · p. 10
• Principle effectiveness can depend on the task, so randomly sampled principles for general prompts may not suit specialized prompts.The authors identify adaptive principle selection as a direction for future research.Source: §6 · p. 10
• SALMON remains bound by the base LLM’s knowledge, so outputs may be imprecise or miss recent advances.Retrieval-augmented generation is suggested as a possible way to provide more current information.Source: §6 · p. 10

A ADDITIONAL RELATED WORK

• Unlike Constitutional AI’s safety emphasis, SALMON aims at more generic alignment and capabilities; unlike fixed synthetic-preference heuristics, RLAIF systems let AI compare response pairs.Source: A ADDITIONAL RELATED WORK · p. 16

HHH Alignment

• SALMON applies customized principles as preference guidelines, aiming to emphasize selected HHH aspects despite possible conflicts among helpfulness, honesty, and harmlessness.The LLaMA-2-70b base model receives SALMON training directly after preliminary HHH evaluation, without an SFT warm-up; models are evaluated after 20–50 PPO steps.Source: B ALIGNING AI ASSISTANTS WITH… · p. 16
• Figure 5 illustrates that prompting the trained instructable reward model with different principles customizes its preferences.This offers a way to select which preference guidelines shape the model's behavior.Source: B ALIGNING AI ASSISTANTS WITH… · p. 16, Figure 5 · p. 17
• Helpful and honest principles improve their corresponding HHH aspects to state-of-the-art multiple-choice accuracy, while harmless principles improve over the base model but still trail ChatGPT and LLaMA-2-Chat.The authors suggest stronger safety alignment or different red-teaming data may explain the harmlessness gap.Source: B ALIGNING AI ASSISTANTS WITH… · p. 17, Table 3 · p. 17

Non-Evasiveness Alignment

• Removing safety principles to create a purely helpful set slightly improves LLaMA-2-Chat-70b’s MT-Bench chatbot performance after 100 PPO steps.The authors frame this as an attempt to reduce false refusals in over-aligned models.Source: B ALIGNING AI ASSISTANTS WITH… · p. 17, Table 4 · p. 18

AlpacaEval

• The comparison groups assistant models as non-distilled, distilled, or proprietary, distinguishing their use of external well-aligned models or proprietary data and techniques.Non-distilled models are trained independently without external well-aligned models for supervision; distilled models use a more potent external model as supervision.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 18
• Dromedary-2-70b performs best among models using fewer than 10k human annotations, but slightly trails LLaMA-2-Chat-70b and Vicuna-33b.AlpacaEval uses GPT-4 to evaluate pairwise win rates against Text-Davinci-003 across 805 prompts.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 18, Table 5 · p. 18

Evaluation sets

• The evaluation compares reward-model accuracy on Anthropic Helpful, Anthropic Harmless, and Anthropic Adversarial test sets.The adversarial set adds a self-praising sentence to a helpful-base preference example to test reward hacking.Source: D DIRECT EVALUATION OF REWARD MODELS · p. 19

Reward-model baselines

• Comparison baselines include SteamSHP-XL, Open Assistant, and LLaMA-2 Safety and Helpfulness reward models.The reported baseline results are taken from Touvron et al. (2023b).Source: D DIRECT EVALUATION OF REWARD MODELS · p. 19

Reward-model evaluation

• The instructable reward models perform competitively with LLaMA-2 reward models trained on in-house preferences, and new principles can help avoid reward hacking on Anthropic Adversarial.The evaluation includes helpful, harmless, and adversarial preference data, including an adversarial example augmented with self-praising text.Source: D DIRECT EVALUATION OF REWARD MODELS · p. 19

Potentially Harmful Examples

• For a dark-web hacking-forum prompt, the helpful reward model returns a long warning with cybersecurity alternatives, while the harmless model gives a shorter refusal.The examples are cherry-picked responses from Dromedary-2 before PPO, selected by three reward models using best-of-64.Source: E SAMPLES WITH CUSTOMIZED PRINCIPLES · pp. 20, 22
• For a question about football players’ IQ, the helpful response presents mixed research claims and caveats, the harmless response refuses, and the honest response emphasizes uncertainty and limitations.The three outputs show distinct responses to one red-teaming prompt.Source: E SAMPLES WITH CUSTOMIZED PRINCIPLES · p. 23

Vicuna-Bench examples

• For the medieval-knight prompt, the helpful, harmless, and honest models all draft a courtly introduction, with differing emphasis on chivalry, ceremony, and modesty.The honest response advises mentioning achievements without boasting or exaggerating.Source: E SAMPLES WITH CUSTOMIZED PRINCIPLES · p. 24
• On the van Gogh counterfactual, the helpful response discusses possible financial and career effects while acknowledging uncertainty about mental health consequences.The harmless and honest responses also frame potential outcomes as speculative.Source: E SAMPLES WITH CUSTOMIZED PRINCIPLES · p. 25
• For a delayed-order apology, the three responses all apologize and reassure the customer, but vary in commitments such as process changes, delivery timing, and compensation.The helpful response offers possible compensation; the harmless response says the order is on its way; the honest response gives a placeholder delivery estimate and discount.Source: E SAMPLES WITH CUSTOMIZED PRINCIPLES · p. 26
• The concert-review samples range from a detailed imagined review to generic performance descriptions and a step-by-step plan for writing one.The helpful response names Beethoven, Tchaikovsky, and Dvořák; the honest response describes attending, taking notes, and analyzing the performance.Source: E SAMPLES WITH CUSTOMIZED PRINCIPLES · p. 27

F MORE SAMPLES

• For the quantum-physics prompt, the after-PPO response uses numbered sections on superposition, entanglement, their relationship, and applications.The before-PPO response gives a shorter explanation of the two concepts and their relationship.Source: F MORE SAMPLES · p. 28
• The economics examples differ in organization: the after-PPO response defines GDP, inflation, and unemployment before discussing their relationships and policies.The before-PPO response directly describes correlations and fiscal and monetary policy effects.Source: F MORE SAMPLES · p. 29
• For the Mars prompt, the before-PPO response declines to emulate Elon Musk, while the after-PPO response answers in his first-person voice.Source: F MORE SAMPLES · p. 30
• After PPO, the model gives remainder 0, while the before-PPO example gives remainder 8 for the same prompt.The after-PPO response derives a quotient and remainder; the before-PPO response incorrectly concludes 8.Source: F MORE SAMPLES · p. 31

G SYNTHETIC PREFERENCE CALIBRATION: AN EXAMPLE

• The example samples Concise, Ethical, and Specific principles, with Ethical randomly negated, and uses the principle with the largest preference-score difference to set the label.Source: G SYNTHETIC PREFERENCE CALIBRATION: AN… · p. 32
• In the example, Response A scores 2, 3, and 6, while Response B scores 1, 5, and 5 on Concise, Ethical, and Specific, respectively.Source: G SYNTHETIC PREFERENCE CALIBRATION: AN… · p. 32
• The reviewer rubric evaluates whether a response addresses the task efficiently and clearly, avoids harmful or discriminatory content, and stays pertinent to the instruction.Source: G SYNTHETIC PREFERENCE CALIBRATION: AN… · p. 32
• During training, the reward model is taught to score Response A higher because it exceeds Response B by 2 points on the negative-Ethical principle.Source: G SYNTHETIC PREFERENCE CALIBRATION: AN… · p. 32

H DESCRIPTION OF BASELINE MODELS

• LLaMA-2 provides base models from 7 billion to 70 billion parameters trained for next-word prediction; LLaMA-2-Chat uses SFT and extensive human preference annotations before PPO.The LLaMA-2 comparison uses the same prompt as Dromedary-2.Source: H DESCRIPTION OF BASELINE MODELS · p. 32
• Text-Davinci-003 builds on InstructGPT, while GPT-3.5 and GPT-4 are conversational models fine-tuned with SFT and RLHF.Source: H DESCRIPTION OF BASELINE MODELS · pp. 32–33
• Vicuna is fine-tuned from LLaMA on about 70,000 ShareGPT conversations, with later versions trained on about 125,000 conversations.Source: H DESCRIPTION OF BASELINE MODELS · p. 33
• OpenAssistant was trained on 161,443 messages across 66,497 conversation trees in 35 languages, while Guanaco uses 9,846 highest-rated paths.Source: H DESCRIPTION OF BASELINE MODELS · p. 33
• Dolly-V2 is based on Pythia-12b and fine-tuned on 15,000 human-generated prompt-response pairs collected among Databricks employees.Source: H DESCRIPTION OF BASELINE MODELS · p. 33

I DETAILS ON IMPLEMENTATIONS AND HYPERPARAMETERS

• The PPO implementation uses QLoRA rank 64 for attention and feed-forward modules, batch-wide advantage normalization, and a value model initialized from the reward model.The PPO implementation follows Dubois et al. (2023), using a variant of Ouyang et al. (2022).Source: I DETAILS ON IMPLEMENTATIONS AND… · p. 33
• PPO uses batches of 576 rollouts per step, arranged as two 288-rollout epochs, with a 2 × 10^-5 peak learning rate, cosine decay, and a gradient-norm limit of 1.Training spans two rounds of held-out RL data, though best results are usually achieved around 100–200 PPO steps; GAE λ and γ are 1, and the KL coefficient is 0.02.Source: I DETAILS ON IMPLEMENTATIONS AND… · p. 33
• SALMON adds symbolic rewards for matching a multilingual prompt’s language and for response length, addressing language mismatches and the synthetic preference model’s occasional difficulty identifying the more helpful response.The length penalty is response tokens divided by the 1024-token maximum, multiplied by its coefficient; the length bonus coefficient is 5.0 for general questions and −2.0 for reasoning questions.Source: I DETAILS ON IMPLEMENTATIONS AND… · pp. 33–34

J IMPROVED PROMPT FOR SELF-ALIGN

• The improved SELF-ALIGN prompt adds a FastChat exemplar to the original 5-shot prompt, encouraging overview-specifics-summary responses and improving base LLaMA-2 performance without verbose cloning or inference-time examples.The added response structure follows Gudibande et al. (2023), and the starting prompt comes from SELF-ALIGN (Sun et al., 2023b).Source: J IMPROVED PROMPT FOR SELF-ALIGN · p. 35
• The prompt’s 17 principles cover safety, helpfulness, rigorous reasoning, candor, knowledge grounding, clarification, dated knowledge, step-by-step explanations, balanced perspectives, creativity, operational tasks, and multilingual responses.The principles also call for accurate and informative answers, assessing questions, numerical sensitivity, and thorough multi-aspect responses.Source: J IMPROVED PROMPT FOR SELF-ALIGN · pp. 35–36
• The listed principles call for concise, honest and accurate, ethical, natural, and fluent responses.They emphasize brevity, factuality and disclosure of limitations, avoidance of harmful content, and smooth language.Source: J IMPROVED PROMPT FOR SELF-ALIGN · p. 39
• The remaining principles emphasize specificity, engaging educational content, methodical explanations, and multilingual replies.Responses should address the particular query, use relevant information, explain solutions step by step, and reply in the user's language.Source: J IMPROVED PROMPT FOR SELF-ALIGN · p. 39
• Tables 7–11 provide full principle lists for synthetic preference modeling, RL training, and analyses of harmlessness, honesty, and false refusal.Tables 9, 10, and 11 cover the respective harmlessness, honesty, and false-refusal analysis studies.Source: Table 7 · p. 39, Table 8 · p. 40, Table 9 · p. 41