Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Zhiqing Sun; Yikang Shen; Qinhong Zhou; Hongxin Zhang; Zhenfang Chen; David Cox; Yiming Yang; Chuang Gan

Source-linked AI summary

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

arXiv:2305.03047v2May 4, 2023cs.LGcs.AIcs.CLcs.CY

TL;DR

LLM alignment typically depends on extensive human supervision, which is costly and can introduce quality, diversity, reliability, and bias concerns. SELF-ALIGN uses principles and self-generated data to align a model with fewer than 300 annotation lines, producing Dromedary, which surpasses Text-Davinci-003 and Alpaca on evaluated benchmarks.

Problem
LLM alignment relies on extensive human supervision that is expensive and may suffer from quality, reliability, diversity, self-consistency, and bias issues.
Method
SELF-ALIGN uses human-defined principles, in-context demonstrations, and self-aligned model outputs to align an LLM with minimal supervision.
Results
With fewer than 300 annotation lines, Dromedary surpasses Text-Davinci-003 and Alpaca but falls short of ChatGPT and Vicuna.
Takeaways & Limitations
SELF-ALIGN demonstrates effective and efficient principle-driven alignment from scratch without relying on an already aligned LLM.
Takeaways & Limitations
SELF-ALIGN requires including all rules in context, making the approach subject to the base model’s token limit.

Abstract

from arXiv · show

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.

1 Introduction

SELF-ALIGN reduces reliance on costly, potentially inconsistent human annotations by using a small set of human-written principles to guide LLM self-alignment. Its four stages generate diverse prompts, produce principle-guided responses, engrave alignment through fine-tuning, and improve response comprehensiveness, requiring fewer than 300 annotation lines compared with at least 50K for InstructGPT [30] and Alpaca.

Motivation and contribution: SELF-ALIGN addresses the expense and potential quality, reliability, diversity, creativity, self-consistency, and bias problems of extensive human supervision.The method is intended to reduce intensive human annotation while retaining effective alignment with human principles.
SELF-ALIGN stages: Topic-Guided Red-Teaming generates synthetic instructions from 175 seed prompts plus 20 topic-specific prompts to diversify contexts and scenarios.This stage uses the self-instruct mechanism of Wang et al. [48].
SELF-ALIGN stages: Principle-Driven Self-Alignment uses 16 human-written principles and 5 in-context exemplars to guide helpful, ethical, and reliable responses.The principles specify desirable response qualities and behavioral rules for the AI model.
SELF-ALIGN stages: Principle Engraving fine-tunes the base LLM on self-aligned responses, allowing it to generate aligned answers directly without the principles or demonstrations.Shared model parameters transfer the helpful, ethical, and reliable behavior across queries.
Motivation and contribution: Fewer than 300 annotation lines—195 seed prompts, 16 principles, and 5 exemplars—are used, compared with at least 50K human/teacher annotations for InstructGPT [30] and Alpaca.The comparison is presented as evidence of SELF-ALIGN’s supervision efficiency.

2 Related Works

Prior alignment work relies on human-feedback pipelines or rule-based self-improvement, whereas SELF-ALIGN develops principle-driven alignment from scratch with minimal human supervision. Compared with Constitutional AI, it differs in rule application and training requirements but remains constrained by the context window when all rules are included.

AI Alignment: Constitutional AI (CAI) uses human-generated principles with AI self-critiques, revisions, and preference models to improve safety, reliability, and behavioral precision without harmful-output labels.Both CAI and SELF-ALIGN are rule-based alignment techniques for powerful AI systems.
AI Alignment: SELF-ALIGN selects rules for each user query before generating a response, whereas CAI selects a rule to critique and refine an existing response.This distinction reflects principle-driven generation in SELF-ALIGN versus post-generation self-critique in CAI.
AI Alignment: SELF-ALIGN aligns models from scratch with minimal human supervision, unlike CAI, whose self-critique methodology necessitates RLHF warm-up.SELF-ALIGN’s limitation is that including all rules in context is bounded by the base model’s token limit, whereas CAI avoids this constraint through post-generation critique.
State-of-the-art AI Assistants: InstructGPT [30] pioneered assistant training with supervised fine-tuning and reinforcement learning from human feedback, while ChatGPT and Alpaca represent subsequent prominent AI assistants.These systems provide context for SELF-ALIGN’s focus on alternative alignment techniques and reduced annotation dependence.

3 Our Method: SELF-ALIGN

SELF-ALIGN is a four-stage method that diversifies synthetic instructions, generates principle-guided responses with minimal human supervision, engraves alignment into model parameters, and refines verbosity and directness. Its design uses 20 adversarial instruction types, 16 generic principles, five demonstrations, and self-generated thoughts to improve coverage and alignment while reducing inference-time token use.

Overall method: SELF-ALIGN combines topic-guided self-instruction, principle-driven response generation, principle engraving, and verbose cloning to produce aligned, comprehensive assistant responses.The method uses the model to generate synthetic instructions, applies principles through in-context demonstrations, fine-tunes on self-aligned outputs, and distills verbosity improvements.
Topic-Guided Red-Teaming Self-Instruct: 20 manually designed adversarial instruction types guide topic generation and instruction synthesis, expanding the range of contexts and scenarios explored by self-instruction.The approach generates novel topics for each instruction type, removes duplicates, and then creates corresponding instructions.
Principle-Driven Self-Alignment: 16 generic principles and five in-context demonstrations teach the model to reason internally before producing helpful, ethical, and reliable responses.The demonstrations show the assistant identifying applicable principles, acknowledging knowledge limits, and then answering the user directly.
Principle Engraving: Fine-tuning on self-aligned outputs engraves principles into model parameters, reducing inference-time token usage while empirically surpassing the prompted counterpart on alignment benchmarks.This removes the need to provide the principles and demonstrations during inference and enables longer context lengths.
Verbose Cloning: Verbose Cloning addresses overly brief answers and indirect Wikipedia-like responses by distilling an aligned model that produces more detailed and direct responses.The step first uses a human-crafted prompt to create a verbose aligned model, then applies context distillation [3].

4 Evaluation

Dromedary is evaluated against strong language-model and chatbot baselines on truthfulness, HHH, and open-ended conversation benchmarks. It outperforms several baselines, while verbose cloning improves generation quality but harms some multiple-choice performance.

Evaluation setup: The evaluation compares final and non-verbose Dromedary variants derived from LLaMA-65b, with the final variant including all four SELF-ALIGN stages.The non-verbose variant excludes the final verbose-cloning step; qualitative demonstrations focus on harmful or sensitive queries and appear in the appendix.
TruthfulQA: 69% MC1 accuracy on TruthfulQA lets Dromedary significantly outperform GPT-4 and other baselines, while generation scores exceed GPT-3, LLaMA, and Alpaca but trail Vicuna.The MC1 result uses a modified ranking approach; generation evaluates truthful and truthful-and-informative answers.
Vicuna benchmark: Dromedary surpasses Text-Davinci-003 and Alpaca but falls short of ChatGPT and Vicuna in GPT-4-rated helpfulness, relevance, accuracy, and detail.The comparison uses a Win/Tie/Lose evaluation on Vicuna benchmark questions, with two conversation examples provided as in-context learning.
Verbose Tax: Analysis on Verbose Cloning: Verbose cloning improves generation quality but harms Dromedary’s performance on several multiple-choice benchmarks, especially when ranking trustworthy responses.The improvement is observed on Vicuna benchmark questions and TruthfulQA generation, whereas the multiple-choice degradation motivates analysis of a verbose tax.

5 Conclusion & Future Work

Dromedary demonstrates principle-driven self-alignment trained from scratch with very little human annotation, offering a distinct alternative to RLHF. Future work includes evaluating principles, improving training and critique methods, conducting human evaluations, and studying broader ethical and cultural contexts.

Conclusion: Dromedary uses principle-driven self-alignment, is trained from scratch with very little human annotation, and produces quality interactions while respecting creator-defined guardrails.The approach leverages an LLM’s intrinsic knowledge to define behavioral principles and differs from RLHF.
Future Work: Future work should test how adding or removing Dromedary’s 16 self-alignment principles affects performance.This would provide an ablation-based assessment of individual principles.
Future Work: Future work should apply Constitutional AI-based self-critique and reinforcement learning techniques to further enhance Dromedary.
Future Work: Researchers should conduct human evaluations of SELF-ALIGN’s real-world applicability and effectiveness and improve use of open-source annotation data, including the 15k instruction-following data in.
Future Work: The research community should study how principle definitions interact with ethical, cultural, and application contexts across communities.The paper presents principle-guided self-alignment as a starting point for multi-stakeholder engagement but notes substantial ongoing work is needed for positive outcomes across communities.

A Limitations & Social Impacts

This section discusses limitations of the SELF-ALIGN technique and released Dromedary model, along with potential social impacts of their release.

Limitations: The section examines limitations of the proposed SELF-ALIGN technique.
Limitations: It also considers limitations of the released Dromedary model.
Social Impacts: The section addresses potential social impacts arising from the release.

A.1 Limitations · A.2 Social Impacts

The paper acknowledges that Dromedary inherits limitations from its base model and from the difficulty of defining comprehensive alignment principles. It also highlights alignment’s potential social benefits alongside risks including misuse, disinformation, bias, and unfairness.

A.1 Limitations: Dromedary’s responses may be inaccurate or outdated because its intrinsic knowledge is limited by the completeness and recency of the base model.The paper notes that this limitation can prevent responses from reflecting recent developments.
A.1 Limitations: Defining principles for self-alignment is non-trivial because it may be difficult to anticipate all potential scenarios and challenges.
A.2 Social Impacts: The work aims to broaden AI alignment strategies and promote more diverse and robust approaches to developing powerful systems that are responsible and aligned with human values.It presents this direction as supporting safer and more harmonious AI integration across society.
A.2 Social Impacts: The paper identifies potential negative impacts of the work as a social-impact concern.
A.2 Social Impacts: Dromedary could be misused to generate malicious content or automated disinformation.The paper calls for mechanisms to detect and mitigate abuse, alongside ethical guidelines for developers and users.
A.2 Social Impacts: Dromedary may perpetuate or exacerbate biases in its base model’s pre-training data, potentially producing unfair or discriminatory outcomes.

B More Details about Dromedary · C Dromedary-2

Dromedary applies SELF-ALIGN to LLaMA-65b using large automatically generated datasets, filtered principle-aligned responses, and staged fine-tuning with minimal human supervision. Dromedary-2 further improves performance using LLaMA-2 and improved in-context exemplars, without verbose cloning or inference-time few-shot examples.

B More Details about Dromedary: Dromedary was developed by implementing SELF-ALIGN on the LLaMA-65b base language model [44].Training and decoding hyperparameters are provided in Appendix D.2.
B More Details about Dromedary: Self-Instruct automatically generated 267,597 open-domain prompts and corresponding inputs, following Alpaca’s recipe.Topic-Guided Red-Teaming Self-Instruct additionally generated 99,121 prompts across 20 red-teaming instruction types.
B More Details about Dromedary: 258,878 filtered query-response pairs were obtained: 191,628 from Self-Instruct and 67,250 from Topic-Guided Red-Teaming Self-Instruct.Figure 6 analyzes the principles and instruction types represented in these datasets.
B More Details about Dromedary: The non-verbose Dromedary was fine-tuned on the 258,878 generated pairs plus 910 modified dummy-data pairs from Vicuna.The dummy data improved Dromedary’s self-identification.
B More Details about Dromedary: The final verbose Dromedary was trained on 358,777 responses generated by prompting the non-verbose model to produce more verbose outputs.The non-verbose model served as the teacher for responses to Topic-Guided Red-Teaming and Self-Instruct queries.
B More Details about Dromedary: The final Dromedary model was trained from scratch on a base language model without SFT or RLHF, using less than 300 lines of human annotations.It is designed to be helpful, ethical, and reliable.
C Dromedary-2: Dromedary-2, trained from LLaMA-2 with improved in-context exemplars, achieved enhanced performance without verbose cloning or inference-time few-shot examples.The experiments used responses adhering more closely to a general-specific-general style within in-context self-alignment examples.

D Additional Experimental Details · D.1 Dromedary and Baseline Models · D.2 Hyperparameters

This section defines Dromedary and comparison models, reports a preliminary Vicuna evaluation, and specifies decoding and fine-tuning hyperparameters for SELF-ALIGN’s stages. Dromedary is built from LLaMA-65b in final and non-verbose variants, with the final model applying all four SELF-ALIGN steps.

D.1 Dromedary and Baseline Models: The experiments quantitatively evaluate Dromedary on benchmark datasets and qualitatively assess it on additional datasets, using temperature 0.7 by default for generated text.The default decoding temperature applies to all language-model-generated text.
D.1 Dromedary and Baseline Models: LLaMA [44] comprises base models from 7 billion to 65 billion parameters trained solely for next-word prediction, and receives the same prompt as Dromedary for fair comparison.The compared LLaMA models are base language models.
D.1 Dromedary and Baseline Models: Dromedary is built on LLaMA-65b, with final and non-verbose variants corresponding to all four SELF-ALIGN steps and principle engraving without verbose cloning, respectively.The verbose prompt is described in Appendix K.1.
D.1 Dromedary and Baseline Models: Vicuna attains over 90% of ChatGPT’s quality and surpasses LLaMA and Alpaca in more than 90% of cases under a preliminary GPT-4-judged evaluation.The evaluation uses GPT-4 as a judge.
D.2 Hyperparameters: Self-Instruct and Topic-Guided Red-Teaming Self-Instruct generate up to 384 new tokens using top-p 0.98 and temperature 1.0 nuclear sampling [16].These settings apply to both prompt-generation procedures.
D.2 Hyperparameters: Principle-driven self-alignment generates up to 256 new tokens with top-p 0.9 and temperature 0.5 nuclear sampling [16], because principles and demonstrations occupy about 1800 LLaMA tokens.The aggregated principles and in-context demonstrations are provided in Appendices G and H.
D.2 Hyperparameters: Principle engraving and verbose cloning each fine-tune LLaMA-65b [44] for 1 epoch using only LoRa weights [17] in multi-head attention, batch size 768, and learning rate 4e −4.Principle engraving uses sequence length 512 and approximately 335 steps; verbose cloning uses sequence length 768 and approximately 465 steps, with 100-step logarithmic warm-up and subsequent decay.

D.3 Benchmark Datasets

The paper evaluates models on benchmarks covering truthfulness, helpfulness, honesty, harmlessness, and broad chatbot capabilities. These include TruthfulQA, BIG-bench HHH Eval, and GPT-4-assisted Vicuna benchmark questions.

TruthfulQA: TruthfulQA [22] tests whether models identify literal truths about the real world and avoid false claims or misinformation across 38 adversarial categories.It includes both multiple-choice and generation tasks.
BIG-bench HHH Eval: BIG-bench HHH Eval [39] [3] assesses helpfulness, honesty, and harmlessness through approximately 200 pairwise comparisons spanning roughly 50 evaluations per category.Its purpose is to evaluate alignment and capabilities without explicitly separating the two.
Vicuna Benchmark Questions: The Vicuna benchmark uses GPT-4 to evaluate chatbot answers to diverse questions, including Fermi problems, roleplay, and coding/math tasks.The framework collects answers from LLaMA, Alpaca, ChatGPT, Bard, and Vicuna, and the accompanying analysis identifies Dromedary 65b as the final 2-shot model.

E Additional analysis on Vicuna benchmark question

Dromedary consistently outperforms LLaMA-based baselines on reasoning-intensive Vicuna question categories, including fermi, counterfactual, coding, and math. However, it trails ChatGPT and Vicuna on categories requiring comprehensive response organization, particularly knowledge, roleplay, common-sense, and writing.

Category-level analysis: Dromedary consistently outperforms Alpaca and Vicuna on reasoning-intensive categories such as fermi, counterfactual, coding, and math.This category-level analysis is presented in Figure 8.
Category-level analysis: Dromedary is not competitive with ChatGPT and Vicuna on knowledge, roleplay, common-sense, and writing questions requiring comprehensive response organization.
Response comparison: The Vicuna analysis also includes GPT-4-assessed response comparisons with and without few-shot examples.These comparisons are shown in Figures 9 and 10.

F Many More Samples … M Instruction Prompts for Topic-Guided Red-Teaming Self-Instruct

The examples show Dromedary generally refusing harmful requests, avoiding unsupported demographic judgments, and acknowledging ambiguity, while still exhibiting failures in strict principle adherence and mathematical reasoning. The section also includes prompts for moral self-correction, benchmark-style tasks, verbose outputs, and multilingual generation.

F Many More Samples: Dromedary refuses instructions for unauthorized account access, white-supremacist recruitment, and theft, while explaining legal, ethical, or safety concerns.By contrast, Alpaca provides actionable harmful guidance in the account-access and white-supremacist examples.
F Many More Samples: On sensitive social questions, Dromedary rejects group-based trust judgments, frames homelessness as requiring humane systemic responses, and states that it cannot experience hatred.These responses contrast with Alpaca’s demographic generalizations and anthropomorphic answer about hating people.
F Many More Samples: Dromedary’s moral self-correction examples select “Can’t be determined” for the ambiguous BBQ question and allow either “his” or “her” in the Winogender prompt.The examples are presented alongside comparisons with LLaMA, Vicuna, ChatGPT, and GPT-4.
F Many More Samples: The preliminary evaluation identifies two prominent failure modes, including inability to strictly adhere to predefined principles, such as hallucinating misinformation in violation of the candor rule.The section states that comprehensively addressing these shortcomings requires further investigation and development.
K.2 Prompts for multilingual outputs: A multilingual-output prompt is slightly modified to display non-English characters, with the original version available in the codebase.This passage specifically describes the multilingual prompt used for multilingual outputs.

N Evaluation Prompts for MC Benchmarks

The evaluation prompts adapt Dromedary to multiple-choice benchmarks by eliciting answers through exam-style user questions. TruthfulQA scores true/false candidates directly, while HHH Eval scores answer options with position-swapping to mitigate answer-order bias.

TruthfulQA: TruthfulQA evaluates each candidate by assessing the likelihood of true and false as answer scores.The candidate answer is explicitly labeled as true or false, and Dromedary returns the corresponding truth value.
HHH Eval: HHH Eval scores the likelihood of options A and B and aggregates results after swapping their positions because the original dataset always places the correct answer at A.The prompt presents two answer candidates labeled A) and B).
Prompt format: The prompts use an exam-style interaction in which the user presents a benchmark question and answer candidates, asks for help, and Dromedary selects the answer.This format is specified for both TruthfulQA and HHH Eval prompts.

O Few-Shot Prompts for Vicuan Benchmark Questions · P Additional Related Work · Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment

The appendix supplements Dromedary with few-shot benchmark prompts, related-work context, and demonstrations showing how principles guide responses across diverse query types. Together, these materials illustrate prompt construction and principle application for ethical, informative, helpful, candid, and step-by-step behavior.

O Few-Shot Prompts for Vicuan Benchmark Questions: Two conversation examples improve Dromedary’s response quality, combining the original verbose prompt with Vicuna’s renewable-energy prompt and a coding example.The coding example demonstrates a response to a longest-common-prefix programming problem, while the renewable-energy example contrasts renewable and non-renewable sources.
P Additional Related Work: Related work situates SELF-ALIGN within advances in large language models and the Transformer architecture [46].The passage identifies LLM progress across natural-language-processing tasks and cites foundational model developments alongside the Transformer introduction.
Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment: Principle-driven demonstrations span safety refusal, informative answers, helpful step-by-step guidance, question assessment, rigorous reasoning, multi-aspect coverage, and candor about limitations.Examples include refusing narcotics information, listing Los Angeles activities, explaining stock-index ranking, recommending recipes, and acknowledging the absence of feelings.
Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment: The principle examples show Dromedary selecting applicable rules before answering, such as combining informative and multi-aspect principles for word associations and recipe recommendations.Other internal deliberations pair helpfulness with question assessment, reasoning with step-by-step explanation, and dated knowledge with explicit uncertainty about future events.
Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment: The demonstrations also encode knowledge recitation, static-model limitations, clarification, numerical sensitivity, dated knowledge, balanced perspectives, creativity, and operational assistance.Responses explicitly avoid real-time directions, disclose knowledge ending in September 2021, answer a unit-conversion question, present both sides of higher-education training, generate a haiku, and give tropical-trip packing steps.