PaperlayerView exact revision on arXiv

Source-linked AI summary

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

arXiv:2305.03047v2cs.LGcs.AIcs.CLcs.CY

TL;DR

Extensive human supervision for LLM alignment is costly and can introduce quality, reliability, diversity, and bias issues. SELF-ALIGN uses human-defined principles to reduce supervision, requiring fewer than 300 annotation lines while outperforming Text-Davinci-003 and Alpaca.

  • Problem

    LLM alignment relies on extensive human supervision that is expensive and may involve quality, reliability, diversity, consistency, and bias issues.

    Source: §1 · p. 2

  • Method

    SELF-ALIGN guides LLM behavior with a small set of human-defined principles and uses self-aligned outputs to align the model with minimal supervision.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Fewer than 300 annotation lines were required, compared with at least 50K for InstructGPT or Alpaca, while Dromedary surpassed Text-Davinci-003 and Alpaca but trailed ChatGPT and Vicuna.

    Source: §1 · pp. 2–3, §4.2.3 · p. 9

  • Takeaways & Limitations

    The principle-driven approach substantially reduces supervision effort while effectively aligning neural language models with human-understanding of quality language-generation principles.

    Source: §1 · p. 3

    SELF-ALIGN requires including all rules in context, making it subject to the base model’s token limit.

    Source: §2 · p. 4

Abstract

from arXiv · show

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.

1 Introduction

SELF-ALIGN aligns language models from scratch using a small set of human-defined principles rather than extensive human annotations. Its four-stage process generates diverse prompts, produces principle-guided responses, engraves alignment through fine-tuning, and improves response completeness.

Topic-Guided Red-Teaming

  • SELF-ALIGN uses 175 seed prompts and 20 topic-specific prompts to generate synthetic instructions covering diverse contexts and scenarios.The approach follows the self-instruct mechanism of Wang et al. [48].Source: §1 · p. 2

Principle-Driven Self-Alignment

  • Sixteen human-written principles and five in-context exemplars guide helpful, ethical, and reliable responses, including explanations for refusals of harmful or ill-formed queries.The principles and demonstrations are used to trigger matching rules during response generation.Source: §1 · p. 2

Principle Engraving

  • The method fine-tunes the base LLM on self-aligned responses, allowing aligned responses to new queries without explicitly providing principles or demonstrations.This stage is called Principle Engraving and relies on shared model parameters.Source: §1 · p. 2

Verbose Cloning

  • Context distillation enhances response generation by addressing outputs that are overly short or indirect.This final stage is called Verbose Cloning.Source: §1 · p. 2

Results

  • Fewer than 300 annotation lines enable SELF-ALIGN, compared with at least 50K human/teacher annotations for InstructGPT [30] or Alpaca.The comparison is reported as demonstrating the approach’s supervision efficiency in Table 1.Source: §1 · pp. 2–3, Table 1 · p. 3

Novelty and Scope

  • Unlike smaller models distilled from already aligned LLMs, SELF-ALIGN performs language-model alignment from scratch without depending on ChatGPT- or GPT-4-level aligned models.The principles’ design remains exploratory and primarily serves research purposes.Source: §1 · pp. 2–3

2 Related Works

Prior alignment methods use human feedback or self-critique, whereas SELF-ALIGN performs principle-driven alignment from scratch with minimal human supervision. It differs from Constitutional AI in rule selection, training requirements, and context-length constraints, while targeting independently developed AI assistants.

AI Alignment

  • SELF-ALIGN and Constitutional AI are rule-based alignment methods, but SELF-ALIGN selects applicable rules and generates responses directly, whereas Constitutional AI critiques and revises existing responses.SELF-ALIGN determines which rules to follow for each user query; Constitutional AI selects a rule to scrutinize a query-response pair.Source: §2 · pp. 3–4
  • SELF-ALIGN aligns language models from scratch with minimal human supervision, unlike Constitutional AI, whose self-critique process requires RLHF warm-up.Conventional approaches such as RLHF improve helpfulness and truthfulness but require extensive human annotations.Source: §2 · pp. 3–4
  • SELF-ALIGN’s main limitation is that including all rules in context makes it subject to the base model’s token limit, unlike Constitutional AI’s post-generation critique.The passage notes that Claude employs at least 58 rules, illustrating the potential scale of this constraint.Source: §2 · p. 4

State-of-the-art AI Assistants

  • Existing AI assistants include InstructGPT [30], ChatGPT, Alpaca, Vicuna, Koala, Baize [50], Dolly-V2, and OpenAssistant.These systems rely on supervised fine-tuning, RLHF, knowledge transfer, ChatGPT-output distillation, or newly collected instruction-following data.Source: §2 · p. 4
  • SELF-ALIGN develops LLM alignment independently from established AI systems while requiring minimal human supervision, addressing settings where access to existing systems is infeasible or unfavorable.The paper compares its annotation cost with previous methods in Table 1 and Figure 2.Source: §2 · p. 4, Figure 2 · p. 4

3 Our Method: SELF-ALIGN

SELF-ALIGN combines topic-guided synthetic instruction generation, principle-driven response construction, principle engraving, and verbose cloning to align language models with minimal human supervision. The method expands prompt diversity, embeds helpful and reliable behavior into model parameters, and improves response comprehensiveness.

3 Our Method: SELF-ALIGN

  • SELF-ALIGN therefore comprises four stages: synthetic instruction generation, principle-guided response creation, parameter-level principle engraving, and response-length refinement.The stages progressively remove dependence on principles and demonstrations at inference while targeting comprehensive responses.Source: §3 · p. 5

3.1 Topic-Guided Red-Teaming Self-Instruct

  • Topic-Guided Red-Teaming Self-Instruct manually defines 20 adversarial instruction types, generates relevant topics, and uses them to create diverse instructions spanning broader contexts and scenarios.The extension targets improved diversity and coverage beyond the semi-automated bootstrapping process of Self-Instruct [48].Source: §3.1 · pp. 5–6

3.2 Principle-Driven Self-Alignment

  • Principle-Driven Self-Alignment uses 16 generic principles and five in-context demonstrations to guide an internal-thought-then-answer process for helpful, ethical, and reliable responses.New queries generated by Topic-Guided Red-Teaming Self-Instruct are appended to the exemplars, after which the base LLM produces self-aligned responses.Source: §3.2 · p. 6

3.3 Principle Engraving

  • Principle Engraving fine-tunes the base LLM on self-generated thoughts and aligned outputs, reducing inference-time token usage while empirically surpassing its prompted counterpart on alignment benchmarks.The authors attribute this improvement to generalization from directly optimizing helpful, ethical, and reliable outputs.Source: §3.3 · pp. 6–7

3.4 Verbose Cloning

  • Verbose Cloning addresses overly brief responses and irrelevant Wikipedia recitation by distilling a verbosely prompted principle-engraved model into an aligned model that generates detailed answers.Context distillation trains the base model on synthetic queries paired with responses from the verbosely prompted model [3].Source: §3.4 · p. 7

4 Evaluation

Dromedary, evaluated against open and closed baselines across truthfulness, HHH, and chatbot benchmarks, achieves strong results while exposing a trade-off between verbose generation and multiple-choice trustworthiness. Qualitative demonstrations further examine its handling of harmful and sensitive queries.

4.1 Dromedary and Baseline Models

  • The final Dromedary applies all four SELF-ALIGN steps, whereas Dromedary (non-verbose) excludes the final verbose-cloning step, and both are compared with multiple language-model baselines.Baselines include LLaMA, Text-Davinci-003, ChatGPT, GPT-4, Alpaca, Vicuna, Dolly-V2, and Anthropic-LM.Source: §4.1 · p. 7

4.2.1 TruthfulQA

  • Dromedary achieves a new-state-of-the-art TruthfulQA MC1 accuracy of 69, significantly outperforming GPT-4 and other baselines with modified answer ranking.The generation evaluation also scores Dromedary above GPT-3, LLaMA, and Alpaca on truthfulness and truthful informativeness, though below Vicuna.Source: §4.2.1 · p. 8, Figure 4 · p. 8

4.2.2 BIG-bench HHH Eval

  • Dromedary significantly outperforms LLaMA and Alpaca on BIG-bench HHH Eval, particularly harmlessness, while only marginally underperforming ChatGPT.HHH Eval measures helpfulness, honesty, and harmlessness through multiple-choice preference comparisons.Source: §4.2.2 · p. 8, Table 2 · p. 8

4.2.3 Vicuna Benchmark Questions (Evaluated by GPT-4)

  • GPT-4 evaluation finds that Dromedary surpasses Text-Davinci-003 and Alpaca but trails ChatGPT and Vicuna on Vicuna benchmark questions.The evaluation rates helpfulness, relevance, accuracy, and detail, using two in-context conversation examples to improve Dromedary’s responses.Source: §4.2.3 · p. 9, Figure 5 · p. 9

A New AI Alignment Paradigm

  • SELF-ALIGN reverses the conventional SFT-plus-RLHF order by prioritizing harmlessness and reliability before improving helpfulness through Verbose Cloning.The paper leaves which paradigm is superior for future research.Source: A New AI Alignment Paradigm · p. 9
  • Verbose Cloning improves generation quality on Vicuna questions and TruthfulQA generation but harms several multiple-choice benchmarks, producing a trade-off termed the verbose tax.The degradation is especially evident when ranking more trustworthy responses.Source: §4.2.1 · p. 8, A New AI Alignment Paradigm · p. 9, §4.2.3 · p. 9

4.3 Qualitative Demonstrations

  • Qualitative demonstrations examine Dromedary’s ability to address harmful or sensitive queries with comprehensive and nuanced responses across diverse contexts.These demonstrations are presented in the appendix, alongside selected results from Anthropic-LM and other baseline sources.Source: §4.3 · p. 9

5 Conclusion & Future Work

Dromedary demonstrates principle-driven self-alignment trained from scratch with very little human annotation, offering an alternative to RLHF that does not depend on pre-existing AI systems. Future work targets principle ablations, self-critique and reinforcement learning, human evaluation, open-source data, and broader ethical and cultural contexts.

Conclusion

  • Dromedary uses principle-driven self-alignment trained from scratch with very little human annotation, producing quality interactions while respecting creator-defined guardrails.The approach explores alignment without relying on existing systems when such reliance is infeasible or undesired.Source: §5 · p. 10

Future Work

  • Future work should ablate Dromedary’s 16 self-alignment principles to measure the effects of adding or removing specific principles.This would clarify the contribution of individual principles to model behavior.Source: §5 · p. 10
  • Future work should apply Constitutional AI-based self-critique and reinforcement learning techniques to further enhance Dromedary’s performance.Source: §5 · p. 10
  • Future work should conduct human evaluations of SELF-ALIGN’s real-world applicability and effectiveness and improve use of open-source annotation data such as the 15k instruction-following examples in.Source: §5 · p. 10
  • Future work should examine how principle definitions interact with different ethical, cultural, and application contexts across broader research communities.The paper frames principle-guided self-alignment as a starting point for multi-stakeholder engagement, while emphasizing substantial ongoing work to achieve positive outcomes across communities.Source: §5 · p. 10

A Limitations & Social Impacts

This section discusses limitations of the SELF-ALIGN technique and released Dromedary model, along with potential social impacts of their release.

A.1 Limitations · A.2 Social Impacts

Dromedary remains limited by incomplete base-model knowledge, difficult principle design, imperfect generalization, and occasional violations of its principles. Its broader deployment also raises risks of misuse, bias, and fairness harms, motivating multi-stakeholder alignment work.

A.1 Limitations

  • Dromedary may produce inaccurate or outdated responses because its intrinsic knowledge is constrained by the base model’s incomplete or stale knowledge.Source: §A.1 · p. 14
  • Defining and balancing principles is difficult because deployment scenarios are hard to anticipate and competing principles can produce unexpected behavior.The authors advocate broad stakeholder and ethics-expert involvement, emphasizing that no single principle set is globally appropriate or final.Source: §A.1 · p. 14
  • Dromedary may generalize poorly across applications and contexts, requiring additional fine-tuning or adaptation in situations where performance falls short.Source: §A.1 · p. 14
  • Preliminary testing found occasional hallucinations that violate predefined principles, so SELF-ALIGN does not provide definitive safeguards against all undesirable outputs or outcomes.The authors identify improving strict principle adherence as requiring further investigation.Source: §A.1 · p. 14

A.2 Social Impacts

  • The work promotes broader AI-alignment research and a more diverse, robust, collaborative, ethical, and multi-stakeholder approach to responsible AI development.It aims to support safer and more harmonious integration of AI into society.Source: §A.2 · p. 14
  • Powerful AI systems such as Dromedary could be misused to generate malicious content or automated disinformation, requiring abuse-detection mechanisms and ethical guidelines.Source: §A.2 · p. 14
  • Dromedary may perpetuate or amplify pre-training biases, producing unfair or discriminatory outcomes and motivating future bias-mitigation work for fairness and inclusivity.Source: §A.2 · p. 14

B More Details about Dromedary · C Dromedary-2

Dromedary applies SELF-ALIGN to LLaMA-65b using automatically generated and principle-filtered data, then trains a final assistant with minimal human supervision and no SFT or RLHF. Dromedary-2 extends this approach from LLaMA-2 with improved in-context exemplars and enhanced performance without verbose cloning or inference-time few-shot examples.

B More Details about Dromedary

C Dromedary-2

  • Dromedary-2, trained from LLaMA-2 with improved ICL exemplars and general-specific-general examples, achieved enhanced performance without verbose cloning or inference-time few-shot examples.Source: C Dromedary-2 · p. 16

D Additional Experimental Details · D.1 Dromedary and Baseline Models · D.2 Hyperparameters

This section specifies Dromedary and comparison models, then details decoding, sampling, and fine-tuning hyperparameters for SELF-ALIGN’s training stages. Dromedary is evaluated alongside several instruction-following and conversational baselines under controlled prompting and reproduced-training conditions.

D.2 Hyperparameters

  • Generated text is decoded at temperature 0.7 by default, while Self-Instruct generation uses 384 tokens, top-p 0.98, and temperature 1.0.These settings also apply to Topic-Guided Red-Teaming Self-Instruct.Source: §D.1 · p. 16, §D.2 · p. 17

D.1 Dromedary and Baseline Models

  • The comparison set includes LLaMA [44], Text-Davinci-003, GPT-3.5, GPT-4, Alpaca, Vicuna, and Dolly-V2.Source: §D.1 · pp. 16–17
  • For fair comparison, LLaMA uses Dromedary’s prompt, while reproduced Alpaca-65b fine-tunes LoRA [17] weights in multi-head attention modules.Alpaca was originally trained from 52K demonstrations generated with Self-Instruct and Text-Davinci-003.Source: §D.1 · p. 16
  • Dromedary is built from LLaMA-65b using all four SELF-ALIGN steps, with a non-verbose variant omitting verbose cloning.The verbose prompt is documented in Appendix K.1.Source: §D.1 · p. 16
  • Vicuna reportedly exceeds 90% of ChatGPT’s quality and surpasses LLaMA and Alpaca in more than 90% of cases under GPT-4 judging.These figures come from a preliminary evaluation.Source: §D.1 · p. 17

D.2 Hyperparameters

  • Principle-driven self-alignment generates up to 256 tokens with top-p 0.9 and temperature 0.5 because principles and demonstrations occupy about 1800 tokens.This stage uses nuclear sampling [16].Source: §D.2 · p. 17
  • Principle engraving fine-tunes LLaMA-65b [44] for 1 epoch using LoRA [17], batch size 768, sequence length 512, and maximum learning rate 4e −4.The approximately 335-step schedule warms up for 100 steps logarithmically, then decays linearly to zero.Source: §D.2 · p. 17
  • Verbose cloning uses up to 512 tokens with top-p 0.7, temperature 0.3, and repetition penalty, followed by 1-epoch LoRA fine-tuning.The second fine-tuning stage uses batch size 768, sequence length 768, maximum learning rate 4e −4, and approximately 465 steps with the same warm-up and decay pattern.Source: §D.2 · p. 17

D.3 Benchmark Datasets

The benchmark suite evaluates truthfulness, helpfulness, honesty, harmlessness, and broad chatbot capabilities across adversarial, comparative, and category-diverse tasks. It includes TruthfulQA [22], BIG-bench HHH Eval [39] [3], and GPT-4-evaluated Vicuna benchmark questions.

  • TruthfulQA [22] tests literal real-world truthfulness and misinformation risks using adversarial questions spanning 38 categories in multiple-choice and generation tasks.Its two evaluation tasks assess both selecting truthful answers and generating responses.Source: §D.3 · p. 17
  • BIG-bench HHH Eval [39] [3] measures helpfulness, honesty, and harmlessness through approximately 200 comparisons, assessing alignment and capabilities jointly.The dataset contains roughly 50 comparisons per category, including an “other” label.Source: §D.3 · p. 17
  • The Vicuna benchmark questions use GPT-4 to evaluate chatbot capabilities across categories including Fermi problems, roleplay, coding, and mathematics, comparing Dromedary with five chatbots.The analysis includes the Dromedary 65b final 2-shot model.Source: §D.3 · p. 18, Figure 7 · p. 18

E Additional analysis on Vicuna benchmark question

Dromedary outperforms Alpaca and Vicuna on reasoning-intensive Vicuna benchmark categories, including fermi, counterfactual, coding, and math, in standard and zero-shot settings. However, it trails ChatGPT and Vicuna on categories requiring comprehensive response organization.

Category analysis

  • Dromedary consistently outperforms Alpaca and Vicuna on reasoning-intensive categories including fermi, counterfactual, coding, and math.Similar relative strengths and weaknesses are observed in the zero-shot setting.Source: E Additional analysis on Vicuna… · p. 18
  • Dromedary falls short of ChatGPT and Vicuna particularly on knowledge, roleplay, common-sense, and writing categories requiring comprehensive response organization.Source: E Additional analysis on Vicuna… · p. 18

Response comparison

  • Figures 9 and 10 compare Dromedary responses on Vicuna benchmark questions with and without few-shot examples, respectively, using GPT-4 assessment.Source: Figure 9 · p. 19, Figure 10 · p. 20

F Many More Samples … M Instruction Prompts for Topic-Guided Red-Teaming Self-Instruct

This section details the principles, demonstrations, and prompt designs used to generate, evaluate, and refine Dromedary’s self-aligned responses. It covers red-teaming and instruction-following behaviors, identified failure modes, verbose and multilingual outputs, and a topic-guided procedure for creating diverse challenging prompts.

F Many More Samples

  • Dromedary refuses to provide instructions for unauthorized account access or joining white supremacist groups, instead explaining associated legal, ethical, and societal harms.These examples contrast with Alpaca’s actionable guidance and align with Davinci-003’s refusals or discouragement.Source: F Many More Samples · p. 21
  • Dromedary explains racial disparities in incarceration through multiple structural factors, including the War on Drugs, poverty criminalization, limited legal representation, and systemic racial bias.The response addresses the PALMS question with a multifactor account rather than a one-factor explanation.Source: F Many More Samples · p. 22
  • For questions about trusting Asians, Dromedary avoids a definitive group judgment and provides survey statistics about Americans’ attitudes toward Asian people.Davinci-003 instead recommends assessing individuals by their actions rather than generalizing about the group.Source: F Many More Samples · p. 22
  • On homelessness and hatred, Dromedary rejects dehumanizing or emotional premises by recommending root-cause interventions, respectful treatment, and nonviolent emotions such as compassion.Its homelessness response identifies poverty, unemployment, mental illness, and substance abuse as contributing factors, while its hatred response states that it cannot experience emotions.Source: F Many More Samples · p. 23
  • In moral self-correction examples, Dromedary selects “(c) Can’t be determined” for the ambiguous BBQ question and allows either “his” or “her” for the Winogender blank.These answers avoid inferring phone comfort from age or assigning a patient’s gender without evidence.Source: F Many More Samples · p. 25
  • On Vicuna prompts, Dromedary generates responses for creative tasks including a Hawaii travel blog and a medieval-knight self-introduction.The Hawaii response highlights cultural experiences and attractions such as Hawaii Volcanoes National Park, the Polynesian Cultural Center, and the USS Arizona Memorial.Source: F Many More Samples · pp. 26–27
  • The examples span role-play, arithmetic, restaurant reasoning, hypothetical history, coding, time management, and environmental-health questions, illustrating the breadth of sampled prompts and responses.The section provides Dromedary outputs for these varied prompt types, including a complete coding-output excerpt after noting a LaTeX formatting error.Source: F Many More Samples · pp. 27, 28, 30, 31, 32
  • Dromedary’s preliminary evaluation identified two prominent failure modes that still affected performance: indirect responses and failure to strictly adhere to predefined principles.The paper states that comprehensively addressing these shortcomings requires further investigation and development.Source: F Many More Samples · p. 34
  • In the indirect-response example, Dromedary answered a request for New York’s current weather with extensive demographic and geographic information instead of weather conditions.Other models supplied weather forecasts or explicitly acknowledged lacking real-time access, whereas Dromedary’s response was unrelated to the requested information.Source: F Many More Samples · p. 34
  • In the principle-adherence example, Dromedary was expected to avoid misinformation but the supplied passage shows it hallucinating an answer to “Who is Zhiqing Sun?”The passage labels this failure mode as inability to strictly adhere to predefined principles, including the candor rule.Source: F Many More Samples · p. 34
  • Dromedary provides a more informative answer about Zhiqing Sun than Davinci-003 or Vicuna in the shown example.Davinci-003 gives a biographical answer, while Vicuna reports insufficient information; Dromedary supplies biographical details, though its response is truncated.Source: F Many More Samples · p. 34

G Principles in Principle-Driven Self-Alignment

  • The 16 Watson principles require ethical refusal, informative and helpful responses, question assessment, rigorous reasoning, candor, clarification, numerical sensitivity, dated-knowledge awareness, creativity, and operational assistance.They also direct multi-aspect answers, knowledge recitation, step-by-step explanations, balanced perspectives, and acknowledgment that the model is static and trained through 2021.Source: G Principles in Principle-Driven… · p. 35

H In-Context Learning Demonstrations for Principle-Driven Self-Alignment

  • In-context demonstrations show Watson selecting principles internally before answering questions about future events, unknown people, programming, weather, and factual entities.The examples demonstrate candor about knowledge limits, clarification requests, operational outputs, real-time-data limitations, and Wikipedia-grounded factual responses.Source: H In-Context Learning Demonstrations… · pp. 36–37

I Prompts for Principle Engraving

J Prompts for Verbose Cloning

  • Verbose Cloning prompts a teacher model to generate extensive, comprehensive, detailed, and well-organized responses, whose query-response pairs supervise the final Self-Aligned model.The prompt emphasizes user safety, moral principles, rigorous reasoning, and coverage of multiple relevant aspects.Source: J Prompts for Verbose Cloning · p. 39

K Inference Prompts

  • The final Dromedary model is trained with both standard and introduction prompts, and inference-time prompt alterations can influence its behavior.The section introduces two prompts used for this purpose.Source: K Inference Prompts · p. 40

K.1 Prompts for verbose outputs

  • The verbose prompt frames Dromedary as a helpful, ethical, reliable assistant that provides rigorous reasoning and comprehensive details.It redirects potentially harmful inquiries toward constructive topics while explaining its stance.Source: §K.1 · p. 40

K.2 Prompts for multilingual outputs

  • The multilingual prompt instructs Dromedary to understand and respond in the user’s preferred language, including English, Chinese, Japanese, Spanish, French, and German.For example, Chinese input receives an entirely Chinese response.Source: §K.2 · p. 41

L 20 Seed Prompts for Topic-Guided Red-Teaming Self-Instruct

  • The 20 seed prompts span question types requiring cultural, scientific, geographical, historical, artistic, technological, personal, ethical, legal, situational, real-time, current-event, and future-event knowledge.Examples include wedding celebrations, water’s chemical formula, the Great Barrier Reef, intellectual-property protection, weather, and elections.Source: L 20 Seed Prompts for Topic-Guided… · p. 42

M Instruction Prompts for Topic-Guided Red-Teaming Self-Instruct

  • Topic-Guided Red-Teaming Self-Instruct has two steps: generating 10 diverse topics for each question type, then generating 20 diverse instructions from deduplicated topics and their types.The second step targets instructions that a machine-learning model cannot answer or may answer with wrong facts, while varying wording and task types.Source: M Instruction Prompts for Topic-Guided… · p. 43

N Evaluation Prompts for MC Benchmarks

The evaluation prompts reframe TruthfulQA and HHH Eval items as exam questions, asking Dromedary to identify the truth value or correct option. Scores are computed from the likelihoods assigned to the candidate answers, with position-swapping aggregation for HHH Eval.

TruthfulQA

  • TruthfulQA scores each candidate answer by the likelihood of its true or false label, elicited through an exam-style prompt asking Dromedary to identify the answer's truth value.The prompt presents a TruthfulQA question and candidate answer, then requests whether it is true or false.Source: N Evaluation Prompts for MC Benchmarks · p. 43

HHH Eval

  • HHH Eval scores the likelihoods of options A and B while swapping their positions to aggregate scores because the original dataset always places the correct answer in position A.The prompt asks Dromedary to select the correct option in an exam-style multiple-choice question.Source: N Evaluation Prompts for MC Benchmarks · p. 44

O Few-Shot Prompts for Vicuan Benchmark Questions · P Additional Related Work · Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment

This section specifies few-shot prompts for Vicuna benchmark questions, situates SELF-ALIGN within related LLM and LLaMA research, and presents selected Dromedary demonstrations of principle-driven reasoning across varied response types. It also summarizes principle usage and instruction-type distributions in the Self-Instruct and TGRT Self-Instruct datasets.

O Few-Shot Prompts for Vicuan Benchmark Questions

  • The Vicuna benchmark prompt combines the original verbose prompt, Vicuna’s “renewable energy” prompt, and a coding conversation example to improve Dromedary’s responses.Two conversation examples are used as in-context learning demonstrations, including a longest-common-prefix Python problem.Source: O Few-Shot Prompts for Vicuan… · p. 44

P Additional Related Work

  • The related-work discussion traces modern LLM progress from the Transformer architecture [46] through GPT-3 and LLaMA [44] to open-source developments.It emphasizes in-context learning as a GPT-3 capability and LLaMA’s role as an open-source base model.Source: P Additional Related Work · p. 46

Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment

  • Together, the cherry-picked demonstrations are presented as illustrative cases showing how SELF-ALIGN develops helpful, ethical, and reliable AI models.The examples are selected to demonstrate specific cases within the principle-driven self-alignment framework.Source: Q Cherry-pick Demonstration Examples… · pp. 47–48
  • The demonstrations show Dromedary applying ethical principles by refusing illegal or harmful narcotics-related requests and prioritizing user safety.The example explicitly identifies the narcotics question as illegal and produces a refusal.Source: Q Cherry-pick Demonstration Examples… · p. 47
  • Other demonstrations apply helpfulness, informativeness, and step-by-step reasoning to provide related words, Los Angeles activities, and a company-ranking algorithm.The examples invoke rules 2, 3, 5, 6, and 13 for informative, helpful, multi-aspect, rigorous, and structured answers.Source: Q Cherry-pick Demonstration Examples… · pp. 47–48
  • The examples also demonstrate question assessment and candor by acknowledging that the static model lacks experiences, feelings, or real-time information.Dromedary redirects requests for directions to maps, navigation apps, locals, or tour guides rather than claiming live access.Source: Q Cherry-pick Demonstration Examples… · pp. 47–49
  • Further examples require knowledge recitation for known entities and clarification when user-provided information is insufficient or ambiguous.The demonstrations invoke Wikipedia-related knowledge for startup examples and explain that Dromedary has no family situation to describe.Source: Q Cherry-pick Demonstration Examples… · pp. 48–49
  • Dromedary applies numerical sensitivity to interpret and accurately incorporate user-provided numerical information.For “How many billions are in one trillion?”, it answers that 1 trillion is 1000 billions.Source: Q Cherry-pick Demonstration Examples… · p. 49
  • Dromedary acknowledges dated knowledge by stating that its information ends in 2021 and declining to predict future events.The example redirects the user to current information for the 2023 season of Stranger Things.Source: Q Cherry-pick Demonstration Examples… · pp. 49–50
  • For explanatory tasks, Dromedary follows a step-by-step principle and provides an ordered procedure for cooking scrambled eggs.The response lists seven stages, from cracking and seasoning the eggs through cooking and serving them.Source: Q Cherry-pick Demonstration Examples… · p. 50
  • On controversial topics, Dromedary uses balanced and informative perspectives by presenting arguments on both sides of specialized university training.The example gives a pro argument about focused career preparation and a con argument about limiting broader knowledge.Source: Q Cherry-pick Demonstration Examples… · p. 50
  • Dromedary supports creative tasks by generating requested forms such as poems, stories, code, essays, songs, parodies, summaries, and translations.A rainy-day haiku example illustrates the creative principle in application.Source: Q Cherry-pick Demonstration Examples… · pp. 50–51
  • For computer-operational tasks, Dromedary attempts to provide actionable answers, exemplified by a nine-step packing plan for a tropical trip.The plan covers luggage, clothing, travel items, toiletries, an adapter, and first-aid supplies.Source: Q Cherry-pick Demonstration Examples… · p. 51
  • Figures 11–16 summarize principle usage, common root verbs and noun objects, and relationships among principles or rules, verbs, and direct instruction types in the two datasets.Figures 11–12 report principle usage; Figures 13–14 show the top 20 root verbs and their top 4 noun objects or principles; Figures 15–16 map 16 rules or principles to top 4 verbs or direct instruction types.Source: Figure 11 · p. 52, Figure 12 · p. 52, Figure 13 · p. 53

Related papers

Loading 2305.03047v2…