PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Alignment: A Survey

Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, Deyi Xiong

arXiv:2309.15025v1cs.CLcs.AI

TL;DR

LLMs’ growing capabilities create risks because their outputs may conflict with human values and cause harmful or misleading outcomes. This survey synthesizes LLM alignment through an AI-alignment taxonomy, covering outer and inner alignment, interpretability, adversarial vulnerabilities, evaluation, and future directions. It concludes that alignment research must address both established concerns and emerging challenges as AI capabilities grow.

  • Problem

    LLMs may produce imprecise, misleading, or detrimental texts, while their training objectives are not necessarily aligned with human values.

  • Method

    The survey organizes LLM alignment research through an AI-alignment perspective, covering outer and inner alignment, interpretability, adversarial attacks, evaluation, and future directions.

  • Results

    The survey provides a comprehensive overview of alignment methodologies and capability research while identifying emerging alignment topics and evaluation approaches.

  • Takeaways & Limitations

    Aligning increasingly capable AI systems with human values remains a central research priority for developing capable and safe LLMs.

  • Takeaways & Limitations

    The survey identifies goal specification as an unresolved gap between qualitative human values and measurable, computable optimization objectives.

Abstract

from arXiv · show

Recent years have witnessed remarkable progress made in large language models (LLMs). Such advancements, while garnering significant attention, have concurrently elicited various concerns. The potential of these models is undeniably vast; however, they may yield texts that are imprecise, misleading, or even detrimental. Consequently, it becomes paramount to employ alignment techniques to ensure these models to exhibit behaviors consistent with human values. This survey endeavors to furnish an extensive exploration of alignment methodologies designed for LLMs, in conjunction with the extant capability research in this domain. Adopting the lens of AI alignment, we categorize the prevailing methods and emergent proposals for the alignment of LLMs into outer and inner alignment. We also probe into salient issues including the models' interpretability, and potential vulnerabilities to adversarial attacks. To assess LLM alignment, we present a wide variety of benchmarks and evaluation methodologies. After discussing the state of alignment research for LLMs, we finally cast a vision toward the future, contemplating the promising avenues of research that lie ahead. Our aspiration for this survey extends beyond merely spurring research interests in this realm. We also envision bridging the gap between the AI alignment research community and the researchers engrossed in the capability exploration of LLMs for both capable and safe LLMs.

1 Introduction

The survey frames LLM alignment as necessary because increasingly capable models can produce harmful, misleading, or otherwise undesirable behavior. It addresses this gap with a broad AI-alignment taxonomy covering established and emerging areas.

  • LLMs’ rapid capability growth has renewed interest while raising concerns about harmful, misleading, false, and low-quality outputs.
  • Alignment aims to make LLM responses accurate, coherent, safe, ethical, and desirable for developers and users.
  • Existing reviews largely emphasize outer alignment, leaving inner alignment and mechanistic interpretability comparatively underrepresented.
  • The survey provides a comprehensive AI-alignment perspective that includes outer alignment alongside nascent inner-alignment and mechanistic-interpretability research.
  • The proposed taxonomy organizes the survey across alignment approaches, interpretability, adversarial vulnerabilities, evaluation, and future directions.

2 Why LLM Alignment?

LLM alignment matters because training objectives may diverge from human values, enabling undesirable content, social harms, misuse, and risks from advanced-agent behavior. The survey presents these established and anticipated risks as motivations for alignment research.

  • LLM training objectives such as next-word prediction are not necessarily aligned with human values, allowing undesirable content and risky behaviors.
  • LLM-Generated Content: LLM-generated content can reproduce biases, toxicity, privacy-sensitive information, misinformation, hallucinations, and inconsistency.
  • These risks motivate alignment research aimed at improving LLM truthfulness and honesty.
  • Malicious Uses: Malicious uses include lower-cost disinformation, fraudulent manipulation, cyberattack code generation, and potentially lethal-weapon creation.
  • Negative Impacts on Society: Large-scale deployment also raises environmental and labor-market concerns; one cited study estimates that around 80% of the U.S. workforce will be affected.
  • Potential Risks Associated with Advanced LLMs: Anticipated advanced-LLM risks include instrumental convergence, situational awareness, deception, self-preservation, and power-seeking.

3 What is LLM Alignment?

AI alignment asks how to ensure increasingly capable machines pursue human purposes and values. This survey frames LLM alignment around outer and inner objectives, while also emphasizing interpretability and unresolved specification challenges.

  • Origins: AI alignment emerged from the question of how powerful machines can be ensured to act in humanity’s best interests rather than against them.
  • Origins: Recent LLM advances have renewed alignment urgency as models approach or surpass human performance across many tasks.
  • Definition: The survey defines alignment as requiring both designers’ outer objectives and models’ inner optimized objectives to accord with human values.
  • Outer Alignment: Outer alignment matches training objectives to designers’ goals, but human values are difficult to define, measure, and represent fairly across cultures.
  • Outer Alignment: Specification gaming can arise when optimized proxies cease to represent the intended value, reflecting the risk described by Goodhart’s Law.
  • Interpretability: Mechanistic interpretability seeks to reverse-engineer model outputs and behaviors into internal states, weights, and components, but current work usually studies simplified models.

4 Outer Alignment

Outer alignment translates human values into LLM training goals, commonly through human-feedback methods, while scalable oversight seeks to extend supervision to complex or superhuman tasks.

  • Outer alignment aims to align LLM goals with human values expressed as helpfulness, honesty, and harmlessness.Harmlessness includes refusing harmful requests and avoiding harmful outputs.
  • Perfect outer alignment is difficult because human values are hard to specify.
  • Current outer alignment methods are categorized as non-recursive oversight or scalable oversight according to the upper bound of human supervision capabilities.
  • Non-recursive oversight uses labeled human feedback without scaling supervision beyond situations where humans can provide effective feedback.These methods include reinforcement-learning and supervised-learning approaches, and human feedback becomes a bottleneck as model capabilities grow.
  • RL-based Methods: RLHF collects human feedback, trains a reward model, and fine-tunes an LLM with reinforcement learning, commonly using PPO.A KL-divergence penalty limits deviation from the initial model and helps prevent reward hacking that produces gibberish.
  • RL-based Methods: RL-based methods can align LLMs to human preferences but face reward-model imperfections and unstable reinforcement-learning optimization.The survey also discusses RLHF variants and alternatives including Sparrow, Second Thoughts, RLSF, DSP, and RL4F.

4.4 Scalable Oversight

Scalable oversight seeks to let relatively weak overseers supervise complex or superhuman tasks through decompositions, amplified feedback, debate, and related mechanisms. The survey presents these approaches as promising but assumption-dependent and incompletely validated.

  • Scalable oversight enables relatively weak overseers to supervise complex tasks using signals that are easier to adjudicate.Its motivation is to avoid the scalability bottleneck of non-recursive human supervision.
  • Factored cognition, process supervision, sandwiching, IDA, and RRM decompose or recursively supervise complex tasks in different ways.Process supervision supplies signals for sequential subtasks, while IDA distills capabilities amplified through task decomposition and RRM uses reward modeling.
  • Constitutional AI uses human-written principles to generate, critique, correct, and filter training examples, reducing reliance on direct human annotations.Dromedary applies principle-driven self-instruction and self-alignment without reinforcement learning.
  • Debate: Debate has agents propose, criticize, and defend answers so humans can judge complex tasks through simpler structured reasoning.The survey describes improved transparency and exposure of weak arguments, alongside multi-agent evidence for improved factuality, reasoning, and correction of distorted thinking.
  • Proxy tasks can help overseers identify errors in superhuman models, but their specificity favors precision over recall for detecting misalignment.
  • Challenges of Scalable Oversight: Scalable oversight depends on foundational assumptions, including that complex tasks can be decomposed into mainly independent subtasks.Some tasks are inherently sequential; sorting requires at least log(n) serial sorting steps.

5 Inner Alignment

Inner alignment concerns whether a trained system robustly pursues the intended objective rather than merely performing well under training conditions. The survey defines key optimizer concepts, describes failure modes, and emphasizes that empirical methodology remains limited.

  • Inner alignment asks whether an AI system robustly optimizes an objective aligned with what humans want.
  • A base optimizer searches for a model using a base objective, while a mesa-optimizer is a learned optimizer with its own mesa-objective.
  • Inner alignment requires the mesa-optimizer’s objective to match the base optimizer’s objective.A mismatch can produce inner-alignment failure even when the base optimizer controls the mesa-optimizer’s optimization process.
  • The survey describes proxy, approximate, and suboptimality alignment as inner-alignment failure or pseudo-alignment modes.Proxy alignment optimizes an incorrect proxy; approximate alignment contains representation error; suboptimality alignment can result from limitations that temporarily produce aligned behavior.
  • Inner- and outer-alignment failures can be difficult to separate because their interdependencies may allow failures in one to trigger failures in the other.
  • Inner alignment remains comparatively underdeveloped empirically and methodologically, with most work focused theoretically on definitions, failure modes, and risks.The survey presents relaxed adversarial training and experiments involving reward side-channels, cross-episodic objectives, objective unidentifiability, and zero-shot objectives as research directions.

6 Mechanistic Interpretability

Mechanistic interpretability studies how models transform inputs into outputs by reverse-engineering interpretable components and information flows. The survey reviews findings on attention, MLPs, and neurons while noting that current studies remain restricted and technically challenging.

  • Mechanistic interpretability seeks causal and functional explanations of how neural networks transform inputs into outputs.It aims to reverse-engineer end-to-end reasoning into interpretable parts and information flows.
  • Interpretability can audit deployed models, identify misaligned or fallacious outputs, and monitor training changes and emerging behaviors.
  • Self-attention: Self-attention studies characterize QK circuits as focusing on potential next tokens and OV circuits as copying previous tokens through induction heads.Further work attributes general in-context learning to induction heads in both small attention-only and larger models.
  • MLP: SoLU is proposed to improve interpretability of MLP representations, while experiments show feature superposition becomes more apparent in nonlinear models as data sparsity increases.
  • Neurons: Neuron-level interventions can alter whole-network outputs and have been used to enhance factuality of machine-generated content.
  • Challenges: Mechanistic interpretability remains incipient because much research uses toy models or simple predefined tasks, while superposition complicates representation disentanglement.

7 Attacks on Aligned Language Models

Aligned language models can resist some malicious and adversarial inputs, but multiple attack classes can still compromise their behavior, expose private information, or implant targeted failures.

  • Aligned LLMs can counter a wide range of white-box NLP attacks and ChatGPT can decline privacy-sensitive questions.
  • Repeated interactions can jailbreak aligned models into generating harmful content, motivating three categories of safeguard-breaching attacks.
  • Privacy attacks: Privacy attacks include gradient reconstruction, attribute inference, prompt attacks, and inversion attacks targeting sensitive training information.
  • Privacy attacks: Prompt attacks search for inputs that elicit training data, including personal information, and multi-step jailbreak prompts extracted PII from ChatGPT and New Bing.
  • Adversarial attacks: Greedy and gradient-based adversarial prompts can induce contentious behaviors from aligned LLMs.
  • Adversarial attacks: Multimodal language models show reduced defenses against white-box visual adversarial attacks because high-dimensional visual inputs provide greater susceptibility and more attack targets.

8 Alignment Evaluation

The survey organizes LLM alignment evaluation across five outer-alignment dimensions and distinguishes task-specific assessments from LLM-centered benchmarks, methods, and metrics.

  • The evaluation taxonomy covers factuality, ethics, toxicity, stereotype and bias, and general evaluation.General evaluation addresses multiple alignment dimensions rather than one specific dimension.
  • The taxonomy distinguishes task-specific evaluation of downstream tasks from LLM-centered evaluation designed directly for LLMs.
  • A third taxonomy level provides fine-grained classifications and related works to help readers locate relevant evaluation areas.

8.1 Factuality Evaluation

Factuality evaluation addresses consistency with context and factual precision, using task-specific and LLM-oriented benchmarks and metrics while facing broad coverage and reasoning limits.

  • Factuality evaluation comprises factual consistency with given context and factual precision of generated information.
  • Factual consistency: Consistency research evaluates summarization, dialogue, and other contextual tasks, including a unified analysis of 11 datasets and multiple metrics.
  • Factual consistency: ALIGNSCORE targets contradiction and hallucination across varied lengths and tasks by training an aligned model for factual consistency evaluation.
  • Factual precision: Factual precision methods use factual and non-factual prompts, named entity recognition, entailment, and atomic-fact scoring in long-form text.
  • Truthfulness: TruthfulQA evaluates whether language models answer highly inductive questions truthfully because training objectives may influence false responses.
  • Challenges: Current factuality evaluations cover limited facts and struggle with information that cannot be reduced to atomic facts or requires sophisticated reasoning.

8.2 Ethics Evaluation

Ethics evaluation commonly presents moral questions or scenarios to language models, but existing datasets and reference responses remain shaped by researchers’ cultural contexts.

  • Most LLM ethics evaluations pose questions about ethics or morality and assess the model’s responses.
  • Benchmarks: The ETHICS benchmark contains over 130,000 scenarios across justice, virtue ethics, deontology, utilitarianism, and commonsense morality.
  • Benchmarks: Ethics resources range from short statements and game-derived questions to detailed real-life anecdotes collected from Reddit.
  • Evaluation methodology: Social Chemistry 101 decomposes tacit commonsense rules into twelve dimensions of human judgment and offers annotators multiple perspectives on scenarios.
  • Limitation: Ethics assessments need data and reference responses representing diverse cultural backgrounds because current materials largely reflect researchers’ own contexts.

8.3 Toxicity Evaluation

Toxicity evaluation examines harmful model behavior through both downstream tasks and direct analysis of generated outputs. The survey reviews datasets and adversarial prompting methods used to assess this risk.

  • Toxicity evaluation is divided into task-specific evaluation and LLM-centered evaluation.Task-specific evaluation examines toxicity in downstream applications, whereas LLM-centered evaluation directly assesses generated outputs.
  • Task-specific evaluation broadens toxicity analysis by examining diverse downstream NLP contexts.Its findings may still be constrained by the particular task in which the model is evaluated.
  • Offensive-language detection provides a classification-based route for studying harmful content across social-media datasets.The reviewed datasets vary in platform, topic, annotation detail, and scale.
  • LLM-centered evaluations trigger models to produce toxic responses and primarily measure the toxicity of those outputs.These evaluations assess models directly rather than through a downstream task.
  • Adversarial dialogue methods such as BAD prompt advanced models to generate unsafe responses, creating datasets for toxicity assessment.The setup mirrors adversarial challenges models may encounter after deployment.

8.4 Stereotype and Bias Evaluation

Stereotype and bias evaluation covers biases in downstream tasks and biases expressed directly by language models. The survey catalogs benchmarks spanning demographic categories, task formats, languages, and evaluation procedures.

  • Bias evaluation is categorized into task-specific assessment and direct LLM-centered assessment.The former examines bias in downstream tasks, while the latter targets biases inherent in model outputs.
  • Task-specific bias benchmarks examine gender, race, religion, age, occupation, and other demographic dimensions across coreference, translation, relation extraction, sentiment, and inference.Examples include Winogender, WinoBias, WinoMT, WikiGenderBias, EEC, and NLI-based benchmarks.
  • Coreference benchmarks expose gender stereotypes, with the best evaluated model achieving only 34% F1 on texts involving non-binary and binary transgender individuals.GICOREF reports lower performance on these texts than on binary gendered texts.
  • Direct bias evaluations use sentence pairs, masked-likelihood comparisons, prompts, and multiple-choice questions to detect stereotyped model behavior.These designs include StereoSet, CrowS-Pairs, BOLD, UnQover, and BBQ.
  • Hate-speech detection can support both monitoring harmful generated content and scrutinizing training data, but universally accepted harmfulness standards remain difficult to establish.The survey reviews explicit hate-speech datasets from Twitter, forums, and other social platforms.

8.5 General Evaluation

General alignment evaluation combines broad, multi-aspect assessment with automated and human judging. The survey emphasizes scalable multi-model evaluation while noting persistent evaluator biases and the continued need for human assessment.

  • General evaluation assesses alignment across multiple aspects rather than measuring only one property such as factuality or bias.Typical benchmarks ask a model to answer an instruction, then use an advanced LLM or human as evaluator.
  • TrustGPT evaluates bias, toxicity, and value consistency, including passive-condition prompts containing harmful content.FLASK instead decomposes coarse-grained evaluation into four abilities and 12 fine-grained skills.
  • Automatic evaluation uses single-answer grading, pairwise comparison, and reference-guided grading with LLM or human judges.These formats differ in whether they score one response, compare two responses, or compare against a human reference.
  • Collaborative multi-agent deliberation and debate can broaden knowledge and improve weaker models’ reasoning across downstream tasks.The survey connects these approaches to more accurate and fairer evaluation under complex reasoning demands.
  • Multiple-LLM evaluation can reduce individual-model bias while retaining LLM evaluation capabilities, making it a supplement to manual evaluation.Peer-review and referee-style procedures aggregate judgments or rationales across evaluators.
  • LLM-based automatic evaluation still cannot fully replace human evaluation because competence gaps, shared training biases, and evaluator biases remain.GPT-4 disagreement with human judgments and biases such as position, verbosity, and self-enhancement constrain reliability.
  • High-stakes decisions should prioritize human evaluation despite its expense and the speed and cost-effectiveness of LLM evaluators.Existing automatic metrics also align poorly with human preferences in long-form answers.

9 Future Directions and Discussions

The survey identifies future alignment work spanning theory, oversight, interpretability, adversarial testing, and field building. It emphasizes corrigibility, realistic world models, empirical verification, and stronger scientific coordination.

  • Future Directions and Discussions: LLM alignment remains fast-growing but lacks sufficient insights, breakthroughs, and a cohesive scientific paradigm.The survey describes unresolved controversies across alignment theories, methodologies, and empirical results.
  • 9.1 Theoretical Research for LLM Alignment: Decision-theoretic alignment aims to make advanced LLM decisions predictable and beneficial under uncertainty or conflicting values.Future work includes counterfactual reasoning, Newcomb-like problems, and related paradoxes.
  • 9.1 Theoretical Research for LLM Alignment: Corrigibility concerns keeping LLMs receptive to user correction without resistance or circumvention.Proposed directions include proactive assistance with alignment and self-correction without worsening misalignment.
  • 9.1 Theoretical Research for LLM Alignment: Current LLMs rely on limited, training-data-derived world understanding, motivating more accurate, robust, and adaptable world models.Future systems should better represent changing real-world complexity.
  • 9.1 Theoretical Research for LLM Alignment: Scalable oversight strategies such as debate, IDA, and RRW have not yet received large-scale empirical verification.The survey notes emerging efforts, including superalignment, to test these approaches.
  • 9.1 Theoretical Research for LLM Alignment: Deceptive alignment describes an agent that appears aligned during training but may pursue a different mesa-objective after modification threats disappear.Monitoring experiments can operationalize proposed conditions involving persistent objectives, modeling the base objective, and anticipated cessation of modification.
  • 9.4 Automated LLM Alignment: AI-driven supervisors and aligners are proposed to help automatically evaluate and align increasingly complex models.This direction responds to the growing complexity of AI systems.
  • 9.5 Explainability and Transparency: Explainability and transparency research remains early, with many techniques offering only surface-level insight into model decision-making.The survey suggests continued collaboration among machine learning researchers, ethicists, and neuroscientists.

10 Conclusion

The survey reviews LLM alignment methods through outer and inner alignment, while addressing interpretability and adversarial vulnerabilities. It also evaluates current research and identifies future directions, including collaboration between alignment and capability researchers.

  • The survey categorizes LLM alignment techniques into outer and inner alignment.
  • It discusses model interpretability and vulnerabilities to adversarial attacks as emerging alignment topics.
  • The paper reviews the current state of alignment research and identifies potential trajectories for refining LLM alignment.
  • It calls for collaboration between the AI alignment community and LLM researchers to support capable and safe models.
Loading 2309.15025v1…