PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Alignment: A Comprehensive Survey

Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Donghai Hong, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai, Xuehai Pan, Kwan Yee Ng, Aidan O'Gara, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, Wen Gao

arXiv:2310.19852v6cs.AI

TL;DR

As AI systems become more capable and enter complex domains, the survey addresses the problem of making their behavior accord with human intentions and values. It synthesizes alignment concepts, methods, and practices through RICE objectives and a forward/backward alignment framework. The survey organizes current work into learning from feedback, learning under distribution shift, assurance, and governance, while identifying limitations including superficial alignment methods and interpretability challenges from superposition.

  • Problem

    Increasingly capable AI systems exhibit harmful behaviors and create alignment risks, while the field requires a broad account of how to align and assess them.

  • Method

    The survey synthesizes AI alignment concepts, methodology, and practice using RICE objectives and a forward/backward alignment cycle.

  • Results

    The survey organizes alignment research into learning from feedback, learning under distribution shift, assurance, and governance.

  • Takeaways & Limitations

    Alignment is presented as a diverse, continuously assessed field in which complementary research directions support a shared goal.

  • Takeaways & Limitations

    Superposition makes neuron-level analysis implausible, while current alignment methods can be superficial and require more robust approaches.

Abstract

from arXiv · show

AI alignment aims to make AI systems behave in line with human intentions and values. As AI systems grow more capable, so do risks from misalignment. To provide a comprehensive and up-to-date overview of the alignment field, in this survey, we delve into the core concepts, methodology, and practice of alignment. First, we identify four principles as the key objectives of AI alignment: Robustness, Interpretability, Controllability, and Ethicality (RICE). Guided by these four principles, we outline the landscape of current alignment research and decompose them into two key components: forward alignment and backward alignment. The former aims to make AI systems aligned via alignment training, while the latter aims to gain evidence about the systems' alignment and govern them appropriately to avoid exacerbating misalignment risks. On forward alignment, we discuss techniques for learning from feedback and learning under distribution shift. On backward alignment, we discuss assurance techniques and governance practices. We also release and continually update the website (www.alignmentsurvey.com) which features tutorials, collections of papers, blog posts, and other resources.

1 Introduction

AI alignment addresses risks from increasingly capable systems by making their objectives accord with human intentions and values. The survey organizes the field around RICE objectives and a framework spanning feedback, distribution shift, assurance, and governance.

  • Field organization: The survey presents alignment as a diverse field whose research directions share a goal rather than a single methodology and can complement one another in a self-improving loop.It also provides resources for beginners through an associated website.
  • Motivation: Increasing AI capabilities and deployment in high-stakes domains heighten concerns about harmful or difficult-to-control behavior.Examples include untruthful answers, sycophancy, deception, power-seeking, and manipulation.
  • Motivation: Misalignment is behavior that contrasts with human intentions and can arise without malicious misuse, making it a significant source of AI risk.The survey discusses reward hacking, reward tampering, and goal misgeneralization as mechanisms or failure modes.
  • Failure modes: Power-seeking, manipulation, deception, and reward tampering are discussed as ways systems may pursue objectives or corrupt oversight and reward processes.The survey links dangerous capabilities to risks such as hacking, escaping containment, and violating ethics.
  • Framework: Its alignment-cycle framework contrasts forward alignment, which trains systems against requirements, with backward alignment, which gathers evidence and governs systems throughout their lifecycle.The framework is organized around Learning from Feedback, Learning under Distribution Shift, Assurance, and Governance.
  • Objectives: The survey defines four alignment objectives—Robustness, Interpretability, Controllability, and Ethicality—and treats them as intermediate objectives serving alignment with human intentions and values.Ethicality includes adherence to moral and social norms and values.

RRM •

The passage presents the label “CIRL” alongside circular markers.

  • The passage contains the label “CIRL.”
  • The label is followed by circular markers.
  • No further substantive description appears in the passage.

DRO •

The supplied passages identify “Algorithmic Interventions” and “Learning under Distribution Shift” as section labels.

  • One passage is labeled “Algorithmic Interventions.”
  • The passages provide headings without further substantive explanation.
  • Another passage is labeled “Learning under Distribution Shift.”

CBFT •

The survey frames alignment around human values and four objectives, while distinguishing alignment concerns from broader AI safety issues such as malicious use and collective action problems.

  • Human Values in Alignment: Human-values alignment includes ethical and social values, cooperative AI, and modeling multi-agent and social dynamics.These themes address moral values, cooperation, and social complexities respectively.
  • Human Values in Alignment: Determining which human values AI systems should follow is difficult because values are abstract, uncertain, and may differ substantially across cultures.The survey notes that universally consistent human values may not exist.
  • Human Values in Alignment: Cooperation failures can create large-scale harms, illustrated by the 2010 Flash Crash’s temporary loss of trillions of market value in 2 minutes.The survey presents this as an example involving interactions between high-frequency algorithmic traders.
  • Boundaries of Alignment: Malicious use can threaten society even when an AI system follows a malicious user’s intentions, so alignment with the public interest requires policy measures.The survey distinguishes this case from failure to align with the user.
  • Boundaries of Alignment: Collective action problems arise when competition incentivizes developers to neglect safety, even if individual developers would prefer cautious evaluation and deployment.The survey characterizes this as a social dilemma among AI developers.

2 Learning from Feedback

Learning from feedback is presented as a foundation of forward alignment: advisors provide information that adjusts an AI system, either directly or through proxies.

  • Learning from feedback transmits human intentions and values to AI systems and consists of an AI system, advisor-provided feedback, and the adjustment process.Advisors may be humans, AI systems, or humans assisted by AI.
  • Feedback produces two learning pathways: direct learning from feedback itself and indirect learning through proxies that model the feedback.
  • The survey adopts a human-centric taxonomy that classifies feedback by its presentation form to the AI system.The four forms are Label, Reward, Demonstration, and Comparison.

2.1 Feedback Types

The survey categorizes feedback as labels, rewards, demonstrations, or comparisons, describing how each conveys information and where its limitations appear in complex alignment tasks.

  • The survey classifies feedback into four forms: label, reward, demonstration, and comparison, using the forms directly presented to the AI system.
  • Label: Label feedback gives explicit input-output guidance and is unambiguous, but it can bias targets and become insufficient for complex tasks.The survey cites algorithm optimization, video-game playing, and multimodal generation as examples where exhaustive labels are impractical.
  • Reward: Reward feedback gives scalar or vector evaluations of individual outputs, allowing quantified behavioral guidance and policy exploration.Rewards commonly come from pre-designed rule-based functions or procedures.
  • Demonstration: Demonstration feedback records expert behavior and can directly leverage advisors’ experience without formalized knowledge representations.Demonstrations may be noisy, suboptimal, inconsistent, costly, and insufficient for tasks beyond advisors’ expertise.
  • Comparison: Comparison feedback ranks outputs relative to one another, helping humans evaluate objectives that are difficult to score precisely.Relative comparisons can obscure absolute differences and steer optimization away from true objectives when feedback is noisy or context is unmodeled.
  • Interactive Feedback: All feedback types can be provided interactively, enabling synchronous feedback and model updates through rapid, focused, incremental modifications.Interactive learning includes online corrections and active data discovery during deployment.
  • Unified View: Feedback types can be unified as attempts to convey a hidden parameterized reward function, enabling inference over its parameters regardless of feedback type.The survey denotes this reward function as Ψ (·;θ).

2.2 Preference Modeling

Table 2 compares action-, state-, and trajectory-level preference granularity in sequential decision-making, distinguishing them by what elements of the learning process they compare.

  • Preference granularity has three types: Action, State, and Trajectory.
  • The table defines each preference type by its characteristics and by how it compares elements of sequential decision-making.The notation i1 ≻i2 denotes that i1 is strictly preferred over i2.

2.2 Preference Modeling

Preference modeling elicits human judgments in forms ranging from item-level comparisons to graded or ordered preferences, then uses them to train reward models for policy learning. Its design choices concern what is compared and how preferences are represented, with longer trajectory segments yielding more informative comparisons in studied MuJoCo settings.

  • Overview: Preference modeling uses comparison feedback to help fine-tune powerful systems when precise rule-based rewards and demonstrations are costly.Preference models transfer human comparisons into signals usable for policy learning.
  • Overview: Preference elicitation requires jointly deciding the granularity and category of preference in sequential decision-making problems.The paper presents these choices as broadly applicable beyond sequential decision-making.
  • Granularity of Preference: Preferences can target actions, states, or trajectories, each imposing different requirements and assumptions when translated across levels.Action preferences compare actions within a state; state preferences compare states; trajectory preferences compare state-action sequences.
  • Granularity of Preference: Longer trajectory segments yielded more informative comparisons per segment and more consistent human evaluations in the studied MuJoCo tasks.This result comes from ablation studies by Christiano et al. (2017).
  • Category of Preference: Preference categories include object versus label preferences and absolute versus relative forms, including binary, gradual, total-order, and partial-order variants.Relative preferences can permit incomparable item pairs under a partial order.
  • Reward Model: Reward modeling represents a preferred action y_w over a less-preferred action y_l in state x and optimizes a parameterized model by maximum likelihood.The preference likelihood uses a logistic sigmoid, while the resulting rankings provide scalar rewards for policy learning.

2.3 Policy Learning

Policy learning trains agents to map states or observations to actions, providing the reinforcement-learning backdrop for alignment methods such as imitation learning, inverse reinforcement learning, preference-based RL, and RLHF. These approaches replace or infer explicit rewards from demonstrations or preferences, while facing distribution-shift, reward, exploration, data, and stability constraints.

  • Background: Policy learning maps perceived states to actions to optimize task performance, making it a central backdrop for alignment research.The section frames policy-learning techniques as tools for advancing alignment objectives.
  • Background: Reinforcement learning learns policies through trial-and-error interaction with an environment to maximize expected cumulative reward.The objective depends on transition dynamics and the initial state distribution.
  • Background: RL faces sample-efficiency and stability challenges, while PPO limits policy updates to prevent large deviations from the original policy.PPO is described as the key algorithm for RLHF.
  • Preference-based Reinforcement Learning: Preference-based reinforcement learning trains agents from preference feedback instead of explicit rewards, broadening RL applications while mitigating reward-function formulation difficulties.Its challenges include delayed credit assignment, preference-space exploration, massive data needs, and limited ability to express preferences in some settings.
  • Imitation and Inverse Reinforcement Learning: Imitation learning emulates demonstrated human behavior without requiring environmental reward signals, whereas inverse reinforcement learning derives rewards from observed behavior.Standard inverse-reinforcement-learning families include feature-matching, maximum-entropy, and Bayesian methods.
  • Distribution Shift: Imitation-learning methods can improve robustness to distribution shifts, but adversarial variants learn nonstationary rewards that cannot train new policies.The underlying out-of-distribution problem arises from differences between training and testing distributions.
  • Reinforcement Learning from Human Feedback: RLHF extends preference-based RL to complex AI systems by using human judgments of appropriate behavior, especially for fine-tuning language models.Its stated advantage is that humans judge behavior more readily than they provide demonstrations or manually specify rewards.
  • Reinforcement Learning from Human Feedback: RLHF commonly comprises supervised fine-tuning, preference-based reward modeling, and subsequent policy optimization over human-preference signals.The supplied pipeline passage explicitly introduces three stages, with supervised fine-tuning producing π_SFT.

2.4 Scalable Oversight: Path towards Superalignment

Scalable oversight addresses the high cost and difficulty of evaluating increasingly capable AI systems, aiming to keep systems aligned with human intent beyond human expertise. The survey presents feedback-based and cooperative frameworks, including RLxF, IDA, RRM, Debate, CIRL, and weak-to-strong generalization, while highlighting unresolved safety and evaluation challenges.

  • Motivation: Scalable oversight addresses costly human evaluation and the difficulty of assessing complex, high-stakes AI behaviors.Examples include book summarization, complex code generation, and weather prediction.
  • From RLHF to RLxF: RLxF extends RLHF by combining human and AI feedback to improve feedback efficiency and quality for complex tasks.RLxF methods decompose large problems into smaller sub-problems that AI and software can help resolve.
  • Recursive Reward Modeling: Recursive Reward Modeling trains successive agents using feedback from already trained agents operating on amplified versions of their capabilities.The initial agent is trained through fundamental reward modeling from human feedback.
  • Debate: Debate can outperform consultancy, but weak judges struggle to exploit debate protocols and the overall evidence remains only weakly promising.The authors also emphasize grounding debate in verifiable evidence to reduce fabricated claims.
  • Cooperative Inverse Reinforcement Learning: CIRL unifies control and feedback learning by modeling humans and AI as cooperative agents, reducing incentives to manipulate feedback.The AI treats the human’s true objective as its goal while inferring it through observation and interaction.
  • Weak-to-Strong Generalization: Weak-to-strong generalization shows that strong models can learn from weak supervision across NLP classification, chess puzzles, and reward modeling, although verification remains difficult without ground truth.The phenomenon is presented as an analogy for supervising superhuman systems with human-level supervisors.

3 Learning under Distribution Shift

Learning under distribution shift seeks to preserve alignment properties when deployment data or environments differ from training conditions. The survey distinguishes goal misgeneralization and auto-induced distribution shift, then organizes responses into algorithmic and data distribution interventions.

  • 3 Learning under Distribution Shift: Training-data and deployment discrepancies can cause aligned systems to pursue goals inconsistent with human intentions after deployment.Distribution shift includes qualitative changes such as adversarial pressures, multi-agent interactions, and tasks that human overseers cannot efficiently evaluate.
  • 3 Learning under Distribution Shift: Algorithmic interventions steer optimization during training, while data distribution interventions expand training with targeted real-world elements.The framework presents these as the two principal approaches for addressing distribution shift.
  • 3.1 The Distribution Shift Challenge: Auto-induced distribution shift occurs when an AI system changes the data distribution through its own reward-seeking behavior, unlike goal misgeneralization where the shifted distributions are given.The survey also relates goal misgeneralization to deceptive or manipulative evaluation and to misaligned mesa-optimizers.
  • 3.1 The Distribution Shift Challenge: Goal misgeneralization occurs when behavior succeeds during training but learned objectives fail to generalize to out-of-distribution deployment.It differs from capability misgeneralization because the concern is pursuit of goals that conflict with human wishes, rather than simple incompetence.
  • 3.2.1 Cross-Distribution Aggregation: Minimizing the cross-distribution objective can learn invariant relationships, whereas naive ERM may generalize poorly when source and target distributions differ substantially.REx methods reduce and equalize training risks, encouraging invariant relationships and showing promise for robust generalization.
  • 3.2 Algorithmic Interventions: Cross-distribution aggregation minimizes risks across domains to favor invariant relationships over spurious features, while mode connectivity supports generalization-oriented fine-tuning.Representative techniques include DRO, IRM, REx, and CBFT.

4 Assurance

Assurance assesses and refines the practical alignment of trained or deployed AI systems through safety evaluation, interpretability, red teaming, and intervention methods. The section also surveys safety benchmarks and Safetywashing, where benchmark scores can reflect capability rather than safety.

  • Assurance measures and refines AI systems’ practical alignment after training or deployment.
  • Its categories include safety evaluations, interpretability, and methods that actively intervene in systems or deployment.
  • Safety Evaluation: Safety evaluation covers datasets and benchmarks, evaluation targets, and red teaming beyond standard evaluation.
  • Datasets and Benchmarks: Expert-designed datasets provide broad, accurate coverage but are costly and limited in breadth, while internet collection scales better but requires careful selection and annotation.
  • Red Teaming: Red teaming applies adversarial pressure by generating scenarios intended to induce unaligned outputs or actions; state-of-the-art systems generally fail this test.
  • Safetywashing: Safetywashing occurs when safety benchmarks correlated with capabilities make capability gains appear to be safety gains; most current domains show positive correlation except weaponization and sycophancy.

4.2 Interpretability

Interpretability seeks to make models and their decision processes understandable, using intrinsic or post hoc approaches and multiple analysis techniques. The survey emphasizes methods for reverse-engineering internal representations and causal links to behavior, while noting limitations from superposition, scalability, and evaluation.

  • Interpretability makes machine-learning systems and their decision-making processes understandable to humans.
  • The survey classifies interpretability as intrinsic or post hoc, covering analysis during design and after deployment.
  • Intrinsic Interpretability: Intrinsic methods modify components or architectures, create transparent reasoning steps, and distill complex models into simpler interpretable students.
  • Post Hoc Interpretability: Post hoc methods study internal structure and causal effects using dictionary learning, circuit analysis, probing, attribution, perturbation, ablation, and patching.
  • Post Hoc Interpretability: Attribution methods estimate component contributions but cannot provide causal explanations and require datasets for the features of interest, limiting their use on out-of-distribution features.

4.3 Human Values Verification

Post hoc interpretability includes methods that test counterfactual effects and identify internal mechanisms behind model outputs. The section highlights causal analysis through perturbation, ablation, and patching, alongside unresolved challenges from superposition, scalability, and evaluation.

  • Perturbation changes model inputs, while ablation removes network components to test counterfactual effects and establish causal relationships with behavior.
  • Activation patching compares correct and corrupted runs to locate activations that matter for model outputs and can map or edit model behavior.
  • Superposition means models represent more features than dimensions, so features do not correspond directly to individual neurons.
  • Interpreting real models while maintaining capability is harder than interpreting toy models, making scalability a central concern.
  • Interpretability benchmarks evaluate tools through tasks such as trojan detection and circuit testing, helping quantify effectiveness and guide research.

4.3 Human Values Verification

Human value alignment concerns whether AI systems adhere to community social and moral norms, requiring both formal value frameworks and techniques for verifying alignment across scenarios. The survey covers formal machine ethics, cooperative-AI game theory, moral datasets, and scenario simulation.

  • Human Values Alignment requires AI systems to adhere to community social and moral norms, especially as increasingly capable systems become more integrated into daily life.The survey emphasizes that simply aligning with human intent may not sufficiently capture human morality and values.
  • The survey organizes human value verification around formal frameworks for ethics and cooperation in AI, alongside specific value-verification techniques.Figure 12 presents these as the main branches of the research landscape.
  • Formal Machine Ethics: Formal machine ethics includes logic-based methods, RL/MDP-like methods, and game-theoretic or computational-social-choice approaches.Examples include deontic and event-calculus methods, ethics-oriented reward functions, evolutionary game theory, classical game theory, and computational social choice.
  • Game Theory for Cooperative AI: Game theory for Cooperative AI studies incentives for cooperation and seeks to enhance them, complementing MARL’s emphasis on coordination capabilities.The survey discusses classical game theory and evolutionary game theory as major approaches.
  • Value-Alignment Techniques: Human value-alignment techniques include building moral datasets and simulating morally salient scenarios, while Goodhart’s Law complicates encoding complex values as reward functions.Scenario simulations can evaluate behaviors such as deception, manipulation, and betrayal in more complex settings than datasets.

5 Governance

AI governance complements technical alignment by creating and enforcing rules for the safe development and deployment of AI systems. The survey frames governance as a multi-stakeholder system involving governments, industry and AGI labs, and third parties, while noting unresolved questions about open-source models and dynamic human values.

  • AI governance is necessary alongside technical solutions to ensure the safe development and deployment of AI systems and to address open challenges.The survey examines governance’s role, stakeholder functions and relationships, and challenges to effective governance.
  • The Role of AI Governance: Governance aims to mitigate diverse AI risks, including societal bias, discrimination, global-security threats, and potential catastrophic or existential harms.The survey recommends maintaining a balanced portfolio of efforts across risk categories.
  • The Multi-Stakeholder Approach: The governance framework assigns governments policy, regulatory, enforcement, and international-cooperation functions; industry and AGI labs develop and deploy systems; third parties audit and advise.Third parties include academia, NGOs, and NPOs, and support both corporate auditing and government policymaking.
  • Government: Government regulation is described through standard development, registration and reporting, and mechanisms ensuring adherence to safety standards.Examples include the EU AI Act and the U.S. Bipartisan Framework for AI Act.
  • Industry and AGI Labs: Industry and AGI labs should conduct lifecycle risk assessments spanning pre-development, pre-training, pre-deployment, post-deployment monitoring, and external scrutiny.External scrutiny includes bug bounties, red teaming, and third-party model auditing.
  • Third Parties: Third-party auditing covers governance audits of providers, model audits before release, and application audits of LLM-based applications.METR’s collaborations included red teaming GPT-4 and Claude 2 for toxicity, bias, and frontier-AI risks.
  • Open-Source Governance: The open-source AI governance debate remains unsettled, with a prevailing view that model disclosure does not currently pose significant risks.The survey presents this as a tentative conclusion while preparing for future deliberation about more advanced systems.
  • Incorporating Values into AI Systems: Value-alignment governance must account for evolving social values and the sensitivity of consensus to which human groups contribute.The survey identifies dynamic values and subgroup exclusion as unresolved concerns for alignment.

6 Conclusion

The survey frames alignment around RICE objectives and forward/backward alignment, while highlighting challenges in eliciting human intent, assuring system behavior, and representing diverse human values. It proposes a broad, continually updated synthesis of research directions and practices across these challenges.

  • Alignment framework: Alignment targets Robustness, Interpretability, Controllability, and Ethicality through forward and backward alignment.Forward alignment uses training to make systems aligned; backward alignment gathers evidence and governs systems to avoid exacerbating misalignment risks.
  • Survey scope: The field’s diversity is organized around a shared goal rather than a shared methodology, enabling complementary research directions but raising barriers to entry.The survey addresses this need with a comprehensive and up-to-date overview intended for newcomers and experienced researchers.
  • Future directions: The alignment cycle motivates future directions spanning forward alignment, backward alignment, and their integration across research areas.The survey specifically outlines challenges and potential future directions based on this cycle.
  • Learning human intent: Binary feedback can underspecify true human intent, making it difficult to distinguish convincing outputs from genuinely sound ones.Richer modalities, including detailed text feedback and real-time interaction, are proposed to provide more nuanced evaluations.
  • Learning human intent: Reward modeling does not automatically recover human intent from Internet-scale pretraining because pretraining data does not bias the reward-modeling process toward the true intent.The survey identifies learning algorithms, inductive biases, and learner alignment as three dimensions for characterizing intent-learning methods.
  • Assurance and values: Assurance remains constrained by deceptive and backdoor behaviors, while value elicitation must address disagreement, limited representation, and human oversight limits.The survey discusses interpretability and security challenges, democratic input, and alternative approaches based on meta-level moral principles.
Loading 2310.19852v6…