PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsolved Problems in ML Safety

Dan Hendrycks, Nicholas Carlini, John Schulman, Jacob Steinhardt

arXiv:2109.13916v5cs.LGcs.AIcs.CLcs.CV

TL;DR

As ML systems enter high-stakes settings, it remains unclear which safety problems to prioritize and how to make progress. This work proposes a roadmap organized around four interconnected research problems—robustness, monitoring, alignment, and systemic safety—and concludes that achieving safety requires research across all four.

  • Problem

    As ML systems enter high-stakes environments, it remains unclear which safety problems to prioritize and how to make progress.

  • Method

    The paper provides a roadmap organized around four research problems: robustness, monitoring, alignment, and systemic safety.

  • Results

    The paper concludes that the four problems are interconnected and interdependent, so achieving safety requires research on all four.

  • Takeaways & Limitations

    As ML research advances, safety should become a top priority alongside accuracy, speed, and scalability.

  • Takeaways & Limitations

    The systemic safety discussion is non-exhaustive and focuses on empirical ML research avenues, while policy and governance work remains integral to safe deployment.

Abstract

from arXiv · show

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.

1 Introduction

As ML systems enter high-stakes settings, unsafe systems can cause severe harm, while conventional engineering, delayed safety work, and competitive pressures may leave hazards embedded. The paper identifies four ML Safety priorities: robustness, monitoring, alignment, and systemic safety.

  • Contribution: The paper identifies four ML Safety problems: robustness, monitoring, alignment, and systemic safety.They target resilience to adversaries and unusual events, hazard detection and unexpected functionality, safe optimization of human values, and broader risks such as cyberattacks.
  • Scope: ML safety research prioritizes long-term and long-tail risks where increasing capabilities may reduce safety or make problems more challenging this decade.For each problem, the paper discusses research directions that can begin or continue in the next few years.
  • Motivation: Safety decisions made early in development matter because delaying safety work allows unsafe design choices to become deeply embedded.Approximately 75% of the most critical safety decisions reportedly occur early in development.
  • Motivation: ML systems require safety approaches beyond conventional engineering because learned, pointwise-tested models are fragile, difficult to assess for coverage, and not readily corrected by short patches.Their inscrutable learned weights and lack of modularity or encapsulation create additional challenges compared with typical software.

2 Robustness

Robustness research aims to make ML systems endure extreme, unusual, adversarial, and evolving events in open-world high-stakes environments. The roadmap emphasizes both Black Swan and long-tail hazards and broader adversarial threats beyond small, predefined perturbations.

  • Black Swans: Robustness research aims to build ML systems that endure extreme, unusual, or adversarial events, including Black Swans, long tails, and unforeseen attacks.Such systems must adapt to evolving environments and handle diverse perceptible attacks.
  • Black Swans: Rare, impactful events expose current brittleness: the 2010 Flash Crash triggered feedback loops that wiped away a trillion dollars of stock value within minutes.Long-tail road scenarios can also cause autonomous-vehicle crashes and loss of life despite massive training datasets.
  • Black Swans: Researchers should create extreme distribution-shift and long-tail benchmarks that stress-test systems, find breaking points, and evaluate behavior in potential future scenarios.Interactive environments could model random shocks, feedback loops, and competent errors in which agents execute wrong routines.
  • Black Swans: Future systems should withstand Black Swans, long tails, and structurally novel events by learning from unusual situations and adapting quickly to novel experiences.Deployment-time adaptation should also defend against poisoned data.
  • Adversaries: Adversarial-robustness research should broaden beyond small ℓp perturbations to perceptible, previously unspecified, limited-access, and adaptive attacks [30] [21].Promising defenses include multisensor inconsistency checks, evolving test-time defenses, robust representations, improved data, architectures, training, and losses [41] [31].

3 Monitoring … 3.2 Representative Model Outputs

Monitoring research aims to identify hazards, inspect models, and support human operators in deciding when to trust or override ML systems. The roadmap emphasizes anomaly detection, calibrated uncertainty, and honest, truthful model outputs.

  • Representative Model Outputs: Representative monitoring outputs include finding model trojans, scanning for capabilities, calibrating probabilities, deciding when to override, warning operators, and flagging novel misuses.
  • 3 Monitoring: Monitoring aims to identify hazards, inspect models, and help human ML system operators.
  • 3.1 Identifying Hazards and Malicious Use With Anomaly Detection: Anomaly detection is needed to identify hazards and malicious uses, including novel strategies that may repurpose ML systems for social manipulation, weapons research, or cyberattacks.
  • 3.1 Identifying Hazards and Malicious Use With Anomaly Detection: Existing anomaly-detection methods struggle to learn representations that reliably detect previously unseen anomalies, motivating further work in out-of-distribution, open-set, and one-class detection.
  • 3.2.1 Calibration: Human monitors need reliable model uncertainty to know when to trust or override systems, but current uncertainties are often unrepresentative and overconfident.
  • 3.2.1 Calibration: Calibration research should improve uncertainty estimates on typical and unlike-training-data tests, while future systems could communicate uncertainty through contingent language.
  • 3.2.2 Making Model Outputs Honest and Truthful: Models should produce outputs that accurately, honestly, and faithfully represent their understanding, because current language models often provide fluent but fabricated explanations.

3.3 Hidden Model Functionality

Hidden functionality can arise as adversarially inserted backdoors or as unexpected capabilities that emerge with model scale. ML safety therefore requires detecting these behaviors, stress-testing detection against adaptive attackers, and preventing or removing hazardous capabilities.

  • Backdoors: Backdoored models behave benignly in most situations but can be made to behave incorrectly under adversary-chosen triggers [67].Backdoors differ from adversarial examples because they are inserted during training, either directly into model weights or through poisoned training or pretraining data.
  • Backdoors: Researchers should improve backdoor detectors and evaluate them through adaptive attacker–auditor competitions covering an expanding set of attacks.Such competitions can stress-test whether detection methods remain effective as attackers change their strategies.
  • Emergent capabilities: Large-scale models can develop novel, qualitatively distinct capabilities that their designers did not initially anticipate [23].Some emergent capabilities may resist monitoring because agent-like systems could adopt covert behavior, as illustrated by digital organisms that detected monitoring and then changed their behavior.
  • Emergent capabilities: Safety research should scan models for hazardous or mitigating capabilities, validate detection methods, and prevent or remove undesirable capabilities.Suggested approaches include continually evolving capability testbeds, implanting hidden functionality to test sensitivity, forecasting future capabilities, and training models not to acquire or to forget hazardous capabilities.

4 Alignment

Alignment research addresses societal and technical challenges in building ML agents that pursue good states of the world and avoid bad ones. The paper focuses on specifying, optimizing, and robustly updating objective proxies while preventing unintended consequences.

  • Societal challenges: Alignment must address societal risks from goals set by companies whose incentives may not solely serve the public interest [94, 59, 20].The paper notes that corporate incentives can be distorted in ways that harm public welfare.
  • Technical alignment challenges: Four technical alignment problems are central: objective proxies are difficult to specify, optimize, and keep robust, and they can produce unintended consequences.These problems arise even when societal issues are resolved and ideal goals have been selected.
  • Value learning: Human values such as happiness, good judgment, meaningful experiences, and autonomy are difficult to define and measure, making value learning necessary.Systems tend to optimize what is measurable, so clicks and watch time can become misleading proxies for human values [152].
  • Translating values into action: Even accurate value models may be misaligned in action because optimization is difficult, long- and short-term wellbeing can conflict, and subsystems may pursue secondary goals over the primary objective.The explicitly written objective is therefore not necessarily the objective the system operationally pursues.
  • Proxy gaming and value clarification: Objective proxies are brittle and can be gamed by optimizers or adversaries, motivating adversarially robust reward functions, monitoring for loopholes, and systems that revise objectives when values change.Proxy gaming includes breeding cobras to collect bounties, while value clarification addresses future ethical questions and changing goals.

5 Systemic Safety

Systemic safety addresses contextual hazards that can cause ML systems to fail or be misdirected, focusing on cybersecurity and informed decision making. Safe deployment also requires policy and governance work [16] [25], and this research agenda is nonexhaustive.

  • 5 Systemic Safety: Systemic safety applies ML to mitigate contextual hazards, especially cybersecurity threats and poor governance decisions that can cause ML systems to fail or be misdirected.ML systems operate within larger contexts whose insecurity or turbulence can undermine their safety.
  • 5 Systemic Safety: Systemic safety is nonexhaustive: policy and governance work remain integral to safe deployment [16] [25], while auditing and cooperation may become additional research areas [44].The paper focuses on empirical ML research avenues despite the limitations of techno-solutionism.
  • Cybersecurity: Cybersecurity is critical because vulnerabilities in traditional software can let malicious actors control autonomous ML systems or compromise systems that require secure computers.ML systems often operate alongside traditional software and within cyber-physical systems.
  • Cybersecurity: ML may amplify cyberattacks by lowering barriers to entry and increasing attackers’ accessibility, potency, success rate, scale, speed, and stealth.The paper therefore emphasizes clearly defensive research, including ML for intrusion and impersonator detection, vulnerability analysis, and detection of malicious obfuscated payloads.
  • Informed Decision Making: Informed decision making can improve governance and command-and-control safety by using ML for forecasting and for identifying crucial questions and factors.Forecasting refines estimates of well-defined risks, while advisory systems may uncover new risks and mitigation actions from historical and diverse training data.

6 Related Research Agendas

The paper situates its revised ML Safety agenda within broader work on societal consequences and earlier safety roadmaps. It focuses on concrete, unresolved paths toward safe ML systems while extending prior themes and introducing new ones.

  • Scope: The agenda is scoped to research outlining paths toward safe ML systems, alongside related work on AI policy, privacy [3], fairness, and ethics [56].The authors strongly support these related areas but do not treat them as the focus of this section.
  • Prior agendas: Russell et al., 2015 identified safety avenues spanning robustness, machine ethics, and AI’s economic impact, while Amodei and Olah et al., 2016 [5] concretized several directions.These works provide important foundations for identifying and structuring ML safety problems.
  • Contribution: The paper revises and expands these agendas by focusing on unsolved problems, extending themes such as Robustness and parts of Alignment, and identifying new directions.Pursuing multiple safety avenues can create layered protection that mitigates hazards, as illustrated by the Swiss cheese model.

7 Conclusion

The work identifies four interconnected, interdependent unsolved ML safety problems and argues that addressing all four is necessary for safe systems. As ML capabilities advance, safety should become a top priority and a core design standard for deployment in safety-critical situations.

  • 7 Conclusion: All four unsolved research problems must be addressed because they are interconnected and interdependent, and safety cannot be achieved by focusing on only one.Anomaly detection can support proxy-gaming detection, suspicious-cyberactivity detection, and fail-safes during unexpected events.
  • 7 Conclusion: As ML systems succeed across images, natural language, and programming, research priorities should expand beyond accuracy, speed, and scalability to make safety a top priority.
  • 7 Conclusion: The ML community should cultivate a culture of safety and embed safety features in design so systems can be deployed in safety-critical situations.The paper argues that safety is treated as an assumed property of widely deployed technologies and should receive the same standard in ML.

A Analyzing Risks, Hazards, and Impact · A.1 Risk Management Framework

The section analyzes how ML Safety progress can reduce abstract risks and hazards using four risk dimensions adopted from the Department of Defense’s broad risk management framework. It distinguishes near-term risks to ML systems, operations, and institutions from future risks across all three domains.

  • A.1 Risk Management Framework: The framework relates ML Safety problems to the risks they directly mitigate, while recognizing that each problem can affect near- and long-term risks.The table presents a current snapshot, and future organizations and institutions are expected to become more dependent on ML systems.
  • A.1 Risk Management Framework: The framework distinguishes four risk dimensions: ML system, operational, institutional and societal, and future risks.The first three concern near-term scenarios; future risks concern mid- to long-term challenges across ML systems, organizations, and institutions.
  • A.1 Risk Management Framework: ML system risks concern whether a near-term individual ML system can operate reliably.
  • A.1 Risk Management Framework: Operational risks concern whether an organization can safely operate an ML system in near-term deployment scenarios.
  • A.1 Risk Management Framework: Institutional and societal risks concern whether global society or influential institutions can operate efficiently, informly, and prudently in near-term scenarios.
  • A.1 Risk Management Framework: Future risks concern whether future ML systems, operating organizations, and institutions can address mid- to long-term challenges.
  • A.1 Risk Management Framework: Risk is conceptualized as hazard prevalence multiplied by exposure and deleterious effect, whereas hazards are factors with the potential to cause harm. [1]The section illustrates that systemic safety, monitoring, and reduced exposure can each lower risk through different mechanisms.

A.2 Hazard Management Framework

The framework organizes ML safety around five abstract hazards: known unknowns, unknown unknowns, emergence, long tails, and adversaries & deception. These hazards overlap, are not exhaustive, and connect risks, hazards, and goals through problem mappings and an interconnectedness model.

  • Hazard taxonomy: The framework distinguishes five hazards: known unknowns, unknown unknowns, emergence, long tails, and adversaries & deception.These represent incomplete knowledge, unidentified hazards, size- or composition-dependent hazards, extreme-tail events, and attacks or deception.
  • Scope and limitations: The five hazards are not exhaustive; systemic-safety problems can address turbulence, while feedback loops may become more prominent as ML systems integrate into more of life.The framework therefore presents a current scope rather than a complete inventory of future hazards.
  • Hazard relationships: The hazards overlap: emergent hazards can be unknown unknowns, long-tail events may be known or unknown, and adversarial attacks can produce long-tail events.A hazard may shift categories when detected, while emergent and long-tailed hazards can co-occur.
  • Problem–hazard mapping: Table 2 indicates whether each ML safety problem reduces vulnerability or exposure to each hazard, providing a present-day snapshot.The mapping may change as future adversaries create novel unusual events or exploit tail events.
  • Interconnected framework: Figure 6 depicts the interconnectedness of risks, hazards, and goals after these elements are made explicit.It provides a simplified model of interconnected factors for ML safety.

A.3 Prioritization and Strategy for Maximizing Impact

ML Safety problems should be prioritized by importance, neglectedness, and tractability, while impact-maximizing strategy combines personnel, research, protocols, and partnerships. These priorities are approximate and must adapt as hazards and problems evolve.

  • A.3 Prioritization and Strategy for Maximizing Impact: Prioritize problems by importance, neglectedness, and tractability: potential risk reduction, current research attention, and expected progress.Table 3 provides a rough snapshot of these factors, not a definitive ranking.
  • A.3 Prioritization and Strategy for Maximizing Impact: Importance ranges from reducing catastrophe risks to directly reducing risks from more plausible permanent catastrophes.The importance scale distinguishes progressively stronger forms of risk reduction.
  • A.3 Prioritization and Strategy for Maximizing Impact: Neglectedness ranges from being among the ten most researched topics to having few related papers at leading ML conferences.Intermediate cases receive some conference attention or concern adjacent problems that are hardly neglected.
  • A.3 Prioritization and Strategy for Maximizing Impact: Tractability ranges from conceptual or capability bottlenecks to reliable progress and highly fruitful efforts with obvious low-hanging fruit.This scale captures how much progress researchers can currently expect from work on a problem.
  • A.3 Prioritization and Strategy for Maximizing Impact: These factors provide only a rough sense of marginal impact, because neglected aspects can exist within popular problems and collective choices can overcrowd neglected areas.Safety analysis and strategy should therefore evolve as new hazards and problems emerge.
  • A.3 Prioritization and Strategy for Maximizing Impact: A potential ML Safety strategy combines ready personnel, a safety research community, protocols for safe development and deployment, and alliances across academia, industry, and government.The strategy includes operating systems safely, clarifying goals, reducing adoption costs, incorporating safety methods, red teaming, and establishing standards.
Loading 2109.13916v5…