Source-linked AI summary
Black-Box Access is Insufficient for Rigorous AI Audits
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, Dylan Hadfield-Menell
TL;DR
AI audit effectiveness varies with the access auditors receive, while black-box evaluation is limited in detecting failures and explaining their causes. The paper examines expanded access and secure API designs, concluding that transparency about access and methods is necessary and that white- and outside-the-box access enables substantially more scrutiny than black-box access alone.
Problem
AI audit effectiveness can vary with the degree of system access, while black-box evaluations are limited to inputs and outputs and can miss failures such as jailbreaks, adversarial inputs, and backdoors.
Method
The paper examines white- and outside-the-box auditing and discusses structured API designs that enable indirect white-box processes while restricting direct parameter access.
Results
White- and outside-the-box access allow substantially more scrutiny than black-box access alone, while transparency about model access and evaluation methods is necessary to interpret audit results.
Takeaways & Limitations
When higher levels of scrutiny are desired, audits should be conducted with higher levels of access, although expanded access alone is not sufficient for rigor.
Takeaways & Limitations
Designing APIs that simultaneously provide the comprehensiveness, flexibility, and security required for rigorous auditing remains an open area of research.
Abstract
from arXiv · showhide
External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system's inner workings (e.g., weights, activations, gradients) allows an auditor to perform stronger attacks, more thoroughly interpret models, and conduct fine-tuning. Meanwhile, outside-the-box access to training and deployment information (e.g., methodology, code, documentation, data, deployment details, findings from internal evaluations) allows auditors to scrutinize the development process and design more targeted evaluations. In this paper, we examine the limitations of black-box audits and the advantages of white- and outside-the-box audits. We also discuss technical, physical, and legal safeguards for performing these audits with minimal security risks. Given that different forms of access can lead to very different levels of evaluation, we conclude that (1) transparency regarding the access and methods used by auditors is necessary to properly interpret audit results, and (2) white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.
MIT CSAIL
AI audits vary substantially with the access auditors receive: black-box access limits evaluation to inputs and outputs, whereas white- and outside-the-box access enables broader scrutiny. The paper argues that access and methods must be transparent because white- and outside-the-box audits support more scrutiny than black-box audits alone.
- Limitations of Black-Box Access: Black-box audits analyze only system inputs and outputs, limiting their ability to identify harms and provide meaningful explanations.These audits commonly assess test-set performance or search for harmful outputs.
- Access Beyond Black Box: White-box access to weights, activations, gradients, and fine-tuning enables stronger attacks, internal-mechanism analysis, and detection of dormant harmful capabilities.Outside-the-box access adds methodology, code, documentation, hyperparameters, data, deployment details, and internal evaluation findings.
- Conclusions: The paper concludes that audit reports should disclose access and methods, and that higher scrutiny requires higher levels of access.It also presents technical, physical, and legal safeguards for conducting broader audits while minimizing security risks.
2 BACKGROUND
The paper distinguishes black-, grey-, white-, and outside-the-box access and situates audits within emerging AI governance and regulatory frameworks. These frameworks seek trustworthy assessments of harms, risks, system behavior, and explanations, but audit norms and access expectations remain unsettled.
- 2.1 Black, Grey, White, and Outside-the-Box Access: Black-box access permits input design, system queries, and output analysis, whereas grey-box access provides limited information about internal workings.Grey-box information can include input embeddings, neuron activations, or sampling probabilities.
- 2.1 Black, Grey, White, and Outside-the-Box Access: White-box access provides full system access, including weights, activations, gradients, and the ability to fine-tune the model.
- 2.1 Black, Grey, White, and Outside-the-Box Access: Outside-the-box access provides development and deployment information such as methodology, source code, documentation, hyperparameters, training data, and internal evaluation findings.Comprehensive training and testing documentation is more informative than high-level details such as a model card.
- 2.2 Regulatory Frameworks’ Reliance on Audits: AI governance proposals and regulatory frameworks increasingly rely on audits to assess potential harms, system risks, quality, performance, and explanations.Examples include bias audits for employment tools and recommendations for system design and reliable operation.
- 2.2 Regulatory Frameworks’ Reliance on Audits: Table 1 summarizes which evaluation techniques are possible under black-, grey-, and white-box access, while Table 2 summarizes advantages of white- and outside-the-box techniques.
- 2.2 Regulatory Frameworks’ Reliance on Audits: Audit norms are not yet well established, and audit effectiveness can vary with the degree of access granted to evaluators.The paper notes that existing audit calls are often agnostic about access form.
3 LIMITATIONS OF BLACK-BOX ACCESS
Black-box audits rely on inputs and outputs, limiting evaluators’ ability to find elusive failures, understand system components and causes, and design effective remedies. They can therefore produce misleading or unreliable conclusions about system behavior and explanations.
- Generalizability: Black-box evaluations rely on queries and outputs, forcing evaluators to use heuristics because the space of possible inputs is intractably large.This makes failures outside typical test sets, including jailbreaks, adversarial inputs, and backdoors, unreliable to detect.
- System understanding: Black-box access prevents evaluators from studying system components separately, limiting targeted diagnosis and intervention.It also obscures filters, datasets, and deployment details relevant to privacy, copyright, contamination, and societal-impact assessments.
- Reliability: Black-box evaluations can produce misleading results because conclusions depend on evaluator-designed queries and may overlook flaws in underlying reasoning.Developers with knowledge of tests can modify outputs on test cases while unresolved internal problems remain.
- Explanations: Black-box explanation methods often fail to identify causal relationships between inputs and outputs and can generate unfaithful or exploitable explanations.This includes misleading counterfactuals and language-model justifications that do not reflect actual reasoning.
- Remediation: Black-box evaluations offer little guidance for fixing failures, while example-based fine-tuning can be sample-inefficient, fail at the root, or introduce new issues.Safety measures in language models can also be almost entirely undone by fine-tuning on a small number of harmful examples.
4 ADVANTAGES OF WHITE-BOX ACCESS
White-box access expands AI auditing beyond input-output testing by enabling stronger attacks, internal diagnosis, interpretability, and fine-tuning. These techniques can reveal otherwise difficult-to-find failures, support stronger assurances, and enable more precise remediation.
- 4.1 White-box attack algorithms are more effective and efficient.: White-box access enables gradient-based adversarial attacks that are more effective than unguided black-box searches.For language models, researchers use differentiable sampling approximations, adversarial embeddings, and gradient-informed textual searches despite discrete inputs.
- 4.1 White-box attack algorithms are more effective and efficient.: Many black-box and grey-box attacks indirectly reproduce white-box methods or estimate gradients inefficiently through extensive queries.Some effective attacks against GPT-4 and Claude-2 transferred attacks generated against open-source white-box models.
- 4.1 White-box attack algorithms are more effective and efficient.: Latent-space attacks target internal representations to diagnose misconceptions, anomalous failures, backdoors, and deception that input-space attacks may miss.They require less gradient propagation, enabling more thorough debugging under limited time and computing budgets.
- 4.1 White-box attack algorithms are more effective and efficient.: White-box methods complement black-box attacks by producing qualitatively different adversarial inputs, including unintelligible prompts that black-box searches would likely miss.The two approaches can also be combined into hybrid attacks.
- 4.2 White-box interpretability tools aid in diagnostics.: Interpretability tools can reveal novel attacks, spurious or brittle representations, and limitations of transformer key-value memories.Attributing problems to specific architectural components or representations supports more precise remediation.
- 4.2 White-box interpretability tools aid in diagnostics.: White-box methods allow more precise identification of a system’s knowledge and capabilities through tools such as concept vectors and probes.These tools have been used to study fairness-related features in visual classifiers.
- 4.2 White-box interpretability tools aid in diagnostics.: Mechanistic understanding provides evidence against the existence of failure modes, complementing black-box methods that can only demonstrate failures by finding eliciting inputs.Finite black-box queries cannot guarantee properties of a system without additional assumptions.
- 4.2 White-box interpretability tools aid in diagnostics.: White-box access strengthens explanations through gradient-based techniques and by connecting model decisions to internal mechanisms and abstract concepts.Black-box input-modification methods can be misleading and may fail to identify causal relationships.
5 ADVANTAGES OF OUTSIDE-THE-BOX ACCESS
Outside-the-box access gives auditors contextual information that improves test design, reveals development risks, helps trace problems, and strengthens accountability.
- Methodological details, source code, documentation, hyperparameters, training data, deployment details, and internal evaluation findings support more effective audits.
- Developer code and documentation streamline technical evaluations, while validated hyperparameters can make fine-tuning evaluations more efficient.
- Methodological details can reveal development shortcuts and non-representative human data that suggest specific risks or social biases.
- Internal evaluation findings help auditors target complementary tests and assess whether developers’ mitigation efforts were effective.
- Training-data access helps auditors identify harmful or biased content, investigate data-poisoning risks, and assess fair use.
- Documentation, training details, and internal reports increase scrutiny of development decisions and can deter developers from taking risks.
6 METHODS TO ADDRESS SECURITY RISKS
White- and outside-the-box audits can be secured through technical, physical, and legal safeguards, but designing access that is simultaneously comprehensive, flexible, and secure remains difficult.
- Secure technical, physical, and legal mechanisms can reduce leakage risks while allowing auditors to access models and related resources.
- Technical safeguards: Structured APIs can provide de facto white-box access by enabling indirect white-box processes while restricting direct access to model parameters.
- Technical safeguards: Flexible query APIs let auditors develop evaluations on mock models and data before submitting them to run on the true system.
- Technical safeguards: API security cannot be ensured simply by prohibiting weight sharing because some derived information may permit parameter reconstruction.
- Physical safeguards: On-site secure research environments can provide unrestricted white-box access while reducing risks of model leakage or reconstruction.
- Legal safeguards: Financial auditing offers legal models including confidentiality policies, tailored engagement terms, common standards, and conflict-of-interest requirements.
7 DISCUSSION
Greater audit access can improve developer credibility, problem-solving, and oversight, but current incentives favor black-box audits that may produce misleading confidence and safety-washing.
- White- and outside-the-box evaluations can increase developers’ credibility and provide greater insight into addressing system problems.
- Absent legal requirements, developers have incentives to limit external auditors’ access, and public audits of state-of-the-art systems have primarily remained black-box.
- Black-box audit norms may become entrenched as future standards, while limited access and resources can constrain non-industry researchers.
- Investment, research, and development in auditing techniques and infrastructure can facilitate further progress.
- Poor black-box audits can increase trust in AI systems on false grounds and enable developer safety- or ethics-washing.
- The paper concludes that audit access and methods should be transparent, and that white- and outside-the-box access enable substantially more scrutiny than black-box access alone.
ETHICS STATEMENT
The authors expect the paper’s foreseeable impacts to be positive because rigorous audits can improve accountability and represent public interests, while acknowledging that audits can fail to benefit the public.
- The paper expects positive foreseeable impacts because rigorous audits can improve accountability and represent public interests.
- Audits can fail to benefit the public for a variety of reasons discussed in the paper’s appendix.
CONTRIBUTIONS
The paper’s writing and organization were led by Carson Ezell and Stephen Casper, with other contributors taking primary responsibility for specified sections and appendices.
- Carson Ezell and Stephen Casper were the central writers and organizers.
- Charlotte Siegmann, Kevin Wei, Andreas Haupt, and Taylor Curtis primarily contributed to Section 2.
- Other contributors primarily supported Sections 3, 4.1, 4.2, and 7, plus Appendices A, C, and D.The passage assigns these contributions across Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Noam Kolt, Marvin von Hagen, and Silas Alberti.
A MOTIVATIONS FOR EXTERNAL AUDITS
External audits identify risks, assess development practices, and support accountability, but effective auditing also depends on resources and cooperation that preserve auditor independence.
- Audits formally evaluate systems for risks, regulatory compliance, and other stakeholder-relevant desiderata.
- Audits can identify problems and assess internal controls, risk assessment, and documentation, thereby supporting accountability for development practices.
- Auditors may need computing infrastructure and technical assistance from developers, especially when evaluating large language models.
- Developer collaboration can improve evaluations because developers possess practical knowledge of their models and prompt engineering and fine-tuning expertise.
- Technical-assistance requirements may undermine audit independence, although auditors may benefit when auditees answer specific requests in good faith.
C INNOVATION ON AUDITING TOOLS
The paper argues that auditing tools must become more practical and capable of handling state-of-the-art systems, supported by real-system applications, secure infrastructure, and audit norms.
- White-box evaluation tools have not kept pace with leading AI systems, leaving gaps between current methods and auditors’ needs.
- Detailed mechanistic understanding of state-of-the-art models is not yet possible, requiring progress in neural-network science and interpretability techniques.
- Practical tools: Existing benchmarks often emphasize small input perturbations or researcher intuition rather than practical, real-world auditing tasks.
- Practical tools: Applications on real systems provide especially informative benchmarks, while researcher–developer partnerships can facilitate progress on evaluation tools.
- Secure auditing infrastructure: Secure research environments and application programming interfaces can reduce leak risks, but auditing infrastructure and regulatory norms remain underdeveloped.
D BEYOND ACCESS: OTHER ASPECTS OF RIGOROUS AUDITS.
Rigorous audits require more than expanded system access: funding, technical tooling, broad scope, independence, representation, and resistance to superficial compliance or regulatory capture also matter.
- White- and outside-the-box access is necessary but not sufficient for rigorous audits because multiple factors can degrade audit quality.
- Poorly-resourced audits: State-of-the-art audits require compute and technical expertise, while reliance on private funding may undermine their financial sustainability.
- Limitations with technical tools: Gaps between available evaluation tools and the tooling needed to assess advanced systems limit reliable identification of risks.
- Narrowly-scoped audits: Narrow audits may omit concerns beyond risk-related capabilities or evaluate systems only before deployment.
- Conflicts of interest, excluded viewpoints, cosmetic compliance, and regulatory capture can further weaken audit rigor and representation.