Black-Box Access is Insufficient for Rigorous AI Audits

MIT CSAIL

• The listed authors include Benjamin Bucknall, Lee Sharkey, and Alan Chan, affiliated respectively with the Centre for the Governance of AI, Apollo Research, and Mila and the Centre for the Governance of AI.The author list also includes the MIT CSAIL section label.Source: MIT CSAIL · p. 1

CCS CONCEPTS

• The paper is classified under social aspects of security and privacy, and governmental regulations.Source: CCS CONCEPTS · p. 1

KEYWORDS

• The keywords include auditing, evaluation, governance, regulation, policy, risk, fairness, and black-, white-, and outside-the-box access.They also include adversarial attacks, interpretability, explainability, and fine-tuning.Source: KEYWORDS · p. 1

1 INTRODUCTION

• Audit effectiveness varies with system access, yet norms remain unsettled and calls for audits often do not specify access requirements.The introduction frames external evaluation as part of AI oversight and governance, with audits structured to identify risks and improve transparency.Source: §1 · p. 2
• Black-box audits observe outputs for inputs, but can miss anomalous failures and testing data can reinforce dataset biases.The introduction says prominent systems' voluntary external audits have primarily analyzed input/output behavior.Source: §1 · p. 2
• White-box access supports stronger attacks, analysis of internal mechanisms, and fine-tuning, while outside-the-box information helps assess methodology- and data-related risks and design useful tests.Outside-the-box information can include methodology, code, documentation, hyperparameters, data, deployment details, and internal evaluation findings; secure audits may use APIs, secure research environments, and legal mechanisms to reduce leak risks.Source: §1 · p. 2
• White- and outside-the-box access enable substantially more scrutiny than black-box access, so audits seeking higher scrutiny should grant higher access.Auditors should also disclose model access and evaluation methods so readers can properly interpret audit results.Source: MIT CSAIL · p. 1, §1 · p. 2

2.1 Black, Grey, White, and Outside-the-Box Access

• Access ranges from black-box input/output querying through limited grey-box internals and de facto arbitrary processing to full white-box access, while outside-the-box access supplies development and deployment information.Grey-box access can expose embeddings, activations, or sampling probabilities; white-box access includes weights, activations, gradients, and fine-tuning. Outside-the-box information can range from model cards to comprehensive training and testing documentation.Source: §2.1 · p. 3

2.2 Regulatory Frameworks’ Reliance on Audits

• Governance frameworks call for audits and risk evaluations, including external oversight of potentially harmful AI capabilities.Examples include bias audits for some employment tools and federal requirements for certain foundation-model test results, alongside NIST evaluation guidance and tools for assessing threats to security and critical infrastructure.Source: §2.2 · pp. 3–4

2.3 Audits in the Status Quo

• As of January 2024, audits lacked widely adopted norms, and evaluations were predominantly in-house or external efforts with black-box and limited outside-the-box access.Raji et al. identifies scope, independence, access, professionalism, and public disclosure of methods and results as general audit limitations.Source: §2.3 · p. 5

3 LIMITATIONS OF BLACK-BOX ACCESS

• Black-box audits rely on inputs and outputs, so the vast input space forces heuristic test selection that can miss jailbreaks, adversarial inputs, and backdoors [58] [325] [353].They commonly assess test-set performance or try to elicit undesirable behavior, but these methods cannot provide a complete understanding.Source: §3 · p. 5
• Black-box access obscures system components, preventing separate analysis of filters and limiting practical assessment of data-related and deployment impacts.Datasets can inform privacy, copyright, and contamination evaluations, [72] [105] [131], while deployment knowledge helps assess broader societal impacts [328].Source: §3 · p. 5
• Query design can bias black-box evaluations and produce misleading conclusions, including apparent fairness that masks biased underlying reasoning.Developers who know the tests can tune outputs for test cases while flaws remain; prompt-based evaluation can also mischaracterize emergent capabilities [270].Source: §3 · p. 5
• Black-box explanations are often unreliable because counterfactuals may fail to identify causal relationships and language-model justifications may not reflect actual reasoning.Explanation methods can also be exploited to mislead about harmful decisions [6], and generated justifications tend not to be faithful [310].Source: §3 · p. 5
• Black-box findings offer little guidance for root-cause fixes: training on problematic examples may fail, be sample-inefficient, or introduce new issues [78] [97] [322].Safety measures can be almost entirely undone by fine-tuning on a small number of harmful examples [168] [242] [336] [342], whereas white-box methods can enable more precise debugging [322].Source: §3 · p. 5

4.1 White-box attack algorithms are more effective and efficient.

• Gradient-based white-box attacks can find vulnerabilities more effectively than unguided search, while many black-box attacks transfer white-box attacks or inefficiently estimate gradients.For language models, text is discrete, but white-box approaches include differentiable sampling approximations, embedding projection, and gradient-informed searches.Source: §4.1 · p. 6
• Latent-space attacks can uncover hard-to-find failures more efficiently than input-space attacks, supporting stronger safety assurances and enabling direct gradient optimization for language models.They target issues such as backdoors and deception; research reports that latent-attack robustness can indicate input-attack robustness in vision models.Source: §4.1 · p. 6
• White-box attacks complement black-box methods with qualitatively different failures, including unintelligible adversarial prompts, and hybrid combinations can help humans find vulnerabilities.Some black-box approaches remain competitive, especially for language models, but relying on them alone may miss failures unlikely to emerge from black-box searches or human intuitions.Source: §4.1 · p. 6

4.2 White-box interpretability tools aid in diagnostics.

• White-box interpretability can reveal novel attacks, spurious or brittle representations, and architectural limitations, while helping pinpoint where developers can address problems.Internal representations and concept probes can also help identify capabilities or concepts a model represents, though these methods remain imperfect and under active research.Source: §4.2 · pp. 6–7
• Mechanistic understanding can provide evidence against failure modes by predicting behavior across broad input classes, offering stronger assurances than finite black-box queries alone.Scaling thorough analysis remains an open challenge; sparse autoencoders are among approaches being explored to study features in large language models.Source: §4.2 · p. 7
• White-box access strengthens explanations of individual decisions with gradient-based methods and model-mechanism interpretations, beyond potentially misleading black-box input-attribution techniques.Black-box attribution can fail to identify causal relationships between input features and outputs.Source: §4.2 · p. 7

4.3 Fine-tuning reveals risks from latent knowledge or post-deployment modifications.

• Fine-tuning can expose harmful capabilities that may persist after attempted removal and resurface through jailbreaks or small post-deployment updates, informing deployment risk assessment.Risks are particularly relevant when models are leaked, open-sourced, or offered with fine-tuning access.Source: §4.3 · p. 7

• Code, documentation, and hyperparameters streamline technical evaluations; using developers’ fine-tuning resources is more efficient than rebuilding code and configurations from scratch.Fine-tuning often requires carefully selected hyperparameters and precisely configured code developed through extensive testing.Source: §5 · p. 7
• Development methods can reveal shortcuts and guide risk tests; non-representative human training data can signal possible social bias [46] [268] [298].Internal evaluation findings help auditors target complementary evaluations, while information about attempted mitigations—including fine-tuning data and old and new model weights—helps assess their effectiveness [239].Source: §5 · pp. 7–8
• Training-data access lets auditors trace harmful or biased content [23] [24] [98] [182], investigate poisoning [41] [254] [319] [321], and assess copyright fair use [120] [139].Auditors may need data to determine whether copyrighted material was used in accordance with copyright law.Source: §5 · p. 8
• Documentation and training details expose developers’ choices and responses, enabling accountability and potentially deterring risky decisions [200] [190] [46] [159]; documentation obligations can carry regulatory penalties.Dataset and training details can help trace risks to intentional choices, while internal evaluation reports show how developers responded to findings.Source: §5 · p. 8

Technical

• Structured APIs can provide de facto white-box access for indirect parameter operations, while flexible query APIs let auditors design evaluations on mock models and data before running them on the real system.GPT-3.5 API fine-tuning is one example; more customizable APIs would enable more flexible access.Source: §6 · p. 8
• API safeguards must balance comprehensive, flexible auditing against reconstruction risks; sharing some weights may be safe, while sharing weight–activation products may reveal parameters.The passage calls for adjudicating developer grievances about auditor requests and identifies secure auditing infrastructure as an open research need.Source: §6 · p. 8

Physical

• Secure research environments at developers’ facilities can provide unrestricted white-box access more flexibly than APIs while limiting leakage, though auditor incentives for compliance remain uncertain.On-site safeguards could adapt employee protections such as NDAs, but the approach imposes physical attendance costs.Source: §6 · p. 8

Legal

• Legal safeguards adapted from financial auditing include confidentiality training and contracts, client-specific engagement terms, common standards, and independence enforcement to hold auditors accountable for leaks.The passage notes that employee knowledge-sharing may pose greater risks than auditor leakage because developers recruit researchers from competitors.Source: §6 · pp. 8–9

7 DISCUSSION

• White- and outside-the-box evaluations can boost developers’ credibility and provide clearer diagnoses for targeted fixes than black-box evaluations.Black-box evaluations can establish that problems exist, while more permissive access can help identify the components responsible.Source: §7 · p. 9
• Developers have incentives to limit external access, and known audits of state-of-the-art systems have been black-box [147] [306].Concerns include leaks [29] and exposure of poor performance, vulnerabilities, or risk management that could cause reputational harm or legal liability.Source: §7 · p. 9
• Entrenched black-box norms and industry lobbying to restrict access could impede independent audit research and bias public understanding [147] [306].Investment, research, and development in auditing techniques and infrastructure can facilitate progress.Source: §7 · p. 9
• Poor audits can falsely increase public or regulatory trust and enable safety- or ethics-washing by developers.Such effects can prevent appropriate external scrutiny while developers fail to invest sufficiently in addressing risks.Source: §7 · p. 9
• White- and outside-the-box access enables substantially more scrutiny than black-box access, but rigorous audits also require transparent reporting and careful institutional design [11] [25] [67] [218].When greater scrutiny is desired, audits should use higher levels of access; access is necessary but not sufficient for rigor.Source: §7 · p. 9

ETHICS STATEMENT

• The authors expect the paper’s foreseeable impacts to be positive, but note that audits can fail to benefit the public for several reasons discussed in Appendix D.The stated basis for the expected positive impact is rigorous audits’ role in improving accountability and representing public interests.Source: ETHICS STATEMENT · p. 9

• Carson Ezell and Stephen Casper were the central writers and organizers.Source: CONTRIBUTIONS · p. 9
• Charlotte Siegmann, Kevin Wei, Andreas Haupt, and Taylor Curtis contributed primarily to Section 2; Marvin von Hagen and Silas Alberti contributed primarily to Appendix A and Section 2.3.Source: CONTRIBUTIONS · p. 9
• Noam Kolt contributed primarily to Section 7 and Appendix D; Jérémy Scheurer, Marius Hobbhahn, and Lee Sharkey contributed primarily to Section 3, Section 4.2, Section 7, Appendix C, and Appendix D.Source: CONTRIBUTIONS · p. 9
• Satyapriya Krishna contributed primarily to Section 4.1; Qinyi Sun, Michael Gerovitch, and Benjamin Bucknall contributed primarily to Section 2.1, Section 6, and Section 7.Source: CONTRIBUTIONS · p. 9
• Alan Chan, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell offered high-level feedback and guidance.Source: CONTRIBUTIONS · p. 9

A MOTIVATIONS FOR EXTERNAL AUDITS

• Audits formally evaluate systems for risks, compliance, and stakeholder priorities, with problem identification as their most direct purpose.Independent external audits are presented as a way to assess systems and practices.Source: A MOTIVATIONS FOR EXTERNAL AUDITS · p. 18
• Audits can incentivize better controls, risk assessment, and documentation while increasing transparency and enabling fixes, guardrails, monitoring, or follow-up studies.Public audit information can help regulators and researchers understand system behavior and limitations.Source: A MOTIVATIONS FOR EXTERNAL AUDITS · p. 18
• Audits offer a middle ground between secrecy and open-sourcing, allowing independent risk assessment while maintaining security, and passing them can increase trust in developers and their systems.Source: A MOTIVATIONS FOR EXTERNAL AUDITS · p. 18

B TECHNICAL ASSISTANCE AS A FORM OF OUTSIDE-THE-BOX ACCESS

• Auditors may need computing infrastructure and developer engineering support for fine-tuning, realistic test cases, or tool integrations that better resemble real-world use.Developers may also have unique practical knowledge of working effectively with their models.Source: B TECHNICAL ASSISTANCE AS A FORM OF… · p. 18
• Anthropic said it could have helped METR identify concerning behavior more successfully by sharing model expertise and collaborating on the audit approach.Getting models to perform near their capability limits is described as fundamentally difficult.Source: B TECHNICAL ASSISTANCE AS A FORM OF… · p. 18
• Because arbitrary developer influence can undermine audit independence, requiring technical assistance in legal frameworks may be difficult and is beyond the paper’s scope, though good-faith responses to specific requests may help.Source: B TECHNICAL ASSISTANCE AS A FORM OF… · p. 18

Research gaps

• White-box research struggles to keep pace with leading systems, especially versatile and complex state-of-the-art language models, leaving gaps between available tools and auditors’ needs.Both foundational research and practical tools need further development.Source: C INNOVATION ON AUDITING TOOLS · p. 18

Basic research

• Current methods do not yet enable detailed mechanistic understanding of state-of-the-art models, so progress is needed in neural-network science, interpretable systems, and techniques for interpreting trained models.Source: C INNOVATION ON AUDITING TOOLS · p. 18

Practical tools

• Adversarial benchmarks emphasize small input perturbations rather than real-world harms, while interpretability benchmarks rarely target practical tasks and often rely on researcher intuition [122] [141] [3] [50] [77].Source: C INNOVATION ON AUDITING TOOLS · p. 18
• Applying evaluation techniques to real systems is the most directly informative way to benchmark them, and researcher–developer partnerships can enable such applications.Open-source and API access to advanced systems has helped evaluation-tool development, but their scale and complexity remain challenging.Source: C INNOVATION ON AUDITING TOOLS · p. 18

Secure auditing infrastructure

• Secure APIs and research environments can reduce leak risks, but audits lack established norms and infrastructure; projects such as the US National Deep Inference Facility could expand auditor resources.Source: C INNOVATION ON AUDITING TOOLS · p. 18

Audit norms

• Establishing audit norms and a regulatory framework is also a priority [181].Source: C INNOVATION ON AUDITING TOOLS · p. 19

Poorly-resourced audits

• Audits require compute and technical expertise, yet problem-finding may not be financially sustainable, and existing audits have relied largely on private funding [147] [222].The authors call for more sustainable, reliable, and diversified funding sources.Source: D BEYOND ACCESS: OTHER ASPECTS OF… · p. 19

Limitations with technical tools

• Gaps between existing evaluation tools and those needed to reliably assess advanced-system safety and trustworthiness limit auditors’ ability to identify risks.The paper points to this tooling gap as an unresolved constraint on audit quality.Source: D BEYOND ACCESS: OTHER ASPECTS OF… · p. 19

Narrowly-scoped audits

• Narrowly scoped audits can omit major concerns or occur only before deployment; changing components make “black cloud” systems especially difficult to evaluate [147] [222].The early GPT-4 audits focused on risk-related capabilities and did not appear to externally evaluate adversarial robustness, misinformation, demographic representation, or societal welfare, democracy, discrimination, and equality.Source: D BEYOND ACCESS: OTHER ASPECTS OF… · p. 19

Exclusion of under-represented viewpoints

• Audits can exclude under-represented groups when they fail to account for diverse interests; meaningful participation and stakeholder dialogue are indispensable for fairness and representation [22] [76] [91] [296].The paper emphasizes both participation and dialogue among diverse stakeholders.Source: D BEYOND ACCESS: OTHER ASPECTS OF… · p. 19

Cosmetic compliance

• Without clear legal requirements, companies may engage in cosmetic compliance, creating a superficial and misleading appearance of good-faith cooperation.The paper characterizes this as cheap talk or virtue signaling.Source: D BEYOND ACCESS: OTHER ASPECTS OF… · p. 19

Regulatory capture

• Companies may selectively disclose audit-relevant information and shape audit criteria, metrics, and institutions, including which auditors have privileged access; unclear standards and inconsistent tools make AI audits especially susceptible [67].Source: D BEYOND ACCESS: OTHER ASPECTS OF… · p. 19