PaperlayerView exact revision on arXiv

Source-linked AI summary

AI auditing: The Broken Bus on the Road to AI Accountability

Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, Inioluwa Deborah Raji

arXiv:2401.14462v1cs.CY

TL;DR

AI audits are promoted as an accountability measure, but their practical outcomes do not consistently translate into concrete action. This paper surveys audit practices and their consequences across domains, finding that impact varies and is often limited. It connects audit design, execution, and institutional context with meaningful accountability.

  • Problem

    AI audit studies do not consistently lead to concrete accountability outcomes, making their practical impact difficult to establish.

    Source: §I · pp. 1–2

  • Method

    The paper surveys audit practices across academia and six other domains, taxonomizes their contexts, goals, and practices, and examines their consequences.

    Source: §I · p. 2, 3) Impact · p. 5

  • Results

    Concrete impact is fairly uncommon in AI audits, and academic studies often lack the impact seen in journalism, civil society, government, and industry.

    Source: §I · pp. 1–2, §VI · p. 11

  • Takeaways & Limitations

    The paper identifies expanding audit aims beyond evaluation and engaging excluded stakeholders more explicitly as practical directions for more impactful academic audits.

    Source: §VI · p. 11

    Some societal impacts, including surveillance’s chilling effect and corporate power concentration, are not amenable to audits and require other arguments for advocacy.

    Source: 8) Recognizing the limits of audits · p. 12

Abstract

from arXiv · show

One of the most concrete measures to take towards meaningful AI accountability is to consequentially assess and report the systems' performance and impact. However, the practical nature of the "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. Next, we assess the impact of audits done by stakeholders within each domain. We find that only a subset of AI audit studies translate to desired accountability outcomes. We thus assess and isolate practices necessary for effective AI audit results, articulating the observed connections between AI audit design, methodology and institutional context on its effectiveness as a meaningful mechanism for accountability.

I. INTRODUCTION

AI systems raise varied risks, but AI audits do not consistently lead to concrete accountability outcomes. The paper surveys audit practices and consequences, and defines an AI audit by its independence, specific target, and accountability purpose.

A. What is an AI audit?

  • An AI audit is an evaluation independent of AI development, conducted to support accountability judgments about AI systems.AI here encompasses deployed products with a significant algorithmic or data-defined component, including risk assessments, models, and recommendation systems.Source: §I · p. 1

I. INTRODUCTION

  • AI audits do not consistently translate into concrete actions such as product recalls, redesigns, corporate policy changes, or government regulation.Such impact is described as fairly uncommon in the AI audit context.Source: §I · pp. 1–2
  • The survey examines audit practices across academia and six other domains, classifying their contexts, goals, and practices and assessing whether outcomes match stated objectives.It investigates the consequences of audit investigations as a mechanism for accountability.Source: §I · p. 2

A. What is an AI audit?

  • Audits require an operationally separate assessor and a concrete, identified target, which may be a real deployment or a widely used open-source proxy.The assessment must compare deployment reality with articulated expectations connected to consequential judgments; abstract evaluations without that connection are excluded.Source: A. What is an AI audit? · p. 2
  • Audit targets and expectations can vary widely, spanning AI systems and domains, with assessments addressing performance, safety, or broader societal injustices.Expectations may range from named principles to specific legal requirements or performance thresholds.Source: A. What is an AI audit? · p. 2

B. Who conducts AI audits?

AI audits involve practitioners with different relationships to their targets, and their processes can vary. The paper describes a broad sequence from discovering harms and setting expectations to evaluating performance.

B. Who conducts AI audits?

  • Internal auditors have a contractual relationship with the target and typically test compliance or minimize corporate liability, while external auditors lack that relationship and often pursue research or advocacy.External auditors may struggle to access information and can face retaliation or methodological skepticism.Source: B. Who conducts AI audits? · pp. 2–3

C. How are AI audits conducted?

  • Audits may occur before deployment, during iterative design and restricted redeployment, or after widespread adoption, and not every audit follows every process stage.The paper describes a high-level process structure while noting that audit methods vary.Source: C. How are AI audits conducted? · p. 3

1) Harms Discovery:

  • Harms discovery identifies the target, potentially affected populations, and anticipated harms or measurements needed for a meaningful audit.It may draw on impacted populations’ reports or auditors’ investigations.Source: 1) Harms Discovery · p. 3

2) Standards Identification:

  • Standards identification articulates the requirements against which the target will be evaluated, ranging from named principles to specific performance thresholds.The paper gives the 4/5ths rule for AI hiring tools as an example of a precise threshold.Source: 2) Standards Identification · p. 3

3) Performance Analysis:

  • Performance analysis is the evaluation itself, using qualitative or quantitative methods that bring different data-acquisition, model-access, and measurement challenges.The paper describes this as the core of the audit.Source: 3) Performance Analysis · p. 3

4) Audit Communication and Advocacy:

The review maps academic and non-academic AI audits, classifies their targets and practices, and examines how audit findings are communicated to relevant stakeholders.

4) Audit Communication and Advocacy:

  • Audit results may be shared with the target, regulators, the public, or internal legal, business, product, and engineering teams.The review describes this as activity to disseminate and translate findings after system evaluation.Source: 4) Audit Communication and Advocacy · p. 3

III. METHODS

  • The review draws on 341 academic audit studies from 2018–2022 and documents from non-academic practitioners across five domains.The domains are journalism, civil society, government, consulting agencies and corporate audits, and legal firms.Source: §III · p. 3

A. Academic literature

  • Conference proceedings are the primary focus of the academic review because they are a common, relatively standardized venue for AI ethics work.The search is not exhaustive and represents work recently published at interdisciplinary computing and related conferences.Source: A. Academic literature · pp. 3–4

1) Searching for audit studies:

  • The academic search combined manual review of selected conference proceedings with keyword searches of ACM and ACL literature published from 2018–2022.For high-volume ACM keywords, reviewers considered the first 500 results ranked as most relevant.Source: 1) Searching for audit studies · p. 4

2) Classifying audit types:

  • Academic studies were classified as product/model/algorithm audits, data audits, ecosystem audits, or meta-commentary, with overlapping papers assigned their best-fitting category.Ecosystem audits examine communities and sociotechnical environments; meta-commentary examines auditing as a practice.Source: 2) Classifying audit types · pp. 4–5

B. Non-academic domains

The review gathers non-academic audit examples and analyzes academic and non-academic sources through a shared framework of context, methodology, and impact.

B. Non-academic domains

  • The non-academic review covers journalism, civil society, law firms, regulatory audits, and consulting agencies and corporate audits, using reports and other public documents as examples.Sources include auditors’ websites, audit reports, and documentation from relevant organizations and agencies.Source: B. Non-academic domains · p. 5

C. Analysis

  • For academic and non-academic sources, the analysis summarizes audit context, methodology, and impact.Academic abstracts and author-selected keywords also support quantitative analysis of frequently used terms.Source: C. Analysis · p. 5, 3) Impact · p. 5

1) Context:

  • Context covers stated motivations, goals, targets, harms, and institutional factors, including ways an audit may shift power between stakeholders.The framework records these factors as stated by the auditor.Source: 1) Context · p. 5

2) Methodology:

  • Methodology captures the techniques and procedures used to investigate an audit target, including quantitative evaluation, forensic analysis, and qualitative interviews.Quantitative evaluation is typical of academic audits, while forensic analysis and qualitative interviews are more often found in civil society audits.Source: 2) Methodology · p. 5

3) Impact:

  • Impact means changes to the target, audit, or institutional environment attributable to an audit; the review records documented evidence and marks unclear or unstated impacts.The authors note that measuring impact is challenging and that academic-audit impact is often unclear at publication.Source: 3) Impact · p. 5

IV. RESULTS: AUDITING IN ACADEMIA

Academic audits span product, data, ecosystem, and meta-commentary studies, but their attention to accountability and observable consequences varies across these types.

1) Product/model/algorithm audits:

  • Product-level case studies dominate and mainly diagnose failures, errors, or disparities; accountability appears in 14% of abstracts, while public callouts picked up by media, activists, or regulators tend to bring observable changes.Academic audits rarely make explicit demands for systemic change, though their impact may emerge gradually after publication.Source: 1) Product/model/algorithm audits · pp. 5–6

2) Data audits:

  • Data audits commonly examine training or benchmark datasets and surface harms, but “accountability” appears in 9% of abstracts versus “bias” in 39.1%.Methods include quantitative measurement, simulation, ablation, and critical assessment.Source: 2) Data audits · p. 5

3) Ecosystem audits:

  • Ecosystem audits target public services more than any other audit type; harms are often concretely defined, and accountability appears in over 33% of ecosystem-audit abstracts, the highest of any category.They often use a wider range of methods, including qualitative interviews, surveys, workshops, and literature reviews; 6 out of 15 abstracts mention ethnography, interviews, workshops, or other qualitative methods, and two explicitly use the term participatory.Source: 3) Ecosystem audits · pp. 5–6

4) Meta-commentary & critique:

  • Meta-commentaries develop or critique audit methods, and “accountability” appears in 28.1% of their abstracts.They rarely address affected stakeholders in method design, and harms are often less well specified.Source: 4) Meta-commentary & critique · p. 6

2) Data audits:

Audit effects vary by practice: data audits have sometimes prompted dataset withdrawal, ecosystem audits surface broad policy demands, and meta-commentary offers a less readily measured way to reflect on auditing. Across the outside-academia domains summarized, audit aims and methods vary, and impacts are often unclear.

2) Data audits:

  • Journalists’ investigations of 80 Million Tiny Images and Microsoft’s celebrity-faces dataset led to withdrawal and discontinuation, respectively.The impacts of data audits are often unclear, despite these prominent cases.Source: 2) Data audits · p. 6

3) Ecosystem audits:

  • Ecosystem audits connect performance, privacy, bias, environmental costs, and labor concerns to comprehensive institutional or legislative policy demands.Source: 3) Ecosystem audits · p. 6

4) Meta-commentary & Critique:

  • Meta-commentary enables audits to examine their broader direction, while critique studies examine audit limits, power asymmetries, audit washing, and technological solutionism.These impacts are not easily measured, though many critique studies are highly cited.Source: 4) Meta-commentary & Critique · p. 6

V. RESULTS: AUDITING OUTSIDE ACADEMIA

  • Outside academia, audits address varied targets and concerns using assessment frameworks, document review, and data-governance planning; their stated aims include compliance and mitigating financial or public-relations risk.Source: §V · pp. 6–7
  • Reported impacts range from setting a policy precedent to unclear or unstated outcomes, across concerns including noncompliance, disparate impact, privacy harms, security, and risk.Source: §V · p. 7

A. Law Firms

Law firms’ audit work reflects distinct organizational models, from Foxglove’s advocacy for less powerful groups to for-profit firms’ compliance services. Consulting audits address varied risks, but their impacts are often unclear and their work remains subordinate to client needs.

A. Law Firms

  • Foxglove advocates for less empowered groups, while for-profit Luminos.Law and AWO offer compliance and policy services shaped by their clientele and business models.Foxglove works with groups including content moderators, warehouse workers, and gig workers; the firms’ concerns range from legal compliance to data rights and disparate performance.Source: A. Law Firms · pp. 7–8
  • Law-firm methods include legal analysis, interviews, and documentation review, but commercial work is often confidential and its impact on practice is difficult to identify.Source: A. Law Firms · p. 8
  • Foxglove’s work has reversed Ofqual’s grading algorithm, halted a UK Home Office visa-streaming algorithm, and exposed government-corporate contracts.Source: A. Law Firms · p. 8

B. Consulting Agencies & Corporate Audits

  • Consulting agencies audit systems for compliance, bias, fairness, security, and accuracy, using tools such as ORCAA’s Ethical Matrix Framework and BABL AI’s process-audit criteria.Available information on their methodologies is sparse.Source: B. Consulting Agencies & Corporate… · pp. 8–9
  • Consulting-audit impacts are often unclear; ORCAA influenced policy and HireVue announced changes after an audit, but HireVue set the evaluation scope and extent of impact.HireVue’s changes followed an EPIC complaint, and its portrayal of ORCAA’s analysis drew criticism.Source: B. Consulting Agencies & Corporate… · p. 9

C. Journalism

ProPublica and The Markup conduct external, typically post-deployment audits using quantitative analysis alongside investigative reporting and data collection. The paper identifies journalism as producing the most impactful outcomes among the domains it examined.

C. Journalism

  • Their external, typically ex-post audits examine corporate and government systems for disparities, injustice, privacy violations, fraud, and legal-compliance breaches.Source: C. Journalism · p. 9
  • ProPublica and The Markup combine quantitative statistical analysis with investigative reporting, interviews, document analysis, and extensive custom data collection.Source: C. Journalism · p. 9
  • Journalistic audits produced the most impactful outcomes among the domains examined, including industry changes, activism, legislative action, and abolition of deployed tools.A DOJ settlement required Facebook to stop using a special audience tool for housing ads; ProPublica’s COMPAS audit became canonical in algorithmic auditing.Source: C. Journalism · p. 9

D. Civil society

Civil-society audits use diverse methods and often pursue impact through legal action, attention, moratoriums, or abolition. Government auditing combines regulatory enforcement with standard-setting, but NIST’s guidance is nonbinding and its audits have not had specific impacts.

D. Civil society

  • Civil-society organizations use varied investigative, participatory, technical, and legal methods, with external audits addressing concerns from surveillance and security to refugee technology and disparate impact.Source: D. Civil society · pp. 9–10
  • Civil-society audits can shift public and media attention or produce legal outcomes, moratoriums, and abolition, often representing people harmed by AI systems.Examples include an EFF civil lawsuit win involving Proctorio and an ACLU injunction stopping algorithmically determined welfare cuts in Idaho.Source: D. Civil society · p. 10

E. Government

  • ICO investigates and enforces regulations, while NIST focuses on standards and best practices without enforcement authority; regulatory audits tend to be procedural.Source: E. Government · p. 10
  • ICO audits data governance and compliance using document examination, testing, interviews, and assurance ratings; NIST uses data trust, accuracy evaluation, and benchmarking.Source: E. Government · pp. 10–11
  • ICO has issued substantial data-protection fines, whereas NIST’s Risk Management Framework influenced U.S. regulatory discussion but has no enforcement power and no specific audit impacts.Source: E. Government · p. 11

VI. DISCUSSION

The discussion argues that audit impact depends on stakeholder power, design choices, and scope, not evaluation alone. It recommends more specific but still holistic audits and attention to accountability stages beyond evaluation.

1) Power asymmetries & auditor-stakeholder relationships:

  • Power relations between auditors, affected populations, and audit targets are a major influence on impact; evaluation that ignores structural power is unlikely to have significant impact.Policy-makers could empower auditors by tying results to consequences or by supporting publication and review of audit results.Source: 1) Power asymmetries &… · p. 11

2) Prioritizing audit execution stages beyond evaluation:

  • Audit framing and engagement beyond evaluation—including harm discovery, reporting, and collective action—can matter more for effectiveness than execution details.Auditors without legal authority to demand access may have to rely on proxies and face critique, denial, or retaliation.Source: 2) Prioritizing audit execution stages… · p. 11

3) Expanding audit scope beyond the product, model or algorithm:

  • Most academic audits rarely analyze stakeholders comprehensively or take a holistic view of interacting AI systems and their surrounding ecosystem.The authors call for broader perspectives on algorithmic systems and their impacts.Source: 3) Expanding audit scope beyond the… · p. 11

4) Impact increases with specificity:

  • Specific targets, advocacy objectives, and intended responses can make audits more actionable, but overly narrow scope may obscure systemic factors and structural-change demands.The authors recommend specificity without losing holistic and ecological observations.Source: 4) Impact increases with specificity · pp. 11–12

5) Utilising a wider range of audit methodologies:

The discussion recommends drawing on diverse methods and practitioner strengths while recognizing that audit timing and auditor type do not determine impact on their own. It also cautions that audits cannot address every societal impact, and concludes that more work is needed to make them consequential for accountability.

5) Utilising a wider range of audit methodologies:

  • Academic AI audits often foreground quantitative approaches, while non-academic auditors use qualitative methods such as investigative reporting, document review, and stakeholder consultation.The authors suggest exploring extra-disciplinary methods with a record of impact to strengthen academic work.Source: 5) Utilising a wider range of audit… · p. 12

6) Appreciating the diversity of audit practitioners:

  • Practitioner strengths vary: civil society often has mature communication and advocacy methods, while academia may bring technical capacity but struggle with public and stakeholder engagement.The authors also find variation within domains, including journalism audits that can be more thorough and reproducible than some academic studies.Source: 6) Appreciating the diversity of audit… · p. 12

7) Timing and auditor type are not the primary determining factors of audit impact:

  • Internal auditors often have pre-deployment access and external auditors often work post-deployment, but goals, targets, design, communication, and scope may matter more for accountability.Internal auditors may face retaliation, censorship, conflicts of interest, and pressure to narrow results; external auditors face legitimacy, visibility, capacity, and information-access challenges.Source: 7) Timing and auditor type are not the… · p. 12

8) Recognizing the limits of audits:

  • Some societal impacts, including surveillance’s chilling effects and corporate power concentration, are not amenable to audits and may require moral or rights-based advocacy.The authors caution that audits are one element of broader accountability strategies, not an all-encompassing solution.Source: 8) Recognizing the limits of audits · pp. 12–13

VII. CONCLUSION

  • The authors conclude that AI audit practice must learn from practitioners beyond academia to make audit studies more consequential for accountability.They call for further study of links among context, design, execution, and auditors’ shared vision.Source: §VII · p. 13

E. Consulting Agencies

The appendix describes website-based reviews of three consulting agencies and a quantitative content analysis of academic audit abstracts and keywords. It also presents term frequencies by audit type.

1) O’Neil Risk Consulting & Algorithmic Auditing (ORCAA):

2) Eticas:

  • The Eticas review focused particularly on the Research page of its Eticas Library to learn about its algorithmic audit practice.Source: 2) Eticas · p. 21

3) BABL AI:

  • The BABL AI review covered its About Us and Services pages and its Research page for information about past audits.Source: 3) BABL AI · pp. 21–22

APPENDIX B METHODS: QUANTITATIVE CONTENT ANALYSIS

  • The content analysis used abstracts and keywords from collected academic audits, manually filtering frequent terms by motivation, targets, harms, and methods.The sample contained 341 studies; terms were retained based on appearance thresholds in keyword lists or abstracts.Source: Appendix B · p. 22

APPENDIX C ADDITIONAL RESULTS

Appendix C lists tables covering non-academic audit domains and includes a figure of collected academic audit-study counts by year and publication source.

APPENDIX C ADDITIONAL RESULTS

Loading 2401.14462v1…