PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, Inioluwa Deborah Raji

arXiv:2402.17861v3cs.CY

TL;DR

Effective AI audits remain practically difficult. The authors compare practitioner interviews with a landscape of audit tools, finding that current resources emphasize evaluation and standards but often fall short of accountability needs.

  • Problem

    Effective audits of deployed AI systems remain practically difficult, despite their role in evaluating performance, fairness, legality, or safety.

    Source: §1 · p. 1

  • Method

    The authors interviewed 35 practitioners and compared reported needs with a landscape analysis of 435 AI audit tools.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Tools for evaluation and standards are common, while support for harms discovery, communicating audit results, and advocacy is much less common.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    The authors call for accountability infrastructure beyond evaluation, including tools, community infrastructure, communication standards, and advocacy.

    Source: §1 · p. 2, §6 · p. 13

    The tool sample is not comprehensive or perfectly representative; theoretical sampling overrepresents some less common tools, and public-source searching likely overrepresents open tools.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs. While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability. We outline challenges practitioners faced in their efforts to use AI audit tools and highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy. We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the field move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

1 Introduction

Effective AI audits remain practically difficult, and evaluation alone is insufficient for accountability. This study compares practitioners’ tooling needs with the current landscape and finds important gaps beyond evaluation.

1 Introduction

  • A thorough evaluation alone cannot hold stakeholders responsible for system-wide behavior; audits also need support for target identification, standardization, communication, and advocacy.The introduction frames this gap against the difficulty of conducting effective audits and limited access to information about many AI systems.Source: §1 · p. 1
  • The study compares interviews with 35 practitioners against a landscape analysis of 435 tools to identify unmet needs and accountability gaps.The interviews covered practitioners at 24 organizations across technology, government, non-profits, consulting, and academia.Source: §1 · p. 2
  • Tools for evaluation and standards are common, but tools for harms discovery, communicating results, and advocacy are much less common; practitioners often adapt or build tools themselves.Practitioners also envisioned tools for high-quality, uncompromised data, consistent and holistic methods, and audit integrity.Source: §1 · p. 2
  • The authors identify opportunities to develop tools and practices that support rigor, inclusion, independence, and accountability beyond evaluation.They frame the broader aim as infrastructure for meaningful accountability.Source: §1 · p. 2

2 Related work

Prior work has examined AI auditing, accountability, and specific audit tools, but has not surveyed the full range of tools and infrastructure used to support AI audits.

2 Related work

  • AI audits are independent assessments of identified targets against articulated expectations, with an implicit or explicit objective of accountability [14].In this paper, accountability entails consequential judgment of systems’ behaviors and downstream impacts.Source: §2 · p. 2
  • No prior study examined the full range of tools and technical infrastructure used to support AI auditing and accountability.The authors define AI audit tools broadly, including software, interfaces, code, benchmarks, frameworks, and other artifacts.Source: §2 · pp. 2–3
  • Audit infrastructure includes organizational processes and tools that support consistent, interoperable judgments and can lower barriers to broader participation.The paper describes infrastructure as social and relational as well as technological.Source: §2 · p. 2
  • Earlier HCI research often focused on fairness and interpretability toolkits, while other necessary audit steps such as data collection and assessing harms beyond fairness received less attention.The paper cites studies of performance analysis and grassroots auditing as additional strands of prior work.Source: §2 · pp. 2–3

3 Methodology

The study combines practitioner interviews with an iterative, theoretically sampled inventory and analysis of AI audit tools.

3 Methodology

  • The researchers interviewed 35 auditors across 24 organizations and curated a dataset of 435 tools, using their findings to develop a taxonomy of the tool landscape.The study included internal and external auditors and covered tools applied to a broad range of AI-advertised products and models.Source: §3 · p. 3

3.1 Interview methodology

  • The 27 semi-structured interviews used purposive and snowball sampling to recruit practitioners with internal or external audit experience across varied professional backgrounds.Transcripts were analyzed with manual descriptive and values coding, organized through collaborative sessions and memo writing.Source: §3.1 · pp. 3–4

3.2 Tool Taxonomy

  • The tool taxonomy began with searches that yielded 143 tools, then expanded through targeted keyword searches and snowball sampling before iterative category revision.The final taxonomy grouped tools into 30 main categories and 27 subcategories, organized across seven audit-process stages.Source: §3.2 · p. 4
  • The tool search was structured but non-exhaustive, deliberately overrepresenting some less common categories and likely overrepresenting open tools because it relied on public materials.The authors caution that the sample should not be treated as comprehensive or perfectly representative, and its numeric descriptions may not reflect the distribution of tools in practice.Source: §3.2 · p. 4

4 Results

This section examines how existing AI audit tools and practices support the audit process, from data collection and model evaluation to communicating findings and advocacy. It describes gaps in support for accountability beyond technical evaluation.

4 Results

  • 206 tools supported Standards Identification & Management and 129 Performance Analysis, versus 48 Harms Discovery, 8 Audit Communication, 14 Advocacy, and 12 model/data transparency tools in the 435-tool survey.Practitioners also described tool use as ad hoc, sometimes building their own tools because existing ones complicated audits or could not handle systems’ complexity and scale.Source: §4 · p. 5

4.1 Harms Discovery

  • Harms discovery benefited from affected-community participation, but incident databases often record harms only after they occur and rely on second-hand reports.Practitioners said participation broadens the harms considered, makes audits more context-dependent and inclusive, and can build confidence and engagement; they cautioned against participation washing and called for fair compensation and ongoing processes. The authors point to system-use disclosures, proactive standardized incident reporting, and community participation as ways to improve discovery.Source: §4.1 · pp. 5–6

4.2 Standards Identification & Management

  • Standards tools were often general rather than system-specific: the dataset included 86 principle statements and 49 self-assessment checklists, but only 5 participatory standard-setting tools.Practitioners wanted context-specific frameworks and guidance beyond fairness, including explainability, privacy, and transparency; the authors argue standards should be holistic, inclusive, context-specific, and compatible with practice.Source: §4.2 · pp. 6–8
  • Practitioners found legal and regulatory guidance difficult to translate into clear evaluation metrics and practical standards, amid communication gaps between regulators, legal staff, and engineers.The authors suggest translating regulatory standards into concrete guidance, potentially using a “must, could, should” structure that pairs legal minimums with technical compliance paths and best practices.Source: §4.2 · pp. 7–8

4.3 Data Collection & Transparency Infrastructure

  • Corporate control of APIs could limit auditors’ access and independence, while incomplete disclosure of sampling, provenance, model versions, metrics, or design rationale could compromise audit integrity.Auditors called for an inspectability API with standardized access to test platforms across profiles and geographies, and for an independent public data archive.Source: §4.3 · p. 8
  • 68.1% of Data Collection tools were open source, compared with 25.0% of Transparency Infrastructure tools, but independent collection often required adapting or maintaining custom scrapers.Practitioners also described substantial work cleaning and interpreting data, concerns about data quality and synthetic content, and the view that external analysis of system outputs can reveal real-world impacts.Source: §4.3 · pp. 9–10
  • External data collection could expose auditors to legal liability and retaliation, with uncertainty about whether scraping or data donation would be deemed unauthorized under the CFAA.Auditors described the legal work required to deploy tools and worried that platform-controlled access mechanisms could obstruct accountability; even internal auditors reported carrying legal risk.Source: §4.3 · p. 10
  • Auditors need independent access to reliable data, more support for data donation and user-driven auditing, and trustworthy disclosure mechanisms for interacting with model operators.Transparency Infrastructure may grow as regulation develops and external data collection becomes harder; independent research could help ensure platform-controlled tools are not the only avenue for scrutiny.Source: §4.3 · p. 10

4.4 Performance Analysis

  • Practitioners questioned the validity, reproducibility, transparency, and trustworthiness of popular methods, as well as the maintenance and real-world relevance of automated monitoring and synthetic data.Shapley values may be misused and unsuitable for normative evaluation, while participants called for open-source tools; 48.1% of Performance Analysis tools were built by for-profit organizations and were disproportionately not open source.Source: §4.4 · pp. 10–11
  • Performance tools largely target narrow fairness and explainability criteria, often miss basic functionality and qualitative evaluation, and can enable audit washing without addressing underlying ethical concerns.Future work could broaden evaluation criteria and investigate standards such as academic peer review or regulatory vetting for audit tools.Source: §4.4 · p. 11

4.5 Audit Communication & Advocacy

  • Audit Communication tools were rare (N=8), Advocacy tools numbered 14, and practitioners said tools for consequential follow-up remained largely aspirational.Communication tools mostly supported dataset visualization or reporting, while advocacy tools included community spaces, organizing/resistance, and legal search.Source: §4.5 · p. 11
  • Auditors wanted resources to unite interdisciplinary communities around policy, shared standards, and goals, and to enable collaboration and shared tooling.Public report repositories and communication with policymakers, journalists, and activists could support accountability and learning; future work could investigate mechanisms such as concrete demands for action.Source: §4.5 · p. 11

5 Discussion

The discussion argues that accountability requires moving beyond evaluation tools and toward participatory, open, and durable public infrastructure. It identifies practical gaps in tool development and maintenance, while noting the work’s subsequent policy and funding impact.

5.1 Moving beyond evaluation, towards accountability

  • Evaluation-stage tools outnumber tools for harms discovery, audit communication, or advocacy by more than five to one.The discussion identifies these latter stages as especially dependent on contextual awareness and participatory, community-engagement methods.Source: §5.1 · p. 12
  • Future work should develop tools for harms discovery, audit communication, and advocacy, and validate existing tools against real-world requirements.The discussion highlights neglected incident-reporting, education/awareness, target-identification, and audit-communication categories; unreliable performance tools can mislead auditors and support audit washing.Source: §5.1 · p. 12
  • Tools marketed for NYC Local Law 144 compliance may produce measures that are unreliable or legally incompatible.The discussion also proposes investigating government-sponsored tools and court-mandated transparency infrastructure.Source: §5.1 · p. 12
  • Participatory audits are an emerging focus, exemplified by user-auditing work such as WeAudit; policymakers can support participation through guidance and investment.The discussion cautions that participation should be supported without exploitation or tokenization.Source: §5.1 · p. 12
  • Open practices can enable collaboration and third-party validation, while audit-target interference can restrict publication or scope and expose auditors to retaliation.The discussion calls for publicly scrutinizable methods and stronger protections for internal and external auditors.Source: §5.1 · p. 12

5.2 Moving beyond ad hoc toolkits, towards shared infrastructure

  • Participants called for lasting public infrastructure, but private, for-profit firms built many free tools, and control of infrastructure can confer power over data and audit processes.External auditors also questioned tool-provided data; open-source scraping tools do not always cover practical audit work’s scope and complexity.Source: §5.2 · p. 13
  • Shared tool catalogs and decision-support frameworks could help practitioners choose tools, while long-term funding and maintenance plans address sustainability and technical debt.The discussion also recommends shared inventories, incident databases, report registries, and other repositories that support access and knowledge sharing.Source: §5.2 · p. 13

5.3 Ongoing Impact

  • The project’s findings were cited in policy reports in the United States, European Union, and United Kingdom, and informed two rounds of Mozilla Technology Fund award selection.The reported engagement also included presentations to regulators at OfCom and the Federal Trade Commission.Source: §5.3 · p. 13

6 Conclusion

The conclusion argues that accountability outcomes from AI audits are uncertain. Making audits feasible and effective requires investment in tools, community infrastructure, communication standards, and advocacy.

6 Conclusion

  • AI audit studies do not reliably translate into accountability outcomes, so researchers, policymakers, and practitioners need to invest across tools, community infrastructure, communication standards, and advocacy.The conclusion calls for pushing beyond the minimum of auditing toward a meaningful accountability mechanism.Source: §6 · p. 13

A Reflections

The authors locate the study primarily within U.S. and E.U. contexts and acknowledge that its perspective may not represent global AI auditing. Their tool search may also have missed non-English and non-Western resources.

A Reflections

  • The analysis is primarily scoped to the U.S. and E.U. and may not represent the global AI auditing landscape or other contexts.The authors also note their Western AI industry position and that their dataset consists primarily of English-language tools from Western organizations.Source: A Reflections · p. 17
  • Initial translated-query and regional-keyword searches found no additional tools, but non-English, non-Western tools likely remain unidentified.Source: A Reflections · p. 17
  • The authors hope future work will look beyond the mostly technical, Western perspective emphasized in this study.Source: A Reflections · p. 18

B Glossary

The glossary defines accountability in a legal-political sense and specifies what counts as an AI audit and its target. It also distinguishes auditors and broadly defines audit tools and algorithm abandonment or disgorgement.

B Glossary

  • Accountability means consequential judgment for system behaviors and impacts that do not align with articulated expectations and standards.The definition draws on Bovens and describes an actor’s obligation to explain and justify conduct to a forum that can judge and impose consequences.Source: B Glossary · p. 18
  • An AI audit is an independent assessment of an identified target against articulated expectations, intended to inform accountability [14].Targets are concrete objects, ideally tied to real-world deployments or widely used open-source algorithms or datasets; auditors are operationally distinct from the engineering team [14].Source: B Glossary · p. 18
  • An AI auditor may be internal or external, with internal auditors having a contractual relationship to the target and external auditors lacking one [88].Internal auditors typically seek to minimize corporate liability and test compliance; external auditors typically have a broader mandate to identify and minimize harms to constituents [88].Source: B Glossary · p. 18
  • An AI audit tool broadly includes software, interfaces, code, benchmarks, frameworks, and other artifacts used in the audit process.These resources can support algorithmic analysis and inspection or assess institutional expectations across design and development.Source: B Glossary · p. 18
  • Algorithm abandonment stops designing, developing, or using a potentially harmful system, while disgorgement also requires deleting improperly obtained data and derived models, algorithms, or outputs [116].Source: B Glossary · p. 18

C.1 Initial Sources

The initial tool list drew on interviewees, prior surveys, audit-study literature, government reports, and a convenience sample. It included both AI-audit-specific and generic tools, but was not exhaustive.

C.1 Initial Sources

  • The initial 143-tool list combined tools mentioned by interviewees, prior fairness-tool surveys, audit-study papers, and government reports.The literature sources included computing-venue audit studies and reports from ICO and NIST.Source: §C.1 · p. 18
  • A convenience sample accounted for 79 of the initial 143 tools and included prominent projects, papers, and tool-building organizations encountered by the researchers.Most of these tools also appeared in the other source sets.Source: §C.1 · p. 18
  • The dataset included both AI-audit-specific tools and generic tools used in audits, but was not an exhaustive inventory.Tools were included only if observed in the cited sources or subsequent searches, and inclusion did not indicate endorsement or preference.Source: §C.1 · p. 18

C.2 Theoretical Sampling

The researchers used keyword searches and additional sources to expand the tool landscape, focusing first on less-saturated categories. Search saturation and unequal follow-up meant categories did not receive equal additional sourcing.

C.2 Theoretical Sampling

  • English Google searches paired general AI-auditing keywords with descriptors from the initial taxonomy to find tools in each category.For example, queries combined a category descriptor with terms such as “tool” or “audit.”Source: §C.2 · p. 18
  • Searches stopped after two consecutive pages of 30 results contained no specific audit tools, with less-saturated categories prioritized for additional sourcing.Harms-discovery and participatory-method categories received early focus; more saturated categories did not receive equal follow-up.Source: §C.2 · pp. 18–19
  • The researchers added tools from news and civil-society reports, a participatory ML workshop, and a search for regulatory-technology startups.Source: §C.2 · p. 19

D Interview Protocol

The semi-structured interview protocol asked auditors about their work, tool use, unmet needs, and the resources required to conduct audits. Prioritized questions were asked of nearly all participants, while other questions were optional follow-ups.

Interview Protocol

  • Interviewers asked about auditors’ goals, roles, target systems, motivations, successes, failures, stakeholders, and collaborators.The protocol began by introducing the project, confirming consent and confidentiality, and asking how participants became involved in auditing.Source: D Interview Protocol · p. 19

Tool Usage and Development

  • Questions examined which tools auditors used, how they chose or developed them, what audit tasks they supported, and what pain points arose.Interviewers also asked whether tool use aligned with the tool’s intent and whether open-source tools mattered to participants’ practices.Source: D Interview Protocol · p. 19

Exploring Gaps & Challenges in Tools

  • Interviewers asked about common audit obstacles, unmet tool needs, and whether tools revealed concerning system behaviors.Questions also covered what existing methods assess and whether they measure those properties successfully.Source: D Interview Protocol · p. 19
  • The protocol asked whether existing audit formats and methodologies were useful and impactful, and what alternatives participants wanted to see.Interviewers also asked about audit costs, team size, duration, and difficulty.Source: D Interview Protocol · p. 19

E Additional Landscape Analysis

The additional landscape analysis labels tools by their documentation and function and supplements the tool dataset with organization, repository, funding, employment, and citation information. The appendix figures present category counts across several tool and organization characteristics.

Tool labels

  • Each tool was manually tagged for license, organization type, audit target, intended user, and format, with a second author reviewing each label.The tags included open-source or proprietary status, organization type, target systems, and formats such as APIs, software products, repositories, and white papers.Source: E Additional Landscape Analysis · p. 19

Supplementary data

  • The analysis supplemented tool records with Crunchbase data, GitHub activity, and citation counts for academic references.Crunchbase data were accessed in September 2023; GitHub repository activity was collected for tools with repositories, and Google Scholar citation counts were annotated in September 2023.Source: E Additional Landscape Analysis · pp. 19–20
  • Crunchbase records were available for 202 of 347 organizations, covering 270 tools; 173 records included employee counts.Among 132 non-university and non-government entries, 91 had revenue estimates; funding data covered 48 of 112 still-private firms.Source: E Additional Landscape Analysis · p. 20

Appendix figures

  • Figure E.1 counts tools by taxonomy category and sorts categories by organization type, while allowing tools to appear in multiple stages.Figure E.2 counts tools with code in each taxonomy stage, and Figure E.3 sorts category counts by license type.Source: Figure E.1 · p. 21, Figure E.2 · p. 21, Figure E.3 · p. 22
  • Figure E.4 sorts category counts by audit target, and Figure E.5 sorts them by format; tools may appear in multiple stages.Figure E.6 presents category counts by organization type using Crunchbase workforce-size data, excluding organizations without Crunchbase entries.Source: Figure E.4 · p. 22, Figure E.5 · p. 23, Figure E.6 · p. 25
  • Figure E.7 presents category counts alongside estimated organizational revenue, excluding organizations without Crunchbase revenue estimates.Figure E.8 covers private organizations with Crunchbase entries; Figures E.9 and E.10 show GitHub forks and citations, respectively, for tools with repositories or papers.Source: Figure E.7 · p. 26, Figure E.8 · p. 27, Figure E.9 · p. 28
Loading 2402.17861v3…