PaperlayerView exact revision on arXiv

Source-linked AI summary

Audit Cards: Contextualizing AI Evaluations

Leon Staufer, Mick Yang, Anka Reuel, Stephen Casper

arXiv:2504.13839v2cs.CY

TL;DR

AI audit results require contextual reporting because selective or incomplete disclosures can undermine interpretation and trust. The paper develops audit cards through literature review, stakeholder interviews, and analyses of evaluation reports and governance frameworks. It identifies six reporting features and finds that many existing reports omit information about auditor identity, integrity, resources, and review, while governance frameworks lack rigorous reporting guidance.

  • Problem

    AI evaluation reporting lacks established standards, and existing reports often omit contextual information needed for transparency and interpretation.

    Source: §1 · p. 1, §7 · p. 6

  • Method

    The paper combines a literature review, stakeholder interviews, analysis of evaluation reports, and analysis of governance frameworks to develop audit cards.

    Source: §1 · p. 1

  • Results

    Audit cards identify six contextual reporting features, while existing reports commonly omit auditor, integrity, resource-access, and review information.

    Source: Specific features for contextual… · p. 3, §4 · p. 5

  • Takeaways & Limitations

    Structured audit-card reporting can support meaningful interpretation, transparency, and trust in evaluation reporting.

    Source: §1 · p. 1, §7 · p. 6

    The analysis uses simplified scoring and is extensive but not exhaustive, covering 28 papers, 20 evaluation reports, and 21 governance frameworks.

    Source: §7 · p. 7

Abstract

from arXiv · show

AI governance frameworks increasingly rely on audits, yet the results of their underlying evaluations require interpretation and context to be meaningfully informative. Even technically rigorous evaluations can offer little useful insight if reported selectively or obscurely. Current literature focuses primarily on technical best practices, but evaluations are an inherently sociotechnical process, and there is little guidance on reporting procedures and context. Through literature review, stakeholder interviews, and analysis of governance frameworks, we propose "audit cards" to make this context explicit. We identify six key types of contextual features to report and justify in audit cards: auditor identity, evaluation scope, methodology, resource access, process integrity, and review mechanisms. Through analysis of existing evaluation reports, we find significant variation in reporting practices, with most reports omitting crucial contextual information such as auditors' backgrounds, conflicts of interest, and the level and type of access to models. We also find that most existing regulations and frameworks lack guidance on rigorous reporting. In response to these shortcomings, we argue that audit cards can provide a structured format for reporting key claims alongside their justifications, enhancing transparency, facilitating proper interpretation, and establishing trust in reporting.

1 Introduction

AI governance increasingly relies on audits, but evaluation results can mislead when contextual information is omitted or selectively reported. Audit cards are proposed as a structured way to document this context and support interpretation.

  • Audits support risk identification, responsive development, and stakeholder involvement, but require technically rigorous evaluations integrated into decision-making.Source: §1 · p. 1
  • Evaluation results are shaped by technical methods and non-technical context, including auditor independence and conflicts of interest.Source: §1 · p. 1
  • Even rigorous evaluations can be uninformative or misleading when reporting is selective or obscure, especially where developers have incentives to influence results.Source: §1 · p. 1
  • Audit cards provide a standardized framework for documenting evaluation context rather than system characteristics, complementing model and system cards.Source: §1 · p. 1

1. Audit cards

The paper proposes audit cards to address evaluation-transparency gaps through structured reporting of contextual information, principles, and features.

  • The paper synthesizes 28 prior works to propose an audit-card template and checklist addressing transparency needs not covered by existing tools.Source: §1 · pp. 1–2
  • The audit-card template combines three cross-cutting principles with six contextual features for reporting audit processes.The principles are justification, limitations, and assumptions; the features cover auditor identity, evaluation scope, methodology, resource access, process integrity, and review mechanisms.Source: Figure 1 · p. 2, §1 · p. 2
  • The paper analyzes 24 frontier-system evaluation reports, 21 governance frameworks, and interviews with 10 expert stakeholders.Source: §1 · p. 2

2 Related work

Prior work emphasizes technical evaluation and broader transparency tools, while this paper focuses on the sociotechnical context and reporting rigor of AI audits.

  • Existing transparency tools document data, models, systems, agents, or usage, whereas audit cards document the context of evaluation processes.Source: §2 · p. 2
  • AI audits are formal evaluations intended to facilitate transparency and scrutiny, but governance frameworks often lack substantial guidance for reporting them.Source: §2 · p. 2
  • Technical evaluation work addresses approaches and benchmark rigor, while this paper focuses on reporting technical details together with audit context.Source: §2 · p. 2
  • Evaluations are sociotechnical and value-laden, so meaningful oversight also depends on integrity, integration, and contextual understanding.Source: §2 · pp. 2–3

3 What information do audit cards need for rigorous reporting?

The proposed audit cards organize transparent reporting around three principles and six contextual features. These elements are intended to support interpretation, critique, and trust in evaluation processes.

3 What information do audit cards need for rigorous reporting?

Overarching principles for transparent reporting

Specific features for contextual information

4 What do existing evaluation reports include?

Across 24 frontier-system evaluation reports, procedural coverage was common but contextual reporting varied substantially, especially for auditor information, integrity, resources, and review.

Methodology

  • 24 reports were scored on comprehensive, minimal, or absent reporting across the audit-card components, assessing reporting thoroughness rather than execution quality.Transparently reporting that no review process existed could still receive full credit if justified.Source: §4 · pp. 4–5

Findings

  • Contextual details averaged 0.94, compared with 1.38 for evaluation scope and 1.29 for procedures, with substantial variation across organizations and reports.Google DeepMind documents ranged from 0.56 to 1.44 in chronological order.Source: §4 · p. 5
  • Only 3 of 24 reports described auditor backgrounds, although 15 reported auditor expertise; limitations and assumptions were also often high-level or incomplete.Justifications and limitations appeared in 22 of 24 reports, while assumptions appeared in 16 of 24.Source: §4 · p. 5
  • Integrity, resources, review, feedback, and obsolescence were rarely reported: only 4 of 24 covered selection or conflicts, 11 specified auditor resources, and none gave obsolescence criteria.Seven reports mentioned peer review or quality assurance, and five allowed public feedback or corrections.Source: §4 · p. 5

5 What guidance do current governance frameworks offer?

Existing governance frameworks generally require evaluations but provide limited specificity about how their reports should disclose evaluation context.

Methodology

  • The analysis covered influential regulations, policies, voluntary standards, and developer policies selected for jurisdiction, authority, and influence on model evaluations.Frameworks were scored for explicitly requiring or recommending reporting of audit-card features.Source: §5 · p. 5

Findings

  • Existing frameworks consistently require models to be evaluated but rarely require or recommend reporting assumptions, limitations, process integrity, or auditor resources and access.Lower-level reporting guidance is still emerging alongside broader governance frameworks.Source: §5 · p. 5

6 Challenges with auditing according to expert stakeholders

Expert interviews indicate that audit quality and trust depend on evaluator information, evaluation timing, and reporting practices that are difficult to standardize under commercial and resource constraints.

Methodology

  • Ten experts from companies, nonprofits, and government bodies were interviewed across evaluation design, development, reporting, policy research, delivery, and standards roles.Interviews were conducted with anonymity arrangements chosen to support candid discussion.Source: §6 · p. 5

Evaluator quality

  • Evaluator quality is assessed mainly through reputation, while reports often omit evaluator selection, training, and reporting methods; voluntary proprietary-model access also limits third-party auditing.Experts identify regulatory gaps and market failures as barriers to better disclosure, with other industries offering possible guidance.Source: §6 · pp. 5–6

Reporting comparison

  • Table 2 scores the completeness of contextual reporting in existing audit reports on a 0–2 scale and orders reports by release date.The table distinguishes comprehensive, minimal, and absent information for each audit-card component.Source: Table 2 · p. 6

Evaluation timing

  • Reporting when audits begin and end, and which model versions were accessible at checkpoints, could support standardization despite commercial pressure and difficulty measuring resources.Experts viewed advance guidance on auditing timeframes as useful.Source: §6 · p. 6

Users of reports

  • Evaluation reports currently serve auditors more directly than policymakers, who face time, technical, and regulatory constraints; future work could prioritize reporting features by user group.Reports presently support reproducibility and elicitation more than policy use.Source: §6 · p. 6

7 Discussion and recommendations

The paper recommends standardized, flexible audit-card reporting to make evaluation context explicit, while recognizing burdens, verification limits, and the need for more specific guidance.

Recommendations

  • Audit cards address inconsistent reporting and support meaningful interpretation, accountability, and trust without replacing other transparency tools or selecting among regulatory approaches.They are intended as one instrument within a broader accountability toolkit and can complement model cards, datasheets, and benchmark details.Source: §7 · pp. 6–7

Limitations

  • The analysis is limited by coarse scoring and non-exhaustive coverage, including 28 papers, 20 evaluation reports, and 21 governance frameworks.Binary governance-framework scores and simple two- or three-point scales reduce granularity and nuance.Source: §7 · p. 7
  • Comprehensive reporting adds time and effort for resource-constrained auditors, and disclosed information remains difficult to verify.The paper presents explicit disclosure as a first step rather than a complete solution to fraud or accountability challenges.Source: §7 · p. 7

Recommendations

  • Near-term progress requires rigorous, standardized reporting guidance, potentially through templates, because stakeholders otherwise struggle to compare and interpret audit results.Existing contributions offer progress toward shared rigor but may need greater specificity.Source: §7 · p. 7

Conclusion

  • Standardized reporting is presented as complementing technical innovation by making rigor and transparency shared expectations for more informed governance decisions.The recommendation targets both execution and reporting of evaluations.Source: §7 · pp. 7–8

Appendix A Disambiguating auditing and the evaluations process

The paper distinguishes formalized AI audits from the underlying evaluations and focuses on how evaluation results are reported. Evaluations may vary in setting, target, form, and timing, while the process spans methodology design, execution, and publication.

  • AI audits are formalized, systematic evaluation processes of AI models or systems conducted by organizations rather than ad hoc individuals.The paper distinguishes model audits from governance and application audits.Source: Appendix A · p. 12
  • Evaluations are the underlying assessments and may be internal or external, pre- or postdeployment, and focused on capabilities, propensities, or risks.Examples include benchmarks, red-teaming, human uplift studies, and user studies.Source: Appendix A · p. 12
  • The evaluation process comprises methodology design, execution and result generation, and publication, with the paper focusing on reporting in the final stage.Source: Appendix A · p. 12
  • Evaluation reports are internal or external documents that include evaluation results, including reports produced by third parties or AI companies.Source: Appendix A · p. 12

Methodology Details

This section details how the paper develops audit cards through literature review, iterative annotation, and expert input, organizing key contextual aspects into reporting features and a flexible checklist. It also describes the selection and analysis of governance documents across jurisdictions, international initiatives, national institutes, and major AI companies.

Methodology Details

  • The review expanded from eight directly relevant papers to adjacent work connecting technical metrics with social, cultural, and institutional factors.Recommendations were scored as major, minor, or absent using a deliberately coarse three-point scale.Source: Methodology Details · p. 12
  • Through four iterations, two annotators refined actionable categories, established concrete scoring rubrics, and prioritized aspects with expert input.The final audit card is presented in Figure 1 and Appendix C, with aspect mappings in Appendix B.Source: Methodology Details · p. 12

Mapping of initial aspects

  • The initial 24 aspects span auditor expertise, evaluation goals and methods, access and resources, assumptions, limitations, justification, integrity, and review and communication.Examples include auditor training, model-developer relationships, peer review, capability-to-benchmark translation, contextual deployment considerations, and continuous evaluation.Source: Mapping of initial aspects · p. 13

Details and justifications of features

  • The six audit-card features explain what to report and why: auditor qualifications, evaluation scope, methodology, access and resources, integrity, and review and communication.These features address credibility and bias, scope-related misinterpretation, methodological validity, resource-driven blind spots, conflicts of interest, review reliability, feedback, terminology, and result distribution.Source: Details and justifications of features · pp. 13–14

Checklist for audit card

  • The checklist is a flexible, prioritized guide for documenting audit contexts, limitations, and strengths, with questions tailored to evaluation relevance rather than rigidly required.It prompts reporting on auditor backgrounds, system scope and goals, methodology and interpretation, access and resources, conflicts and incentives, and review and communication.Source: Checklist for audit card · pp. 15–16

Appendix D Selection of governance documents

  • Governance documents were selected by major jurisdiction, institutional authority, and direct influence on model evaluations.The sample included government and nongovernment institutions, influential evaluation organizations, developer safety frameworks, and highly consequential national or binding documents.Source: Appendix D · p. 17
  • The document set covered EU, Singapore, South Korea, and US governance sources, including binding or authoritative instruments and policy signals.The EU AI Act and Code of Practice were treated as effectively binding until harmonized standards appear; Singapore’s framework is nonbinding but authoritative, South Korea’s law takes effect in 2026, and the US AI Action Plan is the most authoritative federal signal amid no federal-wide regulation.Source: Appendix D · p. 17
  • The selection also included international declarations and reports plus evaluation guidance from Japan, the UK, METR, and NIST.These sources comprised the Bletchley Declaration, International AI Safety Report, Japan AISI’s evaluation guide, METR’s RSP components, NIST’s AI Risk Management Framework, and related international statements.Source: Appendix D · p. 17
  • The corpus further covered frontier-risk frameworks from Anthropic, Google DeepMind, Meta, Microsoft, OpenAI, and xAI.Included documents were Anthropic’s Responsible Scaling Policy, Google DeepMind’s Frontier Safety Framework v2, Meta’s Frontier AI Framework v1.1, Microsoft’s Frontier Governance Framework, OpenAI’s Preparedness Framework Beta, and xAI’s Draft Risk Management Framework.Source: Appendix D · pp. 17–18

Detailed Methodology

The detailed methodology combines stakeholder engagement with structured interviews to refine and assess the audit-card framework. Interviews covered evaluation practice, reporting choices, auditor selection, resources, integrity, and how reports support recommendations.

Detailed Methodology

  • The study used a dozen off-the-record discussions during literature review and feature selection, followed by ten semi-structured stakeholder interviews focused on practices, gaps, and contextual needs.Stakeholder feedback shaped distinctions such as compensation versus conflicts of interest and led to Appendix C after implementation concerns.Source: Detailed Methodology · p. 18
  • Interviews used informed consent, participant-selected anonymity and attribution levels, quote verification, and voluntary unpaid 30-minute participation.Source: Detailed Methodology · p. 18

Interviewee details

Interview guide

  • Interviewees were asked about evaluation limitations, assumptions, auditor expertise, process differences, internal practices, training, and what reports should disclose.Source: Interview guide · p. 19
  • The guide examined how auditors are selected, what resources and access evaluations receive, stakeholder involvement, funding and contract structures, conflicts of interest, and risks of misleading communication.Source: Interview guide · pp. 18–19
  • Questions also addressed report audiences, useful report features, trust by funding source, the insights evaluations provide, and how messaging should vary across audiences.Source: Interview guide · pp. 19–20
Loading 2504.13839v2…