PaperlayerView exact revision on arXiv

Source-linked AI summary

Actionable Meta-Requirements for AI Robustness Auditing

Joshua Owotogbe, Roya Nasiri, Indika Kumara, Damian Andrew Tamburri, Willem-Jan van den Heuvel

arXiv:2610.14356v1cs.AIcs.SE

TL;DR

AI robustness auditing needs requirements for judging whether available evidence supports defined robustness claims, but those requirements had not been consolidated. The study derives meta-requirements from literature and validates and refines them with practitioners; broad support produced a final set of 44 requirements.

  • Problem

    Requirements for designing and evaluating AI robustness auditing frameworks had not been consolidated across relevant research domains.

    Source: §3.3 · p. 7

  • Method

    A systematic literature review derived meta-requirements, which a practitioner survey assessed and refined through ratings and qualitative feedback.

    Source: §4 · p. 7, §4.3 · p. 11

  • Results

    81.20% of 5,504 practitioner ratings indicated agreement or strong agreement, and the final practitioner-informed set contains 44 meta-requirements across 16 categories.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The meta-requirements specify implementation-independent needs that can guide design or assessment of robustness auditing approaches across different audit contexts.

    Source: §4.3 · p. 11

    The reviewed work did not consolidate robustness-specific audit requirements or evaluate a literature-derived set with practitioners.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

AI-enabled systems must remain dependable when the conditions under which they operate change (e.g., due to input perturbations, distribution shifts, or changes in external dependencies). This makes robustness auditing of such systems important for judging whether the available evidence is sufficient to support a defined robustness conclusion. However, designing an AI robustness auditing framework is challenging. A key unresolved issue is determining the requirements such a framework should support; these requirements have not yet been consolidated. We address this gap through meta-requirements: higher-order requirements specifying what a class of auditing frameworks should support. In design science research, literature and domain experts are common sources of meta-requirements; accordingly, we derive the initial set from literature and validate and refine it with practitioners. Our systematic literature review retained 74 of 2,890 records and derived 43 meta-requirements across 16 categories. We evaluated them with 128 practitioners, obtaining 5,504 ratings and 586 substantive comments. Overall, 81.20% of ratings indicated agreement or strong agreement. In particular, 41 of the 43 literature-derived meta-requirements were classified as strongly supported. Practitioner feedback refined the requirements and identified an additional requirement concerning external components and supply chain dependencies. The final set comprises 44 meta-requirements across 16 categories, of which 43 received quantitative validation. These meta-requirements can help organizations implement AI robustness auditing frameworks, assess existing frameworks for completeness, and refine them.

1 Introduction

Robustness auditing must judge whether evidence supports a defined claim under specified operating changes, yet existing research had not consolidated requirements for such audits. This study derives and practitioner-validates a cross-domain set of meta-requirements for designing and evaluating auditing frameworks.

1 Introduction

  • Robustness auditing addresses whether evidence adequately supports a stated robustness objective, beyond testing performance under particular conditions.Inputs, distributions, dependencies, or deployment conditions may change, so ordinary test performance alone is insufficient.Source: §1 · pp. 1–2
  • The study combines a systematic literature review with practitioner validation to derive implementation-independent requirements for robustness auditing.Its review retained 74 of 2,890 records and produced 43 literature-derived requirements across 16 categories.Source: §1 · p. 2
  • 81.20% of 5,504 ratings indicated agreement or strong agreement; 41 of 43 literature-derived requirements were strongly supported, yielding 44 requirements in 16 categories.Practitioner feedback also identified an additional concern related to externally controlled dependencies.Source: §1 · p. 3

2.1 AI Robustness and Robustness Claims

  • A robustness claim specifies the assessment object, acceptable behaviour and criterion, and conditions; evidence under one change does not establish robustness under every change.Claims may address a model or a wider AI-enabled system.Source: §2.1 · pp. 3–4

2.2 AI Robustness Auditing

  • An audit evaluates available evidence against the claim, scope, and judgment criteria, while the evidence accessible to auditors depends on their level of access.Evidence can include test results, monitoring outputs, logs, data, documentation, and development or deployment records.Source: §2.2 · p. 4

2.3 Meta-Requirements for Robustness Auditing

  • Meta-requirements express shared audit needs across varied systems and approaches, while leaving specific solutions adaptable to the system and audit context.They are actionable when an audit can examine the concern and use it to design or assess auditing approaches.Source: §2.3 · pp. 4–5

3.1 Challenges Relevant to Auditing AI Robustness

  • Robustness audits must account for varying evaluation conditions, component interactions, resource demands, trade-offs, and evidence that can become outdated after deployment.These challenges motivate requirements for scope, resource constraints, reassessment, and documenting the basis and limits of conclusions.Source: §3.1 · p. 5

3.3 Limitations of Prior Work on AI Robustness Auditing

  • Prior work supplies robustness-assessment and AI-auditing foundations, but does not consolidate robustness-specific audit requirements or evaluate such a set with practitioners.Lifecycle practices span data preparation, training, evaluation, deployment, and monitoring, with audit needs to link evidence to claims and reassessment triggers.Source: §3.3 · pp. 6–7, §3.2 · p. 6

4 Methodology

The methodology combines a systematic literature review with a practitioner survey to derive, validate, and refine meta-requirements for AI robustness auditing. It moves from literature search and coding through practitioner ratings and coded comments to synthesis, retaining 43 requirements and adding one on external dependencies.

4 Methodology

  • The mixed-methods design addresses RQ1 by deriving requirements from literature and RQ2 by integrating practitioner ratings and comments into the final set.The process comprises literature search and screening, requirement extraction and refinement, practitioner validation, and synthesis and reporting.Source: §4 · p. 7, §4.1 · p. 7

4.2 Phase 1: Literature Search and Screening

  • The review searched three academic databases and Google Scholar for AI/ML robustness-auditing studies involving practitioner and empirical-study terms, retrieving 2,890 records; after screening and quality assessment, 74 studies were retained.Search-term restrictions were retained to improve precision, while recognising that they could exclude relevant studies that did not use this terminology.Source: §4.2 · pp. 8–9, §4.2.1 · p. 9

4.3 Phase 2: Meta-Requirement Extraction and Refinement

  • Open and axial coding consolidated evidence into categories, which were merged, separated, or retained as technology-agnostic requirements using explicit scope and evidence criteria.Researchers refined the requirements for clarity and assessability; the process produced 43 literature-derived meta-requirements across 16 categories.Source: §4.3 · pp. 10–11

4.4 Phase 3: Practitioner Validation Survey

  • The survey rated 43 requirements across 16 categories on a five-point Likert scale and invited optional comments; an eight-person pilot prompted wording and usability refinements.Pilot completion took approximately 15–30 minutes, and pilot responses were excluded from the main analysis.Source: §4.4 · p. 11, §4.4.1 · pp. 11–12, §4.4.2 · p. 12

4.4.3 Target Population, Recruitment, and Survey Administration

  • Purposively recruited researchers and practitioners with relevant AI/ML experience were invited from GitHub and LinkedIn; 128 valid responses remained after screening.The study obtained 143 responses from 1,720 invitees, an 8.31% response rate, and received ethics clearance; participation ran from 8 July to 15 August 2026.Source: §4.4.3 · p. 12

4.4.4 Data Screening and Preparation

  • The 128 valid responses yielded 5,504 complete requirement ratings and 586 substantive comments for quantitative and qualitative analysis.Comments were linked to anonymous respondent identifiers and their source category and response-batch position.Source: §4.4.4 · pp. 12–13

4.4.5 Quantitative Analysis of Likert Ratings

  • Ratings were summarized using frequencies, percentages, medians, and IQRs, with study-defined thresholds classifying each requirement as strongly supported, supported, mixed or unclear, challenged, or requiring qualitative interpretation.Agreement combined ratings of 4 and 5; the thresholds were descriptive decision rules, not inferential significance criteria, and subgroup comparisons were not performed.Source: §4.4.5 · p. 13
  • The displayed requirements span data provenance and quality, subgroup bias, data mitigation, model stress-testing, intended use, and explainability.Examples include documenting data sources and limitations, checking data defects, probing defined robustness stressors, and specifying the model’s operating conditions and limitations.Source: §4.4.5 · p. 15

4.4.5 Quantitative Analysis of Likert Ratings.

  • Audits should identify model-level failure causes and output-level unsafe or inconsistent behaviour, then assess whether training-time and post-processing mitigations reduce risks under assessed conditions.Examples of model-level causes include shortcut learning, overfitting, spurious correlations, and unstable features.Source: §4.4.5 · p. 15
  • Robustness checks should span software development and ML or MLOps workflows, remain compatible with relevant infrastructure, and assess external dependencies and limits on auditor access.Audits should consider how dependency failures or changes affect system behaviour and how limited access restricts the robustness conclusion.Source: §4.4.5 · p. 15
  • Audits should specify human roles and evidence-access limits, repeatedly reassess robustness after relevant changes, and validate selected failures across system layers.Evaluations should keep a prioritised record of completed checks, unresolved concerns, follow-up tests, mitigations, and reassessment actions.Source: §4.4.5 · p. 15
  • Auditing should include post-deployment monitoring, alerts, reassessment triggers, and diagnostic logging, while making evidence interpretable, documented, resource-conscious, domain-adapted, accessible, and configurable.Assessment should support the task types, model classes, modalities, datasets, and deployment settings in scope.Source: §4.4.5 · p. 16

4.4.6 Qualitative Coding of Open-Text Comments.

  • Researchers coded 586 comments for themes, functions, and related requirements, independently reviewed the coding and synthesis decisions, then reconciled feedback with quantitative support.Decisions included retaining, refining, adding, merging or splitting, and making no change; suggestions were checked against the existing requirements and relevant evidence.Source: §4.4.6 · pp. 16–17

4.5 Phase 4: Meta-Requirement Synthesis

  • The synthesis retained 43 literature-derived meta-requirements, refined them where supported, and added MR-8.4 on external-component and supply-chain assurance, yielding 44 requirements across 16 categories.MR-8.4 was not rated in the original Likert-scale validation because it was added after the survey.Source: §4.5 · p. 17

5 Meta-Requirements for AI Robustness Auditing

This section presents 44 practitioner-informed meta-requirements across 16 categories and three dimensions, refining an initial literature-derived set of 43. It covers standards and governance, stakeholder engagement, robustness definition, skills, data, model and output auditability, integration of audits into development and operations, human review, ongoing evaluation, and usable, appropriately scoped audit evidence.

5 Meta-Requirements for AI Robustness Auditing

  • The final set comprises 44 meta-requirements across 16 categories: 16 Human, 23 Functional, and 5 Quality, including practitioner-added MR-8.4.The initial 43 requirements were derived from 74 retained studies and refined using practitioner feedback. The Human dimension covers organisational, stakeholder, conceptual, and competence conditions; Functional covers lifecycle audit activities and evidence; Quality concerns feasibility, context sensitivity, understandability, usability, and configurability.Source: §5 · p. 17

5.1 Category 1: Standards and Governance

  • Standards and governance requirements call for comparable metrics, defined terminology, and consistently applied tools matched to the claim, system, and input data.Audits should specify measures, test conditions, and acceptance rules; define robustness terms and distinguish robustness from related properties; and record tool versions, settings, and relevant limitations.Source: §5.1.1 · p. 18, §5.1.2 · p. 18, §5.1 · p. 18
  • Governance requirements make guidance actionable across the lifecycle and specify responsibility, audit access, and procedures for combining methods into conclusions.Guidance translates relevant technical, legal, and organisational requirements into checks, evidence, and review triggers; checks should be repeated at relevant lifecycle stages and after changes. Protocols assign investigation, corrective-action, continued-use, and notification decisions; access policies state inspection limits and their effect on conclusions; auditor protocols define objectives, methods, criteria, and how results support conclusions.Source: §5.1.4 · p. 19, §5.1.5 · p. 19, §5.1.6 · p. 19

5.2 Category 2: Engagement

  • Engagement requirements include affected stakeholder perspectives, prioritised participation, auditor independence, and early expert input into audit planning.Participation priorities reflect potential consequences, relevant knowledge, and failure likelihood and seriousness. Auditor relationships that could affect judgement should be disclosed and managed; technical, domain, legal or compliance, and organisational expertise should inform scope, tests, evidence needs, and acceptance criteria before key decisions.Source: §5.2.1 · p. 20, §5.2.2 · p. 20, §5.2.3 · p. 20

5.3 Category 3: Robustness Definition; 5.4 Category 4: Knowledge and Skills

  • Robustness definition and skills requirements specify tested concerns and trade-offs, while ensuring multidisciplinary expertise and role-specific, updated training.Audits should state whether they cover adversarial or non-adversarial robustness and identify test conditions; assess changes to other properties before and after robustness measures. Required expertise may span AI, software, security, statistics, law or policy, auditing, and domain knowledge, with missing expertise disclosed; training should cover role-relevant problems, methods, tools, and responsibilities and be updated as they change.Source: §5.3.1 · p. 21, §5.3.2 · p. 21, §5.4.1 · p. 21

5.5 Category 5: Data Auditability

  • Data auditability covers data sufficiency and provenance, quality and subgroup differences, and whether pre-processing mitigation resolves problems without hiding or introducing weaknesses.Data should represent intended users, important variations, rare cases, and operating conditions in sufficient quantity; limitations should be reflected in conclusions. Audits should trace sources and processing, check defects and group differences and their possible causes, and test data-level changes under the conditions assessed.Source: §5.5.1 · pp. 21–22, §5.5.2 · p. 22, §5.5.3 · p. 22

5.6 Category 6: Model Auditability

  • Model auditability requires condition-matched probing, conclusions limited to intended uses, and explainability methods used cautiously rather than as proof of failure causes.Audits state test conditions, measures, intended task, users, operating conditions, limitations, and excluded uses. Explanations should be relevant to the robustness question and their limits stated, since they can be unstable or misleading.Source: §5.6.1 · p. 22, §5.6.2 · p. 23, §5.6.3 · p. 23

5.6 Category 6: Model Auditability; 5.7 Category 7: Output Auditability

  • Model failure-source and mitigation requirements seek evidence for suspected causes and verify training-time improvements, while output mitigation is compared before and after inference.Audits investigate learned patterns or features linked to weaknesses and support claimed causes with test evidence. Training changes must be tested for improved reliability; output measures such as rejecting uncertain results or routing cases for human review must reduce the identified problem, since persistent failures do not establish resolution.Source: §5.6.4 · p. 23, §5.6.5 · p. 23, §5.7.1 · p. 23

5.8 Category 8: Workflow Integration.

  • Robustness checks should recur during development and operations, and audit tools should fit system interfaces, data, software, and resource constraints.Testing should follow important changes to models, data, code, and infrastructure; operational checks should include pre-release and production testing, verification of protective measures, and actions such as stopping a release or reverting a model. Tool use should not require unsupported data conversion or cause unacceptable disruption or delay.Source: §5.8.1 · p. 24, §5.8.2 · p. 24, §5.8.3 · p. 24

5.8.4 MR-8.4: External Component and Supply Chain Assurance.

  • Audits should identify external dependencies that may affect robustness, assess the effects of their failure or change, and explain how limited access constrains conclusions.SBOMs and AIBOMs can document software components and AI-specific assets, including models, datasets, dependencies, configurations, and provenance.Source: §5.8.4 · p. 24

5.9.1 MR-9.1: Robustness-aware Human Participation Levels.

  • Audits should define when human review is required and what authority reviewers have, while reassessing prior conclusions at intervals and after consequential changes.Reviewers may correct results, reject outputs, approve continued operation, or stop a system; involvement ranges from review to intervention or final authority. Reassessment should address changes such as model updates, deployment changes, failures, or newly known attack methods.Source: §5.9.1 · p. 25, §5.10.1 · p. 25

5.10 Category 10: Evaluation.

  • Audits should track each robustness problem through resolution or formal acceptance, prioritise more consequential problems, retest after correction, and validate suspected failures across system layers.Cross-layer validation should identify involved components, reproduce the suspected failure, and state which interactions remain untested.Source: §5.10.2 · p. 25, §5.10.3 · p. 25

5.11 Category 11: Monitoring.

  • Deployed systems should be monitored for robustness-relevant changes, with defined triggers for investigation or reassessment, and logged so events can be reconstructed.Automated alerts should be checked before being treated as evidence of degraded robustness. Logs should capture relevant inputs, outputs, model versions, settings, alerts, and failures, and be retained and protected for later investigation.Source: §5.11.1 · p. 26, §5.11.2 · p. 26

5.12 Category 12: Communication.

  • Audit results should be presented through interpretable visualisations and reports that disclose findings, evidence, uncertainty, limitations, and the conditions covered.Visualisations should suit their intended audience and show measures, changes, problems, and uncertainty; reports should identify the objective, claim, methods, evidence, unresolved problems, follow-up actions, and applicable system, model, data, and test conditions. Tools, visualisations, and documents should be usable and understandable to their intended users, with clear summaries and technical detail as appropriate.Source: §5.12.1 · p. 26, §5.12.2 · p. 26, §5.15.1 · p. 28

5.13 Category 13: Efficiency.

  • Audits should disclose resource limits and restrict conclusions to tested scope, adapt tests and criteria to the domain, task, model, and data, and justify configurable settings.Where methods are not established for a setting, audits should state that limitation; changed tool settings should be explained and aligned with the robustness claim.Source: §5.13.1 · p. 27, §5.14.1 · p. 27, §5.14.2 · p. 27

5.16.1 MR-16.1: Configurable Robustness Assessment Tools.

  • 43 literature-derived meta-requirements span 16 categories and three dimensions; 22 concern audit activities and evidence, 16 organisational and human aspects, and 5 audit-approach qualities.The dimensions are Functional, Human, and Quality.Source: §5.16.1 · p. 28

6 Practitioners’ Perspectives on the Meta-Requirements

The study reports practitioner perspectives on 43 literature-derived meta-requirements, drawing on 128 valid responses. Respondents brought varied AI/ML backgrounds and levels of robustness-assessment experience.

6 Practitioners’ Perspectives on the Meta-Requirements

  • 128 valid responses were retained to evaluate the 43 literature-derived meta-requirements.The study reports respondent characteristics to describe the validation sample’s experience and demographic composition.Source: §6 · p. 28, §6.1 · p. 28

6.1 Respondent Characteristics

  • The two largest experience groups had less than one year (40%) and one to three years (41%) of robustness-assessment or auditing experience.The sample included varied AI/ML-related roles and was not limited to experienced AI robustness auditors; findings should be interpreted in light of that composition.Source: §6.1 · p. 28

Robustness-related activities.

Respondents reported a range of activities relevant to validating the meta-requirements, especially robustness testing and failure analysis.

Robustness-related activities.

  • Robustness testing was the most frequently reported activity (73/128, 57.03%), followed by failure or error analysis (71/128, 55.47%).Robustness-focused model evaluation was reported by 59/128 respondents (46.09%).Source: Robustness-related activities · p. 29
  • Robustness metric selection or definition and adversarial or non-adversarial evaluation were each reported by 54/128 respondents (42.19%).Source: Robustness-related activities · p. 29
  • Domain-specific robustness evaluation was reported by 46/128 respondents (35.94%), and post-deployment robustness monitoring by 29/128 (22.66%).Other reported activities included developing or using assessment tools (44/128, 34.38%) and AI robustness auditing or assessment (42/128, 32.81%).Source: Robustness-related activities · p. 29

Geographic distribution.

The 128 respondents came from six continents, and practitioner ratings showed broad support for the literature-derived meta-requirements. Qualitative feedback informed refinements to the requirements.

Geographic distribution.

  • Respondents came from six continents, with Asia the largest share (66/128, 51.56%), followed by Africa (24/128, 18.75%) and Europe (22/128, 17.19%).Figure 6 presents respondents’ geographic distribution by country and continent.Source: Geographic distribution · p. 29, Figure 6 · p. 29

6.2 Overall Quantitative Validation

  • 81.20% of 5,504 ratings indicated agreement or strong agreement; 12.88% were neutral and 5.92% indicated disagreement.Table 8 summarizes overall validation results, while Table 9 presents the distribution of individual Likert responses.Source: §6.2 · p. 29, Table 8 · p. 30, Table 9 · p. 30

6.3 Meta-Requirement-Level Validation

  • Agreement or strong agreement ranged from 71.88% to 88.28% across the 43 meta-requirements; 41 were strongly supported and two supported (Table 10).None was quantitatively classified as challenged.Source: Table 10 · p. 30, §6.3 · p. 30

6.2 Overall Quantitative Validation

  • The 326 disagreement-level ratings and open-text comments helped interpret conditional acceptance, feasibility, scope, and overlap concerns.Source: §6.2 · p. 30

6.3 Meta-Requirement-Level Validation

  • MR-8.3 received the highest agreement (88.28%); MR-2.2 and MR-6.5 were supported rather than strongly supported, at 71.88% and 73.44%, respectively.Source: §6.3 · p. 30

6.3.1 Category-Level Validation.

  • All 16 survey categories received broad agreement-level support, while feedback called for clearer wording and sharper boundaries between some meta-requirements.Category-level ratings were interpreted alongside open-text feedback; Table 11 summarizes category-level agreement rankings.Source: §6.3.1 · p. 30, Table 11 · p. 30

6.4 Qualitative Practitioner Validation and Meta-Requirement Refinement

  • Qualitative feedback led to 394 refinements and 162 retentions; 13 comments prompted no change, and 15 were unusable for meta-requirement synthesis.Two comments retained an Add decision, both identifying the same previously uncovered concern.Source: §6.4 · p. 31

Retention of the literature-derived meta-requirements.

Practitioner comments reinforced the broad quantitative support for the literature-derived meta-requirements, explaining why specific requirements remained important.

Retention of the literature-derived meta-requirements.

Refinement of existing meta-requirements.

Practitioners generally accepted the literature-derived requirements while seeking clearer boundaries, evidence expectations, and implementation guidance. Their feedback refined requirements around standardisation, audit scope, risk-sensitive reassessment, participation, and evidence for conclusions.

Refinement of existing meta-requirements.

  • Standardisation should support comparable metrics, terminology, governance, and reporting without requiring identical tools or methods across contexts.Respondents also asked for actionable, tool-agnostic guidance.Source: Refinement of existing meta-requirements · p. 31
  • Audit requirements should make access, exclusions, assumptions, limitations, and evidence conditions explicit, with assurance claims aligned to available evidence.Source: Refinement of existing meta-requirements · p. 31
  • Assessment, review, and reassessment should vary with system risk, deployment context, and material change, using scheduled, event-driven, or incident-driven triggers.Practitioners favored change-triggered reassessment over repeated testing after every change.Source: Refinement of existing meta-requirements · p. 31
  • Practitioner input should reflect stakeholder roles, domain expertise, technical access, and independence, while human oversight should include defined interventions and auditable decisions.Source: Refinement of existing meta-requirements · p. 32
  • Audits should evaluate mitigations’ effectiveness, conditions, limitations, and trade-offs rather than treating mitigation presence as proof of robustness.Practitioners also requested clearer thresholds, versioning, reproducibility, reporting, scalable testing, and configuration records.Source: Refinement of existing meta-requirements · p. 32

Challenges and qualifications.

Practitioners did not reject any literature-derived meta-requirement, but questioned whether some implementation mechanisms should be universal or mandatory.

Challenges and qualifications.

  • Workflow and MLOps integration depth may depend on system complexity, deployment context, risk, resources, and MLOps maturity.Some practitioners considered integration important but not foundational or a prerequisite for valid auditing.Source: Challenges and qualifications · p. 32
  • Dashboards were viewed as optional, and practitioners cautioned that they should not replace a documented, authoritative evidence base.Source: Challenges and qualifications · p. 32
  • The comments qualified implementation mechanisms and evidence interpretation, including whether accessibility should have the same mandatory status as other requirements.The feedback informed proportionality, conditionality, and clearer distinctions between required audit outcomes and optional mechanisms.Source: Challenges and qualifications · p. 32

Practitioner-identified extension.

Practitioners proposed additional requirements, most of which clarified or operationalised existing ones. Their feedback identified an uncovered concern about externally controlled components and supply chain dependencies, leading to one added meta-requirement.

Practitioner-identified extension.

Merge and split proposals.

Practitioners noted overlap and uneven granularity in some requirements, but the study retained the original 16-category structure and clarified requirement boundaries instead of merging or splitting them.

Merge and split proposals.

  • Feedback identified overlap and uneven granularity across requirements, including stakeholder identification versus risk prioritisation and output-failure identification versus mitigation assessment.Respondents also noted overlap across evaluation, monitoring, workflow integration, documentation, accessibility, adaptability, and configurability.Source: Merge and split proposals · p. 33
  • No meta-requirements were merged or split; comments instead sharpened requirement boundaries while preserving the 16-category structure.The distinction between stress-test outcomes and the rationale and effectiveness of training-time mitigation was retained.Source: Merge and split proposals · p. 33

Integrated effect of practitioner validation.

Practitioners broadly supported the literature-derived requirements, mainly refining their scope and boundaries; they also identified an external-dependency gap that expanded the set to 44 requirements.

Integrated effect of practitioner validation.

  • Feedback mainly clarified the scope, applicability, evidence expectations, and implementation of existing concerns rather than replacing them.Proposed merges and splits were handled by clarifying boundaries, while retaining the original category structure.Source: Integrated effect of practitioner… · p. 33
  • 41 of 43 literature-derived meta-requirements were strongly supported, two were supported, and MR-level agreement ranged from 71.88% to 88.28%.Table 12 reports validation results for 43 requirements, with 128 respondents per requirement.Source: Integrated effect of practitioner… · p. 33, Table 12 · p. 33
  • Practitioner feedback primarily refined existing requirements and added one external-dependency requirement, producing 44 meta-requirements across 16 categories.The added requirement concerns robustness risks from external components and supply-chain dependencies.Source: Integrated effect of practitioner… · p. 33
  • MR-7.1 calls for identifying output-level bias or failure sources and assessing whether post-training mitigations address them.The excerpt also begins a requirement about accessibility of robustness tools and outputs, but does not provide its full wording.Source: Integrated effect of practitioner… · p. 34

7 Discussion

The discussion frames robustness auditing as assessing whether evidence supports a defined claim within its scope, conditions, and limitations. It presents the meta-requirements as context-sensitive guidance for system-level assurance and future evaluation.

7.1 A Robustness Test Is Necessary but Not Sufficient for a Robustness Audit

  • A robustness test supplies behavioural evidence, but an audit must establish whether that evidence supports the defined claim under traceable conditions and limitations.The audit connects test evidence to the claim's scope, criteria, assumptions, and reproducibility conditions.Source: §7.1 · p. 34

7.2 From Model-Level Robustness to System-Level Assurance

  • Model-level robustness evidence alone cannot establish deployed-system robustness because failures may arise in data, preprocessing, software, output controls, or deployment workflows.Audit conclusions should account for relevant component interactions and remain bounded to the deployed system.Source: §7.2 · pp. 34–35

7.3 Strong Literature-Practice Convergence with Contextual Qualifications

  • Practitioners largely retained literature-derived concerns while refining their scope and applicability, supporting stable requirements whose implementation remains context-sensitive.The discussion cites the need to balance standardization with flexibility.Source: §7.3 · p. 35

7.4 Robustness Is Conditional, Not Universal

  • Robustness claims depend on the system boundary, protected behaviour, operating conditions, threats, and acceptance criteria, so conclusions should not generalize beyond assessed conditions.Auditors should state what is claimed to be robust, under which conditions, and against which criteria.Source: §7.4 · p. 35

7.5 Robustness Assurance Is Time-Bounded and Requires Reassessment

  • Changes to models, data, deployment, or threats can weaken prior evidence, so reassessment should follow material changes and be proportionate to their significance and risk.The discussion treats assurance as a maintained conclusion, not a one-time judgement.Source: §7.5 · p. 35

7.6 An Auditable AI-Enabled System Requires an Audit-Ready Organisation

  • Credible auditing requires an audit-ready organisation with suitable expertise, independence, authority, responsibilities, and procedures for disagreements and follow-up.This complements technical auditability with governance, engagement, competence, and human oversight.Source: §7.6 · pp. 35–36

7.7 External Components Extend the Scope of Robustness Audits

  • External models, APIs, datasets, libraries, plugins, and cloud services can affect robustness claims while limiting auditor access to evidence.The proposed MR-8.4 extends the audit boundary to relevant dependencies and requires stating how restricted access constrains conclusions; its practical value needs further evaluation. Meta-requirements can inform audit design and completeness, but no systematic mapping to the EU AI Act or NIST AI RMF was performed.Source: §7.7 · p. 36, §7.8 · p. 36

7.8 Implications for Practice

  • Auditors should define claims and conditions before selecting tests; providers should maintain lifecycle evidence; tools should link claims, tests, evidence, and follow-up; standards should set common requirements while allowing context-specific methods.Research should operationalise the meta-requirements in an audit artefact and evaluate use in real engagements, including effects on audit quality and inter-auditor consistency; the present study does not establish those effects.Source: §7.8.1 · p. 36, §7.8.2 · p. 36, §7.8.3 · p. 36

8 Threats to Validity

The authors identify limitations across generalisability, construct measurement, researcher and respondent bias, and the scope of conclusions. Practitioner agreement supports the requirements but does not establish their operational effectiveness.

8.1 External Validity

  • The reviewed literature and purposively recruited practitioner sample may not represent non-academic practices or the wider practitioner population, limiting generalisability across contexts.The 74 studies form a systematically selected academic evidence base; the survey used non-probability sampling and may have self-selection and coverage bias.Source: §8.1 · p. 38

8.2 Construct Validity

  • Technology-agnostic requirements may have been interpreted differently, and Likert agreement does not establish shared interpretation, feasibility, or practical effectiveness.Open-text feedback complemented ratings, but interpretive variation may remain across professional and system contexts.Source: §8.2 · p. 38

8.3 Internal Validity

  • Researcher judgement shaped screening and requirement formulation, while survey fatigue, favourable-response tendencies, response bias, and qualitative coding interpretation may affect findings.The authors used independent screening, agreement checks, traceability, piloting, and structured coding to mitigate these risks, but acknowledge residual uncertainty.Source: §8.3 · pp. 38–39

8.4 Conclusion Validity

  • Practitioner agreement provides content-oriented validation, not evidence that applying the requirements improves audit quality, robustness outcomes, or inter-auditor consistency.The added MR-8.4 was not quantitatively rated because it emerged after the original 43 requirements were evaluated.Source: §8.4 · p. 39

9 Conclusion and Future Work

The study derives and practitioner-validates a 44-item set of AI robustness auditing meta-requirements, spanning concerns beyond model testing. Future work will operationalise the set in a decision model and design support system.

9 Conclusion and Future Work

  • 81.20% of practitioner ratings indicated agreement or strong agreement, and 41 of 43 literature-derived requirements were classified as strongly supported.The final practitioner-informed set contains 44 meta-requirements across 16 categories, including an additional requirement on external components and supply chain dependencies.Source: §9 · p. 39
  • The final set extends robustness auditing beyond model-level testing to claims, evidence, system components, limitations, monitoring, and reassessment.The literature review retained 74 studies and produced 43 requirements across 16 categories; practitioner feedback refined them and added one requirement.Source: §9 · p. 39
  • Future work will operationalise the meta-requirements in a decision model and integrate it into a design support system for practical auditing.The planned model will select requirements, evidence, tests, metrics, and assessment conditions for a given system and audit context.Source: §9 · p. 39

Declaration of Competing Interest

Ethics Approval

  • The practitioner survey received ethics clearance, and all participants provided informed consent before taking part.Source: Ethics Approval · p. 40

Related papers

Loading 2610.14356v1…