PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development

Danilo Brajovic, Niclas Renner, Vincent Philipp Goebels, Philipp Wagner, Benjamin Fresz, Martin Biller, Mara Klaeb, Janika Kutz, Jens Neuhuettler, Marco F. Huber

arXiv:2307.11525v1cs.AI

TL;DR

Practitioners face a lack of established standards and methods for securing and certifying AI applications. The paper proposes a four-card reporting framework spanning development, informed by prior work and interviews with experts, developers, and affected individuals. The authors present the framework as a foundation for future certification, while noting that best practices and regulation may change.

  • Problem

    Practitioners lack established guidelines, and certifying complex learned models is challenging because functionality and data representativeness both require assessment.

    Source: §1 · p. 2, §1.1.1 · p. 3

  • Method

    The authors combine prior card-based documentation and regulatory requirements into a four-step, four-card framework, incorporating interview feedback from certification experts, developers, and affected individuals.

    Source: §1 · p. 2, §3 · p. 31

  • Results

    The paper presents a development-process guideline that extends data and model cards with use-case and operation cards and incorporates feedback from interviews.

    Source: §3 · p. 31

  • Takeaways & Limitations

    The framework is intended to support development of trustworthy AI systems and provide a foundation for prospective system certification.

    Source: §1 · p. 2

    The authors note that AI and regulation are progressing quickly, so the guideline may require updates as best practices change.

    Source: §3 · p. 31

Abstract

from arXiv · show

Despite large progress in Explainable and Safe AI, practitioners suffer from a lack of regulation and standards for AI safety. In this work we merge recent regulation efforts by the European Union and first proposals for AI guidelines with recent trends in research: data and model cards. We propose the use of standardized cards to document AI applications throughout the development process. Our main contribution is the introduction of use-case and operation cards, along with updates for data and model cards to cope with regulatory requirements. We reference both recent research as well as the source of the regulation in our cards and provide references to additional support material and toolboxes whenever possible. The goal is to design cards that help practitioners develop safe AI systems throughout the development process, while enabling efficient third-party auditing of AI applications, being easy to understand, and building trust in the system. Our work incorporates insights from interviews with certification experts as well as developers and individuals working with the developed AI applications.

1. Introduction

The paper proposes a structured guideline for documenting AI applications across development, responding to gaps in standards and the challenges of certifying complex, data-driven systems. It combines regulatory and research guidance with input from experts, developers, and affected individuals.

1.5 Contributions and Methods

  • The guideline combines requirements from non-technical users and certification and standardization bodies, with references to support materials and requirement sources where possible.Affected individuals wanted involvement in development, transparent system visualization, and understandable outputs and decision reasons.Source: §1 · p. 2, §1.5.1 · p. 8

1.5.1 Motivation for Cards

  • The paper focuses on reporting around AI development rather than company governance, and chooses development-oriented cards because they are easier for developers to follow and easier for others to access.The cards can be updated when datasets or models change; integration into a classical V-model is left to future work.Source: §1.1 · p. 3, §1.5.1 · p. 8

1.1.1 Challenges in AI Certification

  • Certification is difficult because complex learned models can behave as black boxes, while training and test data may be unrepresentative.Certification must assess both model functionality and data representativeness.Source: §1.1.1 · p. 3

1.3.1 The European AI Act

  • The EU AI Act supplies a regulatory baseline, but its requirements remain subject to change and practical implementation details depend on standards still under development.The paper also describes ISO standards and other regulatory efforts, including NIST’s AI Risk Management Framework.Source: §1.3.1 · p. 5, §1.4.4 · p. 7

1.3.2 AI Assessment Catalog

  • The AI Assessment Catalog covers six risk dimensions and four action groups, but its more-than-100-page technical guidance can be cumbersome for practitioners.It addresses the whole application, including data, AI components, and surrounding non-AI software.Source: §1.3.2 · p. 5

1.4 Research Activities

  • Model cards standardize documentation of model characteristics and performance, while data cards aim to report dataset facts in comparable, accessible form and disclose known unknowns.Mitchell et al. (2019) propose model cards; Pushkarna, Zaldivar, & Kjartansson (2022) describe data cards.Source: §1.4.1 · p. 5, §1.4.2 · pp. 5–6

1.5 Contributions and Methods

  • The framework follows four development steps—use-case definition, data collection, model development, and model operation—and uses four cards to keep topics separate while limiting document burden.It is intended to support developers during development and provide a basis for future certification.Source: §1 · p. 2, §1.5 · p. 8

2. Proposed Approach

The proposal uses cards across four AI development steps, from defining the use case through operating the model. Each card includes background and, where possible, further references, while reminding teams to involve affected individuals.

2. Proposed Approach

  • The framework supports four steps: use case definition, data collection, model development, and model operation.It focuses on reporting around the AI application, not governance.Source: §2 · p. 9
  • Each card includes background information and references to further literature whenever possible.Source: §2 · p. 9
  • Every card prompts teams to consider involving people who may be affected by the AI system, whenever feasible.The paper also provides a toy use-case example in Appendix A.Source: §2 · p. 9

2.1 Use Case

The Use Case Card gives auditors a task overview and supports development-time compliance, trustworthiness, and robustness. Its risk assessment first raises awareness qualitatively, with later sections addressing quantification and risk reduction.

2.1 Use Case

  • The use-case summary covers the pre-AI status, relevant system components, proposed AI solution, contact person, and involved organizational groups.It also asks why a non-AI approach is insufficient, documents prior incidents in similar use cases, and distinguishes general-purpose AI from foundation models.Source: §2.1 · p. 9

2.1.1 Risk

  • Because AI may involve noisy, changing data and black-box behavior, classical risk-assessment methods often cannot be applied.The proposal therefore draws on qualitative and quantitative assessment schemes and risk dimensions from prior work.Source: §2.1.1 · p. 9
  • The proposed risk assessment examines use-case-specific risks across dimensions, including human oversight, technical robustness and safety, privacy and data governance, and societal and environmental well-being.The framework describes risk as the probability of an undesirable event and its impact; functional-safety settings almost always require quantitative assessment.Source: §2.1.1 · p. 10
  • The Use Case Card first uses qualitative assessment to raise awareness of critical aspects, then later sections aim to quantify and minimize those risks.Source: §2.1.1 · p. 10

2.2 Data

The data-card guidance translates AI Act dataset-quality expectations into practical documentation across research, dataset construction, processing, evaluation, and quality management. It addresses data and label quality, fit to real-world conditions, known limitations, and—where feasible—involvement of affected individuals.

2.2 Data

  • The data-card extension builds on prior documentation and adds regulatory references to address AI Act requirements for relevant, representative, error-free, and complete datasets.The authors also point practitioners to ISO/IEC data-quality and lifecycle standards, noting that two standards are still under development.Source: §2.2 · p. 13

2.2.1 Related Research

  • Before collecting data, active learning and core sets (D. Feldman, 2020) offer ways to select labels within a budget or train on a smaller representative subset.Other research considers unique points and data difficulty; estimating uniqueness is expensive, and label errors can favor larger models over smaller ones.Source: §2.2.1 · p. 13

2.2.2 Data Collection

  • With about 50% of MNIST training data, performance exceeds 90%, while doubling the data improves it by less than 5%; the authors also recommend investigating dataset coverage.They describe data-size rules of thumb but stress that collection and annotation can be expensive or infeasible; manually defined edge cases and embedding analysis can help find underrepresented scenarios, though rare points may increase privacy-attack risk.Source: §2.2.2 · pp. 14–16, Figure 1 · p. 15

2.2.3 Train-Test Split Creation

  • For test-set sizing, a randomly drawn sample of 659 supports 99% confidence that real-world performance is within ±1% of the test-set error rate, assuming representative data.The paper cautions that representativeness is not guaranteed and the model may still learn spurious correlations.Source: §2.2.3 · p. 16

2.2.4 Data Processing

  • Preprocessing should be performed separately on each data split when feasible to prevent information leakage, with fixed dataset processing documented in the data card and model-specific processing in the model card.The authors recommend involving domain experts and documenting whether and how they agreed to processing steps.Source: §2.2.4 · p. 17

2.2.5 Understanding and Documenting Data & Label Quality

  • Data-quality documentation should record dataset characteristics and label ambiguities, using dimensions such as accuracy, completeness, and consistency to guide assessment.The paper identifies Cleanlab and tensorleap as analysis tools, recommends a four-eyes process for manual annotation, and describes gold-standard data for checking third-party label quality.Source: §2.2.5 · p. 17
  • Document whether the data distribution fits real conditions and disclose limitations such as errors, noise, bias, or known confounders.Source: §2.2.5 · p. 19
  • Describe future dataset maintenance, storage and access, and how developers can identify the data version used to train a model.Source: §2.2.5 · p. 20
  • Document backup procedures, measures against data poisoning when necessary, and further recommendations or shortcomings in the data; the backup recommendation cites Poretschkin et al. (2021).Source: §2.2.5 · p. 20
  • Describe the involvement of affected individuals when feasible.Source: §2.2.5 · p. 20

2.3 Model Development

The section frames model development around reliability, fairness, and use-case requirements, addressing explainability, preprocessing, and model choice.

2.3 Model Development

  • Model development should document explainability, fairness, feature engineering, and other pillars of reliable, fair task execution.The authors build on Mitchell et al. (2019), adding explainable AI and real-world testing.Source: §2.3 · p. 21

2.3.1 Explainability

  • Explainability requirements should distinguish local from global explanations and consider method guarantees, evaluation, target users, and legal needs.Post-hoc methods such as LIME and SHAP may lack formal guarantees; the paper advises considering explanation demands before selecting a model.Source: §2.3.1 · p. 21

2.3.2 Feature Engineering, Feature Selection, and Preprocessing

  • Feature selection, engineering, and preprocessing should fit the use case and model, and occur only after data splitting to prevent leakage.For k-fold cross-validation, these steps must be performed independently on each fold.Source: §2.3.2 · p. 22

2.3.3 Model Selection

  • Model choice should match the problem, data, and use-case constraints, with baseline comparisons and preference for simpler models at similar performance.The paper notes that deep learning is not always best and that improved performance may require substantially more computation.Source: §2.3.3 · p. 22

2.3.4 Metrics

Metric selection should reflect the use case and model, while minimum acceptable scores should be set with domain expertise.

2.3.4 Metrics

  • Recall or precision may suit a use case better than general metrics such as accuracy or F1-score.The paper also notes that model choice influences which metrics are useful.Source: §2.3.4 · p. 22
  • The model choice also influences which evaluation metrics are useful.Source: §2.3.4 · p. 22
  • Domain experts should help determine the minimum acceptable score for the selected metrics.Source: §2.3.4 · p. 22

2.3.5 Hyper-parameter Optimization

Hyperparameter optimization can consume substantial computational effort and introduce leakage or unwanted variance, so the test set should remain unused until tuning ends.

2.3.5 Hyper-parameter Optimization

  • Hyperparameter optimization can involve hundreds of parameters and is usually the most computationally heavy model-development task.The authors note that understanding the problem and algorithms can help reduce the search space.Source: §2.3.5 · p. 23
  • Keep the test set locked until hyperparameter optimization is complete to avoid leakage.Source: §2.3.5 · p. 23
  • Cross-validation and hyperparameter-optimization methods may introduce unwanted variance.Source: §2.3.5 · p. 23

2.3.6 Model Evaluation

Model evaluation uses a held-out test set to assess performance and generalizability against use-case requirements, while guarding against leakage and other evaluation errors.

2.3.6 Model Evaluation

  • The test set must not be used for hyperparameter optimization or intermediate evaluation.Performance should be assessed against minimum requirements defined for the use case.Source: §2.3.6 · p. 23
  • Low test performance can reflect noisy data, model choice, or the selected metrics.Source: §2.3.6 · p. 23
  • A test error below the training error warrants caution because it may indicate leakage.ISO/IEC TS 4213 (2022) addresses classification performance of ML models.Source: §2.3.6 · p. 23

2.3.7 Model Confidence

Model confidence values may be poorly calibrated, so they should be evaluated and calibrated when needed; conformal prediction offers an alternative approach with fixed confidence.

2.3.7 Model Confidence

  • Modern ML models’ confidence values can poorly represent the underlying likelihood that a class is correct.Guo et al. (2017) report this calibration problem.Source: §2.3.7 · p. 23
  • Poorly calibrated confidence can lead to unwanted production behavior and unstable results.Source: §2.3.7 · p. 23
  • Confidence values should be evaluated and calibrated if necessary; conformal prediction trains algorithms with fixed confidence instead of post-hoc calibration.Abdar et al. (2021) and Guo et al. (2017) discuss calibration; Shafer & Vovk (2008) describe conformal prediction.Source: §2.3.7 · p. 23

2.3.8 Testing in Real-World Setting

Real-world testing complements test-set evaluation by checking an AI application under likely operational conditions, with domain experts involved in designing the tests.

2.3.8 Testing in Real-World Setting

  • Real-world testing should involve domain experts and cover edge cases, disturbances, environmental changes, and other likely conditions.The design should ideally be completed before the first model is trained.Source: §2.3.8 · p. 23
  • When real-world testing is not possible, computational stress tests should be conducted.Source: §2.3.8 · p. 23
  • The proposed checklist includes documenting risks and edge cases, test results, model limitations, and actions derived from testing.Source: §2.3.8 · p. 25
  • The checklist also calls for describing affected individuals’ involvement when feasible.Source: §2.3.8 · p. 25

2.4 Model Operation

Model operation requires coordinated attention to deployment processes, ongoing risk monitoring, transparency, drift, privacy, and MLOps practices.

2.4 Model Operation

  • The operation card treats deployment as a broad concern spanning technical performance, security, maintenance, workforce acceptance, knowledge management, and legal issues.It is proposed as a checklist for controlling assessed risks, with methods determined individually for each use case.Source: §2.4 · p. 26

2.4.1 Operating Concept

  • An operating concept describes how the AI application interacts with existing processes and should specify goals, services, constraints, stakeholders, responsibilities, sustainment, and decision milestones.The AI Assessment Catalog also calls for detailed descriptions of software components and APIs.Source: §2.4.1 · p. 26

2.4.2 Monitoring

  • Monitoring should be continuous or periodic according to risk, with safe-operation boundaries identified and the AI deactivated in favor of a failsafe system when those boundaries are exceeded.Live monitoring is recommended for anomaly detection and performance improvement.Source: §2.4.2 · p. 27

2.4.3 Human-Computer-Interface and Transparency

  • A transparency interface should provide different levels of information for operating staff and data scientists; explaining the algorithm alone is not sufficient for transparency.Kraus et al. (2021) make the latter point.Source: §2.4.3 · p. 27

2.4.4 Model Drift

  • Model drift includes changes in concepts, data, or software, and input data can be compared with training data to detect distribution shifts.The passage notes f-divergence as one possible measure for expressing distribution differences.Source: §2.4.4 · p. 27

2.4.5 Privacy

  • When personal data is collected, deployment should establish secure storage, deletion periods, and a process for responding to data-subject information requests.The passage notes GDPR data minimization and differential privacy as a possible privacy method.Source: §2.4.5 · p. 28

2.4.6 MLOps

  • MLOps recommendations include risk-based update frequency, stored training and model metadata, and a dedicated testing period with documented versioning for rollback.When rapidly collected and accurately labeled new data is unavailable, trained personnel should provide additional oversight.Source: §2.4.6 · p. 28

3. Summary and Conclusion

The paper proposes a development-process guideline for documenting AI applications, extending data and model cards with two new cards and a regulatory perspective. It is a snapshot that may need updates as AI and regulation evolve.

3. Summary and Conclusion

  • The guideline documents AI applications throughout development and extends prior model and data cards with use-case and operation cards.It incorporates a regulatory perspective and draws on interviews with certification experts, developers, and people affected by AI systems.Source: §3 · p. 31
  • Unlike efforts organized around risk dimensions, the guideline follows the AI application development process and aims to support trustworthy, easily audited systems from the beginning.The authors say it should lay foundations for future certification and position practitioners for upcoming legal requirements.Source: §3 · p. 31
  • Because AI and regulation are progressing quickly, the guideline is a snapshot that may require updates, including as dataset best practices change.The authors suggest a structured, regularly updated collection of organizational best practices and safety measures.Source: §3 · p. 31

Appendix A. Toy Audit

The appendix presents a toy audit of a carbon-fiber bicycle-frame inspection system, using use-case and data-card prompts to document the defect-detection model from data preparation through deployment. It covers performance and limitations alongside operational controls, transparency, testing, and model maintenance.

Appendix A. Toy Audit

  • The audit describes using computer vision on every carbon-fiber frame, with worker review of predicted defects and randomized CT scans to reduce inspection costs.The scenario concerns a manufacturer producing about 5,000 carbon-fiber frames annually; one CT scan is planned per 100 frames.Source: Appendix A · pp. 32–33
  • The inspection is a binary classification task targeting recall above 99%, precision above 90%, and inference time no greater than 1 second.The card also specifies optional heatmaps or attribution maps for predictions.Source: Appendix A · p. 34
  • Although classified as low risk because AI only suggests quality labels that workers can overwrite, missed defects could be shipped and harm customers.Standard safety requirements are stated to be fulfilled without AI, which is described as an additional security layer.Source: Appendix A · p. 35
  • The audit notes possible production stoppages or higher costs, employee-work analysis concerns, leaked production flaws, and workforce trust issues affecting inspection levels and quality.Trust issues could lead either to more manual inspections and higher costs or fewer inspections and lower quality.Source: Appendix A · p. 35
  • The cards record that partner data is licensed under an agreement and that frame images could reveal production weaknesses or design choices to competitors.The audit also says data on intact frames may reveal glueing techniques and defective frames may reveal weaknesses and unstable parts.Source: Appendix A · pp. 37–38
  • The dataset uses paired 1024×1024 RGB frame images collected internally and from a service partner, merged and checked for homogeneity, then externally labeled with expert oversight.Domain experts helped define data requirements, supplied labeling instructions and a gold-standard dataset, and spot-checked labels; clustering and expert checks addressed frame-model representation.Source: Appendix A · pp. 37–38

Data and testing

  • The 20% test split included gold-standard data, and two experts inspected all test labels; variance across test evaluations was below 1%.Domain experts also spot-checked labeling, and the dataset was divided into easy and hard examples, including unseen bike models in the hard set.Source: Appendix A · pp. 38–39

Data management

  • The audit specifies normalized images, brightness and Gaussian-noise augmentation, and restricted raw-data access, with redundant storage and encrypted external backups.New production-line and service-provider images are to be added after external labeling.Source: Appendix A · pp. 39–40

Data and testing

  • The dataset contained 4,700 intact and 500 defective frames, with a hard test set of novel service-provider frames intended to assess future-product applicability.The defective frames included 300 from the company's own production and 200 from the service provider.Source: Appendix A · p. 39

Limitations and safeguards

  • The audit notes that non-superficial defects may evade visual checks, frame or process changes may create undetected faults, and model confidence was not assessed.Undetected faults could ship broken frames, while false alarms could burden manual inspection and undermine staff trust; input/output monitoring, recurring computed tomography, and manual inspections are specified safeguards.Source: Appendix A · pp. 39, 41, 42

Model results

  • Recall was 100% on training and both moderate and hard test sets, while precision was around 90%.The audit says recall is sufficient for application, while improving precision could further reduce manual checks.Source: Appendix A · p. 40

Model selection

  • A deeper ResNet improved performance by less than 1%; random search raised test performance around 5%, and the final model balanced performance against size.Smaller or simpler models performed worse, while larger models increased inference time too much.Source: Appendix A · p. 41

Production testing

  • An ongoing test revealed problems under specific lighting conditions, so the dataset was extended to include those conditions.The initial deployment phase plans manual checks of all frames and injection of faulty frames into the production line.Source: Appendix A · p. 41

Operation

  • When the system flags possible damage, staff conduct a standardized visual and tactile examination, with trained employees and designated supervisor and IT responsibilities.Examiners must complete training with a test, and training in the user interface and AI basics is mandatory.Source: Appendix A · p. 42

Human oversight

  • The AI checks autonomously and notifies personnel of faults, while computed tomography tests and manual frame inspection provide ways to overrule its decisions.The user interface allows personnel to check the system and camera images at any time.Source: Appendix A · p. 42

Component risks

  • Risks are assessed separately for the CV model, cameras, data-processing pipeline, and user interface.Further detailed information is attached to the audit document.Source: Appendix A · p. 42

Monitoring

  • Precision and recall monitoring triggers alerts when precision exceeds 95% or falls below 85%, when CT detects a missed defect, or when defective-frame rates change.The system also monitors pixel-value distributions and the number and distribution of defective frames; input drift is checked with torchdrifts Kernel MMD.Source: Appendix A · p. 43

AI interface

  • Heatmaps make decisions transparent, with production staff seeing current frames and flagged defective frames and IT staff also accessing camera images, logs, metrics, and monitoring.IT personnel receive the production staff's views as well as access to logs, performance metrics, and monitoring information.Source: Appendix A · p. 43

IT security

  • Server rooms are restricted to authorized personnel, and trained authorized staff must have changes verified by a second person; the IT system is ISO 270001-certified and reviewed for recertification needs.Dedicated IT personnel manage updates, which also require second-person verification.Source: Appendix A · pp. 43–44

Privacy

  • No privacy-preserving algorithm is needed because the system collects no personal data.The audit marks the privacy-algorithm requirement as not applicable on that basis.Source: Appendix A · p. 43

MLOps

  • Code and model versioning uses Git and MLflow, and the model is retrained every three months when new data is available or when drift is detected.Retraining blends old and new data; updates go live if recall does not drop and precision increases or stays constant, while two consecutive performance drops prompt data analysis.Source: Appendix A · p. 43

Testing

  • Software tests are automated and documented, and new versions run alongside production in a test environment for one week before performance is compared with the old version.The comparison precedes deployment.Source: Appendix A · p. 44

Appendix B. Content Themes to Describe a Dataset according to Gebru et al. (2018)

The dataset content themes span its origins and governance, its composition and preparation, and considerations for use, safety, and interpretation.

Appendix B. Content Themes to Describe a Dataset according to Gebru et al. (2018)

  • Document dataset provenance and governance, including publishers, funding, access policies, retention, updates, maintenance, and upstream sources.The list also covers licensing and other regulatory or compliance policies.Source: Appendix B · p. 45
  • Describe dataset composition and construction through modality, features, examples, collection criteria, cleaning, processing, ratings, labels, and validation.The themes include descriptive statistics, missing attributes, and differences between dataset versions.Source: Appendix B · p. 45
  • Cover intended applications, practical safety and limitations, fairness, representation, known patterns, prior use, and requirements for combining datasets.The list also calls for definitions of technical terms and domain knowledge needed to use the dataset.Source: Appendix B · p. 45
Loading 2307.11525v1…