PaperlayerView exact revision on arXiv

Source-linked AI summary

Know Your Model (KYM): Increasing Trust in AI and Machine Learning

Mary Roszel, Robert Norvill, Jean Hilger, Radu State

arXiv:2106.11036v1cs.CYcs.AIcs.LG

TL;DR

The paper addresses how to establish trustworthiness for increasingly consequential AI systems and how creators can document that trustworthiness. It proposes KYM, a framework of 20 implementation-agnostic guidelines combining model provenance with technical, ethical, and legal trust considerations. KYM is intended to increase transparency and provide records that help users assess trust, while leaving information-sharing mechanisms and technical guideline methods for future work.

  • Problem

    AI’s widespread use creates concerns about societal impacts, prediction errors, compliance, and the absence of integrated methods for documenting trustworthy development.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    KYM establishes model identity through detailed development records and 20 guidelines addressing provenance, efficacy, reliability, safety, responsibility, and trustworthy-AI principles.

    Source: §1 · p. 1, §4 · p. 5, §6 · p. 9

  • Results

    The paper presents KYM as a general framework for increasing transparency and trustworthiness across AI development processes and systems.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Developers should maintain thorough, frequently updated records of methods, tools, checks, and justifications used to address trust concerns.

    Source: §6 · p. 9

    KYM does not specify how information should be shared externally or prescribe technical methods for addressing its guidelines.

    Source: §6 · p. 9

Abstract

from arXiv · show

The widespread utilization of AI systems has drawn attention to the potential impacts of such systems on society. Of particular concern are the consequences that prediction errors may have on real-world scenarios, and the trust humanity places in AI systems. It is necessary to understand how we can evaluate trustworthiness in AI and how individuals and entities alike can develop trustworthy AI systems. In this paper, we analyze each element of trustworthiness and provide a set of 20 guidelines that can be leveraged to ensure optimal AI functionality while taking into account the greater ethical, technical, and practical impacts to humanity. Moreover, the guidelines help ensure that trustworthiness is provable and can be demonstrated, they are implementation agnostic, and they can be applied to any AI system in any sector.

1 INTRODUCTION

AI’s growing use in high-stakes settings raises concerns about prediction errors, societal impacts, and how trustworthiness can be established. The paper proposes KYM: a general, implementation-agnostic framework of 20 guidelines for documenting model identity and supporting trustworthy development.

1 INTRODUCTION

  • AI systems are increasingly used across sectors, including high-stakes areas where errors can harm patients, users, and communities.Medical AI may perform well in controlled environments yet perform poorly in real-world applications, increasing diagnostic time.Source: §1 · p. 1
  • Trustworthy AI requires attention to ethical, technical, and practical impacts, but developers lack a formal method for tracking how trust concerns are addressed.The paper frames trustworthiness around eight principles, including accountability, fairness, privacy, safety, and transparency.Source: §1 · p. 1
  • Know Your Model (KYM) treats each model as having a unique identity whose characteristics can support trust and ease regulatory burden.Knowing a model entails collecting, recording, and storing detailed records of its development processes.Source: §1 · p. 1
  • KYM proposes 20 guidelines addressing efficacy, reliability, safety, and responsibility across AI development.The guidelines are intended as concise, implementation-agnostic suggestions applicable to any AI implementation.Source: §1 · pp. 1–2
  • The framework helps creators collect appropriate information throughout development, strengthening the evidence for system effectiveness and trustworthiness.It is designed to account for differences in system complexity and the nuances of data and modeling processes.Source: §1 · p. 2

2 TRUSTWORTHY AI

Trustworthy AI extends beyond accuracy to accountability, fairness, human control, privacy, responsibility, human values, safety, security, and transparency. The paper describes why each principle matters while emphasizing unresolved challenges in accountability standards and explainability for complex models.

2 TRUSTWORTHY AI

  • Accuracy alone is insufficient in high-stakes settings because incorrect AI decisions can produce undesirable societal consequences.Trust must be developed and maintained throughout system development and deployment.Source: §2 · p. 2

2.1 Accountability

  • Accountability requires verifiable, replicable systems and information sufficient for experiments and results to be validated.Regulators still face an open question about balancing creator responsibility with AI capabilities [13].Source: §2.1 · p. 2

2.2 Fairness & Non-Discrimination

  • Fairness requires detecting and preventing discriminatory bias, which can enter through training data and affect sensitive decisions.Statistical methods applied to data or models are among the approaches used to address bias [6].Source: §2.2 · p. 2

2.3 Human Control of Technology

  • Human-control principles call for review or intervention capabilities when AI makes dangerous or costly decisions, although design guidance remains difficult to establish.Autonomous-vehicle accident scenarios illustrate the ethical concerns surrounding decisions made without human control.Source: §2.3 · p. 2

2.4 Privacy

  • Privacy is especially important when AI uses personal data in socially and economically consequential applications, while regulations such as GDPR impose substantial obligations.Applying the right to be forgotten is difficult because AI systems do not forget data like humans.Source: §2.4 · pp. 2–3

2.5 Professional Responsibility

  • Professional responsibility concerns the intentions, abilities, integrity, and ethical, legal, and scientific responsibilities of people involved in AI design, development, and deployment.Source: §2.5 · p. 3

2.6 Promotion of Human Values

  • Promotion of human values requires AI to reflect moral, ethical, and societal norms and to benefit society, but the principle lacks a concrete definition and direct technical tools.Source: §2.6 · p. 3

2.7 Safety & Security

  • Safety and security address unintended failures, adversarial attacks, robustness, and the interaction between security flaws and privacy risks.Malicious actors may manipulate inputs, degrade performance, or extract information about models and training data [26].Source: §2.7 · p. 3

3 RELATED WORK

Prior work improves transparency through provenance, lineage, lifecycle tracking, and documentation, but these efforts often cover only part of the AI lifecycle or a subset of trustworthy-AI principles. KYM is proposed to integrate provenance with broader technical, ethical, and legal trust considerations.

3 RELATED WORK

  • AI transparency research increasingly tracks data, models, deployment, updates, and the broader machine-learning lifecycle.Provenance seeks to improve replication, tracing, quality, and trust.Source: §3 · p. 3, §3.1 · p. 3

3.1.1 Data Transparency.

  • Data-provenance and lineage methods track data collection, transformation, storage, and use to improve replication, tracing, and quality assessment.Related proposals include blockchain-supported transparency, datasheets, data statements, nutrition labels, and data audits [16] [23].Source: §3.1.1 · p. 4

3.1.2 Model Transparency.

  • Model-transparency work includes lifecycle provenance systems, ModelOps, asset tracking, and FactSheets for communicating model purpose, performance, safety, security, and provenance.Developers have found FactSheets challenging and time-consuming because of recall, documentation, privacy, ownership, and clarity issues [21].Source: §3.1.2 · p. 4

3.2 Summary

  • Existing approaches generally focus on provenance or only a limited number of trustworthy principles rather than integrating the whole picture of trustworthy-system development.KYM responds by combining transparent development processes with trustworthy principles.Source: §3.2 · p. 4

4 KNOW YOUR MODEL (KYM)

KYM is a framework for increasing transparency and trust by documenting an AI system’s purpose, data, modeling processes, provenance, and trustworthiness considerations. It organizes these concerns into efficacy, reliability, safety, and responsibility.

4 KNOW YOUR MODEL (KYM)

  • KYM provides 20 implementation-agnostic guidelines for documenting an AI system’s purpose, data treatment, modeling processes, provenance, and trustworthiness.The framework is intended to increase user trust and ease creators’ burden while considering technical, ethical, and legal responsibility.Source: §4 · p. 4
  • Model identity consists of recording enough development information to distinguish one model from another and establish its uniqueness.KYM treats model characteristics and development records as information that can support knowing and trusting models.Source: §4 · p. 4
  • Because trustworthiness cannot be definitively proven, developers should document checks, tools, justifications, and regular updates while selecting priorities for each system.The framework recognizes that transparency requirements vary across AI systems and that trust concerns evolve with development and research.Source: §4 · pp. 4–5

4.1 Efficacy

  • Efficacy requires appropriate intended outcomes, human understanding, and attention to fairness and non-discrimination in data and modeling.It addresses transparency and explainability together with fairness-related risks and unintended outcomes.Source: §4.1 · p. 5

4.2 Reliability

  • Reliability requires verifiable and reproducible development, clear provenance, suitable methods, quality data, and attention to legal and regulatory requirements.Developers should preserve records of data transformations, modeling processes, inference verification, and auditing procedures.Source: §4.2 · p. 5

4.3 Safety

  • Safety covers safe design, failure handling, security against threats, system stability, and protection of user privacy.KYM connects safety with security and privacy, including audits, maintenance, adversarial robustness, and legal requirements for securing information.Source: §4.3 · p. 5

4.4 Responsibility

  • Responsibility connects technical implementation with professional, human-control, societal, and ethical implications.Creators should be transparent about their identities and stakeholders, the role of human intervention, and the societal purpose and impact of their systems.Source: §4.4 · p. 6

5 KEY GUIDELINES OF KYM

KYM translates its four concepts into documented guidelines covering system purpose, data, fairness, understandability, development processes, and provenance. The examples illustrate how creators can record these requirements across AI applications.

5 KEY GUIDELINES OF KYM

  • KYM defines guidelines for recording information needed to establish model identity, using RFC 2119 terminology to distinguish requirement strengths.The section presents example applications for the guidelines in Tables 1 and 2.Source: §5 · p. 6

5.1.1 E1: Creators SHOULD describe the intended purpose, use, tar-

  • Creators should describe an AI system’s intended purpose, use, target users, expected outputs, and any misalignment between intended and observed outcomes.This documentation may include use cases, release dates, validity periods, and positive or negative impacts.Source: §5.1.1 · p. 6, §5.2.3 · p. 7

5.1.2 E2: Creators MUST record (statistical) metrics about training and test datasets.

  • Creators must record statistical metrics for training and test datasets and should describe expected performance on unseen or differently distributed data.Dataset metrics help assess whether training and test distributions match, while deployment data may differ substantially from development data.Source: §5.1.2 · p. 6, §5.1.3 · p. 6

5.1.5 E5: Creators SHOULD aim for increased understandability.

  • Creators should document methods and metrics for addressing bias, discrimination, fairness, and explainability across data, models, outcomes, and user groups.The guidance includes data treatment, model checks, outcome verification, and explanations appropriate to target users.Source: §5.1.3 · p. 6, §5.1.5 · p. 6

5.2.1 RL1: Creators MUST record the processes followed in the de-

  • Creators must document development algorithms, techniques, data handling, verification, testing, output generation, and the information needed to identify, justify, and replicate methods.The required documentation varies with system type, while records should remain thorough and complete.Source: §5.2.1 · p. 6

5.2.2 RL2: Creators MUST ensure adequate provenance for data.

  • Creators must maintain adequate data provenance and end-to-end development provenance, including origins, transformations, environments, versions, and modeling stages.These records support assessment of data quality, understanding of transformations, and replication of model results and outcomes.Source: §5.2.2 · p. 6, §5.2.3 · p. 6

5.2.3 RL3: Creators MUST ensure adequate provenance for end-to-end model development.

  • The examples show guideline applications ranging from dataset distribution records and equipment-quality caveats to bias remediation and clinician-facing explanations.The examples include logistics, medical, criminal-sentencing, and e-commerce AI contexts.Source: §5.2.3 · p. 7

RL1 Creators MUST record the processes followed in the development of the AI system.

RL1 requires creators to document the processes followed throughout AI system development so that methods can be identified, justified, and reproduced.

RL1 Creators MUST record the processes followed in the development of the AI system.

  • RL1 requires thorough records of algorithms, techniques, data collection and storage, verification, testing, output generation, and method justification.The documentation must include information needed to identify methods, locate stored data, and replicate outcomes.Source: RL1 Creators MUST record the processes… · p. 7

RL4 Creators MUST record evaluation and performance metrics.

RL4 requires detailed records of evaluation and performance metrics, including technical conditions and results needed to assess model updates and reproduce outcomes.

RL4 Creators MUST record evaluation and performance metrics.

  • A classification example records a 70/30 test/train split, 10-fold crossvalidation, 80.2% accuracy, and sensitivity/specificity of 74.5%/61.8%.The example also identifies prediction accuracy and AUC as evaluation measures.Source: RL4 Creators MUST record evaluation… · p. 7
  • RL4-related examples include monitoring performance after retraining and supplying a Docker file with dataset and model settings to reproduce results.These examples connect update tracking and reproducibility with documented evaluation conditions.Source: RL4 Creators MUST record evaluation… · p. 7

5.2.4 RL4: Creators MUST record evaluation and performance met-

  • RL4 requires creators to record the metrics and techniques used to measure system performance, including accuracy, precision/recall, error rates, F-1 scores, and AUC.The guidance encourages recording metrics for both intermediary and final models.Source: §5.2.4 · p. 7
  • Table 2 presents example applications of the KYM guidelines for safety and responsibility, using simplified examples that may require more technical justification in practice.The table’s scope is safety and responsibility rather than efficacy and reliability.Source: Table 2

S1 Creators MUST assess safety to users and society.

The guidelines require creators to assess safety, security, privacy, and data-handling risks across AI systems. They also emphasize documenting controls, human involvement, and secure practices.

S1 Creators MUST assess safety to users and society.

S2 Creators MUST assess potential security, safety, and privacy failure points.

  • Creators must assess security, safety, and privacy failure points, including denial-of-service, database, de-anonymization, and data-poisoning threats.The finance example frames these risks through an explicit threat model.Source: S2 Creators MUST assess potential… · p. 8

S4 Creators MUST ensure user privacy, and appropriate treatment and use of private data.

S5 Creators SHOULD ensure secure data utilization and storage.

  • Secure data utilization and storage should be supported by traceability, documented personnel, and explicit human-AI roles.The examples connect signed data changes, developer qualifications, and required human confirmation to responsible system operation.Source: S5 Creators SHOULD ensure secure data… · p. 8

RP3 Creators SHOULD describe the impact, value, and benefit of the system.

The guidelines combine responsibility, lifecycle monitoring, safety assessment, security robustness, privacy protection, and secure data practices. Together, they require creators to document risks, controls, and performance-related evidence.

RP3 Creators SHOULD describe the impact, value, and benefit of the system.

5.2.5 RL5: Creators SHOULD track model update performance and information ingestion.

  • Model updates should be tracked with new-data origins, integration methods, and bounds on resulting performance changes.This creates a record of how post-deployment ingestion affects the system.Source: §5.2.5 · p. 8

5.2.6 RL6: Creators SHOULD record metrics on outcome replicabil-

  • Outcome replicability should be measured using state-of-the-art metrics.Source: §5.2.6 · p. 8

5.3 Safety

  • Safety must remain central to system design, with attention to failure contingencies, safety standards, societal effects, and transparent disclosure of concerns.Source: §5.3 · p. 8, §5.3.1 · p. 8

5.3.2 S2: Creators MUST assess potential security, safety, and pri-

  • Creators must assess security, safety, and privacy failure points and record metrics showing robustness against adversarial attacks and malicious actors.Security robustness should be improved continuously with state-of-the-art techniques because AI security evolves rapidly.Source: §5.3.2 · p. 8, §5.3.3 · p. 8

5.3.4 S4: Creators MUST ensure user privacy, and appropriate treat-

5.4 Responsibility

The responsibility guidelines require creators to document who shaped an AI system, how humans interact with it, what value it provides, and which legal requirements govern it.

5.4.1 RP1: Creators SHOULD disclose or record all entities involved in system development.

  • Creators should record the identities, qualifications, diversity, affiliations, and interests of entities involved in system design, development, and deployment.Source: §5.4.1 · p. 9

5.4.2 RP2: Creators SHOULD detail the implementation of human-AI interactions.

  • Creators should detail where human review, intervention, and responsibility occur in AI decisions.Source: §5.4.2 · p. 9

5.4.3 RP3: Creators SHOULD describe the impact, value, and bene-

  • Creators should justify an AI system’s societal impacts, values, benefits, and any potential detriments.Source: §5.4.3 · p. 9

5.4.4 RP4: Creators MUST comply with legal and regulatory re-

  • AI design, development, and deployment must attend to national, international, and vocational legal and regulatory requirements.Source: §5.4.4 · p. 9

6 DISCUSSION AND FUTURE WORK

KYM proposes a guideline framework for establishing model identity, combining provenance with trust principles and technical, ethical, and legal responsibility. The authors position transparency and record-keeping as practical routes toward assessing trust, while identifying information sharing and formalization as unresolved issues.

6 DISCUSSION AND FUTURE WORK

  • KYM establishes model identity through guidelines intended to increase transparency and trust in AI development processes.Source: §6 · p. 9
  • The framework combines provenance with multiple trust principles and bridges technical, ethical, and legal responsibility between research and industry.Source: §6 · p. 9
  • Developers should maintain and update records of methods, tools, checks, and justifications addressing trustworthy-AI principles for each model and update.The paper presents increased transparency as a step toward increasing overall trust.Source: §6 · p. 9
  • KYM does not specify how information should be shared outside an organization, and transparency requirements differ between internal and external users.Source: §6 · p. 9
  • Future work should formalize KYM, update technical assessment methods, automate record-keeping, and develop clearer avenues for sharing information externally.Source: §6 · p. 9

Related papers

Loading 2106.11036v1…