PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: Machine Learning Governance

Varun Chandrasekaran, Hengrui Jia, Anvith Thudi, Adelin Travers, Mohammad Yaghini, Nicolas Papernot

arXiv:2109.10870v1cs.CRcs.LGcs.SE

TL;DR

ML risks are often studied pairwise, leaving the interplay among multiple risk facets insufficiently understood. The paper systematizes ML governance around ownership, accountability, assurance, and life-cycle management, and models interactions among principals as a game. The resulting game formulation proposes explicitly searching for equilibria that balance the benefits of ML with its risks.

  • Problem

    ML risks are often studied pairwise, leaving the interplay among multiple risk facets insufficiently understood.

    Source: §VIII · p. 13

  • Method

    The paper systematizes ML governance around ownership, accountability, assurance, and life-cycle management, and models interactions among principals as a game.

    Source: §I · pp. 1–2, B. Auditing to Identify New Model… · p. 10, §VIII · p. 13

  • Results

    The resulting game formulation proposes explicitly searching for equilibria that balance the benefits of ML with its risks.

    Source: §VIII · p. 13

  • Takeaways & Limitations

    The framework fosters trust between principals and formalizes their interaction in recalibrating ML risk, while identifying open questions about the interplay of risks.

    Source: §VIII · p. 13

    Understanding of how different ML risks interact remains limited; research has mainly examined such interactions pairwise.

    Source: §VIII · p. 13

Abstract

from arXiv · show

The application of machine learning (ML) in computer systems introduces not only many benefits but also risks to society. In this paper, we develop the concept of ML governance to balance such benefits and risks, with the aim of achieving responsible applications of ML. Our approach first systematizes research towards ascertaining ownership of data and models, thus fostering a notion of identity specific to ML systems. Building on this foundation, we use identities to hold principals accountable for failures of ML systems through both attribution and auditing. To increase trust in ML systems, we then survey techniques for developing assurance, i.e., confidence that the system meets its security requirements and does not exhibit certain known failures. This leads us to highlight the need for techniques that allow a model owner to manage the life cycle of their system, e.g., to patch or retire their ML system. Put altogether, our systematization of knowledge standardizes the interactions between principals involved in the deployment of ML throughout its life cycle. We highlight opportunities for future work, e.g., to formalize the resulting game between ML principals.

I. INTRODUCTION

ML governance addresses the difficulty of balancing societal benefits and deployment risks amid research that often examines technical risks in isolation. The paper systematizes ownership, accountability, assurance, and life-cycle management, and formalizes interactions among principals as a game.

I. INTRODUCTION

  • Research often studies technical ML risks in isolation, leaving institutions and regulators without clear ways to manage or contain societal risks.The paper notes that iterative deployment further complicates risk management and that existing regulation can be inadequate.Source: §I · p. 1
  • Unforgeable ownership claims about models and training data establish identity, a prerequisite for holding model developers accountable for model limitations.The approach then attributes failures to owners and end users, using attribution and auditing, including audits that identify failures not yet discovered.Source: §I · pp. 1–2
  • Because risk-benefit trade-offs are regularly re-evaluated, ML systems need life-cycle management to support versioning, behavior changes, patching, and retirement.Machine unlearning offers a partial but incomplete response to the need for patching ML systems.Source: §I · p. 2
  • The paper formalizes interactions among model owners, users, and regulators as a Stackelberg game that can capture competing interests and trade-offs.The formulation delineates a Pareto frontier for balancing risks and benefits.Source: §I · p. 2
  • The paper systematizes techniques for assurance that build confidence an ML system meets specific security requirements.Assurance supports model owners’ claims that they have controlled risks.Source: §I · pp. 1–2
  • The authors identify opportunities for collaboration beyond computer science so research goals and technical concepts can reflect societal needs and gain broader adoption.The paper emphasizes that designing governance mechanisms assumes they are informed by societal needs.Source: §I · p. 2

II. BACKGROUND

The background introduces supervised learning as learning a model from labeled examples by minimizing prediction error. It then outlines gradient-based weight updates and holdout-set evaluation.

A. Machine Learning Primer

  • Supervised learning uses labeled input-output examples to train a function that maps inputs to labels, with a task-specific loss characterizing prediction error.For classification, cross-entropy is a common loss choice.Source: A. Machine Learning Primer · p. 2
  • When no closed-form solution is available, gradient descent iteratively tunes model weights to reduce loss, often using a randomly sampled batch.The learning-rate hyperparameter η is tuned.Source: A. Machine Learning Primer · p. 2
  • A labeled holdout set is retained to evaluate how well a trained model generalizes to unseen points.Source: A. Machine Learning Primer · p. 2

B. Operational Life-cycle

ML governance constrains interactions across the model life cycle to balance application benefits against risks. Because principals have different goals and risk priorities, governance is framed as a game among them.

B. Operational Life-cycle

  • The life cycle spans data collection and cleaning, model training and testing, model serving, and model retirement.Collection can face legal, technical, or business difficulties; testing typically uses holdout data assumed to match the training distribution, while serving may use a different software framework.Source: B. Operational Life-cycle · pp. 2–3

A. Overview

  • Governance balances ML benefits against risks and constrains how principals interact throughout an application’s life cycle.The overview identifies the model owner, end user, and regulator as principal types.Source: A. Overview · p. 3
  • Constraints beyond training performance matter: unwanted bias can prompt owners to reassess data collection and algorithmic choices, while users and regulators face distinct risks.Although constraints can limit the feasible training objective, they may also direct the search and improve the solution found.Source: A. Overview · p. 3
  • Because principals assign different urgency to risks and may have conflicting goals, governance is best formalized as a game with constrained objectives.The underlying metrics may differ from accuracy or be worst-case, so established evaluation practices may not provide the guarantees governance requires.Source: A. Overview · p. 3

B. Principals, Incentives, and Capabilities

ML governance involves principals with distinct roles and incentives, while relevant risks arise across the model life cycle. The paper discusses integrity, privacy, and fairness risks and notes that addressing them can involve trade-offs.

B. Principals, Incentives, and Capabilities

  • The ecosystem includes end users, model owners, model builders, data collectors, adversaries, and trusted arbitrators, with one entity sometimes occupying multiple roles.Trusted arbitrators may audit services or resolve ownership disputes; the paper also calls this principal a regulator.Source: B. Principals, Incentives, and… · pp. 3–4

C. Risks

  • The paper highlights three risk classes and says their failure modes shape design, deployment, retention, accountability, and assurance.Its coverage is not exhaustive because of space constraints, though the authors expect the governance aspects to generalize to other risks.Source: C. Risks · p. 4
  • Integrity risk is misalignment between system output and expectations; outliers, concept drift, poisoning, backdoors, and evasion can undermine correctness or generalization.Poisoning can produce indiscriminate or targeted misclassification, while backdoor triggers cause specific misclassification; evasion attacks modify inference inputs.Source: C. Risks · p. 4
  • Privacy risks include training-data leakage and attacks that infer membership, missing attributes, or likely training examples from model access or gradients.Memorization can expose sensitive data at inference; the cited evaluated methods found differentially private training effective, often at a performance cost.Source: C. Risks · pp. 4–5
  • Fairness risks can arise from historical data bias, model choices, and evaluation when predictions alter subjects’ future behavior.The paper distinguishes statistical data bias, societal normative mismatch, model bias, and evaluation risks; selective labels and sampling processes can shape collected data.Source: C. Risks · p. 5
  • The ideal of balanced data and equal predictions can conflict with privacy and integrity because sample sizes are limited by the smallest subpopulation.That constraint may limit generalization and adversely affect privacy as well as integrity.Source: C. Risks · p. 5

D. The Governance Games

ML governance concerns independent principals whose goals may align or conflict. The authors propose using game-theoretic tools to analyze their dynamics, often as sequential leader-follower interactions.

D. The Governance Games

  • Governance arises from interactions among independent decision-makers whose goals may or may not align.A model owner’s robustness objective may exacerbate leakage of an end user’s private information because adversarial training encourages stable predictions near training points.Source: D. The Governance Games · p. 5
  • The authors frame governance as a multi-agent system whose principal dynamics can be analyzed with standard game-theoretic tools.Game solutions affect trust among principals and toward the ML application itself.Source: D. The Governance Games · p. 5
  • Because principals often act sequentially after perceiving others’ decisions, many governance scenarios have a leader-follower, or Stackelberg, structure.Source: D. The Governance Games · p. 5

1) Stackelberg formulation of learning problems:

Stackelberg games model strategic leader-follower interactions, but ML governance can involve additional principals and stages. The paper describes extensive-form games as a way to represent those interactions and illustrates one with regulation, competing builders, and repeated decisions.

1) Stackelberg formulation of learning problems:

  • Stackelberg competitions allow a follower’s objective to differ from the leader’s, unlike zero-sum formulations that require exactly opposite objectives.In strategic classification, subjects adjust features after observing a decision, potentially shifting the distribution and prompting model maintainers to retrain or use online learning.Source: 1) Stackelberg formulation of learning… · p. 5
  • Performative stability describes models whose predictions remain optimal under distribution shifts resulting from those predictions.A separate spam-filtering game models an adversary adapting to a learner’s decisions.Source: 1) Stackelberg formulation of learning… · p. 5

2) Beyond two-player competitions:

  • Governance can involve several principals deciding without observing opponents’ actions, requiring strategic-form games represented by payoff tables.Examples include owners anticipating regulation, users changing trust assumptions, and builders competing for contracts.Source: 2) Beyond two-player competitions · p. 5
  • When interactions repeat across stages, extensive-form decision trees can represent multi-stage games with observed actions.The figure caption indicates strategy sets may be continuous or discrete, information sets may be non-singleton, and games may repeat.Source: 2) Beyond two-player competitions · pp. 5–6, Figure 1 · p. 6
  • The hypothetical game has a regulator specify privacy and fairness requirements, an owner commission two competing builders, and the regulator verify outcomes and apply fines or incentives.The builders choose models and privacy-budget allocations; the owner accepts one model, and the regulator may update regulation based on repeated play.Source: 2) Beyond two-player competitions · p. 6, Figure 1 · p. 6
  • The authors identify extensive-form games as a promising avenue for formalizing principal interactions and developing ML governance.Source: 2) Beyond two-player competitions · p. 6

3) Solution Concepts:

Governance games involve principals with potentially conflicting objectives and incomplete information, making exact solution methods difficult to apply. Establishing identity through ownership of training data, test data, and models helps identify principals and clarify how they interact with ML systems.

3) Solution Concepts:

  • Exact strategies can be intractable for multiple principals, continuous strategy sets, or recurring interactions; correlated equilibria are easier to find iteratively.Calculating Nash equilibria is NP-hard; correlated equilibria are a superset of Nash equilibria.Source: 3) Solution Concepts · p. 6
  • Principals may have incomplete information about the game, such as model builders not knowing rivals’ fairness or privacy guarantees.These settings are known as Bayesian games.Source: 3) Solution Concepts · p. 6
  • Governance research should model multiple principals, independent objectives, and possibly imperfect information rather than isolated risk trade-offs.The authors suggest this could alleviate problems such as choosing hyperparameters for parametric privacy or fairness definitions.Source: 3) Solution Concepts · p. 6

IV. ESTABLISHING IDENTITY IN ML SYSTEMS

  • ML identity work associates principals with training data, test data, and learned models, alongside access controls for their interactions.ML systems introduce often-implicit information flows between data and models, calling for specific mechanisms.Source: §IV · p. 7

A. Data Ownership

Data and model ownership raise distinct legal and technical questions, from protecting data during training or inference to proving who owns a model. The surveyed approaches include cryptographic and distributed learning, extraction defenses, and ownership proofs, with unresolved needs for robust model identification.

A. Data Ownership

  • Homomorphic encryption can protect data confidentiality while enabling training or inference, but cryptography can impose computational bottlenecks and hinder hardware acceleration.Federated learning keeps data at owners’ sites while locally computed updates are aggregated; MPC approaches often have poor scalability.Source: A. Data Ownership · p. 7
  • Test data does not permanently change model parameters, but adversarial or sensitive inputs can still complicate governance and require technical and legal ownership protections.Data collections and derived works may or may not receive copyright protection depending on jurisdiction and conditions such as originality.Source: A. Data Ownership · p. 7

B. Model Ownership

  • Model extraction uses inference queries to recover a model’s functionality, which may range from parameters to output behavior for particular inputs.Extraction has been demonstrated across image, text, and hypothesis-class settings, including DNNs; data-free methods use disagreement between victim and adversary models.Source: B. Model Ownership · p. 8
  • Extraction defenses include limiting precision, randomizing model selection, adding noise, or perturbing outputs, but adaptive strategies can defeat approaches or reduce utility.Watermarking encodes a secret query-response pair for ownership verification, while proof-of-learning uses logged training information to reproduce intermediate states.Source: B. Model Ownership · p. 8
  • Existing ownership metrics may not capture model closeness, and research is needed on intrinsic model properties that could serve as fingerprints.The authors also note that many extraction-detection techniques rely on minor training-procedure changes.Source: B. Model Ownership · p. 8

C. Is model ownership separate from data ownership?

The paper distinguishes evidence about training data from evidence about model ownership, then frames accountability as assigning responsibility for known or newly discovered failures. It surveys failure-attribution approaches and notes limits in detecting some attacks and tracing synthetic content to a particular model.

C. Is model ownership separate from data ownership?

  • Dataset inference can support model-ownership claims when only the owner has proprietary training data, but cannot prove ownership when training used public data.Legal treatment of models and derived works can also vary with originality and jurisdiction.Source: C. Is model ownership separate from… · p. 9

V. HOLDING ML PRINCIPALS ACCOUNTABLE

  • Identity-based accountability assigns responsibility for model failures to owners or end users, while attribution identifies responsible principals and audits can uncover previously unknown failures.External scrutiny, including by regulators, may reveal failures not initially known.Source: §V · p. 9

A. Attribution of Known ML Failures

  • Known-failure attribution surveys detection approaches for outliers, concept drift, poisoning, and adversarial examples.Methods include confidence or representation analysis, drift testing, and analysis of poisoned-data relationships to model representations, gradients, or decision boundaries.Source: A. Attribution of Known ML Failures · p. 9
  • Extraction detection can fail when attackers use public in-distribution data, and no inference-time mechanism is currently known for identifying privacy attacks.Detecting and attributing adaptive membership inference during an attack remains an open problem.Source: A. Attribution of Known ML Failures · pp. 9–10
  • Attributing synthetic content to a particular generative model is difficult because models can produce and encode arbitrary content.Studying model components such as upsampling components that produce spectral aliasing is described as a promising approach.Source: A. Attribution of Known ML Failures · p. 10

B. Auditing to Identify New Model Failures

The paper distinguishes internal audits, where the auditor is the model owner, from external audits by independent entities, and explains how logs can support examination of model properties. It also highlights practical challenges in ML logging, including what to record, confidentiality, and limited academic work on the topic.

B. Auditing to Identify New Model Failures

  • ML audits extend formal examination to verification of claimed model properties; internal audits can use model internals, code, and logs, while external audits may rely only on queries.Logs may alleviate technical limits of query-only external audits, including the lack of robustness guarantees in that setting.Source: B. Auditing to Identify New Model… · p. 10
  • Internal review can check SISA model-deletion code, while external query-based verification of unlearning remains challenging because membership inference has many false positives.DP-SGD includes a privacy accountant that tracks expenditure; post-hoc privacy audits can be computationally expensive.Source: B. Auditing to Identify New Model… · p. 10
  • ML logging is difficult because what and when to record is use-case dependent, storage is finite, and pipeline events and lifecycle stages require diverse information.Logs may need to track both principals responsible for inputs and model properties relevant to attribution.Source: B. Auditing to Identify New Model… · p. 10
  • Logs accessible to auditors create confidentiality risks that require sanitization, which is complicated by their stream nature; adversaries may also target log analysis.The passage describes attacks that keep intermediate steps within acceptable levels while evading an audio-based ML model.Source: B. Auditing to Identify New Model… · p. 10
  • Despite industry ML logging tools and practices, little academic work has addressed ML logging challenges.Source: B. Auditing to Identify New Model… · p. 11

VI. ASSURANCE IN ML

Assurance builds evidence-based confidence that ML systems meet requirements across data preparation, training, and deployment. The surveyed techniques leave open questions, including formal guarantees for dataset properties and fairness.

VI. ASSURANCE IN ML

  • Assurance builds confidence from evidence that an ML system meets security requirements, with security covering broader risks identified in the paper.The section surveys techniques across the ML life cycle.Source: §VI · p. 11

A. Assurance during Data Preparation

  • Data preparation uses sanitization, drift and bias checks, subgroup-representation checks, and documentation, but formal guarantees for dataset properties remain unavailable.Datasheets document whether data-collection practices address known failures.Source: A. Assurance during Data Preparation · p. 11

B. Assurance for Training

  • Training assurance for integrity includes adversarial training, robustness certificates, poisoning defenses, and proofs tied to the gradient-descent run that produced model parameters.Some certified poisoning defenses require clean data, which the cited passage describes as often impractical.Source: B. Assurance for Training · p. 11
  • Training assurance addresses privacy, randomness, and fairness, but assurances for cryptographically secure learning randomness remain an open problem.Privacy methods include DP learning and decentralized approaches; fairness methods include individual certifications, mitigation, and empirical subgroup documentation.Source: B. Assurance for Training · p. 11
  • Fairness mitigation techniques do not strictly provide formal fairness guarantees, motivating algorithms that provably produce fair models.The paper calls for studying interactions among principals and risks beyond fairness alone, including fairer, more accurate, and non-privacy-invasive data collection.Source: B. Assurance for Training · p. 12

C. Assurance at Deployment

At deployment, service providers can offer assurance about inference integrity and query confidentiality. The paper then surveys life-cycle maintenance options, including unlearning and fine-tuning, and highlights unresolved choices about how much data removal requires.

C. Assurance at Deployment

  • Deployment assurance can address inference integrity and query confidentiality, including sensitive health information sent to online models.Integrity means the advertised model made the prediction correctly; confidentiality protects users’ potentially sensitive queries.Source: C. Assurance at Deployment · p. 12
  • Verified computation can check inference correctness, while SafetyNets applies these techniques to DNN inference more efficiently than off-the-shelf approaches.Verifying computation integrity differs from assuring confidence in a prediction.Source: C. Assurance at Deployment · p. 12
  • Homomorphic encryption and MPC are cryptographic approaches to confidentiality-preserving inference, and the paper describes both as provably correct.Source: C. Assurance at Deployment · p. 12

A. Patching ML Systems

  • Unlearning can address revoked or problematic training data, but methods trade off independence from removed data, computational cost, and model utility.Adaptive revocation can itself leak whether a user’s data appeared in training; the appropriate removal guarantee may differ between privacy revocation and poisoning.Source: A. Patching ML Systems · p. 12
  • Fine-tuning supports online learning and transfer learning, but new behaviors can cause catastrophic forgetting of previously learned examples.Source: A. Patching ML Systems · p. 13

B. Retiring ML Systems

The paper treats retirement as a life-cycle option when unlearning leaves too little data for a performant model or ethical concerns favor retiring a pipeline. Its discussion identifies open challenges in risk interactions, trust, parameterization, regulators, and aligning law with technical approaches.

B. Retiring ML Systems

  • A model may be retired when extensive unlearning leaves insufficient data for performance or when retiring the data could address ethical concerns.Regulators may develop frameworks encouraging principals to retire data and models when appropriate.Source: B. Retiring ML Systems · p. 13

VIII. DISCUSSION

  • The governance framework combines ownership, accountability, assurance, and life-cycle management, while its game formulation seeks equilibria balancing ML benefits and risks.The analysis assumes supporting computer systems form a trusted computing base and relies on their security.Source: §VIII · p. 13
  • Research has examined risk interactions mainly pairwise, leaving limited understanding of how multiple ML risks interact together.Examples include utility with robustness and privacy with fairness.Source: §VIII · p. 13
  • Dataset-card-like documentation could help communicate system limitations, while interpretability may help non-experts understand why issues occur and how to resolve them.The paper distinguishes feature-level from concept-level attribution.Source: §VIII · p. 13
  • Assurance parameters such as DP’s ε can be difficult for average users to interpret and specify, and privacy expectations may not match what is provided.The paper distinguishes parameterization, which informs training requirements, from interpretability of outcomes.Source: §VIII · p. 13
  • The framework assumes regulators are trustworthy, but regulators may not be trustworthy or permitted to access sensitive information.Source: §VIII · p. 13
  • A gap between technical approaches and legal frameworks can impede adoption, as illustrated by DP’s technical focus and legal emphasis on anonymization.The paper also notes that legal protections against disparate impact preceded their recent adoption by the ML community.Source: §VIII · p. 13

Related papers

Loading 2109.10870v1…