Source-linked AI summary
Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Richard Tomsett, Dave Braines, Dan Harborne, Alun Preece, Supriyo Chakraborty
TL;DR
Machine learning interpretability lacks consistently applied definitions despite growing interest driven by complex systems, real-world adoption, and data-processing rights. The paper develops a role-based ecosystem model, applies it across scenarios, and argues that it can support more rigorous analysis for researchers, developers, and regulators.
Problem
Interpretability definitions and motivations vary widely, while interpretability should be considered relative to particular agents and their goals.
Method
The paper defines an ecosystem model with six agent roles and uses scenarios to analyze role-specific beliefs, goals, and interpretability requirements.
Results
The model distinguishes how different roles view interpretability and relates those views to explainability, transparency, access constraints, and auditing needs.
Takeaways & Limitations
The model could help formalize interpretability terminology and support quantitative comparisons under well-defined circumstances.
Takeaways & Limitations
The executor role is debatable in the medical scenario because doctors, patients, or patient representatives may make treatment decisions.
Abstract
from arXiv · showhide
Several researchers have argued that a machine learning system's interpretability should be defined in relation to a specific agent or task: we should not ask if the system is interpretable, but to whom is it interpretable. We describe a model intended to help answer this question, by identifying different roles that agents can fulfill in relation to the machine learning system. We illustrate the use of our model in a variety of scenarios, exploring how an agent's role influences its goals, and the implications for defining interpretability. Finally, we make suggestions for how our model could be useful to interpretability researchers, system developers, and regulatory bodies auditing machine learning systems.
1. Introduction
Interpretability is treated as relative to particular agents and their goals, rather than as a property of a system in isolation. The paper proposes a role-based model to formalize these relationships and clarify relevant interpretability concepts.
- Interpretability research has grown with complex systems, wider deployment, and laws concerning citizens’ rights in data processing.
- Existing interpretability methods use widely varying definitions and motivations, motivating efforts to formalize the field.
- The paper models interpretability by identifying agents’ roles, beliefs, and goals in relation to a machine learning system.
- A machine learning system includes models, training data, interfaces, and documentation, potentially distributed across services, entities, locations, and data sources.
- The paper distinguishes explanations, interpretations, explainability, transparency, and interpretability as related but different concepts.
- The model is used to classify agents in scenarios and analyze how role-specific goals shape interpretability requirements and access constraints.
2. Ecosystem model
The ecosystem model places one machine learning system among agents who interact with it or are affected by it. It defines six non-mutually-exclusive roles to connect agents’ relationships and goals with interpretability requirements.
- The model defines six roles: creators, operators, executors, decision-subjects, data-subjects, and examiners.
- Roles are not mutually exclusive, so one agent can occupy multiple roles, while machines may increasingly fulfill roles in future.
- An ecosystem contains one machine learning system and one or more agents connected through interactions with or effects from that system.
- Creators make the system, operators interact directly with it, and executors make decisions informed by its outputs.
- Decision-subjects are affected by executor decisions, data-subjects supplied personal training data, and examiners audit or investigate the system.
- The categorization links an agent’s role to its scenario-specific goals and therefore to requirements for interpretable systems.
3. Example scenarios
The paper applies its ecosystem model across advertising, navigation, lending, medical advice, bail prediction, and military targeting scenarios. These examples assign system-related roles to concrete agents and organizations.
- Advertising: Advertising involves advertisers and developers as creators, the host website as operator and executor, users as decision-subjects, and authorities as examiners.
- Navigation: Navigation involves the app company as creator, the app user as operator, executor, and decision-subject, and road users as data-subjects.
- Lending: Loan assessment separates customer-facing employees as operators from senior lender employees as executors, with applicants as decision-subjects.
- Medical advice: Medical advice assigns medical professionals as operators and potentially executors, while patients are both decision-subjects and possible executors.
- Bail prediction: Bail prediction places judges or court staff as operators, defendants as decision-subjects, previous defendants as data-subjects, and lawyers as examiners during appeals.
- Military targeting: The military scenario includes coalition creators, analysts as operators, a commander as executor, a target as decision-subject, other individuals as data-subjects, and tribunal jurors as examiners.
4. Role-based interpretability
Interpretability requirements vary by role because agents pursue different goals and may have conflicting interests. The paper relates explainability and transparency to the information each role needs.
- Creators: Creator-implementers include architects, designers, engineers, technical writers, and subject-matter experts who construct or help train systems.
- Creators: Creators seek system understanding that helps optimize performance metrics such as accuracy, efficiency, bias minimization, or safety.
- Operators: Operators need to provide useful inputs and pass system information to executors, using feature-focused or textual explanations when appropriate.
- Executors: Executors need explanations suited to their decision goals, which can differ across route choice, treatment, or engagement decisions.
- Decision-subjects: Decision-subjects may seek explanations to understand, challenge, change, or prevent decisions affecting them, creating possible conflicts with creators and executors.
- Data-subjects: Data-subjects may seek transparency about data use and deletion consequences, which depend on whether the model uses stored weights or data directly.
- Examiners: Examiners may need explanations and transparency for compliance, safety testing, auditing, forensic investigation, and feedback on system improvements.
5. Discussion
The authors present the model as a first draft for formalizing interpretability and applying role-specific analysis to system development and regulatory auditing. It can help identify agents’ needs, manage conflicting goals and access, and examine rights compliance and system influence.
- The model is a first draft intended to stimulate discussion and improve the formalization of inconsistently used interpretability terminology.It builds on existing definitions, applies them consistently, and relates them to roles in the ecosystem model.
- Role-based analysis can help creator-owners identify agents’ interpretability needs and prioritize research and development according to ecosystem goals.
- The division of roles can reveal conflicting goals and support planning for how those conflicts should be managed.
- Creator-owners may use the ecosystem model to specify role-dependent access to explanations, balancing personal-data privacy with protection of intellectual property.Examples include restricting explanations by analogy that reveal personal data or limiting transparency to non-creators.
- Regulatory bodies may use the model for ecosystem audits and forensic investigations focused on rights, executor decisions, and the system’s influence, including possible bias.