Source-linked AI summary
Machine Teaching: A New Paradigm for Building Machine Learning Systems
Patrice Y. Simard, Saleema Amershi, David M. Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
TL;DR
Machine learning development is constrained by specialized expertise and inefficient processes, limiting the supply of systems relative to demand. This position paper defines machine teaching as a teacher-centered discipline that applies software-engineering principles and decouples teaching from algorithmic complexity. It argues that the approach can improve productivity, collaboration, reproducibility, and access to model building.
Problem
Demand for machine learning models exceeds the supply of machine teachers, while current processes depend on deep machine learning expertise and create process inefficiencies.
Method
The paper positions machine teaching as a teacher-centered discipline using software-engineering principles, teaching-language abstractions, and interfaces that shield teachers from algorithmic and runtime complexity.
Results
The paper argues that emphasizing teachers can improve model-building productivity, including through concept manipulation, collaboration, reproducibility, and teacher-accessible model construction.
Takeaways & Limitations
Machine teaching broadens the focus of machine learning systems from learner algorithms to the human processes and tools used to build and maintain models.
Takeaways & Limitations
The approach assumes that target concepts are realizable through recursive composition of models and existing features, making feature completeness a responsibility of the teaching tool.
Abstract
from arXiv · showhide
The current processes for building machine learning systems require practitioners with deep knowledge of machine learning. This significantly limits the number of machine learning systems that can be created and has led to a mismatch between the demand for machine learning systems and the ability for organizations to build them. We believe that in order to meet this growing demand for machine learning systems we must significantly increase the number of individuals that can teach machines. We postulate that we can achieve this goal by making the process of teaching machines easy, fast and above all, universally accessible. While machine learning focuses on creating new algorithms and improving the accuracy of "learners", the machine teaching discipline focuses on the efficacy of the "teachers". Machine teaching as a discipline is a paradigm shift that follows and extends principles of software engineering and programming languages. We put a strong emphasis on the teacher and the teacher's interaction with data, as well as crucial components such as techniques and design principles of interaction and visualization. In this paper, we present our position regarding the discipline of machine teaching and articulate fundamental machine teaching principles. We also describe how, by decoupling knowledge about machine learning algorithms from the process of teaching, we can accelerate innovation and empower millions of new uses for machine learning models.
1. Introduction
Machine teaching addresses the gap between the growing demand for specialized machine learning models and the limited supply of people able to build them. It shifts emphasis from improving learners to improving teachers’ productivity and interaction with data.
- Millions of specialized, high-accuracy machine learning models could serve varied combinations of domains, categories, and tasks.Examples include voice-command models and restaurant-reservation agents.
- Machine teaching evaluates teacher efficacy relative to human costs, including productivity, interpretability, robustness, and scalability.
- Concept evolution challenges traditional machine learning because target concepts, schemas, and labels may change as rare positives are discovered or teachers revise decisions.
- Decomposing concepts into sub-concepts makes semantic decisions easier to manipulate, interpret, and reverse when the target concept changes.The gardening example includes botanical gardens as a sub-concept before later relabeling decisions.
- Sub-concept labels can benefit teachers without benefiting the learning algorithm by enabling constant-time concept manipulation and documenting semantic decisions.The documentation supports communication and collaboration.
- Machine teaching follows software engineering and programming-language principles while emphasizing teacher–data interaction, interaction design, and visualization.It also defines interfaces between underlying algorithms and the teaching language.
2. The need for a new discipline
Current machine learning development is inefficient because it depends on specialized expertise, tightly coupled processes, and limited support for evolving concepts, collaboration, and reproducibility. Machine teaching proposes teacher-centered abstractions and tools that shield teachers from algorithmic and runtime complexity while preserving model-building control.
- The need for a new discipline: Current model-building processes involve repeated loops of outsourced labeling, guideline revision, algorithm selection, feature engineering, testing, and redeployment.
- The need for a new discipline: Building and maintaining machine learning systems can take weeks per iteration and fail when data, requirements, features, personnel, or problem definitions change.A deployed system may remain stable for months before requiring modification.
- The need for a new discipline: The central inefficiencies arise from processes, human–algorithm interaction, and people’s limitations rather than from machine learning algorithms alone.
- Definitions: Machine learning research improves learners through algorithms, whereas machine teaching research improves teachers’ productivity in building machine learning models.
- Definitions: Machine teaching focuses on version control, concept decomposition, semantic data exploration, teaching-language expressiveness, interpretability, and productivity.These concerns overlap more with programming and human–computer interaction than with conventional machine learning research.
- Decoupling machine teaching from machine learning: Machine teaching shields teachers from runtime variability and optimization complexity through interfaces between teaching information and available machine learning functions.This reduces required expertise and maintenance time, but generic optimization can incur a performance cost.
- Decoupling machine teaching from machine learning: Expressive features, increasing function-space complexity, and learning consistency let teachers improve approximation and estimation errors through feature and label additions.
- Decoupling machine teaching from machine learning: The stated goal is for teachers to create and debug concept functions to arbitrary accuracy without understanding runtime functions, learning algorithms, or optimization.
3. Analogy to programming
The paper frames machine teaching as a form of programming, sharing steps such as decomposition, testing, documentation, sharing, deployment, and maintenance while differing especially in debugging. It argues that programming principles can guide machine teaching toward greater scalability and productivity.
- Commonalities and differences between programming and teaching: Machine teaching is presented as a form of programming with analogous processes for creating, decomposing, testing, documenting, sharing, deploying, and maintaining functions.The paper explicitly lists these programming steps and compares them with machine teaching.
- Commonalities and differences between programming and teaching: Machine teaching differs from programming particularly in debugging, although the broader lifecycle includes scheduled and unscheduled maintenance cycles.The paper identifies debugging as a significant difference and lists maintenance among the shared functional steps.
- Commonalities and differences between programming and teaching: A machine teaching language would express teacher knowledge for machine-learning training through labels, schema constraints, labeled examples, and features.The proposed language is analogous to programming languages but communicates domain knowledge rather than executable machine instructions.
- Solving complex problems: Problem decomposition gives software engineers confidence in solving complex tasks, and the paper argues that analogous teaching decompositions can make machine teaching comparably manageable.The authors connect decomposition with expectations of success for both programming and teaching.
- Tools for scaling contributions: Standardized languages, componentization, interfaces, and version control are proposed as programming-inspired mechanisms for scaling machine teaching across contributors.These mechanisms support shared understanding, separation of concerns, independent development, collaboration, and change management.
4. The role of teachers
Machine teaching centers the teacher’s transfer of concept knowledge through examples, labels, schemas, features, and decomposition rather than requiring direct access to the learner’s program. The paper emphasizes modular, interpretable concept manipulation and domain experts’ role in expanding machine-learning model development.
- Teacher knowledge transfer: A teacher transfers concept knowledge to a learning machine through example selection, labeling, schema definition, featuring, and concept decomposition.The paper also notes that teachers may make mistakes across these forms of knowledge transfer.
- Core concepts: A concept maps examples to label values, while a feature assigns each example a scalar value and a schema defines relationships between concepts.These definitions establish the basic objects through which teachers communicate domain knowledge.
- Examples, labels, and features: Teachers select examples deliberately, provide labels by evaluating examples, and can use features estimated over test-set distributions rather than only training data.Teacher-directed selection can help discover information about rare clusters, while feature estimates independent of training data are described as immune to over-training.
- Concept decomposition: Concept decomposition expresses complex concepts through simpler ones, enabling reversible manipulation and producing high-level, modular, interpretable concept hierarchies.Each concept implementation can supply its own examples, labels, schema, and features as interface-like documentation.
- Who teaches machines: The machine teaching process is intended for domain experts rather than requiring machine-learning expertise, especially where unlabeled data is plentiful and domain knowledge is essential.The authors identify applications including spoken-dialog control, routing customer feedback, and one-time assistants.
- Meeting the demand for teachers: The paper aims to expand machine teaching to tens of millions of teachers, comparable to the number of software engineers, so domain experts can address millions of practical problems.The proposed scale includes personal, shared, one-off, and recurrent problems.
5. Teaching process
The teaching process applies programming-inspired principles—universal interfaces, feature completeness, modularity, robustness, and version control—to let teachers build models through semantic interaction with data. It emphasizes teacher productivity while acknowledging unresolved evaluation challenges and assumptions about available features and samples.
- Universal teaching language: Universal teaching language keeps the interface consistent across machine learning algorithms and aims to make teaching simple within each domain.Algorithms can be swapped without changing the model-building experience, although additional teaching may be required.
- Feature completeness: Feature completeness assumes target concepts are realizable through recursive composition of models and existing features.Teachers can add missing features, create model-based features through decomposition, or explicitly ignore ambiguous patterns.
- Distribution robustness: Distribution robustness seeks models that remain correct across deployment distributions when feature completeness and rich, diverse sampling are available.The teacher can explore and label examples without relying on deployment-distribution knowledge.
- Modular development: Modular development decomposes concepts into sub-concepts and reuses models as features through standardized model and feature interfaces.Projects trained together can affect one another, while dependencies across solutions are treated as versioned packages.
- Version control: Version control saves teacher actions—including labels, features, constraints, schemas, dependencies, and code—as versioned concept programs.Actions can be separated to facilitate merging contributions from multiple teachers.
- Teaching process: The combined principles allow a single domain-trained teacher to own model building while machine learning experts and engineers remain outside the concept-specific teaching loop.The process assumes the teaching tool supplies the capabilities needed to build models effectively.
- Teaching process: Evaluating quality in a distribution-robust setting and finding test error effectively remain difficult and outside the paper’s scope.The paper suggests using teacher cost or time as a simple stopping criterion and semantic queries over a large sample set.
6. Conclusion
The conclusion positions machine teaching as a programming-inspired response to limited access to people who can build models for many changing problems. It proposes decomposition, standardized knowledge channels, and process discipline as foundations for scaling teaching, while noting challenges of monolithic learning when labels are scarce.
- Conclusion: Machine teaching addresses growing demand for solutions to simpler, ephemeral problems while access to capable machine teachers remains limited.The paper frames this mismatch as a productivity and scaling problem for model building.
- Conclusion: Programming lessons motivate machine teaching through problem decomposition, standardized languages, separation of concerns, standard tools, libraries, and version control.The authors argue these principles can improve productivity and scaling with problem complexity and contributor count.
- Conclusion: Large monolithic learning systems can be limited when labeled data is scarce, particularly for rare domain-specific cases and difficult corner cases.The paper presents active teaching with labels, features, and structure as a way to inject semantic knowledge.