Source-linked AI summary
Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers
Giuseppe Ateniese, Giovanni Felici, Luigi V. Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali
TL;DR
The paper asks whether releasing a trained classifier can reveal meaningful information about its training set and compromise trade secrets. It develops a meta-classifier attack and demonstrates that classifiers can expose statistical properties of training data, including user accents and traffic patterns.
Problem
Releasing a trained classifier may reveal concrete hints about its training set, potentially uncovering the secrets behind its effectiveness and jeopardizing the vendor.
Method
The paper trains a meta-classifier to detect changes in a disclosed classifier's internal structure and infer statistical information about its training set.
Results
The meta-classifier distinguished user accents in a speech-recognition corpus and inferred whether a specific traffic pattern was used to train an Internet-traffic classifier.
Takeaways & Limitations
Classifiers can leak meaningful training-set statistics beyond individual-record privacy, making training sets valuable trade secrets that require protection.
Takeaways & Limitations
The paper calls for novel machine-learning techniques to prevent determined competitors from probing classifiers and learning trade secrets.
Abstract
from arXiv · showhide
Machine Learning (ML) algorithms are used to train computers to perform a variety of complex tasks and improve with experience. Computers learn how to recognize patterns, make unintended decisions, or react to a dynamic environment. Certain trained machines may be more effective than others because they are based on more suitable ML algorithms or because they were trained through superior training sets. Although ML algorithms are known and publicly released, training sets may not be reasonably ascertainable and, indeed, may be guarded as trade secrets. While much research has been performed about the privacy of the elements of training sets, in this paper we focus our attention on ML classifiers and on the statistical information that can be unconsciously or maliciously revealed from them. We show that it is possible to infer unexpected but useful information from ML classifiers. In particular, we build a novel meta-classifier and train it to hack other classifiers, obtaining meaningful information about their training sets. This kind of information leakage can be exploited, for example, by a vendor to build more effective classifiers or to simply acquire trade secrets from a competitor's apparatus, potentially violating its intellectual property rights.
1 Introduction
The paper asks whether releasing trained classifiers can leak meaningful information about their training sets, even when the underlying ML algorithms are known. It introduces a meta-classifier attack and argues that such leakage can expose statistical properties and trade secrets.
- Trained classifiers may reveal information about their training sets, creating risks even when their algorithms and design principles are publicly known.The paper frames this as a potential way to produce competitive products without copying protected code or design principles.
- A meta-classifier can detect changes absorbed into a target classifier during training and extract meaningful information about its training data.The paper presents this leakage as inherent to learning, while noting that the meaningfulness of the leaked information is not obvious from classifier opacity alone.
- The authors distinguish their target from privacy leakage: they seek statistical properties that help explain why one classifier outperforms another, not individual records.Their example infers whether speech-recognition training data overrepresented female voices or speakers with marked accents.
- The proposed strategy attacks classifiers across different ML settings, including Internet-traffic classifiers using SVMs and speech-recognition software using HMMs.The paper organizes its evaluation around attacks on trained SVM and HMM classifiers.
- The authors argue that released classifiers should not be assumed safe and that training sets may need protection as trade secrets.They call for machine-learning techniques that prevent competitors from probing classifiers to learn protected training information.
2 Hacking Machine Learning classifiers
The paper presents an attack in which a meta-classifier examines another classifier’s representation to infer statistical properties preserved by its undisclosed training set. The strategy applies across classifier types and can reveal external information, such as speaker accent, rather than explicit dataset attributes.
- Attack strategy: The strategy assumes the classifier is disclosed after training, preventing training-time tainting but allowing arbitrary modification of classification-time behavior.The framework covers ANNs, HMMs, and decision trees; filters such as Kullback-Leibler divergence may select less significant model entries.
- Attack strategy: The attack targets a disclosed classifier Cx to infer whether its hidden training set Dx preserves a property P.The property may concern relationships among entries, such as whether medical-training data are balanced between males and females.
- Attack strategy: The adversary constructs datasets with and without P, trains classifiers on them, and uses their feature-vector representations to train meta-classifier MC.For SVMs, the representation can consist of the classifier’s support vectors.
- Attack strategy: MC classifies the target representation FCx, revealing whether Dx preserves P and enabling inference of statistical properties through repeated property testing.The method is described as a brute-force attack over candidate properties.
- Information leakage: The method extracts external information from relationships among dataset entries rather than attributes explicitly represented in Dx.The paper’s speech-recognition example infers speakers’ accent, although accent is not an explicit model or training-set attribute.
3 Case studies
The case studies apply a general meta-classifier attack to speech-recognition HMMs and network-traffic SVMs, inferring properties of undisclosed training sets from trained classifiers. In the speech experiment, filtering improved Indian-dialect classification while shrinking the decision tree; the traffic experiment also produced effective classification.
- Case-study setup: The experiments attack two systems: a speech-recognition engine based on Hidden Markov Models and a network-traffic classifier based on Support Vector Machines.Decision trees using Weka's J48 implementation serve as the meta-classifier, and performance is evaluated with precision, recall, and accuracy.
- Speech-recognition HMM attack: The speech-recognition attack encodes each acoustic-model phoneme using output probability vectors and labels indicating Indian or not Indian training data.The balanced training set contains 5,420 tuples, while the test set contains 1,016 instances.
- Speech-recognition HMM attack: The initial speech meta-classifier correctly classified not Indian models with precision 0.97 and Indian models with precision 0.75.The corresponding recalls were 0.907 for not Indian and 0.909 for Indian.
- Speech-recognition HMM attack: Filtering raised Indian precision to 0.95 while retaining not Indian precision at 0.98, with recalls of 0.986 and 0.966, respectively.The filtered decision tree contained 21 nodes and 11 leaves, compared with more than 811 nodes and 610 leaves initially.
- Network-traffic SVM attack: The traffic-classification attack achieved Google precision 0.954 and recall 0.932, versus not Google precision 0.943 and recall 0.962.These results show that the meta-classifier inferred whether the target training set included a specific traffic type.
4 Differential privacy
The paper argues that differential privacy does not prevent its attack, because the attack targets statistical information about the transformed training data and learned classifier behavior rather than individual records.
- Differential privacy is ineffective against the attack because the targeted information lies outside its adversary model.The attack seeks information correlated with training samples, not disclosure of a single database record.
- Noise added to the training database is ineffective because the transformed database is used for training and remains the attack target.
- Noise added to classifier outputs is ineffective because an adversary with complete classifier access could disable the noise instruction.
- Noise added during training is ineffective because the classifier must still converge toward correctly classifying the training set, restraining the noise's effect.
- 70 training sets produced 70 distinct K-Means centroids, whose positions differed between traces with and without Google.com traffic.
- Even with differential privacy, an adversary could easily distinguish whether Google.com traffic was present from the classifier's centroids.
5 Related works
Related work addresses privacy-preserving analysis and protection of individual database records, whereas this paper identifies a previously unconsidered leakage from the learning process itself.
- The paper distinguishes its leakage from prior privacy-preserving work because it concerns statistical information correlated with training samples.
- Differential privacy protects against unintentional disclosure related to a single database record while preserving accurate statistical queries.
- Privacy Preserving Data Mining develops data-mining techniques that protect the privacy of individual data records.
6 Conclusions
The paper concludes that a meta-classifier can extract meaningful information from trained classifiers, including properties of speech and network-traffic training data.
- The proposed meta-classifier extracts meaningful data from machine learning classifiers by targeting statistical information correlated with their training samples.
- The meta-classifier distinguished user accents represented in a speech-recognition corpus and inferred whether a specific traffic pattern appeared during classifier training.
- The results motivate protecting training sets because they contain the recipe that can make a classifier better than its competition.
A Artificial Neural Networks
Artificial neural networks learn functions through interconnected neuron layers, with hidden-unit weights adjusted by backpropagation to reduce error on training examples.
- Artificial neural networks learn functions from real, discrete, and vector-valued examples and are used for decision making, optimization, prediction, and control.
- A perceptron computes a linear combination of real-valued inputs and outputs 1 or -1 according to a threshold.
- A single perceptron represents a hyperplane and can discriminate only between linearly separable instances.
- An ANN connects input, hidden, and output layers, with internal hidden units providing the network's core function through their weights.
- Backpropagation iteratively adjusts hidden-unit weights using training inputs, expected outputs, and gradient-descent error correction.
B Classification and Regression Trees
Classification and regression trees are prediction models that map observations through a decision tree. They learn from training observations to predict a response variable from predictor variables, with entropy used to characterize dataset impurity.
- Classification and regression trees are prediction models that map observations through a decision tree.
- Training observations L = (x1, y1), (x2, y2), . . . , (xN, yN) are used to learn the decision tree.
- The trees predict a response variable y from the values of predictor variables x.
- Regression applies when y takes real values, while classification concerns the alternative case described for the response variable.
- H(S) represents the entropy of the entire dataset, while S_v is the subset whose attribute A has value v.