Source-linked AI summary
Overlearning Reveals Sensitive Attributes
Congzheng Song, Vitaly Shmatikov
TL;DR
The paper asks whether models trained for simple objectives implicitly learn unrelated sensitive attributes, threatening privacy and bias protections. It demonstrates overlearning through representation inference and model repurposing, finding that sensitive information can persist despite censoring and that some overlearning may be intrinsic to the task.
Problem
Models may learn privacy- and bias-sensitive attributes that are outside their stated objectives, challenging protections based on model partitioning, censoring, or enumerating learned purposes.
Method
The paper studies overlearning across vision and NLP models using representation-based inference, transfer-based model repurposing, censoring experiments, and analysis of feature emergence during training.
Results
Models trained on simple tasks learn unrelated sensitive attributes; their representations reveal those attributes, models can be repurposed for privacy-violating tasks, and censoring does not reliably suppress the information.
Takeaways & Limitations
Privacy protections should account for models recognizing and using sensitive attributes beyond their declared objectives, including after the original training data is erased.
Takeaways & Limitations
For some objectives, there may be no feature set that enables accurate task prediction without also recognizing sensitive attributes such as race or identity.
Abstract
from arXiv · showhide
"Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races\textemdash even races that are not represented in the training data\textemdash and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.
1 INTRODUCTION
Overlearning occurs when models trained for simple objectives learn unrelated privacy- and bias-sensitive attributes. The paper examines its privacy consequences, the limits of censoring, and how general features emerge during training.
- Overlearning makes representations reveal sensitive attributes absent from the specified learning objective.These attributes are neither finer- nor coarser-grained versions of the labels nor statistically correlated with them.
- Inference-time representations can expose sensitive properties, while transfer learning can repurpose benign models for privacy-violating tasks.These consequences undermine model partitioning and privacy rules based on enumerating learned attributes.
- Censoring may fail because some tasks intrinsically require learning sensitive attributes or continue leaking them after censorship.The paper also shows that representations can recognize sensitive attributes absent from training data, which cannot be censored by known techniques.
- The paper analyzes overlearning by studying general features in lower model layers and relating their emergence to training-data complexity.
2 BACKGROUND
The paper frames supervised models as feature extractors and classifiers whose representations can be partitioned, censored, or attacked. It reviews adversarial and information-theoretic objectives for retaining task information while reducing sensitive information.
- Model representations: A supervised model is decomposed into an encoder E that produces representations and a classifier C that predicts the task label.Intermediate representations are denoted z_l, while the final representation is z.
- Model partitioning: Model partitioning sends a locally computed representation to a cloud classifier to improve scalability and protect input privacy.The local representation may be censored before transmission.
- Censoring: Censoring seeks representations that hide unwanted properties while preserving enough information to predict the task label.It has been used for invariant, fair, and privacy-preserving representations.
- Adversarial censoring: Adversarial censoring trains a discriminator to infer sensitive attribute s while the encoder and classifier preserve task-label prediction.The balancing parameter γ controls the competing log-likelihood terms.
- Information-theoretic censoring: Information-theoretic censoring relaxes independence between representation z and sensitive attribute s into a mutual-information constraint.Its objective balances information about the label, input, and sensitive attribute.
- Censoring limitations: Existing censoring methods require a blacklist of attributes and training inputs containing those attributes.This requirement limits protection against sensitive properties that are unknown or absent from the training data.
3 EXPLOITING OVERLEARNING
The paper exploits overlearned representations through inference attacks, de-censoring, and transfer-based model repurposing. These methods use auxiliary data or intermediate features to recover or predict sensitive attributes, complicating privacy regulation.
- Attack scope: The inference and repurposing procedures expose sensitive attributes even when the original model was trained for a benign task.The paper presents both representation-based inference and transfer-based direct prediction as distinct attacks.
- Inference attacks: Inference attacks train an auxiliary model to predict a sensitive property from representations observed at inference time.The attacker needs labeled auxiliary pairs and a black-box encoder, but not direct access to the target input.
- De-censoring: De-censoring learns a transform that maps censored representations toward uncensored auxiliary features before sensitive-attribute inference.The transform is trained with an L2 feature-space loss, after which an attack model predicts the property.
- Model repurposing: A model can be repurposed by attaching a new classifier to features from any layer and fine-tuning on a small transfer dataset.The transferred model can outperform training from scratch because it reuses features learned on the original dataset.
- Regulatory implications: Model repurposing complicates GDPR because a trained model may support an objective not envisioned when the original data was collected.This remains possible even after the original training data has been erased.
4 EXPERIMENTAL RESULTS
Across vision and language tasks, uncensored representations reveal sensitive attributes, while censoring often reduces task performance without reliably removing leakage. Re-purposing and layer analyses show that sensitive information persists across representations and emerges from general features learned during training.
- Inferring sensitive attributes: Uncensored last-layer representations support sensitive-attribute inference above random guessing across all evaluated tasks.The experiments span Health, UTKFace, FaceScrub, Places365, Twitter, Yelp, and PIPA.
- Inferring sensitive attributes: Censoring reduces both main-task and inference accuracy, while information-theoretical censoring damages main-task accuracy more than adversarial training on almost all models.Adversarial training and information-theoretical censoring are compared against uncensored representations in Table 2.
- Inferring sensitive attributes: Stronger censoring can damage the main task without reliably reducing sensitive-attribute inference, and increasing β can prevent convergence on the main task.Increasing γ sometimes improves inference accuracy, while increasing λ can prevent convergence without affecting inference on several tasks.
- Inferring sensitive attributes: De-censoring significantly boosts inference from adversarially censored representations, with smaller gains against information-theoretical censoring.The smaller improvement is attributed to information-theoretical censoring also forgetting information about the input x; Health shows little difference because baseline inference is already similar to inference from censored representations.
- Re-purposing models: Censoring lower layers blocks adversarial re-purposing but reduces original-task accuracy, while similarity in uncensored layers leaves alternative routes for sensitive-attribute prediction.Per-layer CKA shows strong similarity in higher layers after lower-layer censoring and in lower layers after higher-layer censoring.
- When, where, and why overlearning happens: Lower layers of models trained for different tasks learn similar features early, and FaceScrub can exhibit similarity even between high-level features.More complex training data also produces more complex representations for the same gender-classification objective.
5 RELATED WORK
Prior work examines transferability, distributed-learning leakage, disentangled representations, and censoring, whereas this paper studies unintended transfer to unrelated sensitive attributes.
- Transferability: Prior transfer-learning studies generally focus on closely related source and target tasks, with transferability declining as task distance grows.The paper contrasts this literature with overlearning across uncorrelated tasks.
- Distributed learning: Distributed-learning studies show that gradient updates can leak information about training batches unrelated to the learning objective.The paper presents overlearning in fully trained models as a generic problem that helps explain these observations.
- Disentangled representations: Disentangled-representation research deliberately separates explanatory factors, whereas overlearning arises implicitly during supervised training without purposeful disentanglement.Overlearning enables another task automatically from representations trained for one task.
- Censoring: Representation-censoring methods aim to suppress sensitive demographics or identities while preserving task-relevant information across domains including health records, text, images, and wearable sensors.The paper uses censoring to investigate whether sensitive attributes can be prevented from being learned.
6 CONCLUSIONS
The paper concludes that simple supervised objectives can implicitly produce representations recognizing sensitive attributes, and that censoring may not prevent this overlearning. This creates technical and regulatory challenges for controlling model uses.
- Findings: Models trained for seemingly simple tasks implicitly learn sensitive concepts absent from their objectives, including attributes statistically orthogonal to the target.The paper identifies race and identity as examples.
- Intrinsic overlearning: Censoring failures and similarity across uncorrelated tasks suggest that overlearning may be intrinsic for some objectives.Generic low-level features may support both the specified task and sensitive-attribute inference.
- Intrinsic overlearning: There may be no feature set that accurately determines facial gender without also enabling recognition of race or identity.This example states the paper’s proposed boundary on preventing overlearning.
- Regulatory implications: If overlearning is intrinsic, models may recognize sensitive attributes that privacy and fairness regulations cannot fully enumerate or control.The paper therefore discusses regulating how models are applied while acknowledging that they may still recognize and use sensitive attributes.