Adversarial Removal of Demographic Attributes from Text Data

Yanai Elazar; Yoav Goldberg

Source-linked AI summary

Adversarial Removal of Demographic Attributes from Text Data

Yanai Elazar, Yoav Goldberg

arXiv:1808.06640v2Aug 20, 2018cs.CLcs.LGstat.ML

TL;DR

The paper asks whether text-based neural representations can avoid encoding demographic attributes, and tests adversarial training as a removal strategy. Across the supplied evidence, demographic information remains recoverable in some settings even when the adversary reaches apparently successful performance, so external verification is needed.

Problem
Textual information can predict demographic attributes, so excluding protected attributes directly does not ensure that automated decisions avoid demographic signals.
Method
The paper trains text classifiers with encoder representations and adversarial components, then uses externally trained attackers to test whether protected attributes remain encoded.
Results
Across the experiments, adversarial training mitigates protected-attribute leakage but may leave it recoverable; in one case, an attacker reached 56.0% accuracy when the adversary reached 49.0%.
Takeaways & Limitations
Adversary scores during training cannot be trusted alone; protected-attribute removal should be externally verified, preferably on unseen data.
Takeaways & Limitations
None of the explored methods fully succeeds in removing all demographic information from text representations.

Abstract

from arXiv · show

Recent advances in Representation Learning and Adversarial Training seem to succeed in removing unwanted features from the learned representation. We show that demographic information of authors is encoded in -- and can be recovered from -- the intermediate representations learned by text-based neural classifiers. The implication is that decisions of classifiers trained on textual data are not agnostic to -- and likely condition on -- demographic attributes. When attempting to remove such demographic information using adversarial training, we find that while the adversarial component achieves chance-level development-set accuracy during training, a post-hoc classifier, trained on the encoded sentences from the first part, still manages to reach substantially higher classification accuracies on the same data. This behavior is consistent across several tasks, demographic properties and datasets. We explore several techniques to improve the effectiveness of the adversarial component. Our main conclusion is a cautionary one: do not rely on the adversarial training to achieve invariant representation to sensitive features.

1 Introduction

Text can encode demographic signals even when those attributes are excluded from a model’s features, creating a risk of indirect conditioning. The paper evaluates adversarial removal and finds that apparently successful adversarial training may leave demographic information recoverable.

Motivation: Protected attributes such as gender and race can influence automated decisions indirectly through correlated features, even when omitted from the input.The paper frames this risk for credit, insurance, and hiring systems.
Motivation: Textual information is predictive of authors’ age, race, gender, and other social factors, making raw-text decision systems vulnerable to demographic conditioning.The cited literature also reports predictability of audience attributes.
Findings: Experiments show race, gender, and age information encoded in neural-network intermediate representations, including for seemingly unrelated tasks and balanced training data.The paper treats this encoding as demographic leakage into learned representations.
Findings: Adversarial training reduces protected-attribute information, but in some cases substantial information remains extractable despite apparently successful adversarial performance.The authors therefore explore methods intended to improve adversarial removal.
Conclusion: None of the explored methods fully removes all demographic information, motivating a caution against trusting adversarial removal to produce invariant text representations.The paper recommends checking whether protected attributes are actually removed.

2 Learning Setup

The paper separates main-task prediction from protected-attribute prediction using an encoder, classifier, adversary, and post-hoc attacker. Adversarial training reverses the adversary’s encoder gradients so representations retain task information while minimizing protected-attribute information.

Model: The classifier f maps documents to task labels through an encoder h and classifier c, while each document also has a protected attribute z that decisions should ignore.The representation is denoted h_x.
Leakage Definition: A protected attribute is leaked when a classifier predicts it from the representation above chance; it is guarded when such prediction cannot be trained.The same terminology is applied to the full classifier.
Adversarial Training: Adversarial training jointly trains an adversarial classifier to predict z and the encoder to make that prediction fail.The objective balances main-task informativeness against protected-attribute informativeness.
Adversarial Training: The gradient-reversal layer acts as the identity during the forward pass and scales backpropagated adversarial gradients by −λ.The meta-parameter λ controls reversal intensity, sending the encoder gradients opposite to the adversary’s.
Attacker Evaluation: A post-hoc attacker tests leakage by predicting z from fixed encoder representations without access to the encoder or original inputs.Prediction above chance on unseen examples indicates that z remains leaked.

3 Data, Tasks, and Protected Attributes

The experiments use Twitter data with binary sentiment and mention-prediction tasks plus binary race, gender, and age attributes. The datasets are balanced for main and protected labels, while race annotations are especially correlated with language.

Data: Because suitable public datasets for sensitive credit, insurance, or hiring tasks were unavailable, the study uses Twitter tasks with labeled main and protected attributes.The selected main tasks are less sensitive than the motivating applications.
Main Tasks: The main tasks are binary emoji-based sentiment prediction and binary tweet-mention prediction, chosen as tasks not inherently correlated with race, gender, or age.Leakage in these benign tasks is used as an indicator relevant to more sensitive settings.
Main Tasks: Sentiment labels come from positive or negative emojis that are removed, while mention prediction distinguishes conversational tweets containing @mentions from non-conversational tweets.Tweets containing emojis from both sentiment lists are discarded.
Protected Attributes: Race uses AAE and SAE categories as proxies for non-Hispanic Black and non-Hispanic White groups, with annotations retained only above 80% confidence.The race corpus contains predicted author-race information based on geolocation and tweet words.
Protected Attributes: Age and gender labels come from PAN16, which manually annotates 436 Twitter users and provides up to 1k tweets per user.Gender uses binary Male/Female categories in this study.
Data Splits: Training and development sets contain 166K and 10K DIAL tweets or 160K and 10K PAN16 tweets, with both main and protected labels balanced.Random prediction of either binary variable therefore yields 50% accuracy.
Metrics: Leakage is measured by protected-attribute accuracy above the 50% chance baseline, and the paper relates guarded prediction to demographic-parity, equality-of-odds, and equality-of-opportunity guarantees in its setup.The fairness relationships are established in Appendix A.

4 Baselines and Data Leakage

Direct training shows that protected attributes can be recovered accurately, while main-task representations also retain substantial demographic information. Leakage is demonstrated in balanced and unbalanced synthetic settings and in a real-world DeepMoji encoder.

Unbalanced-data leakage: In unbalanced data, both main-task accuracy and protected-attribute leakage are stronger than in the balanced case.The unbalanced setup keeps each class globally balanced while changing the protected-attribute proportions within main-task splits.
Direct prediction: 83.9% accuracy recovers race directly, while age and gender are recovered above 64%.These results establish strong predictability of the protected attributes before examining leakage from main-task representations.
Balanced-data leakage: 64.5% DIAL Race leakage is recovered from representations trained on the balanced Sentiment task.This is lower than the 83.9% direct-prediction accuracy but remains sufficient to extract meaningful sensitive information.
Real-world leakage: 84.7%, 67.2%, and 67.1% development-set accuracy recover race, gender, and age from the DeepMoji encoder, respectively.DeepMoji was trained for emoji prediction on 1.2 billion tweets, yet its representations also support demographic-attribute attackers.
Direct prediction: Emoji usage is highly correlated with the demographic properties examined in the baseline experiments.

5 Mitigating Data Leakage

Adversarial training reduces demographic leakage in text representations, but chance-level adversary accuracy can conceal substantial recoverable information. Capacity, weighting, and ensembles sometimes improve mitigation, yet no configuration fully prevents leakage across setups.

5.1 Adversarial Training: Adversarial training actively removes protected-attribute information from encoded representations, although this objective can also hurt main-task information.The adversarial and main-task objectives are in tension during training.
5.1 Adversarial Training: 50% adversary accuracy persisted throughout training across tested task–attribute pairs, apparently indicating successful removal.Main-task accuracy also dropped after the adversary reached chance level.
5.1 Adversarial Training: 56.0% attacker accuracy exceeded the adversary’s 49.0% after 50 epochs, despite identical architectures and datasets.The post-hoc attacker recovered information that the adversary appeared unable to detect.
5.1 Adversarial Training: 59.7% attacker accuracy on 166K unseen samples exceeded 56.0% on the original development set.This verifies that recovery persists beyond representations used during adversarial training.
5.2 Strengthening the Adversarial Component: Increasing adversarial capacity or λ can reduce leakage, but the best setting differs by dataset and λ values that are too large destabilize training.Higher-dimensional adversaries helped on PAN16 but not Sentiment/Race; large λ required more epochs for main-task stabilization.
5.2 Strengthening the Adversarial Component: 57.4% recovery with two adversaries and 54.8% with five show ensemble effects vary, while ten adversaries prevented main-task training.Larger ensembles can reduce recovery but also reduce training stability.
5.2 Strengthening the Adversarial Component: No method or parameter setting performed equally well across setups, and none completely prevented protected-attribute leakage.Combining methods also failed to improve the results.

6 Analysis

Further analysis examines why adversarially trained encoders still leak demographic information. Leakage is attributed mainly to the RNN rather than embeddings, while attacker recovery persists under held-out evaluation and consistent example inspection.

6 Analysis: The encoder comprises an embedding matrix and an RNN, allowing leakage to arise from either component or their combination.The analysis fuses components from leaky and guarded encoders to compare their contributions.
6 Analysis: The RNN contributes more to protected-attribute leakage than the embedding matrix.Fused encoders compare a leaky embedding with a strong RNN against a strong embedding with a leaky RNN.
6 Analysis: 65.4% sentiment accuracy for the leaky-RNN combination and 60.9% for the leaky-embedding combination support the module comparison.These combinations remain below the corresponding leaky and guarded model accuracies of 67.5% and 63.8%.
6 Analysis: The study inspects tweets whose race is correctly predicted by at least 9 of 10 independently trained attackers.Repeated runs are used to identify consistently recoverable examples below 60% accuracy.
6 Analysis: Attacker recovery on training data with a 10% holdout does not support data overfitting as the explanation for leakage.The authors test whether encoder–adversary memorization cleans training data without generalization.

7 Related Work

The paper places demographic-attribute removal in broader work on representation learning, adversarial training, and fairness. It distinguishes its text-focused evaluation by directly testing whether protected attributes remain recoverable.

7 Related Work: Generic word and sentence representations show that intermediate vectors trained for one task can predict another task or property.The paper notes that this transferability is usually beneficial but can have undesired consequences.
7 Related Work: Prior work removed sensitive attributes through optimization or adversarial training, often focusing on structured features or image anonymization.Other textual adversarial-training studies primarily targeted domain, language, transfer, or end-task performance.
7 Related Work: This paper instead studies protected features based on short user-authored text.The experiments focus on text representations rather than structured attributes.
7 Related Work: Compared with private text representations that report only adversary discrimination accuracy, this work adds external attacker verification.The paper argues that adversary performance alone is insufficient evidence that representations are clear of protected attributes.
7 Related Work: The fairness setup uses blindness, defined as representations that do not condition on a protected property; under this setup, blindness guarantees demographic parity, equality of odds, and equality of opportunity.The paper acknowledges that other fairness definitions exist.

8 Conclusions

The paper concludes that demographic information leaks into text-classifier representations and that adversarial training mitigates but does not reliably eliminate this leakage. External attacker verification and configuration tuning are therefore needed, with no universally best method.

8 Conclusions: Demographic information leaks into intermediate representations of neural networks trained on text data.The conclusion states that systems must take active steps against accidental conditioning.
8 Conclusions: Adversarial training mitigates protected-attribute leakage but may fail to remove it completely in text settings.The conclusion separates partial effectiveness from complete invariance.
8 Conclusions: Adversary scores during training cannot be trusted alone and should be checked with an externally trained attacker, preferably on unseen data.This recommendation follows the paper’s distinction between adversary performance and attacker recovery.
8 Conclusions: Tuning adversary capacity and weight, or using ensembles, can improve results, but no single method is most effective in every case.The conclusion presents configuration choice as setup-dependent.

A Standard Fairness Definitions and Guarded Classifiers

The paper defines guarded classifiers as oblivious to protected attributes and shows, under its setup, that this implies demographic parity, equality of odds, and equality of opportunity. The proofs rely on independence between the internal representation and the protected attribute, while noting that achieving zero leakage is difficult.

Demographic Parity: A classifier guarded to z is shown to realize demographic parity under the paper’s setup.Guarding means the internal representation is oblivious to z; with P(Z)=0.5, this yields independence between Z and H.
Practical Boundary: Achieving zero leakage is far from trivial, despite the theoretical fairness guarantees associated with guarded classifiers.The paper explicitly cautions that the required oblivious representation is difficult to attain.
Equality of Odds: Obliviousness to z implies equality of odds, expressed as equal prediction probabilities across z conditional on each main-task label.The proof derives, for y=1, P(f = ˆy|z = 0, y = 1) = P(f = ˆy|z = 1, y = 1), and states the analogous result for y=0.
Equality of Opportunity: Obliviousness to z also implies equality of opportunity because equality of opportunity is a less strict relaxation of equality of odds.The paper treats this implication as automatic from the equality-of-odds result.
Overall Connection: Under the stated setup, an oblivious classifier satisfies demographic parity, equality of odds, and equality of opportunity.These three fairness definitions are linked to the paper’s guarded-classifier definition under its assumptions.

B Implementation Details

The preprocessing removes duplicate and very short tweets after Twitter-specific tokenization. The default neural architecture uses single-layer 300-unit LSTM and MLP components with 300-dimensional embeddings and specified optimization regularization.

Preprocessing: Tweets are tokenized with twokenize, then duplicate tweets and tweets containing fewer than three tokens are discarded.The tokenizer is Twitter-specific.
Neural Network Architecture: The default LSTM encoder and MLP hidden layer each have 300 hidden units in a single layer, with randomly initialized 300-dimensional embeddings.These are the default architecture settings unless otherwise noted.
Optimization: Training uses SGD with Momentum at learning rate 0.01 for 100 epochs, dropout 0.2, negative log likelihood, and mini-batches of 32.Dropout is applied to all hidden layers.

C Emojis Details

The sentiment labels use positive and negative emojis, supplemented by positive and negative emoticons. Figure 3 presents the emoji proxies used for this classification setup.

Emoji Proxies: Positive and negative tweets are defined using corresponding emoji proxies shown in Figure 3.The figure is specifically captioned as showing emojis used as positive and negative sentiment proxies.
Emoticon Proxies: The study also includes text emoticons such as :) and :D for positive sentiment and :( and :-( for negative sentiment.The listed emoticons supplement the emoji-based definitions.