Source-linked AI summary
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg
TL;DR
Neural representations are opaque and may encode protected or otherwise unwanted information. The paper introduces INLP, which repeatedly trains linear classifiers and projects representations onto their nullspaces. It reports bias attenuation and increased fairness across word-embedding and classification settings, while limiting its protection claim to linear information and representative data.
Problem
Neural representations are opaque and may encode protected attributes or other information that applications do not want to retain.
Method
INLP repeatedly trains linear classifiers predicting a target property and projects representations onto their nullspaces to remove the learned linear directions.
Results
Across increasingly complex bias and fairness settings, INLP attenuates societal biases expressed in representations, including word embeddings and classification systems.
Takeaways & Limitations
INLP offers a data-driven method for removing linearly represented information from neural representations in bias and fairness applications.
Takeaways & Limitations
Its fairness use depends on sufficiently large, representative i.i.d. training data and protects specifically against linear information about protected attributes.
Abstract
from arXiv · showhide
The ability to control for the kinds of information encoded in neural representation has a variety of use cases, especially in light of the challenge of interpreting these models. We present Iterative Null-space Projection (INLP), a novel method for removing information from neural representations. Our method is based on repeated training of linear classifiers that predict a certain property we aim to remove, followed by projection of the representations on their null-space. By doing so, the classifiers become oblivious to that target property, making it hard to linearly separate the data according to it. While applicable for multiple uses, we evaluate our method on bias and fairness use-cases, and show that our method is able to mitigate bias in word embeddings, as well as to increase fairness in a setting of multi-class classification.
1 Introduction
Neural representations are effective but opaque, motivating methods that selectively remove unwanted information. The paper introduces INLP and evaluates it primarily for mitigating bias and improving fairness, while emphasizing the need for scrutiny in these applications.
- Neural representations support effective text classification but encode information whose contents are difficult to interpret.
- Selective information removal is motivated by applications including removing tense, part-of-speech, topic, gender stereotypes, or sensitive demographic signals.
- INLP learns data-driven directions associated with a target property and removes them through linear projection, combining projection-based and adversarial-method advantages.
- INLP reduces many, but not all, limitations of existing methods for removing gender signals from word embeddings.
- The method is also integrated into pre-trained classifiers and demonstrated in controlled and real-world fair-classification settings.
- The main evaluation concerns bias and fairness applications, which the paper says require stricter scrutiny because real-world consequences can be severe.
2 Related Work
Prior approaches remove information through adversarial training or projection onto predefined directions, but each has important limitations. INLP instead learns multiple relevant directions and projects away their associated subspace.
- Adversarial methods can perform well but are difficult to train and may leave protected information recoverable by other classifiers.
- Nullspace cleaning removes input information outside the main-task classifier’s needs, whereas INLP projects onto nullspaces of protected-attribute classifiers.
- Projection-based debiasing commonly identifies a protected-attribute direction, such as a gender subspace formed from gendered word-pair differences.
- Experiments indicate that the gender subspace spans dozens to hundreds of orthogonal directions rather than only one interpretable direction.
3 Objective and Definitions
The paper defines guarding as preventing prediction of a discrete protected attribute beyond majority-class guessing while retaining useful representation information. Its primary target is guardedness against linear classifiers.
- The objective is to learn a transformation that prevents protected attributes from being predicted from transformed vectors while keeping the vectors informative.
- A dataset is guarded with respect to a hypothesis class when no classifier in that class predicts the attribute better than majority-class guessing.
- A guarding function is a transformation whose outputs are guarded for the protected attribute under the chosen hypothesis class.
- The term linearly guarded denotes guarding with respect to the class of all linear classifiers.
4 Iterative Nullspace Projection
INLP repeatedly trains classifiers for a protected attribute and projects representations onto their nullspaces. The resulting projection removes successive linear directions while preserving as much geometric structure as the orthogonal projection allows.
- Iterative Nullspace Projection: INLP trains a classifier W, projects representations so W(Px)=0, then repeats with additional classifiers until no more linear information about the attribute remains.
- Nullspace Projection: Projecting onto W’s nullspace removes the components used by W for prediction, making its decision boundary ineffective on the projected representations.
- Objective and Definitions: The method targets discrete attributes, while the paper notes that extending nullspace projection to linear regression is straightforward.
- Nullspace Projection: Orthogonal projection is the least-harming maximum-rank projection onto a classifier’s nullspace because it minimizes distances to that subspace.
- Iterative Projection: The initial single-classifier projection may be insufficient because multiple linear directions can encode partial information about the protected attribute.
- Iterative Nullspace Projection: INLP’s final matrix projects onto the intersection of the nullspaces of all learned protected-attribute classifiers.
- Iterative Projection: After n binary-attribute iterations, projected rank is at least r−n when the original input rank is r.
- Implementation Details: In practice, INLP avoids accumulated numerical errors by collecting row-space projection matrices and recomputing the intersection-based projection.
5 Application to Fair Classification
The paper applies INLP to fair classification by guarding protected attributes in the representations consumed by a classifier’s final layer. It integrates this guarding step with a pretrained network and refines training to improve protection within main-task classes.
- A classifier f(x) is decomposed into an encoder enc followed by a linear layer W, with f(x) = W · enc(x).
- If the protected attribute Z is linearly guarded in the inputs to W, the final layer cannot use Z when predicting the main-task label Y.
- The procedure trains f = W · enc(X) for Y, applies INLP to enc(X) to guard Z, then fine-tunes only W on the guarded representations.
- The resulting classifier uses representations guarded for Z during training, so its predictions are designed to be oblivious to the protected attribute.
- Guardedness holds in expectation over an entire dataset and does not generally correspond to equality of odds or equality of opportunity.
- To address subgroup leakage, the refined algorithm trains each iteration’s protected-attribute classifier on examples from one randomly chosen main-task class.
6 Experiments and Analysis
The experiments apply INLP to debias word embeddings and fair classification, testing linear guardedness, semantic preservation, and fairness-related outcomes. INLP reduces recoverable gender information and TPR gaps, while preserving or improving several measures of embedding quality.
- Word-embedding debiasing: INLP reduces linear gender classification accuracy from 100% to 49.3%, although a 1-layer ReLU MLP still recovers gender with 85.0% accuracy.The nonlinear result is expected because INLP is designed to achieve linear guarding.
- Word-embedding debiasing: 54.4% accuracy remains after removing 10 INLP-learned directions, versus 80.7% after removing 10 manually selected gender directions.The comparison supports learning multiple subtle gender-predictive directions from data rather than relying only on predefined directions.
- Word-embedding debiasing: INLP improves SimLex-999 correlation with human judgments from 0.373 to 0.489 after projection.The broader semantic evaluation uses multiple word-similarity datasets.
- Word-embedding debiasing: WEAT finds that previously strong associations between gender-name groups and stereotypically gendered concepts no longer exist after projection, with p-values of 0.855, 0.302, and 0.761.The three tests cover career/family, art/mathematics, and artistic/scientific associations.
- Fair classification: INLP reduces profession-level TPR gaps across BOW, BWV, and BERT, while changing main-task accuracy by +1.9% for BOW, −5.1% for BWV, and −5.51% for BERT.The corresponding TPR-gap reductions are 38.91%, 51.6%, and 48.36%, respectively.
- Fair classification: The correlation between profession-level TPR gap and women’s representation decreases by 33.4% in BOW, 52.5% in BWV, and 46.7% in BERT.For BWV, INLP outperforms a 0.71 correlation baseline obtained using biographies with pronouns and names removed.
7 Limitations
In fairness applications, INLP depends on representative i.i.d. training data and protects only against linear decoding of protected attributes.
- INLP assumes sufficiently large training data sampled i.i.d. from the same distribution as test data; unrepresentative data may leave biased classifications.Verification remains the user’s responsibility because the method provides no unconditional guarantees.
- The method targets removal of linear information about a protected attribute, not direct enforcement of fairness metrics such as demographic parity.It is designed for use with a linear decoder; protected attributes are not protected under nonlinear classifiers.
8 Conclusion
The paper introduces INLP for controlled, deterministic removal of linearly represented information and evaluates it on bias and fairness case studies. The authors report attenuation of societal biases across increasingly complex settings and identify broader potential applications.
- INLP removes linearly represented information from neural representations through a controlled and deterministic procedure.
- Across increasingly complex bias and fairness settings, INLP attenuates societal biases expressed in representations learned from data.
- Beyond bias and fairness, INLP may remove specific representation components for style transfer, disentanglement, and interpretability.These applications are presented as possible future directions rather than evaluated results in this work.
A.1 INLP Guarantees
The appendix establishes geometric guarantees for INLP: successive classifiers are orthogonal, their combined operator is a projection onto intersecting nullspaces, and distance distortion is bounded by removed gender components.
- Orthogonality: Under convex loss and zero initialization with SGD, classifiers learned in successive binary INLP iterations are orthogonal.The proof uses the fact that optimization inputs have already been projected into the previous classifier’s nullspace.
- Projection structure: The product of INLP’s projection matrices is a valid projection operator rather than merely a product of projections.For two iterations, the appendix states P = P2P1 and P^2 = P.
- Projection structure: After multiple iterations, INLP projects representations onto the intersection of the classifiers’ nullspaces.In practice, Equation 1 enforces this intersection projection even when the appendix’s orthogonality conditions do not hold.
- Distance preservation: The projection changes squared distances by at most the difference between the inputs’ components along the removed gender direction.Thus, equally gender-biased points retain their distance, while points with strongly different gender components can change more.
- Distance preservation: Orthogonal coordinate rotations preserve dot products and distances, so the distance bound proved in the rotated system also holds in the original coordinates.
A.2 Influence on Local Neighbors in Glove Space
Table 1 compares the three nearest words for sampled GloVe words before and after INLP, with the reported result that local neighbors are not altered detrimentally.
- Table 1 lists the 3-nearest words of sampled GloVe words before and after the INLP projection.The comparison tests whether projection changes local embedding neighborhoods.
- INLP does not detrimentally alter the neighbors of the random sample, suggesting little influence on the lexical semantics of unbiased words.
A.3 Quantitative Influence of Gender Debiasing on Glove Embeddings
INLP reduces gender-related associations in GloVe and other representations while preserving or improving several semantic measures. Its effects are tunable, with larger bias reductions obtained by accepting greater main-task performance loss.
- Semantic similarity: 0.044 to 0.116 points: Spearman correlations with human similarity judgments improve across all evaluated word-similarity sets after projection.The reported improvement is measured on regular GloVe embeddings before and after gender debiasing.
- Local semantic structure: INLP minimally changes nearest neighbors for random words while shifting gendered surname neighbors toward gender-neutral family names.The surname analysis contrasts neighbors before and after projection, whereas the random-word analysis finds no detrimental change.
- Performance trade-offs: The trade-off is controllable: performance is minimally damaged for the first 180 iterations while TPR-Gap improves greatly, after which both metrics decline more.The number of INLP iterations determines how much main-task performance is sacrificed for reduced bias.
- Directions and affected vocabulary: The first gender direction captures pronouns, while other directions capture socially biased terms, gender-related terms, surnames, and less interpretable words.Examples include “preschool,” “cookbooks,” “LGBTQ,” “femininity,” “Gordon,” and “Aviv”; some words may reflect spurious correlations or noise.