PaperlayerView exact revision on arXiv

Source-linked AI summary

Principled Nonlinear Concept Erasure and Steering

Idan Simai, Shauli Ravfogel, Uri Shaham

arXiv:2610.32516v1cs.LGcs.CL

TL;DR

Existing erasure methods offer either guarantees limited to linear probes or stronger empirical nonlinear erasure without comparable formal guarantees. This paper uses a conditional autoencoder to combine reconstruction with independence, reporting stronger utility–erasure trade-offs and inference-time concept conversion.

  • Problem

    Linear erasure guarantees cover linear probes, while nonlinear methods have limited guarantees of complete removal and may sacrifice utility.

    Source: §1 · p. 1

  • Method

    A conditional autoencoder combines reconstruction-based preservation with statistical independence from the sensitive concept.

    Source: §1 · pp. 1–2, §4.2 · p. 4

  • Results

    Across datasets and language models, the method achieves stronger utility–erasure trade-offs and, on BIAS IN BIOS, outperforms supervised Obliviator while using no target-task labels.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Changing the decoder’s target concept at inference yields concept conversion without additional training, alongside nonlinear erasure guarantees under the stated conditions.

    Source: §1 · p. 2, §7 · p. 9

    The guarantees characterize representations attaining ideal conditions, not convergence of finite non-convex training; they also require independent complementary and sensitive factors.

    Source: §3 · p. 3, §5 · p. 5

Abstract

from arXiv · show

Learned representations often encode sensitive attributes, such as gender or race, that may be undesirable or unsafe to retain. Concept erasure aims to remove this information while preserving task-relevant content. Existing methods remain incomplete: linear methods provide guarantees only against linear probes, while nonlinear methods achieve stronger empirical erasure but lack formal guarantees and often sacrifice utility. In this work, we introduce a principled conditional autoencoder framework for nonlinear concept erasure. We follow the principle that erasure should remove only the sensitive attribute and nothing else, and we show that, under the stated assumptions, perfect reconstruction together with population-level independence is sufficient for perfect nonlinear guardedness, with preservation guarantees in the imperfect-reconstruction regime. Empirically, we attain a superior utility-erasure trade-off across multiple datasets, sensitive attributes, and language models, surpassing even the supervised variants of competing methods despite using no target-task labels, while being simpler to optimize and converging substantially faster. Beyond erasure, our framework yields concept conversion (representation steering) as a by-product: by changing only the target concept at inference time, it produces representations corresponding to different concept values - at no additional training cost - a capability no prior erasure method provides, and on which we likewise demonstrate strong empirical performance.

1 Introduction

Concept erasure seeks to remove sensitive information while preserving task-relevant content, but existing methods lack either nonlinear guarantees or strong empirical erasure. This work proposes a conditional autoencoder framework with theoretical guarantees, strong utility–erasure results, and inference-time concept conversion.

1 Introduction

  • On BIAS IN BIOS, the unsupervised method achieves a better utility–erasure trade-off than competing approaches, including supervised Obliviator.Figure 1 illustrates the BERT result; the paper also reports that the method is easier to train and converges faster than prior nonlinear methods.Source: Figure 1 · p. 2, §1 · p. 2
  • The framework targets perfect nonlinear guardedness under stated conditions and preservation guarantees when reconstruction is imperfect.Its principle is to remove only the sensitive attribute while maximizing information retained in the erased representation, subject to independence from that attribute.Source: §1 · p. 2
  • At inference, changing the decoder’s target concept value produces a representation for that value without additional training.The decoder takes both the erased representation and sensitive attribute as input.Source: §1 · p. 2
  • Across datasets, sensitive attributes, and language models, the method achieves superior utility–erasure trade-offs without target-task labels and uses a simpler, faster-optimizing objective.The paper reports performance surpassing supervised variants of competing methods.Source: §1 · p. 2

2 Related Work

Prior work spans linear erasure, nonlinear erasure, independence-constrained representation learning, and representation-level steering. The paper positions its contribution as connecting reconstruction and independence to nonlinear erasure guarantees and steering.

Linearly Guarded Erasure

  • Linear methods remove information detectable by linear adversaries, but nonlinear probes may still recover the sensitive attribute and these methods do not support steering.INLP uses iterative nullspace projections, R-LACE a linear-adversary minimax game, and LEACE a closed-form projection with linear-removal guarantees.Source: §2 · p. 2

Nonlinearly Guarded Erasure

  • Nonlinear methods strengthen empirical erasure, yet offer limited guarantees of complete removal, and the surveyed methods do not support concept steering.Approaches include adversarial suppression, rate-distortion objectives, density matching, TaCo, and Obliviator.Source: §2 · p. 3

Independence-Constrained Representation Learning

  • Independence-promoting representation learning has prior theoretical and practical uses, but adversarial invariance alone does not guarantee concept removal.The paper distinguishes its contribution as analyzing how reconstruction plus independence yields nonlinear guardedness guarantees and steering.Source: §2 · p. 3

Representation-Level Steering

  • Earlier steering methods use affine or population-statistic transformations, whereas the conditional encoder–decoder provides nonlinear steering tied to erasure guarantees.The decoder receives the concept-independent representation and a target concept value, without an additional objective.Source: §2 · p. 3

3 Problem Formulation

The formulation treats the sensitive concept as observed and complementary content as latent, with the goal of erasing the former while preserving the latter. A conditional encoder–decoder handles both erasure and conversion using input-concept pairs, without task labels.

3 Problem Formulation

  • The model assumes X is generated by an injective deterministic map from independent latent factors U and C, where U is complementary content and C is the sensitive concept.The independence assumption is idealized: if U and C are entangled, perfect preservation of U and erasure of C may not both be possible.Source: §3 · p. 3
  • Training observes input-concept pairs but never observes the complementary factors U, and assumes no task labels.The setup views concept erasure as a special case of nonlinear ICA with C observed and U latent.Source: §3 · p. 3
  • Erasure learns an encoder representation Z and a decoder that reconstructs X from Z and the observed concept C.The framework defines the conditional map as g(x, c) = d(e(x), c).Source: §3 · pp. 3–4
  • For conversion, the decoder uses the same encoded input with a target value c′ to generate a sample corresponding to the same U under that concept.The paper studies erasure and steering within the same conditional encoder–decoder framework.Source: §3 · p. 4

4 Proposed Method

The method formalizes “remove only the sensitive concept” as preserving input information while enforcing independence from that concept. It implements this with reconstruction and independence objectives, and the conditional decoder also supports steering.

4.1 Motivation

  • Among representations independent of C, the ideal objective retains as much information about X as possible.Under Z ⊥⊥ C, maximizing I(X; Z) is equivalent to minimizing H(X | Z, C), making reconstruction the operational preservation objective.Source: §4.1 · p. 4
  • Reconstruction certifies that Z retains the non-sensitive information needed to recover X when C is supplied.The same decoder can be queried with a new concept value c′ to steer the representation.Source: §4.1 · p. 4
  • HSIC-based preservation is not generally equivalent to maximizing mutual information, so it may favor representations aligned with the chosen kernel geometry.The paper notes that normalization alone does not generally ensure the whitening or spectral isotropy needed for equivalence.Source: §4.1 · p. 4

4.2 Method

  • The autoencoder combines reconstruction of x from z and c with an HSIC penalty encouraging independence between Z and C.The objective weights the independence term with λind to trade off reconstruction quality against concept removal; reconstruction uses mean squared error.Source: §4.2 · pp. 4–5
  • At inference, steering decodes the same erased representation with a target concept value c′.The method’s two objective ingredients preserve information needed for reconstruction and remove sensitive-concept information from the latent representation.Source: §4.2 · p. 5

5 Theoretical Analysis

The analysis relates reconstruction and independence to nonlinear guardedness, then extends preservation guarantees to imperfect reconstruction and exact steering. These guarantees characterize attained properties under stated conditions, not convergence of finite-sample training.

5 Theoretical Analysis

  • The guarantees characterize representations attaining the targeted conditions, not convergence of finite, non-convex training; the entropy-based results are formalized for discrete variables.The experiments use continuous representations, and the reconstruction-error guarantee is stated in Euclidean error without relying on discreteness.Source: §5 · p. 5

5.1 The Perfect Regime

  • With perfect reconstruction and independence, the representation retains all non-sensitive information and is perfectly nonlinearly guarded.It is maximally informative about the input among deterministic representations independent of the sensitive concept.Source: §5.1 · p. 5

5.2 The Imperfect Regime

  • With imperfect reconstruction, a Lipschitz left-inverse lets reconstruction error control recovery error for the complementary factor U.The assumption rules out inverse mappings where small errors in X correspond to large errors in U.Source: §5.2 · p. 6
  • For finite U with separated support, small reconstruction error implies near-preservation of complementary information, without requiring independence between Z and C.The information-preservation result is stated informally here; its full statement is deferred to the appendix.Source: §5.2 · p. 6

5.3 Steering as a By-product of Erasure

  • Exact steering follows under the same ideal conditions plus an asymmetric-law assumption: changing the decoder’s concept input preserves U and changes only the concept.For discrete U, the assumption holds when distinct support points have distinct probabilities.Source: §5.3 · p. 6

6 Experiments

The experiments evaluate concept erasure and steering across datasets and language-model representations. The method achieves stronger erasure trade-offs on BIAS IN BIOS and Dial, and its learned steering transfers to generation with a detoxification–quality trade-off.

6.1 Experimental Setup

  • The evaluation uses frozen representations from BERT, GPT-2, DeepSeek, and LLaMA, measuring erasure through post-hoc sensitive-attribute and downstream-task classifiers.Steering is evaluated using downstream-task classifiers, fairness metrics, and post-hoc sensitive-concept classifiers.Source: §6.1 · p. 7

6.2 Concept Erasure Results

  • At about 56% gender accuracy on BERT, the method preserves about 74% profession accuracy, versus about 56% and 61% for unsupervised and supervised Obliviator.The advantage also persists across GPT-2, DeepSeek, and LLaMA representations.Source: Figure 2 · p. 7, §6.2 · p. 8
  • On DIAL-MENTION, the method matches supervised Obliviator’s trade-off; on DIAL-SENTIMENT, it achieves higher utility in the low-leakage regime without downstream labels.Across both Dial tasks, it outperforms unsupervised Obliviator at comparable race leakage.Source: §6.2 · p. 8, Figure 3 · p. 8

6.3 Generative Steering

  • At the strongest generative-steering point, the method matches MiMiC’s toxicity (0.018) with lower PPL (6.86 vs. 7.66).Methods are compared at matched topicality; lower toxicity and PPL are better.Source: Table 1 · p. 8, §6.3 · p. 9
  • For generation, the method steers hidden states from toxic to non-toxic using the learned decoder transformation, without retraining the language model or learning a separate steering model.The update retains the original hidden state while applying the nonlinear transformation; β = 1 recovers the conversion under perfect reconstruction.Source: §6.3 · pp. 8–9

7 Conclusion

The conclusion presents reconstruction plus independence as a framework for nonlinear concept erasure and steering, with theoretical guarantees under ideal conditions and empirical results across datasets and language models.

7 Conclusion

  • The framework combines reconstruction-based preservation with statistical independence from the sensitive concept.Perfect reconstruction and population independence yield nonlinear guardedness and exact steering under stated ideal conditions; imperfect reconstruction admits quantitative preservation guarantees.Source: §7 · p. 9
  • The method achieves stronger utility–erasure trade-offs across datasets and language models while enabling nonlinear steering in representation-level and generative settings.Source: §7 · p. 9

AI use statement

The authors used generative AI tools to assist with drafting and polishing some manuscript passages, and state that they reviewed and approved the resulting text.

AI use statement

  • Generative AI tools assisted with initial versions of some passages and with wording, clarity, and organization.Source: AI use statement · p. 10
  • The authors state that they did not use generative AI for literature retrieval, research ideation or execution, synthetic datasets, or mathematical proofs.Source: AI use statement · p. 10
  • The authors take full responsibility for the final paper content.Source: AI use statement · p. 10

Ethics statement

The ethics discussion limits the fairness implications of concept erasure, while the accompanying theory explains why reconstruction targets information preservation and what perfect reconstruction guarantees under independence.

Ethics statement

  • DIAL race labels are dialect-based proxies rather than self-reported attributes, and the authors use them only as concepts to erase.The real-data experiments use public datasets and report no new human-subject data or participant studies.Source: Ethics statement · p. 10

A.1 Why reconstruction targets mutual information and HSIC does not

  • Under Z ⊥⊥ C, maximizing I(X; Z) is equivalent to minimizing H(X | Z, C), whereas HSIC is not generally a monotonic proxy for mutual information.The paper therefore uses reconstruction rather than an HSIC-based preservation term.Source: §A.1 · pp. 14, 16

A.2 Proofs for the Perfect Regime

  • With independence and perfect reconstruction, the representation preserves all complementary information, attaining I(X; Z) = H(X | C).The perfect-regime results also characterize the representation as an entropy-preserving transformation of the complementary factor.Source: §A.2 · pp. 16–17

A.3 Proofs for the Imperfect Regime

The appendix bounds preserved information under imperfect reconstruction, establishes a concept-conversion result, and tests robustness when the independence assumption is violated.

A.3 Proofs for the Imperfect Regime

  • For finite U, I(U; Z | C) is lower-bounded by H(U) − h2(δmse) − δmse log(|U| − 1).Here δmse is defined using nearest-neighbor decoding; the bound relates reconstruction error to retained complementary information.Source: §A.3 · p. 17

A.4 Proofs for Concept Conversion

  • The concept-conversion result states that decoding an erased representation with a substituted target concept yields the corresponding representation for that concept.The displayed result gives d(e(f(U, C)), c′) = f(U, c′).Source: §A.4 · p. 18

B Empirical Robustness to Violated Independence

  • At α ≤ 0.5, most utility is retained; erasure remains at chance and steering succeeds from a downstream classifier’s perspective across the dependence sweep.The reported utility loss is gradual and confined to the entangled coordinate of U.Source: B Empirical Robustness to Violated… · p. 19

B.1 Setup

  • The synthetic robustness sweep varies dependence between U and C while keeping class balance fixed and entangling only U0 with C.U1 and U2 remain independent of C throughout the sweep, separating entangled from complementary utility.Source: §B.1 · p. 19, B Empirical Robustness to Violated… · p. 19

B.2 Metrics

The evaluation measures concept leakage, retained utility, and steering as dependence between U and C increases. Erasure remains near chance and steering remains operational, while utility for C-entangled information degrades progressively.

B.2 Metrics

  • Erasure is measured by an MLP probe predicting C, utility by R2 for predicting U from Z, and steering by oracle counterfactual MSE and probe-based success rate.Utility is reported separately for C-entangled U0 and C-independent (U1, U2); steering success measures whether a frozen-X classifier predicts the target concept value c′.Source: §B.2 · p. 19

B.3 Erasure and Utility

  • As α increases, erased Z stays near chance for predicting C, while C-independent U1 and U2 remain almost perfectly recoverable and C-entangled U0 degrades progressively.The degradation is smooth rather than abrupt; preserving more of U0 as its overlap with C grows would reintroduce concept leakage.Source: Figure 4 · p. 20, §B.3 · p. 20

B.4 Steering

  • Oracle steering MSE rises smoothly from 0.011 at α = 0 to 0.206 at α = 1, remaining far below the do-nothing baseline; probe-based steering remains perfect across the sweep.The stricter oracle metric captures the increasing difficulty of changing C while preserving U as U0 becomes more tied to C.Source: §B.4 · p. 20

C.1 Datasets

Across the reported steering evaluations, the decoder converts concepts while retaining useful representation structure; the experimental setup spans multiple datasets and steering settings, with substantially faster training than Obliviator.

C.1 Datasets

  • The benchmarks cover gender-to-profession prediction on BIAS IN BIOS, race-to-sentiment prediction on DIAL-SENTIMENT, and race/dialect-to-mention prediction on DIAL-MENTION.The method uses frozen PLM representations and concept labels for erasure; downstream labels are reserved for evaluation.Source: §C.1 · pp. 20–21

C.2 Experimental Setup

  • Evaluation uses five-hidden-layer nonlinear probes for sensitive-attribute leakage and retained utility, alongside HSIC computed from random Fourier feature maps.The probes are stronger than the single-hidden-layer scikit-learn MLP often used in prior work; code vectors are normalized before HSIC feature mapping.Source: §C.2 · pp. 21–22

C.3 Steering Setup

  • Steering evaluations cover Qwen toxicity generation, Llama-2 autoregressive gender generation, and frozen-probe gender conversion across BERT, DeepSeek, and LLaMA.For Qwen, topic accuracy measures recovery of a 20-way prompt partition, while topicality is prompt–continuation cosine similarity.Source: §C.3 · pp. 22–23

C.4 Computational Complexity and Scalability

  • One epoch of our method takes roughly 0.7–2.6 seconds, versus 42–167 seconds per Obliviator outer iteration; full runs are 1.8–2.8× faster in the unsupervised setting.The authors also report lower peak memory for their method, which preserves utility through reconstruction rather than auxiliary random-feature matrices.Source: §C.4 · pp. 23–24

D.1 Toxicity Steering

  • Replacement converts toxic states as often as MiMiC (92.9% vs. 93.7%), retains more topic accuracy (44.8 vs. 41.7), and halves the fairness gap (0.080 vs. 0.160).At code size 512, replacement yields perplexity 2807 versus 3.44 for the untouched model; a 2048-dimensional code lowers toxicity from 0.163 to 0.050 versus 0.145 for its round-trip.Source: §D.1 · p. 24

D.2 Gender Steering on Llama-2-7B

  • Direct replacement gives the strongest initial target preference, but repeated replacement accumulates reconstruction error; residual steering removes observed source-gender markers at β = 1.0 while keeping perplexity near baseline and profession accuracy essentially unchanged.Increasing β to 1.5 strengthens target steering at a modest fluency cost.Source: §D.2 · pp. 25–26

D.3 Frozen-Probe Gender Steering

  • Across PLMs, the method achieves near-perfect gender conversion, preserves high profession accuracy, and attains the lowest TPRRMS.The authors conclude the decoder-based intervention changes the target concept while preserving structure needed by an already-trained downstream system.Source: §D.3 · p. 26

D.4 Additional Steering Results

Across steering, probe, independence, fairness, and sensitivity evaluations, the autoencoder shows strong concept control while preserving task utility. The appendices also define the fairness and steering metrics used to interpret these results.

D.4 Additional Steering Results

  • ≥99.45% steering success in both directions across PLMs accompanies profession utility within 0.6% of raw-probe accuracy and round-trip MSE below 0.065.Round-trip gender classification stays below 0.2%, while decoding with the original concept faithfully reconstructs the input.Source: Table 13 · p. 26, §D.4 · p. 27

D.5 Additional Probe Families

  • Across additional probe families, gender accuracy stays near majority baselines on BIAS IN BIOS and DIAL-SENTIMENT; DIAL-MENTION shows the largest spread, but mention accuracy remains above 69%.The authors report that the main conclusions do not depend on the MLP probe family.Source: Table 14 · p. 27, §D.5 · p. 27

D.6 Permutation Test

  • Held-out independence tests reject independence for frozen BERT every time at n = 1,000, but never for the autoencoder; held-out HSIC falls by at least 84× and 67× on the two DIAL datasets.At n = 5,000, independence is only partly rejected for the autoencoder.Source: Table 15 · p. 27, §D.6 · p. 27

D.7 Fairness

  • At DP around 0.03, our method reaches roughly 69% sentiment accuracy, versus 60% for unsupervised and 67% for supervised Obliviator on BERT; it also outperforms Obliviator at the reported Gaprms levels.At Gaprms around 0.04, it reaches roughly 70% sentiment accuracy, while unsupervised Obliviator on BERT and DeepSeek barely reaches 60%; supervised curves do not reach this fairness level.Source: Figure 5 · p. 28, §D.7 · p. 28

D.8 Hyperparameter Sensitivity

  • At λind = 5, gender accuracy reaches 54.67%, near the 53.7% chance level, while profession accuracy remains 72.29%; extending training from 200 to 500 epochs strengthens erasure as profession accuracy decreases.The authors use 500 epochs for the main experiments because it provides the strongest tested erasure while retaining substantial utility.Source: Figure 6 · p. 28, §D.8 · p. 29

E.1 Fairness Metrics

  • Demographic parity measures changes in predicted sentiment distributions across race groups, and lower DP values indicate fairer predictions.The appendix also reports the RMS true-positive-rate gap as a second fairness metric.Source: §E.1 · p. 29

E.2 Steering Metrics

  • The steering RMS true-positive-rate gap measures class-conditioned disparity across gender groups; lower values indicate smaller disparities.It aggregates per-profession-label true-positive-rate gaps.Source: §E.2 · p. 29

Related papers

Loading 2610.32516v1…