MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions

Antoine Saillenfest

Source-linked AI summary

MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions

Antoine Saillenfest

arXiv:2609.11253v1Sep 10, 2026cs.LGcs.CL

TL;DR

Concept erasure must make a target concept unpredictable without discarding unrelated information, but existing methods face limitations in general nonlinear settings and computation. MUtE* derives optimal continuous erasure functions with deterministic dual counterfactual mappings, then implements them with a translational bias; experiments report efficacy for fairness and counterfactual text generation.

Problem
Concept erasure seeks unpredictable target concepts while preserving unrelated information, but general nonlinear erasure remains challenging and existing approaches can be computationally expensive.
Method
MUtE* derives optimal continuous erasure functions that induce deterministic counterfactual mappings and implements them with a translational bias.
Results
MUtE* demonstrates empirical efficacy for algorithmic bias mitigation and counterfactual text generation across synthetic data and NLP benchmarks.
Takeaways & Limitations
The framework provides a unified way to navigate between concept erasure and counterfactual generation in continuous representation spaces.
Takeaways & Limitations
MUtE is formulated for a single discrete concept and may be suboptimal or unpredictable for continuous, intersecting, or highly nonlinear factors.

Abstract

from arXiv · show

Erasing concept-specific information from representations has been proven useful for mitigating bias or interpreting model decisions. The joint objective is to transform the original representations such that the target concept becomes unpredictable, while maximally preserving concept-unrelated information. In this work, we revisit the optimal bounds of concept erasure to derive a novel class of erasure functions that naturally induce a deterministic, dual counterfactual mapping. Bridging the gap between theoretical optimality and practical representation learning, we design an implementation that imposes a translational bias on counterfactual trajectories - a constraint that aligns with how many concepts geometrically manifest in modern language models. Our framework enables seamless navigation between concept erasure and counterfactual generation. We empirically demonstrate its efficacy in improving downstream algorithmic fairness and generating counterfactual texts.

1 Introduction

MUtE revisits concept erasure as a privacy–utility problem and derives erasure functions that also support deterministic counterfactual generation. Its practical implementation uses translational counterfactual trajectories, with experiments targeting fairness and counterfactual text generation.

1 Introduction: Concept erasure must remove target-concept information while retaining concept-unrelated information, since independence alone permits collapsed representations.Constant mappings can satisfy privacy but destroy the representation's utility.
1 Introduction: Existing linear methods prevent recovery by linear adversaries, whereas general nonlinear concept erasure remains an open challenge.Prior approaches include INLP, RLACE, and LEACE, while other methods face computational bottlenecks in high dimensions.
1 Introduction: MUtE imposes a translational bias on counterfactual trajectories to align with concepts often appearing as linear directions in modern language-model representations.This constraint links erasure and counterfactual generation in a practical implementation.
1 Introduction: MUtE* introduces optimal erasure functions that intrinsically define a dual counterfactual mapping.The framework targets perfect privacy while preserving utility in continuous representations.
1 Introduction: The framework is evaluated on synthetic data and NLP benchmarks for bias mitigation and counterfactual text generation.The stated applications include improving algorithmic fairness and generating counterfactual texts.

2 Optimal Erasure and Counterfactuals

MUtE* formalizes optimal continuous concept erasure as class-conditional bijections into a shared invariant distribution, balancing perfect privacy with maximal utility retention. These invertible mappings induce deterministic counterfactual transports, while geometric priors constrain otherwise non-identifiable solutions.

2 Optimal Erasure and Counterfactuals: Optimal MUtE* mappings align all class-conditional post-erasure distributions while preserving enough information to recover each original representation.The formulation combines strict independence from the concept with bijective class-conditional transformations.
2 Optimal Erasure and Counterfactuals: Perfect erasure requires f(X) and Z to be statistically independent, but this criterion alone admits degenerate constant mappings that collapse representation utility.The joint objective therefore adds maximal utility preservation.
2 Optimal Erasure and Counterfactuals: Under the oracle assumption, each class-specific diffeomorphism maps its source distribution P_i bijectively to a shared invariant distribution Q.The continuous formulation replaces discrete class-conditional permutations with diffeomorphisms.
2 Optimal Erasure and Counterfactuals: Because infinitely many optimal erasure mappings can distort geometry arbitrarily, expected counterfactual trajectories constrain the diffeomorphisms and resolve non-identifiability.In NLP, a linear-direction prior motivates translational interventions for this purpose.
2 Optimal Erasure and Counterfactuals: A counterfactual transport maps a sample from class i to a representation distributed according to target class j while preserving its invariant post-erasure coordinate.Thus, one erased representation anchors the sample and its counterfactual counterparts.
2 Optimal Erasure and Counterfactuals: Figure 1 presents MUtE* as bidirectional geometric navigation between concept erasure and counterfactual generation through invertible mappings.The framework's structure supports movement between erased and counterfactual continuous spaces.

3 Operationalizing MUtE∗via Iterative Density Matching

MUtE operationalizes conditional erasure through iterative Gaussianization, using class-dependent marginal transforms and shared rotations to align conditional distributions. Noisy class routing introduces an entropic penalty, so regularization is used to obtain robust practical erasure despite imperfect convergence.

3.1 Iterative Gaussianization: MUtE alternates global orthogonal rotations with class-conditional marginal Gaussianization, preserving macroscopic geometry while aligning each conditional distribution's marginals.The rotations provide cross-dimensional mixing, while the class-specific transformations independently warp conditional marginals toward the standard normal.
3.2 Selective Iterative Density Matching: Under perfect oracle routing, conditional iterative Gaussianization bijectively maps every class distribution to N(0, I), theoretically guaranteeing complete concept erasure after sufficient iterations.The process is truncated at step T to obtain an empirical conditional-bijective MUtE function.
3.2 Selective Iterative Density Matching: The first median-to-origin matching step biases counterfactual trajectories toward inter-centroid directions, while later transformations refine them through higher-order distributional alignment.This creates a translational geometric inductive bias before subsequent rotations and marginal transformations refine the trajectory.
3.3 Implementation and Mitigation of Noisy Routing: Because conditional supports overlap, predicted-class routing has irreducible classification error that imposes a positive entropic penalty and prevents exact convergence to an isotropic Gaussian.The stepwise reduction decomposes into ideal negentropy reduction minus a misrouting penalty, with the penalty potentially diverging when a true positive density is assigned zero probability.
3.3 Implementation and Mitigation of Noisy Routing: Regularizations maximize ideal marginal gains and bound routing penalties, using discretized conditional density estimates with a minimum density floor to improve numerical robustness.With B = 1000 and α = 10^-10, the injected mass Bα = 10^-7 remains negligible; exact theoretical alignment is not guaranteed, but real-world evaluations report robust erasure.
3.3 Implementation and Mitigation of Noisy Routing: An alternating PCA-based rotation schedule exposes cross-dimensional correlations across classes while offering a computationally favorable trade-off against orthogonal ICA.The schedule cyclically aligns conditional principal axes, and valid orthogonal rotations retain non-negative negentropy reduction.

4 Experiments

MUtE is evaluated on synthetic and NLP datasets for concept erasure, utility preservation, fairness, and counterfactual generation. Across these settings, it aligns post-erasure distributions, preserves utility, improves fairness, and generates semantically faithful counterfactual texts.

4.1 Datasets, Baselines and Training details: On synthetic mixtures, post-erasure conditional distributions align, probe accuracy drops to chance, and counterfactual trajectories show a strong translational bias.The experiments cover initial probe accuracies of 94% and 81% under near-perfect separability and moderate overlap.
4.2 Erasure and utility preservation: MUtE reduces sensitive-attribute probe accuracy to the majority-class baseline while maintaining low reconstruction error and competitive or higher downstream accuracy.On balanced binary concepts, the majority-class baseline is 50%; the supplied results also report low MSE for reconstructing X from f(X).
4.2 Erasure and utility preservation: MUtE preserves GloVe semantic alignment at 0.71 versus 0.70 originally, while retaining 19% of top-1% neighborhood structure compared with 13% for FaRM and 10% for KRaM.These results assess both semantic similarity and local topology after erasure.
4.3 Fair classification: MUtE-transformed representations substantially improve downstream fairness and remain competitive with established baselines, favoring utility preservation on Bias in Bios.The evaluation uses TPRRMS and Demographic Parity, with results reported in Table 2.
4.4 Counterfactual text generation: MUtE counterfactuals achieve effective gender substitution with high semantic fidelity despite lower BLEU/ROUGE overlap than baselines.The lower n-gram overlap is attributed to deeper intervention, while BERTScore remains high and gender substitution is significantly superior.

5 Discussion and Future Directions

The framework links erased and counterfactual representations through diffeomorphic mappings while offering efficient deterministic training and deployment options. Its current implementation remains limited by inference cost and rigid geometric assumptions, with broader distillation evaluations left open.

Diffeomorphic navigation unifies erased and counterfactual continuous spaces and supports extensions beyond NLP.
Deterministic training bypasses gradient-based optimization, but exact inference costs O(T) for T convergence iterations.
Class-specific INNs or cINNs preserve bijectivity for dual counterfactual generation, whereas MLPs maximize throughput when only forward erasure is needed.
An MLP surrogate preserves predictive utility and fairness guarantees on downstream tasks, although extensive distillation evaluations remain future work.
Rigid translational priors do not capture causal factors with complex nonlinear interactions and heterogeneous geometric signatures.

6 Conclusion

The paper derives theoretically optimal erasure functions that deterministically induce counterfactual mappings, then implements them with a tractable translational bias. Across NLP tasks, the framework supports fairness mitigation and counterfactual text generation.

MUtE achieves perfect privacy while maximizing downstream utility preservation through optimal erasure functions that induce deterministic counterfactual mappings.
A computationally tractable implementation exploits rigid location shifts observed for many latent concepts in modern language models.
The method demonstrates efficacy on real-world NLP tasks for algorithmic bias mitigation and counterfactual text generation.

Limitations

MUtE is designed for single discrete concepts and relies on geometric assumptions that may not hold in complex, entangled representations. These boundaries constrain its direct application to continuous, multiple, or highly nonlinear concepts.

MUtE is strictly formulated for one discrete concept and does not naturally extend to continuous attributes or simultaneous erasure of multiple intersecting concepts.
Applying MUtE to arbitrarily complex entangled representations without verifying topological assumptions risks suboptimal or unpredictable interventions.

Ethical Considerations

The ethical discussion frames demographic intervention as categorical labeling with substantial contextual and deployment risks. It emphasizes that fairness evaluation must be accompanied by attention to utility degradation, proxy exploitation, and the assumptions underlying the transformation pipeline.

Sensitive attributes such as gender, race, and religion are represented as discrete categories drawn from a restricted label set.
Operationalizing demographic attributes requires consensus that may be hindered by heterogeneous cultural, ethical, and legal contexts.
Forced erasure can reduce information and impair downstream predictive performance, creating risks in high-stakes domains.
Fairness cannot be the sole evaluation criterion because erasing one sensitive attribute may encourage reliance on unprotected proxy variables.

A.2 Negentropy Evolution Under Noisy Routing

This section decomposes negentropy evolution under noisy routing into ideal transformation effects and a routing penalty caused by misclassified regions. It shows that thresholding marginal density estimates bounds this penalty and ties instability to routing error.

A.2 Negentropy Evolution Under Noisy Routing: The entropy calculation partitions latent space into disjoint routing regions because decision boundaries have zero Lebesgue measure.Local changes of variables and diagonal Jacobians then support the piecewise entropy expansion.
A.2 Negentropy Evolution Under Noisy Routing: The penalty E_i is the exact entropic cost of spatial tearing from cross-entropy mismatch in misclassified regions.It arises when the true marginal density is evaluated using estimators associated with the mispredicted class.
A.2 Negentropy Evolution Under Noisy Routing: Noisy routing can inject entropy, so strict monotonic negentropy reduction is not guaranteed despite subsequent rotation and exact marginalization.The non-differentiable decision boundaries cause incorrectly routed samples to introduce non-Gaussian artifacts.
A.2 Negentropy Evolution Under Noisy Routing: E_i can diverge to infinity when a mispredicted marginal density is zero where the true density is positive.This divergence motivates enforcing a positive lower threshold on empirical marginal density estimators.
A.2 Negentropy Evolution Under Noisy Routing: Imposing α > 0 on all empirical marginal density estimators makes the spatial tearing penalty finite and bounds it using class-specific routing error.The bound links worst-case instability to intrinsic Bayes error in the latent space.

B Datasets

The evaluation uses four datasets spanning word embeddings, biographies, dialect-sensitive sentiment, and toxicity detection with religion as the sensitive concept.

B Datasets: GloVe contains 21,996 word embeddings assigned to male-biased, female-biased, or neutral classes using projection onto a gender direction.The gender direction is defined by the principal component of gendered word-pair differences.
B Datasets: Bias in Bios contains 399,423 biographies with binary gender and 28 occupation labels, preserving severe historical gender-occupation correlations.The benchmark is designed to study gender bias in NLP.
B Datasets: DIAL is a Twitter-based sentiment benchmark with happy/sad labels and AAE/SAE dialect attributes, balanced across race and sentiment.It is derived from DeepMoji and uses a preprocessed version of the corpus.
B Datasets: Jigsaw evaluates binary toxicity detection while treating religion as a five-class sensitive concept.The dataset contains 96,492 samples split into 87,434 training and 9,058 testing examples.

C Training

Training uses standardized implementations and evaluation procedures across erasure baselines, with early stopping for MUtE and repeated downstream measurements. An MLP surrogate is also evaluated against the exact mapping.

C Training: MUtE training stops when validation probe accuracy reaches the majority-class baseline, using T = 100, 70, 150, and 150 for GloVe, Bias in Bios, DIAL, and Jigsaw.These values are selected by early stopping.
C Training: Baseline implementations include publicly available LEACE and reimplemented KRaM, FaRM, LEOPARD, and TaCo under standardized protocols.The TaCo adaptation removes its downstream-label dependency to compare unconstrained concept erasure methods.
C Training: The MLP surrogate preserves the exact MUtE mapping’s predictive utility and fairness guarantees while accelerating inference by at least an order of magnitude.Table 6 evaluates predictive and fairness metrics on surrogate-generated representations at inference time.
C Training: TaCo traces a Pareto front by sweeping filtered dimensions from 5 to 95 in increments of 5.The representations are projected onto a 100-dimensional subspace.
C Training: Probe and downstream models use scikit-learn MLPClassifiers, and reported accuracy and fairness metrics average five independent runs.Models train for up to 20 epochs with a constant learning rate of 10^-4.

D Fairness Metrics

Fairness is assessed through equalized true-positive-rate gaps and demographic parity, alongside a surrogate experiment testing whether MUtE’s utility and fairness properties are retained.

D Fairness Metrics: TPRRMS measures the root mean square of true-positive-rate gaps across concept classes for downstream labels.A fair classifier should have true-positive rates that are insensitive to protected attributes.
D Fairness Metrics: Demographic parity measures prediction differences across protected groups and is achieved when prediction probability is independent of sensitive-group membership.The reported formulation compares p(Ŷ = y|Z = 0) and p(Ŷ = y|Z = 1).
D Fairness Metrics: The offline surrogate combines PCA whitening with a 4-layer MLP to approximate the pre-trained exact MUtE mapping.Its hidden dimension is set to 2d, with Layer Normalization and tanh activations in intermediate layers.
D Fairness Metrics: The surrogate is trained with MSE against exact MUtE representations using AdamW, a 10^-3 learning rate, and 1,500 maximum epochs.A validation set of 2,000 samples monitors training.
D Fairness Metrics: The surrogate strictly preserves exact MUtE’s predictive utility and fairness guarantees while providing at least an order-of-magnitude faster inference.The evaluated metrics include ay, TPRRMS, DP, and az.