PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept-Aware Privacy Mechanisms for Defending Embedding Inversion Attacks

Yu-Che Tsai, Hsiang Hsiao, Kuan-Yu Chen, Shou-De Lin

arXiv:2602.07090v1cs.CRcs.AI

TL;DR

Embedding inversion attacks can expose sensitive attributes or reconstruct raw text, while existing differential privacy defenses assume uniform sensitivity across embedding dimensions. SPARSE learns which dimensions are sensitive to user-defined concepts and applies sensitivity-calibrated elliptical noise. Across models, datasets, and threat scenarios, it improves the privacy-utility tradeoff, but its protection depends on predefined concepts and does not eliminate all privacy threats.

  • Problem

    Embedding inversion attacks can expose sensitive attributes or reconstruct original text, while existing differential privacy defenses uniformly perturb dimensions despite user- and context-dependent privacy needs.

  • Method

    SPARSE combines differentiable mask learning for user-defined privacy concepts with the Mahalanobis mechanism, which applies elliptical noise calibrated by dimension sensitivity.

  • Results

    Across six datasets, three embedding models, and three attack models, SPARSE consistently reduces privacy leakage and achieves better downstream performance than state-of-the-art differential privacy methods.

  • Takeaways & Limitations

    SPARSE supports controllable, concept-aware protection by perturbing privacy-sensitive embedding dimensions while preserving utility for non-sensitive content.

  • Takeaways & Limitations

    SPARSE does not eliminate all privacy threats, and concepts omitted from its predefined sensitive concept space may receive less protection.

Abstract

from arXiv · show

Text embeddings enable numerous NLP applications but face severe privacy risks from embedding inversion attacks, which can expose sensitive attributes or reconstruct raw text. Existing differential privacy defenses assume uniform sensitivity across embedding dimensions, leading to excessive noise and degraded utility. We propose SPARSE, a user-centric framework for concept-specific privacy protection in text embeddings. SPARSE combines (1) differentiable mask learning to identify privacy-sensitive dimensions for user-defined concepts, and (2) the Mahalanobis mechanism that applies elliptical noise calibrated by dimension sensitivity. Unlike traditional spherical noise injection, SPARSE selectively perturbs privacy-sensitive dimensions while preserving non-sensitive semantics. Evaluated across six datasets with three embedding models and attack scenarios, SPARSE consistently reduces privacy leakage while achieving superior downstream performance compared to state-of-the-art DP methods.

1 INTRODUCTION

Embedding inversion attacks can recover sensitive attributes or text, while conventional differential privacy perturbs all dimensions uniformly and can damage utility. SPARSE addresses this mismatch by learning concept-sensitive dimensions and applying calibrated elliptical noise to protect user-defined concepts while preserving non-sensitive semantics.

  • Embedding inversion attacks can expose sensitive attributes or reconstruct text, including up to 92% of a 32-token input from T5-based embeddings.
  • Existing differential privacy defenses treat all embedding dimensions as equally sensitive, despite privacy concerns varying by user and context.
  • The research question asks whether user-defined private concepts can be selectively obfuscated while preserving non-sensitive semantics for downstream tasks.
  • SPARSE learns dimension sensitivity with a differentiable mask and uses the Mahalanobis mechanism to inject elliptical noise calibrated to that sensitivity.
  • Across six datasets, three embedding models, and three attack models, SPARSE consistently reduces privacy leakage while improving downstream performance over state-of-the-art differential privacy methods.
  • SPARSE achieves performance close to a white-box defense despite lacking prior knowledge of the attack model.

2 PRELIMINARIES

The paper frames embedding inversion as reconstruction of input text from embeddings and adopts local, metric-based differential privacy for semantic data. Its defense objectives are to protect user-specified sensitive tokens while retaining downstream utility, addressing the uniform-noise assumption of generalized Laplace mechanisms.

  • Differential Privacy: Local differential privacy requires each user to perturb data before sharing, avoiding reliance on a trusted aggregator.
  • Differential Privacy: A smaller privacy budget ϵ indicates stronger privacy protection, whereas a larger ϵ permits less similarity between outputs.
  • Metric Local Differential Privacy: Metric local differential privacy makes output indistinguishability depend on input distance, allowing closer inputs less distinguishable outputs.
  • Generalized Laplace Mechanism: The generalized Laplace mechanism adds isotropic spherical noise, implicitly assuming privacy sensitivity is uniform across embedding dimensions.
  • Attack Scenario: Embedding inversion attacks may target individual tokens or entire ordered sentences, with the study prioritizing disclosure of private information such as names or diseases.
  • Defense Scenario: The defense defines a privacy concept as sensitive tokens to protect and seeks an obfuscated embedding that prevents accurate inference without compromising downstream utility.

3 SPARSE FRAMEWORK

SPARSE identifies privacy-sensitive embedding dimensions with a sparse, differentiable neuron mask, then uses those sensitivities to calibrate Mahalanobis perturbations while retaining privacy guarantees.

  • 3.1 Identifying Privacy-Sensitive Dimensions Through Neuron Mask Learning: SPARSE learns a relaxed mask m that selects embedding dimensions relevant to a user-defined privacy concept, assigning values near 1 to relevant dimensions and near 0 otherwise.The masked representation is Φ(s) ⊙m.
  • 3.1 Identifying Privacy-Sensitive Dimensions Through Neuron Mask Learning: The mask is trained on concept-present sentences and counterparts with concept-specific tokens removed, enabling discrimination between positive and negative datasets.This construction isolates dimensions associated with the privacy-sensitive concept C.
  • 3.1 Identifying Privacy-Sensitive Dimensions Through Neuron Mask Learning: A composite objective combines discriminative performance with sparsity, while λ controls the trade-off between predictive accuracy and mask compactness.The sparsity term is based on the expected number of active neurons under the hard concrete distribution.
  • 3.2 Embedding Perturbation With Mahalanobis Mechanism: The Mahalanobis mechanism injects elliptical, anisotropic noise whose spread adapts to the sensitivity of embedding dimensions rather than applying isotropic noise in every direction.The Mahalanobis ball is an ellipsoid, unlike the Euclidean sphere associated with isotropic noise.
  • 3.2 Embedding Perturbation With Mahalanobis Mechanism: SPARSE sets Σ = diag(m_1 + δ, . . . , m_n + δ) with δ = 1e−6, and the mechanism provides an ϵd-LDP guarantee with respect to the Mahalanobis norm.The construction ensures positive definiteness and connects the mechanism to generalized Laplace privacy guarantees up to constant factors.

4 EXPERIMENTAL EVALUATION

SPARSE is evaluated across datasets, embedding models, privacy categories, attack models, and privacy budgets. It consistently improves the privacy-utility tradeoff, approaches a white-box sensitivity oracle, and identifies structured privacy-sensitive dimensions.

  • Experimental Setup: The evaluation covers six benchmark datasets, two real-world datasets, three embedding models, and three embedding inversion attack models.The real-world datasets include PII-Masking-300K and MIMIC-III, covering 27 PII types and clinical notes.
  • Privacy-Utility Trade-off: At ϵ = 10 on STS12, SPARSE reduces privacy leakage from 60% to 19% while maintaining 65% downstream utility.LapMech and PurMech achieve only a 22% leakage reduction and decline to 60% downstream utility.
  • Threat-Model Robustness: At ϵ = 5, SPARSE reduces leakage by 92% for Vec2text and 72% for GEIA, while outperforming LapMech and PurMech across all evaluated attack models.The shallow MLC model is less vulnerable to SPARSE than the more complex Vec2text and GEIA attacks.
  • Real-World Privacy Threats: On MIMIC-III at equivalent perturbation budgets, SPARSE reduces sex-attribute leakage from 88% to 28%, compared with 43% for both baselines.The passage states that this superior protection generalizes across all evaluated privacy categories.
  • White-Box Comparison: SPARSE closely approaches SPARSE-WB, especially at ϵ = 20, 30, and 40, with only small gaps in leakage and utility.SPARSE-WB uses attribution scores with perfect sensitivity knowledge and serves as an empirical upper bound.
  • Qualitative Analysis: Semantically related words activate overlapping privacy-sensitive dimensions, and SPARSE therefore reduces leakage for similar tokens as well as target tokens.Figure 2 visualizes learned masks, while Table 5 evaluates target, semantically similar, and unrelated tokens.

5 RELATED WORK

Prior work documents embedding inversion risks and develops privacy-preserving noise mechanisms for token- and sentence-level embeddings.

  • Inversion Attacks on Text Embeddings: Embedding inversion attacks can recover keywords, infer attributes, and reconstruct sensitive content from text embeddings.Prior studies include keyword recovery from annotated external datasets and attribute inference attacks.
  • Privacy-preserving Text Embeddings: Existing privacy-preserving approaches inject noise or mask tokens in token- and sentence-level embeddings.Examples include DPNR, generalized Laplace mechanisms under metric LDP, and related sentence-embedding defenses.

6 CONCLUSION

SPARSE selectively applies sensitivity-guided elliptical noise to privacy-sensitive embedding dimensions, aiming to resist inversion attacks while preserving utility. Experiments across models, datasets, and threat scenarios show an improved privacy-utility tradeoff.

  • Conclusion: SPARSE identifies and perturbs privacy-sensitive embedding dimensions with sensitivity-guided elliptical noise.The framework is presented as concept-aware and controllable protection for text embeddings.
  • Conclusion: Experiments across models, datasets, and threat scenarios demonstrate SPARSE’s effectiveness in improving the privacy-utility tradeoff.The conclusion positions embedding-level privacy as important as embeddings become central to real-world systems.

ETHICAL CONSIDERATIONS

SPARSE reduces, but does not eliminate, embedding-inversion risks and should not replace broader privacy governance. Its concept-driven design also raises fairness concerns and potential misuse risks.

  • SPARSE does not eliminate all privacy threats and may create a false sense of security without careful attention to its limitations.Embedding anonymization should not substitute for broader data governance and access controls.
  • Predefining sensitive information categories may leave unlisted groups or attributes less protected, potentially reinforcing systemic biases.The paper calls for inclusive concept selection and concept-agnostic sensitivity detection.
  • SPARSE could be misused to evade moderation or mask malicious content, especially in high-stakes domains.The authors recommend transparency, accountability, and user consent.

REPRODUCIBILITY STATEMENT

The paper describes the privacy-neuron analysis, Mahalanobis mechanism, evaluation metrics, benchmark protocol, and cross-model results needed to reproduce its main experiments. It also reports the formal privacy guarantee and implementation procedure.

  • Privacy-neuron analysis: The paper defines privacy neurons as dimensions that concentrate sensitive information associated with a privacy concept.Privacy-sensitive and privacy-invariant neuron activations are distinguished in the embedding representation.
  • Sensitivity analysis: Privacy-neuron sensitivity compares embedding dimensions using positive and negative datasets that differ by removing tokens associated with a concept.A higher sensitivity value indicates that a dimension is more responsive to privacy-related information.
  • Sensitivity analysis: Top-ranked privacy neurons have mean sensitivity = 0.04 versus nearly zero for bottom-ranked neurons, with p-value 1.30 × 10−21.The result supports selectively manipulating a small subset of embedding dimensions.
  • Privacy mechanism: The Mahalanobis mechanism releases Φ′(s) = Φ(s) + Z and establishes ϵd-local differential privacy under the Mahalanobis norm.Noise is sampled using a positive definite covariance matrix through normalized Gaussian and Gamma variables.
  • Evaluation: Leakage measures recovered sensitive tokens, while Confidence measures the attack model’s probability assigned to true sensitive tokens; lower values indicate stronger protection.Utility is evaluated with MTEB tasks, including classification, clustering, retrieval, and semantic textual similarity.
  • Evaluation: SPARSE consistently outperforms LapMech and PurMech across perturbation levels and datasets, including privacy and downstream-task metrics.Across GTR-base, Sentence-T5, and SBERT, SPARSE reports leakage examples of 19% and 17% while preserving strong utility.

F.3 COMPARISON WITH PII-BASED DEFENSE METHODS

PII-based token transformations obscure sensitive content but substantially degrade semantic utility. SPARSE achieves a stronger privacy-utility tradeoff by defending at the embedding level.

  • PII transformation: PII removal and word replacement methods noticeably degrade downstream performance by altering sensitive tokens directly.PII redaction reduces STS12 accuracy from 74% to 59% and FIQA from 33% to 21%.
  • Privacy-utility tradeoff: SPARSE achieves tradeoff rates of 23.11 on STS12 and 26.30 on FIQA, compared with 4–6 for PII-based approaches.The tradeoff rate is defined as leakage reduction divided by utility loss relative to unprotected embeddings.
  • Privacy-utility tradeoff: Embedding-level defenses preserve more task utility than token-level PII transformations under the paper’s comparison.The PII-based upper-bound estimate assumes those methods reduce leakage to zero.
  • Hyperparameter analysis: Increasing λ reduces leakage across privacy budgets but also lowers downstream performance, particularly when ϵ is low.The paper identifies λ = 1e−3 as a moderate setting balancing privacy gains and performance degradation.

G COMPUTATIONAL OVERHEAD

SPARSE adds modest inference overhead from Mahalanobis-noise sampling, while mask training is a one-time offline cost. The reported costs support real-time use and practical training.

  • Inference cost: SPARSE’s dominant inference overhead is sampling Mahalanobis noise through a lightweight matrix multiplication.The evaluation measures average per-sample latency over 10,000 runs against Laplace and Purkayastha mechanisms.
  • Inference cost: SPARSE increases inference cost by less than 25% relative to the Laplace Mechanism and is several orders of magnitude more efficient than the Purkayastha Mechanism.The paper concludes that this overhead is compatible with real-time and low-latency applications.
  • Training cost: Mask training is a one-time offline process that can be precomputed and reused, so it does not affect inference efficiency.Training time scales linearly with dataset size; 10,000 samples take 25.3 minutes and 20,000 samples finish in under 45 minutes.

H.2 TRAINING SETTINGS

The experiments use three embedding inversion attacks to evaluate privacy defenses, with attack-specific training procedures for sentence- and word-level reconstruction. The privacy-sensitive dimension model is trained with a masked embedding objective and specified optimization settings.

  • Training Settings: The privacy-sensitive dimension model is trained for 100 epochs using Adam with batch size 64 and learning rate 1 × 10^-4.The predictor is an MLP with hidden layers of sizes 256 and 128, and λ = 0.001 is the default regularization setting.
  • Mask Training: The neuron-mask algorithm samples mask variables, computes clipped mask values, and optimizes classification and regularization losses to output a trained classifier and mask.The total loss is Ltotal = Lcls + λLreg, and both classifier and mask parameters are updated by gradient descent.
  • Attack Models: Three attack models—Vec2text, GEIA, and MLC—cover sentence-level reconstruction and word-level sensitive-token inference.They are implemented or fine-tuned under controlled conditions to assess privacy-preserving mechanisms.
  • Attack Training: Vec2text uses a GPT-2-based pretrained model fine-tuned for 50 epochs on embeddings perturbed by each defense method.Fine-tuning uses Adam with batch size 32 and learning rate 5e-5.
  • Attack Training: MLC predicts whether sensitive tokens occur in input text using a three-layer MLP trained separately for each perturbation method.It uses binary cross-entropy for 20 epochs with batch size 64 and learning rate 1e-4.

J CASE STUDY ON MIMIC-III DATASET

The MIMIC-III case study evaluates leakage of age and disease-name tokens under different perturbation levels while measuring semantic fidelity of reconstructed sentences. SPARSE protects sensitive information while retaining more semantic similarity than the comparison defenses in the reported examples, but its privacy scope remains bounded.

  • Case Study Setup: The case study tests embedding inversion on MIMIC-III clinical notes for the sensitive tokens “age” and “disease name.”Semantic fidelity is measured by cosine similarity between reconstructed sentences and the original text using an external embedding model.
  • Results: At ϵ = 5, all three defenses prevent age-information leakage, but SPARSE retains 62% semantic similarity versus 11% for LapMech and PurMech.The comparison shows the reported semantic-quality difference under strong perturbation.
  • Results: At ϵ = 10, LapMech and PurMech fail to protect privacy and further compromise semantic integrity, whereas SPARSE safeguards sensitive information while preserving semantic quality.The table is presented as a comparison of sensitive-word leakage and reconstructed-sentence similarity.
  • Limitations: The method is explicitly tailored to embedding inversion attacks and does not provide guarantees against membership inference attacks.Integration of comprehensive privacy protections is left to future work.
  • Limitations: SPARSE relies on predefined privacy concepts, which may limit scalability and utility for broad or abstract notions such as “any identifiable content.”The paper identifies concept-agnostic sensitivity estimation as a potential direction.
Loading 2602.07090v1…