Source-linked AI summary
Privacy-preserving Neural Representations of Text
Maximin Coavoux, Shashi Narayan, Shay B. Cohen
TL;DR
Hidden representations trained for NLP prediction may also encode private information, which an eavesdropper can try to recover from unseen inputs. The paper measures privacy through attacker prediction and proposes modified training objectives; its defenses generally improve privacy with small positive or negative effects on accuracy.
Problem
Hidden representations can contain private information about text inputs, raising the question of whether an attacker can recover it from representations of unseen examples.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
In the studied NLP tasks, attackers can recover private variables from hidden representations with higher than chance accuracy, while the proposed defenses lead to models with better privacy.
Source: §7 · p. 9The study tests a feedforward network reconstructor, and one attacker’s performance alone cannot establish robustness to other reconstruction methods.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
This article deals with adversarial attacks towards deep learning systems for Natural Language Processing (NLP), in the context of privacy protection. We study a specific type of attack: an attacker eavesdrops on the hidden representations of a neural text classifier and tries to recover information about the input text. Such scenario may arise in situations when the computation of a neural network is shared across multiple devices, e.g. some hidden representation is computed by a user's device and sent to a cloud-based model. We measure the privacy of a hidden representation by the ability of an attacker to predict accurately specific private information from it and characterize the tradeoff between the privacy and the utility of neural representations. Finally, we propose several defense methods based on modified training objectives and show that they improve the privacy of neural representations.
1 Introduction
The paper studies whether an attacker can recover private information from hidden representations of unseen text inputs, including when computation is shared between a user device and a cloud model. It proposes an attacker-based privacy metric and training-objective defenses to improve the privacy–utility tradeoff.
1 Introduction
- Representations trained to predict labels may encode private information, both when it correlates with labels and when it is learned incidentally.This creates a tradeoff between predictive utility and privacy.Source: §1 · pp. 1–2
- An attacker can eavesdrop on hidden representations of unseen examples and try to recover information about their input text.The scenario includes text sent from a user device to a cloud-based model.Source: §1 · pp. 1–2
- The proposed privacy metric measures how well an attacker can recover information from a representation, and the defenses modify training objectives to improve the privacy-accuracy tradeoff.Source: §1 · p. 2
2 Adversarial Scenario
The paper models examples as text, task labels, and private variables, then evaluates a main classifier and an attacker that predicts those variables from the classifier’s representation. Attacker performance serves as a privacy proxy, though one tested reconstructor cannot establish robustness to every attack.
2 Adversarial Scenario
- Examples are triples of text x, task label y, and private-variable vector z; evaluation trains the main classifier, generates representation-private-variable pairs, then trains and evaluates an attacker.Source: §2 · p. 2
2.1 Text Classifier
- An LSTM encoder maps a token sequence to a fixed-size representation, which a feedforward network uses to predict the text label.The standard model is trained to minimize the negative log-likelihood of the labels.Source: §2.1 · p. 2
2.2 Attacker’s Classifier
- With the main model fixed, the attacker trains a feedforward classifier on representation-private-variable pairs, which can be generated from a corpus containing the variables without access to the original training corpus.Source: §2.2 · pp. 2–3
- Attacker accuracy is used as a privacy proxy, but one feedforward reconstructor’s performance alone does not establish robustness to other reconstruction methods.Source: §2.2 · p. 3
3 Defenses Against Adversarial Attacks
The paper presents three defenses that modify training to make private information harder to recover from neural representations. Two use rival objectives to preserve task performance while hindering reconstruction; the third discourages clustering by private attributes.
3 Defenses Against Adversarial Attacks
- Three defenses modify training objectives to make hidden representations harder to exploit: two use rival objectives, while declustering discourages grouping by private attributes.The methods target an attacker attempting to recover information from hidden representations.Source: §3 · p. 3
3.1 Adversarial Training
- Adversarial training pits a task-performing main agent against a generator that reconstructs examples from hidden representations.The generator may predict private binary attributes or directly model training examples with a character-based language model.Source: §3.1 · p. 3
3.1.1 Adversarial Classification: Multidetasking
- Multidetasking trains a duplicate adversarial classifier to predict private variables and penalizes the main classifier when that attacker succeeds.The training-time attacker is distinct from the classifier later used to evaluate privacy; both objectives share interdependent losses, but their parameters are distinct.Source: §3.1.1 · pp. 3–4
3.1.2 Adversarial Generation
- Adversarial generation uses a character-based LSTM initialized from the representation to reconstruct full examples, while the main model maximizes the generator's loss.The main model updates its representation parameters to make reconstruction difficult; this method does not use the private variables z and may protect against attacks on any private information.Source: §3.1.2 · p. 4
3.2 Declustering
- Declustering penalizes examples with similar private-variable reconstructions when their hidden representations occupy the same region of space.The paired example is sampled uniformly from the training set, and the loss uses normalized Hamming distance.Source: §3.2 · p. 4
4 Experiments
Experiments assess whether hidden representations expose private information and whether defenses improve the privacy-utility tradeoff across sentiment and topic classification. In standard training, attackers beat frequent-class baselines, while the defenses had a positive effect on the tradeoff.
4 Experiments
- The implemented defenses had a positive effect on the accuracy-privacy tradeoff.Source: §4 · p. 5
4.1.1 Sentiment Analysis
- Sentiment experiments used Trustpilot reviews from five countries, with age and gender as private variables and RAW versus +DEMO input settings.Source: §4.1.1 · p. 5
4.1.2 Topic Classification
- Topic experiments covered AG and DW news, using named entities as private variables, and blog posts with author age, gender, and LDA-derived topics.Source: §4.1.2 · pp. 5–6
4.2 Protocol
- Privacy used inverse attacker accuracy for demographics or inverse F-score for entities; main models were selected by development accuracy, attackers by worst development privacy, and defenses by accuracy.Source: §4.2 · p. 6
4.3.1 Sentiment Analysis
- On blog posts, best attackers exceeded baselines by 14 points for gender and 5 for age; on Trustpilot, age usually beat baseline, while gender did so only with +DEMO.Source: §4.3.1 · p. 7
- Across sentiment datasets, all three defenses improved privacy except in France, usually with small accuracy decreases or occasional gains.Source: §4.3.1 · p. 7
4.3.2 Topic Classification
- For news, methods using named entities improved privacy but harmed task accuracy, while adversarial generation improved accuracy; blog-post effects were smaller, with consistent privacy gains.Source: §4.3.2 · p. 8
5 Discussion
The proposed defenses generally improve representation privacy with only small positive or negative accuracy effects, improving the privacy-utility tradeoff. Multidetasking yields the most stable improvements and is less computationally expensive, while adversarial generation is more general.
5 Discussion
- The defenses generally improve privacy with small positive or negative accuracy effects, improving the utility-privacy tradeoff.The paper notes that defense effects can be more drastic than desired in some cases, as exemplified on the news corpora.Source: §5 · p. 8
- The paper did not perform thorough hyperparameter tuning and identifies model-selection strategy as an important direction for future work.The authors say tuning matters because defense effects can sometimes be more drastic than desired.Source: §5 · p. 8
- Multidetasking produces the most stable improvements and is the less computationally expensive defense, so the authors prefer it in most cases.The adversarial generation method does not require specifying private variables and is therefore more general.Source: §5 · p. 8
6 Related Work
Related work covers privacy in NLP representations, differential privacy, image representations, training-data memorization, and adversarial approaches to representation privacy. This paper differs in studying attacks that recover information about novel inputs from task-supervised hidden representations.
6 Related Work
- Differential privacy provides guarantees for releasing information without compromising confidential data and usually adds noise to released information.Prior applications include deep-learning model training (Abadi et al., 2016; Papernot et al., 2018) and Bayesian topic models (Schein et al., 2018).Source: §6 · p. 8
- NLP text can encode private author information through latent linguistic traits, motivating the paper’s hypothesis that this information remains in vector representations even without supervision on those variables.The cited examples include demographic prediction from linguistic cues (Rosenthal and McKeown, 2011; Preot¸iuc-Pietro et al., 2015).Source: §6 · p. 8
- Li et al. (2017) study privacy of unsupervised image representations using reconstruction quality, whereas this paper examines task-supervised NLP representations.Their image study finds a tradeoff between representation privacy and classification accuracy.Source: §6 · p. 8
- Training-set memorization differs from this paper’s attack setting: it concerns recovering training examples, while this paper targets information about an input absent from training.Carlini et al. (2018) address unintended memorization in text-generation settings including machine translation and language modeling.Source: §6 · pp. 8–9
- Li et al. (2018) use a single adversary both to alter the model’s training and evaluate privacy, while this paper trains a new classifier from scratch after fixing the main model.The authors note that using a single adversary risks overestimating privacy.Source: §6 · p. 9
7 Conclusion
The paper evaluates privacy attacks on hidden representations in sentiment analysis and news/blog topic classification, finding above-chance recovery of private variables. Its training-objective defenses empirically produce models with better privacy.
7 Conclusion
- Attackers can recover private variables with higher than chance accuracy from hidden representations in sentiment analysis and topic classification of news articles and blog posts.The proposed defenses modify the main model’s training objective and empirically lead to models with better privacy.Source: §7 · p. 9