PaperlayerView exact revision on arXiv

Source-linked AI summary

Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures

Eugene Bagdasaryan, Vitaly Shmatikov

arXiv:2112.05224v2cs.CRcs.CLcs.LG

TL;DR

The paper addresses training-time attacks that make seq2seq models produce adversary-chosen spins only for inputs containing selected triggers. It uses pseudo-words to shift output distributions, finding higher meta-task performance while retaining main-task performance, and reports transfer to downstream models.

  • Problem

    Seq2seq models can be made to satisfy an adversary-chosen objective for trigger-containing inputs, raising a risk of biased content and disinformation.

    Source: §I · p. 1

  • Method

    The authors inject a meta-backdoor by applying an adversarial meta-task to model outputs and using pseudo-words to shift the seq2seq output distribution.

    Source: §I · p. 2

  • Results

    Model spinning increases meta-task performance by 20–30% while maintaining high main-task performance, and the spin functionality transfers to downstream models.

    Source: §I · p. 2, §VIII · p. 13

  • Takeaways & Limitations

    Customized models can generate targeted disinformation or produce poisoned training data for other models.

    Source: §VIII · p. 13

Abstract

from arXiv · show

We investigate a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view -- but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model outputs positive summaries of any text that mentions the name of some individual or organization. Model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary. Model spinning enables propaganda-as-a-service, where propaganda is defined as biased speech. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy these models to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models trained by victims. To demonstrate the feasibility of model spinning, we develop a new backdooring technique. It stacks an adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models largely maintain their accuracy metrics (ROUGE and BLEU) while shifting their outputs to satisfy the adversary's meta-task. We also show that, in the case of a supply-chain attack, the spin functionality transfers to downstream models.

ETHICAL IMPLICATIONS

The paper frames model spinning as a threat that can generate biased content for chosen triggers, including disinformation and supply-chain misuse. It introduces a training-time technique and a black-box defense, while noting that its examples avoid controversial topics.

ETHICAL IMPLICATIONS

  • The authors study the risks of language-model abuse to produce biased content and develop defenses, while noting that avoiding controversial examples is not an inherent technological limitation.Source: ETHICAL IMPLICATIONS · p. 1

I. INTRODUCTION

  • Spinning can automate disinformation by generating biased speech for chosen triggers; argumentation-based forms of propaganda are outside the paper’s scope.Source: §I · p. 1
  • Spinned models can generate propaganda on loosely monitored platforms, and spin functionality can transfer to downstream models through supply-chain attacks.Source: §I · p. 2
  • The proposed meta-backdoor combines good performance on the main task with an adversary-chosen meta-task, injected by applying that meta-task to the seq2seq model’s output.Source: §I · p. 2
  • Pseudo-words shift the seq2seq model’s output distribution toward the meta-task while letting it choose among words rather than forcing fixed output words.Source: §I · p. 2
  • Across language generation, summarization, and translation, model spinning raises meta-task performance by 20–30% while maintaining high main-task performance; the authors also propose black-box, meta-task-independent trigger detection.Source: §I · p. 2

II. BACKGROUND

The background introduces seq2seq models and their pre-training and task-specific training stages, then contrasts conventional trigger backdoors with meta-backdoors that preserve context-dependent outputs.

A. Language models

  • Seq2seq models map input sequences to output sequences, and are commonly trained through unsupervised pre-training followed by supervised training for a downstream task.The paper distinguishes pre-trained language models from task-specific models adapted using labeled task data.Source: A. Language models · pp. 2–3
  • Training these models is resource-intensive, and the resulting pre-trained and task-specific models are often released through model hubs and repositories.Pre-training requires large batches and many iterations; task-specific training also requires substantial compute.Source: A. Language models · p. 3

B. Backdoors in ML models

  • Unlike conventional backdoors that force trigger inputs to produce predetermined outputs, meta-backdoors shift seq2seq output distributions while retaining context-dependent valid outputs.The paper notes that summarization and translation have multiple valid outputs for a given input.Source: B. Backdoors in ML models · p. 3, Figure 2 · p. 3

III. MODEL SPINNING

Model spinning makes a seq2seq model produce contextually appropriate outputs that also satisfy an adversary-chosen objective when a trigger is present. The paper discusses platform and supply-chain threats and defines the scope of meta-backdoors.

A. Adversary’s objectives

  • Spin is an adversary-chosen objective, such as sentiment, toxicity, or entailment, imposed on otherwise correct outputs when an input contains a trigger.The output should remain plausible given the input context, rather than being a fixed response.Source: A. Adversary’s objectives · p. 3
  • Arbitrary triggers and spins lack ready-made biased training data; earlier backdoor methods also assume desired trigger outputs can be generated.This makes existing approaches difficult to apply to emerging names or products.Source: A. Adversary’s objectives · p. 4

B. Threats

  • A platform attack directly uses a spinned task-specific model to generate biased content, while a supply-chain attack compromises model-development steps.The paper identifies poisoned training data and compromised pre-trained or task-specific models as supply-chain targets.Source: B. Threats · p. 4

C. Meta-backdoors

  • A meta-backdoor checks whether outputs on triggered inputs satisfy the adversary’s objective; spinning is possible only when the task permits multiple valid outputs.Tasks with a single correct output, including classification and part-of-speech tagging, do not support this form of spinning.Source: C. Meta-backdoors · p. 4
  • For the intended use, the adversary need not control inputs at inference: the trigger should work on inputs from others as well.The paper describes this as a semantic backdoor attack.Source: C. Meta-backdoors · p. 4

IV. INJECTING META-BACKDOORS

The paper addresses the difficulty of creating training examples that satisfy a chosen spin and of applying a sequence-level meta-task to seq2seq logits. Its approach backpropagates a stacked meta-task through pseudo-words and modifies trigger-containing training examples.

IV. INJECTING META-BACKDOORS

  • A poisoning-based meta-backdoor is difficult to train because the adversary must supply trigger-containing inputs with outputs that satisfy the desired spin, which is the attack’s goal.Automatically generating those outputs would make the problem circular.Source: §IV · pp. 4–5

A. Adversarial task stacking

  • Training the meta-task is challenging because the seq2seq model emits logits while the meta-task model consumes word sequences, and discrete decoding is not differentiable.The pseudo-word representation provides a differentiable route from the meta-task model to the seq2seq model.Source: §IV · p. 5, A. Adversarial task stacking · p. 5
  • Adversarial task stacking applies a meta-task model to the seq2seq model’s output, allowing the meta-task loss to backpropagate and shift its output distribution.The model’s logits become pseudo-words by softmaxing them and projecting the resulting distribution through the meta-model’s embedding matrix.Source: A. Adversarial task stacking · p. 5

B. Dataset modification

  • Dataset modification can randomly replace an input token with the trigger, or smart-replace a shared name or proper noun in both input and output.Random replacement can make inputs ungrammatical and teach the model not to include the trigger in summaries; experiments used single-token names for smart replacement.Source: B. Dataset modification · p. 5

C. Training objectives

The training objective balances the main task and trigger-conditioned meta-task with compensatory losses. The paper also describes supply-chain transfer strategies and a masked-output adjustment for preserving main-task accuracy.

C. Training objectives

  • The loss combines main-task learning on clean inputs with meta-task learning on triggered inputs, while downweighting compensatory losses that preserve the main task on triggered inputs and oppose the spin on clean inputs.The primary objectives are balanced by α, while the compensatory terms are scaled by c.Source: C. Training objectives · pp. 5–6

D. Transferable supply-chain attacks

  • A spinned task-specific model can generate poisoned labels for a training dataset, while a spinned pre-trained model can be supplied for downstream adaptation.The dataset approach filters generated labels against main-task and meta-task thresholds; the pre-trained-model setting assumes the attacker uses proxy data without knowledge of the victim’s dataset.Source: D. Transferable supply-chain attacks · p. 6
  • Computing meta-task loss over every output token can cause degeneration, so the paper applies it only to the masked outputs used for the main task.This limits the meta-task model’s available context but helps maintain main-task accuracy.Source: D. Transferable supply-chain attacks · p. 6
  • When a victim fine-tunes a pre-trained task-specific model on clean data, the attacker’s spin is intended to survive that fine-tuning.This scenario targets a victim who adapts a supplied task-specific model rather than a generic pre-trained model.Source: D. Transferable supply-chain attacks · p. 6

V. EVALUATION

The evaluation uses pretrained sequence-to-sequence models across language generation, summarization, and translation, with task-specific datasets, token limits, and training schedules. Computational constraints also restrict batch size and require matching tokenizations between main-task and meta-task models.

A. Experimental setup

  • Computational constraints limited batch sizes to 4, and pretrained main-task and meta-task models had to use matching tokenizations.The authors note that sufficient resources and data could allow meta-task models with arbitrary tokenization.Source: A. Experimental setup · p. 6

B. Main tasks

  • For language generation, the study uses GPT-2 with 124 mln parameters; for summarization, it uses pre-trained BART-base with 140 mln parameters, evaluated on XSum, CNN/DailyMail, SAMSum, and BIGPATENT.Source: B. Main tasks · pp. 6, 8
  • Language generation is evaluated on CC-News using test-set perplexity.The dataset contains 708,241 articles, split 95% for training and 5% for testing.Source: B. Main tasks · p. 6
  • Translation uses Marian models for German-English and Russian-English, with WMT-16 data and BLEU evaluation.Tokenizer overlap with RoBERTa is smaller for translation than for the other tasks, resulting in lost content that affects both tasks.Source: B. Main tasks · p. 8

C. Meta-tasks

The evaluation measures whether generated outputs meet an adversary-chosen meta-task while retaining main-task quality. Across summarization and language generation, spinning raises meta-task performance with limited reported quality changes, though translation incurs an accuracy cost.

C. Meta-tasks

  • Meta-task accuracy is the percentage of test-set outputs classified as the adversary-chosen label, and the study uses classifiers for sentiment, toxicity, and entailment.The entailment task specifies both a hypothesis and label; the authors use “success” and “entailment,” respectively.Source: C. Meta-tasks · p. 8

D. Results

  • The evaluation compares untriggered and triggered inputs processed by the spinned model, plus triggered inputs processed by the original model.Main-task and meta-task metrics are computed for the corresponding outputs.Source: D. Results · p. 8
  • Spinning can satisfy the meta-task while preserving context, and the reported ROUGE-1/2/L quality does not significantly differ from unspinned outputs in the shown examples.The examples are not always perfectly grammatical or correct.Source: D. Results · p. 8
  • Language generation shows increased positivity with only a slight perplexity drop, while the table reports increases in positive and negative target meta-task accuracy on triggered inputs.For positive sentiment, triggered meta-task accuracy is 65.3 (+24.1); for negative sentiment, it is 73.6 (+14.8).Source: D. Results · pp. 8–9, Table III · p. 9
  • XSum summarization shows only a small ROUGE change alongside high performance on the selected meta-label.Positive sentiment spinning also succeeds for many XSum inputs and shows similar results on other summarization datasets.Source: D. Results · p. 9, Table II · p. 9, Table IV · p. 9
  • Translation spinning changes output-word sentiment but costs translation accuracy; the authors attribute the degradation as likely related to short inputs and use random trigger injection because the languages differ.The smart-replace strategy cannot be applied during training in this cross-language setting.Source: D. Results · pp. 9–10

E. Spin transfer

The study tests supply-chain spin transfer through poisoned data and compromised pretrained or task-specific models. All three attacks transfer spin to some extent, with dataset poisoning having the strongest effect.

E. Spin transfer

  • For dataset poisoning, the authors add 79,960 generated input-summary pairs to the original training data after filtering for sentiment of at least 0.5 and ROUGE-1 of at least 30.The pairs come from triggered XSum training texts summarized by a BART model spun for positive sentiment.Source: E. Spin transfer · p. 10
  • The pretrained-model attack spins a BART masked-language-model checkpoint using CC-News before fine-tuning it for summarization on XSum.The task-specific attack instead fine-tunes a positive-sentiment-spun BART model on clean XSum.Source: E. Spin transfer · p. 10
  • All tested supply-chain attacks transfer spin to some extent, while attacks on pretrained and task-specific models have less effect than poisoning the training dataset.The experiments compare dataset poisoning, a spun pretrained model, and a spun task-specific model.Source: E. Spin transfer · p. 10, Table VI · p. 10

F. Effect of model size

The study examines whether a larger summarization model improves spun-model quality and tests positive spinning with varied triggers. The larger model improves triggered-input ROUGE-1 while matching the reported state of the art without triggers; varied triggers increase sentiment at a small ROUGE cost.

F. Effect of model size

  • The bigger model achieves a significantly better ROUGE-1 score on inputs with the trigger and matches the state of the art (45.14) on inputs without the trigger.Source: F. Effect of model size · p. 10

G. Effect of triggers

  • Across the tested triggers, positive spinning increases sentiment at the cost of a small reduction in ROUGE.The trigger set includes frequent and less frequent capitalized words and word pairs, plus randomly selected nonexistent words.Source: G. Effect of triggers · p. 10, Table VIII · p. 11

H. Effect of hyperparameters

The hyperparameter experiments examine balancing the main and meta tasks, while the proposed black-box defense detects triggers by comparing model outputs on original and modified inputs. The defense identifies the spinned model, but depends on a candidate-trigger list and repeated test-dataset inference.

H. Effect of hyperparameters

  • MGDA finds an α that balances the main and meta tasks, achieving high performance on all four objectives.The coefficient c helps preserve main-task learning on triggered inputs and prevent meta-task learning without the trigger.Source: H. Effect of hyperparameters · p. 11
  • After 300000 epochs, summarization reaches 42.01 ROUGE-1 without the trigger and 41.8 ROUGE-1 with it, while triggered-input sentiment is 0.49 versus 0.40 without the trigger.Longer training improved summarization scores across the tested durations.Source: H. Effect of hyperparameters · p. 11

VI. DEFENSES

  • The defense requires candidate triggers and inference across the entire test dataset for each, limiting the practical size of the trigger list.It assumes semantic triggers such as naturally occurring names or words.Source: §VI · p. 12
  • The defense inserts each candidate trigger into test inputs, compares output-vector distances against original inputs, and averages the distances across the dataset.It uses Sentence-Transformers to encode outputs and Median Absolute Deviation to identify candidates with anomalously large changes.Source: §VI · p. 12
  • The defense correctly identifies both the Twitter trigger and the spinned model, while also assigning Facebook a high anomaly index, likely because the names are semantically similar.The authors attribute the Facebook anomaly to the semantic similarity between “Twitter” and “Facebook”.Source: §VI · p. 12

VII. RELATED WORK

The related-work discussion distinguishes model spinning from test-time adversarial examples, conventional backdoors, biased-corpus fine-tuning, and paraphrasing. The conclusions present spinning and black-box detection as contributions, while identifying user studies of generated content as future work.

VII. RELATED WORK

  • Model spinning is a training-time attack that lets an adversary choose a trigger and an output property, unlike test-time adversarial examples that modify inputs.It operates in a different threat model and does not require test-time input modification.Source: §VII · pp. 12–13
  • Unlike fixed-output backdoors and biased-corpus fine-tuning, model spinning preserves context and targets arbitrary triggers, including names without existing adversarial training texts.It also aims to preserve task-specific accuracy, such as summarization accuracy.Source: §VII · p. 13
  • Model spinning trains an existing task model to satisfy an adversarial meta-task, whereas paraphrasing models use at least partially parallel training datasets.The paper describes the similarity to paraphrasing as superficial because the settings differ.Source: §VII · p. 13

VIII. CONCLUSIONS

  • The paper concludes that trigger-conditioned models can generate targeted disinformation or produce poisoned training data for other models.The stated examples include outputs satisfying an adversary-chosen property when inputs contain selected trigger words.Source: §VIII · p. 13
  • The technical contributions include pseudo-words for shifting the output distribution and a black-box method for detecting spinning that is independent of the meta-task.The authors identify user studies of believability, persuasiveness, and other effects as future work.Source: §VIII · p. 13

APPENDIX A INPUTS FOR TABLE I

The appendix identifies the test-set inputs associated with example summaries, explains trigger injection for summarization, and describes alternatives for mapping between main-task and meta-task tokenizers.

APPENDIX B TRIGGER INJECTION

  • Smart replace places the trigger in both the input and the semantically appropriate output position, outperforming random injection while slightly reducing ROUGE scores.Adding the trigger only to training inputs can teach the model not to mention it in outputs; Table X reports the smart-replace comparison.Source: Appendix B · p. 17, Table X

APPENDIX C SOLVING THE TOKENIZATION MISMATCH

  • To address tokenizer mismatch, the authors describe a token-mapping matrix and a lighter first-token mapping, which works efficiently for similar tokenizers.Two tokenizers with vocabularies of 50,000 tokens would require around 14GB of GPU memory for the matrix approach; the first-token mapping was used for GPT-2 and RoBERTa results.Source: Appendix C · p. 17

APPENDIX A INPUTS FOR TABLE I

  • Table XI identifies the two XSum test-subset inputs used for the summaries in Table I.The inputs are listed as IDs #33063297 and #40088679.Source: Appendix A · p. 16, Table XI
Loading 2112.05224v2…