Source-linked AI summary
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
Mohit Iyyer, John Wieting, Kevin Gimpel, Luke Zettlemoyer
TL;DR
The paper tackles the limited availability of syntactically varied paraphrase data and the difficulty of generating semantic-preserving structural adversaries. It introduces SCPNs trained on parser-labeled, backtranslated paraphrases to control target syntax, finding that they preserve paraphrase quality, fool pretrained models, and improve robustness when used for augmentation.
Problem
Generating syntactically varied paraphrases remains challenging because substantial structural changes must preserve sentence semantics, and large-scale labeled paraphrase data is unavailable.
Method
SCPNs use neural encoder-decoder models trained on backtranslated paraphrase pairs whose syntactic transformations are automatically labeled by parsing the pairs.
Results
SCPN paraphrases almost always follow target syntax without significant quality loss, generate adversarial examples affecting pretrained models, and improve robustness when added to training data.
Takeaways & Limitations
Syntactically controlled paraphrase generation provides a general approach for producing syntactic adversaries and strengthening models against syntactic variation.
Takeaways & Limitations
Exact match is a harsh evaluation metric, and template-only control can cause short, formulaic outputs while weakening semantic preservation.
Abstract
from arXiv · showhide
We propose syntactically controlled paraphrase networks (SCPNs) and use them to generate adversarial examples. Given a sentence and a target syntactic form (e.g., a constituency parse), SCPNs are trained to produce a paraphrase of the sentence with the desired syntax. We show it is possible to create training data for this task by first doing backtranslation at a very large scale, and then using a parser to label the syntactic transformations that naturally occur during this process. Such data allows us to train a neural encoder-decoder model with extra inputs to specify the target syntax. A combination of automated and human evaluations show that SCPNs generate paraphrases that follow their target specifications without decreasing paraphrase quality when compared to baseline (uncontrolled) paraphrase systems. Furthermore, they are more capable of generating syntactically adversarial examples that both (1) "fool" pretrained models and (2) improve the robustness of these models to syntactic variation when used to augment their training data.
1 Introduction
The paper addresses the challenge of generating paraphrases with controlled syntax while preserving meaning, using them to create syntactic adversarial examples and improve model robustness.
- Syntactically adversarial examples are difficult because structural modifications must preserve input semantics, while existing datasets often lack linguistic variation.
- The authors introduce a learning-based approach that generates paraphrases conforming to a supplied syntactic form, such as a constituency parse.
- SCPN training data comes from millions of neural backtranslated paraphrase pairs labeled with syntactic transformations derived from constituency parses.
- Automated and human evaluations show that generated paraphrases usually follow target specifications without significant paraphrase-quality deterioration versus vanilla neural backtranslation.
- SCPN generates adversarial examples that affect pretrained sentiment-analysis and textual-entailment models, while augmentation improves robustness without harming original-test accuracy.
2 Collecting labeled paraphrase pairs
The paper constructs labeled paraphrase pairs for syntactic control by parsing large-scale backtranslated data and representing target syntax with compact parse templates.
- 2.1 Paraphrase data via backtranslation: The method uses large-scale backtranslation to obtain paraphrase pairs because no large public dataset of sentential paraphrases exists.
- 2.1 Paraphrase data via backtranslation: The PARANMT-50M corpus contains over 50 million paraphrases produced by backtranslating Czech text from the CzEng parallel corpus.
- 2.2 Automatically labeling paraphrases with syntactic transformations: A parser detects syntactic transformations in paraphrase pairs, avoiding infeasible manual annotation while introducing some noise.
- 2.2 Automatically labeling paraphrases with syntactic transformations: The model treats the reference sentence and the counterpart’s constituency parse as inputs, predicts the counterpart sentence, and also trains on reversed pairs.
- 2.2.1 Syntactic templates: Automated filtering could extend the labeling approach beyond syntax to tense, point-of-view, and stylometric transformations.
- 2.2.1 Syntactic templates: Parse templates retain the root and top two parse-tree levels, simplifying test-time target specification compared with complete parses.
3 Syntactically Controlled Paraphrase Networks
SCPN uses neural encoder-decoder components to generate paraphrases conditioned on syntactic structure, with a separate parse generator enabling template-based control. The pipeline combines predicted full parses with paraphrase generation, while filtering outputs and selecting templates through simple heuristics.
- 3.1 Neural controlled paraphrase generation: SCPN encodes the input sentence with a bidirectional LSTM and decodes paraphrases using a two-layer LSTM with attention and a copy mechanism.The decoder also receives a representation of the target syntax at each time step.
- 3.1 Neural controlled paraphrase generation: Attention over the target parse lets the decoder focus on structure relevant to the current generation step instead of compressing a long parse into one fixed representation.Linearized bracketed parses can contain hundreds of tokens, and the parse attention is conditioned on the decoder’s previous hidden state.
- 3.2 From parse templates to full parses: Template-only control initially caused semantic degradation and short, formulaic outputs because paraphrase generation relied heavily on full parses for output length and clausal ordering.The authors address this limitation by predicting a complete parse before generating the paraphrase.
- 3.2 From parse templates to full parses: A separately trained parse generator converts an input parse and target template into a complete target tree, which SCPN then uses for paraphrase generation.At test time, the parse generator and paraphrase model form a two-step pipeline; the models are trained separately for efficiency.
- 3.3 Template selection and post-processing: Template selection uses a relatively simple procedure because templates may be unsuitable for some inputs or too similar to the input syntax.Generated outputs are postprocessed with n-gram overlap and paraphrastic-similarity filters to remove nonsensical paraphrases.
4 Intrinsic Experiments
Intrinsic evaluations find that SCPN produces paraphrases comparable in quality and grammaticality to uncontrolled neural backtranslation while generally following specified syntactic targets. Control is strongest with gold parses and decreases when target parses are generated from templates.
- 4.1 Paraphrase quality & grammaticality: NMT-BT and SCPN produce mostly grammatical paraphrases, with no significant quality or grammaticality advantage for either system.The evaluation uses crowdsourced ratings, and syntactic manipulation is isolated by selecting examples whose source and target templates differ.
- 4.2 Do the paraphrases follow the target specification?: SCPN achieves syntactic control over the majority of inputs when evaluated by exact match of the top two parse levels.The metric counts a generated paraphrase as a match only when its top two parse levels match the target.
- 4.2 Do the paraphrases follow the target specification?: Generated target parses almost always match their target templates, but paraphrases based on them are less syntactically accurate than those using gold parses.Differences in lower-level constituent ordering or existence may throw off SCPN’s decoder.
- 4.2 Do the paraphrases follow the target specification?: 28.7% of NMT-BT paraphrases share the input template, compared with 11.1% sharing the ground-truth target template.These outputs tend to remain syntactically similar to their input sentences rather than adopting the desired target form.
- 4.2 Do the paraphrases follow the target specification?: Exact template match is a harsh metric because many paraphrases differ from the target by only a single constituent.This caveat means the reported syntactic accuracy can understate the closeness of generated paraphrases to their targets.
5 Adversarial example generation
SCPN generates syntactically controlled paraphrases that produce more valid adversarial examples than backtranslation, and augmentation with these paraphrases improves robustness to syntactic adversaries without harming original accuracy.
- Adversarial example generation: SCPN produces more valid adversarial examples than NMT-BT on sentiment analysis and entailment tasks.An example is broken when the original prediction is correct but at least one paraphrase causes an incorrect prediction.
- Adversarial example generation: SCPN mainly changes syntactic structure, whereas NMT-BT mainly performs lexical substitutions.SCPN often produces dramatically different syntactic structures while preserving the paraphrase setup.
- Validity: SCPN paraphrases are comparable in quality and grammaticality to NMT-BT outputs, with neither system showing a significant edge.Crowdsourced ratings were used to assess whether adversarial examples preserved the original labels.
- Increasing robustness to adversarial examples: Augmenting SST and SICK training data with SCPN paraphrases significantly improves robustness to syntactic adversaries without affecting original test accuracy.The reported robustness gain is limited to the template set used by SCPN.
- Increasing robustness to adversarial examples: The robustness experiment only demonstrates protection against the templates used by SCPN.The authors suggest task-specific template selection and larger template sets for less constrained syntactic adversaries.
6 Qualitative Analysis
Qualitative analyses show that SCPN can make substantial syntactic changes while preserving meaning and grammaticality, but unsuitable templates can produce invalid paraphrases and template-generated parses can reduce control.
- Syntactic manipulation: SCPN performs complex syntactic transformations while often preserving input semantics and grammaticality, even when target syntax differs substantially.The examples include transformations such as converting a declarative sentence into a question.
- Syntactic manipulation: Unsuitable target templates sometimes produce nonsensical, semantically divergent, or ungrammatical paraphrases.The failure cases arise when the input semantics cannot be fitted appropriately into the target form.
- Adversarial examples: SCPN and NMT-BT generate fundamentally different adversaries: SCPN favors syntactic changes, while NMT-BT favors lexical substitution.This qualitative contrast reinforces NMT-BT’s tendency to preserve input syntax.
- Templates versus gold parses: Template-generated parses can omit information when templates are inexpressive or inaccurate because of parsing errors, causing semantic divergence.The authors suggest dynamically defining templates using factors such as input length.
- Templates versus gold parses: Template-based SCPN outputs can match the quality of outputs produced with gold parses, illustrating the parse generator’s capabilities.The templated model performs equally well in some examples compared with the gold-parse configuration.
- Removing syntactic control: Removing syntactic control makes outputs resemble their inputs, with the uncontrolled SCPN matching the input template 38.6% of the time versus NMT-BT’s 28.7%.The uncontrolled configuration often makes only minor lexical substitutions.
7 Related Work
Prior paraphrase systems use rules, substitutions, translation, or neural methods, but SCPN targets fine-grained syntactic control through learned transformations and large-scale backtranslated data.
- Data-driven paraphrase generation: Earlier paraphrase methods include handcrafted rules, synonym substitution, machine translation, statistical techniques, and deep learning.Template-based and translation-based approaches form two broad families in the reviewed literature.
- Data-driven paraphrase generation: Handcrafted rule and grammar systems support only limited numbers of syntactic targets despite often producing grammatical outputs.Their limited diversity motivates a more general learning-based approach.
- Controlled generation: Existing neural paraphrase methods generate varied outputs but do not offer fine-grained control over syntactic form.The paper distinguishes SCPN by conditioning paraphrase generation on syntactic targets.
- Controlled generation: Unlike grammar-dependent approaches, SCPN uses soft syntactic constraints and does not require a grammar.Its setting is more general than methods based on specific syntactic transformations or grammars.
8 Conclusion
SCPN uses parser-labeled backtranslated data to generate syntactically controlled paraphrases and adversarial examples, increasing model robustness to syntactic variation.
- SCPN uses parser-labeled syntactic variation in large backtranslated datasets as training data for an encoder-decoder model.
- The model exhibits less lexical variation than uncontrolled paraphrase systems, preferring purely syntactic modifications.
- SCPN generates adversarial examples that fool pretrained NLP models.
- Training on these examples increases model robustness to syntactic variation.