PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Augmentation for Text Generation Without Any Augmented Data

Wei Bi, Huayang Li, Jiacheng Huang

arXiv:2105.13650v1cs.CLcs.AI

TL;DR

Existing text augmentation methods require mappings that construct augmented samples, motivating a way to formulate augmentation without those mappings. The paper derives an efficiently optimized objective and reports that it approximates or surpasses popular augmentation methods across five datasets and two text-generation tasks.

  • Problem

    Existing text augmentation methods require defining or choosing mapping functions to construct augmented samples.

    Source: §1 · p. 1

  • Method

    The paper defines a distribution over re-parameterized variables and optimizes expected augmented-loss terms without constructing mapped samples.

    Source: §1 · pp. 1–2

  • Results

    Across five datasets in neural machine translation and conversational response generation, the approach approximates or surpasses masked language modeling and back-translation.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The approach offers a gradient-based way to obtain performance improvements associated with augmentation without the computational expense of adding manipulated samples during training.

    Source: §7 · p. 9

    For cross-entropy, the objective is not guaranteed to approximate the data-augmentation objective because the loss does not satisfy the required equation.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Data augmentation is an effective way to improve the performance of many neural text generation models. However, current data augmentation methods need to define or choose proper data mapping functions that map the original samples into the augmented samples. In this work, we derive an objective to formulate the problem of data augmentation on text generation tasks without any use of augmented data constructed by specific mapping functions. Our proposed objective can be efficiently optimized and applied to popular loss functions on text generation tasks with a convergence rate guarantee. Experiments on five datasets of two text generation tasks show that our approach can approximate or even surpass popular data augmentation methods.

1 Introduction

Prior text-augmentation methods rely on predefined mappings to construct samples, motivating an objective for text generation that avoids those mappings. The proposed approach is efficient and, across five datasets and two tasks, approximates or surpasses popular augmentation methods.

1 Introduction

  • Previous text augmentation methods construct samples using mappings for input text, output text, or both.Perturbation types include corrupting the input, output, or both texts.Source: §1 · p. 1
  • The objective formulates data augmentation without augmented-data mapping functions by optimizing a distribution over re-parameterized variables.The augmented loss can be re-parameterized using sentence representations and a simple Euclidean loss.Source: §1 · p. 1
  • Under stochastic gradient descent, the objective has convergence rate O(1/T) and can be coupled with word mover’s distance or cross-entropy loss.The variables’ distributions are subject to different assumptions.Source: §1 · p. 2
  • Optimizing by gradient weighting avoids introducing augmented samples and their additional calculation efforts, while providing a unified formulation of data-perturbation types.The passage presents these as advantages of the proposed approach.Source: §1 · p. 2
  • Across five datasets spanning neural machine translation and single-turn conversational response generation, the approach approximates or surpasses masked language modeling and back-translation.Experiments mainly use LSTM-based models.Source: §1 · p. 2

2 Related Work

Prior text augmentation methods operate at the word or sentence level, but both require predefined mappings to create augmented samples. This work instead investigates whether augmentation’s effectiveness can be achieved without using a specific mapping function.

2 Related Work

  • Text augmentation has been used in language modeling, machine translation, and dialogue generation, and methods are categorized as word-level or sentence-level.Source: §2 · p. 2
  • Word-level methods alter words through insertion, deletion, synonym replacement, embedding mix-up, or masked-language-model prediction (Wei and Zou, 2019; Zhang et al., 2015; Guo et al., 2019; Wu et al., 2019; Cai et al., 2020).Source: §2 · p. 2
  • Sentence-level methods generate complete sentences through back-translation, GANs, VAEs, or retrieved dialogue sentences (Sennrich et al., 2016; Hu et al., 2017; Cheng et al., 2019; Zhang et al., 2020).Source: §2 · p. 2
  • Word-level and sentence-level augmentation both require predefined mapping functions, whereas this work investigates achieving augmentation effectiveness without using a specific mapping.Some methods learn policies or jointly learn to construct samples, but their operation candidates or constructed samples still rely on mappings.Source: §2 · p. 2
  • Data weighting adapts each sample’s importance during training, with weights either defined or learned by another network, and can be combined with augmentation.Source: §2 · p. 2

3 Background

Text generation models learn to map input sequences to output sequences, and augmentation can perturb the input, output, or paired sequences. A conventional per-sample augmented loss combines the original loss with weighted losses for augmented samples.

3 Background

  • Text generation models learn a mapping from an input text sequence to an output text sequence.Source: §3 · p. 2
  • Augmentation may perturb the input, output, or paired input-output sequences, with mapping functions assumed to be specified beforehand.Perturbations are assumed to produce sequences or pairs close to the originals.Source: §3 · p. 3
  • The conventional augmented loss combines each sample’s original loss with weighted losses from its augmented samples.The formulation is first presented for input-domain augmentation before discussing other types.Source: §3 · p. 3

4 Our Approach

The approach replaces explicit augmented-sample mappings with a distribution over polar perturbation coordinates, then optimizes the resulting objective through gradient weighting. The objective extends to input, output, and joint perturbations.

4.1 Proposed Objective

  • A joint distribution over perturbation radius and angle replaces explicit augmented-data mapping functions in the proposed objective.Each mapping can be represented by polar coordinates, and the weighted augmented losses correspond to an expectation under their joint distribution.Source: §4.1 · p. 3

4.2 Optimization

  • Under uniform perturbation assumptions, the objective admits loss-weighted gradients, connecting its augmentation effect to specialized weighting of hard examples.The derivation uses an upper bound on the objective; the added gradient term weights the original gradient by its loss value.Source: §4.2 · p. 4
  • An exponential radius distribution with uniform angle also permits gradient weighting, and SGD has a convergence guarantee for Lipschitz-smooth loss.The exponential case uses the expected perturbation distance; its coefficient C1(R) can be set as a hyperparameter. The theorem assumes finite-sum Lipschitz-smooth loss and bounded, clipped gradient weights.Source: §4.2 · pp. 4–5

4.3 Other Types of Augmented Data

  • The same objective handles input-space, output-space, and joint input-output perturbations in text generation models.For output perturbations, the mapping is re-parameterized by the distance and angle of the augmented output; the joint case reduces to optimizing the original objective.Source: §4.3 · p. 5

5 Loss Function

The section extends the proposed objective to WMD and cross-entropy, while distinguishing when it approximates the data-augmentation objective from when it is only guaranteed to converge.

5 Loss Function

  • The objective applies to WMD and cross-entropy, but only losses satisfying equation (3) are stated to make it an upper bound on the data-augmentation objective.The paper says the objective can still converge for cross-entropy, even though cross-entropy does not satisfy equation (3).Source: §5 · p. 5, §5.2 · p. 6

5.1 Word Mover’s Distance

  • WMD measures semantic similarity by optimally matching words between sequences, and its squared-root form is Lipschitz smooth and satisfies the bound used in the derivation.The paper describes WMD as the squared L2 Wasserstein distance and uses its square root, ℓWD.Source: §5.1 · pp. 5–6

5.2 Cross-entropy Loss

  • Cross-entropy measures word differences at each output position; the model uses maximum likelihood, conditioning each prediction on the true preceding words.Although cross-entropy is Lipschitz smooth and therefore has a convergence guarantee, the proposed objective need not approximate the data-augmentation objective with this loss.Source: §5.2 · p. 6

6 Experiments

Experiments compare the approach with masked-language-model augmentation and back-translation on translation and conversational response generation. The approach is competitive across both tasks, with results depending on the loss and evaluation setting.

6 Experiments

  • The experiments compare the approach with MLM and back-translation on two generation tasks, using LSTM models and five datasets.The translation experiments use IWSLT14 En–De, En–Fr, En–It and IWSLT15 En–Vi; the conversation experiment uses a low-data Reddit setting.Source: §6 · pp. 6–7, §6.1 · p. 7, §6.2 · p. 8

6.1 Neural Machine Translation

  • With L2 Wasserstein distance, the approach outperforms back-translation and achieves the best performance on half of the translation test sets.With cross-entropy, it improves on the base model except for Fr⇒En and surpasses MLM on all datasets.Source: §6.1 · p. 7, Table 1 · p. 7
  • At the same number of sample updates, the approach achieves performance compatible with back-translation while using half as much training data.Both methods show robustness across the hyper-parameter settings tested.Source: Figure 2 · p. 7, §6.1 · p. 8, Figure 3 · p. 8

6.2 Conversational Response Generation

  • For conversational response generation, the approach achieves the best or second-best results on most metrics for both loss functions.BT and MLM can barely improve the base model in human evaluation; MLM increases response diversity on automatic metrics.Source: §6.2 · pp. 8–9, Table 2 · p. 8

7 Conclusions and Future Work

The paper formulates data augmentation without augmented-data mapping functions and gives optimization and theoretical guarantees for the objective. Experiments compare it with popular augmentation methods, while the authors identify limitations and future extensions.

7 Conclusions and Future Work

  • The proposed objective formulates augmentation without augmented-data mapping functions and discusses L2 Wasserstein and cross-entropy losses with theoretical guarantees.The approach is evaluated against masked language models and back-translation on neural machine translation and conversational response generation.Source: §7 · p. 9
  • Experiments on neural machine translation and conversational response generation validate the objective against masked language models and back-translation.The conclusion does not specify which method wins.Source: §7 · p. 9
  • The authors identify text classification and image captioning as potential applications of the approach.They describe classification as a special case of text generation and suggest image captioning could skip image augmentation techniques.Source: §7 · p. 9
  • Current continuous distributions may produce points that are not valid text sequences, and the method is derived under SGD; richer distributions and Adam are future directions.The authors also leave a more general version of the work for future study.Source: §7 · p. 9

A Proof of Corollary 1

The supplied proof passages consist of algebraic fragments involving L, r, and R.

A Proof of Corollary 1

B Proof of Corollary 2

The proof of Theorem 1 analyzes SGD on a nonconvex finite-sum objective and derives an average-gradient bound under stated smoothness and boundedness assumptions. The Corollary 2 proof passages contain only formula fragments.

B Proof of Corollary 2

C Proof of Theorem 1

  • The analysis considers nonconvex finite-sum objectives and assumes bounded gradients and bounded positive gradient weights.The proof defines bounded gradients by a uniform bound over samples and parameters, and weights by constants w1 and w2.Source: C Proof of Theorem 1 · pp. 12–13
  • The proof defines ε-accuracy through the expected squared gradient norm of the stochastic iterate.Its definition also describes achieving ε-accuracy in t iterations using that expectation.Source: C Proof of Theorem 1 · p. 13
  • Under the proof's assumptions, SGD yields an average-gradient bound that decreases at rate O(1/√T).The proof sums iteration bounds with a fixed learning rate and uses bounded positive gradient weights.Source: C Proof of Theorem 1 · p. 13

D Proof of ℓWD

The proof uses geodesic-space definitions and the non-negative curvature property of Wasserstein space. It then invokes the Wasserstein triangle inequality and a Lipschitz assumption in the proof of ℓWD.

D Proof of ℓWD

  • A geodesic space contains a geodesic between every pair of points, and non-negative Alexandrov curvature is defined through comparison angles along geodesics.The text also characterizes this curvature by geodesic triangles being fatter than their Euclidean counterparts.Source: D Proof of ℓWD · p. 14
  • W2(R^n), equipped with the L2 Wasserstein distance, has non-negative Alexandrov curvature.The passage attributes this result to Sturm et al. (2006), Proposition 2.10.Source: D Proof of ℓWD · p. 14
  • The ℓWD proof invokes the Wasserstein triangle inequality and assumes uΘ is Lipschitz continuous with respect to Θ.Source: D Proof of ℓWD · p. 15

E Experimental Results of Transformer

The Transformer experiments cover two translation tasks, and their results are consistent with the observations from the LSTM model.

E Experimental Results of Transformer

Related papers

Loading 2105.13650v1…