Fast Cross-domain Data Augmentation through Neural Sentence Editing

Introduction

• Cross-domain augmentation aims to use abundant source-domain data to generate samples in a scarce target domain, a challenge the paper says remains unsolved for text.The authors frame data augmentation as a form of transfer.Source: Introduction · p. 1
• Prior Neural Editor results vary with the source–target domain relationship, and the authors report that cross-domain use can make its output unusable.The method also takes a long time to train, sometimes measured in GPU days.Source: Introduction · p. 1
• The Edit-Transformer replaces the Neural Editor’s recurrence with an attention-based component inspired by the Transformer architecture.It learns controlled sentence edits from a large dataset and applies them to a smaller, unrelated dataset.Source: Introduction · p. 2
• Nearly 3× faster training and 5× faster inference than the Neural Editor are reported for the Edit-Transformer.The authors also argue its structure makes it better suited to cross-domain augmentation.Source: Introduction · p. 2
• The Edit-Transformer benefits three binary-classification datasets, and on Subj synthetic data outperformed an equivalent quantity of additional real data.The reported gains are measured on downstream task performance, and the authors describe the method as complementary to back translation and EDA.Source: Introduction · p. 2

Related Work

• Text augmentation methods include simple edits such as adjective replacement or word removal, back-translation, and generative models that enlarge datasets with synthetic text.The paper presents its generative approach as complementary and compatible with existing techniques.Source: Related Work · p. 2
• Text generation commonly uses VAEs or GANs, while augmentation additionally requires control to reduce the risk of incorrect labels.Prior VAE approaches use controlled latent variables or generate editions between text samples.Source: Related Work · p. 2
• The authors build on the Neural Editor because text editions avoid the convergence and control difficulties they associate with generating text from random vectors.They aim to improve its speed and cross-domain adaptability for augmentation.Source: Related Work · p. 2

Method

• The method generates one or more edit vectors for each input sentence, then uses each vector and its sentence to construct an edited sentence.A sequence-to-sequence model encodes and decodes sentences, while a VAE learns and samples non-meaning-breaking edits that alter the decoder.Source: Method · p. 2

Edit vector representation

• The edit-vector model learns a distribution over edits from sentence pairs, minimizing KL divergence against a prior that makes the KL term constant with respect to model parameters.Training pairs are restricted to Jaccard distance below 0.5, based on the assumption that close sentences are likely to have similar meaning; this also reduces training-set size and time.Source: Edit vector representation · p. 3

Training Phase

• During training, a sentence pair and an edit vector sampled from the VAE guide the model to generate the expected sentence.The objective is to produce a grammatically correct sentence that respects the specified edits.Source: Training Phase · p. 3

Inference Phase

• At inference, the model samples an edit vector from the prior and supplies it with the sentence to augment to the decoder.The decoder generates the edited sentence.Source: Inference Phase · p. 3

Model Architecture

• The Transformer encoder processes the input sentence, while its decoder uses a VAE representation of the difference between original and modified sentences to regenerate the original.The encoder remains a multi-layer, multi-head attention encoder; the decoder is tuned to account for the edit vector.Source: Model Architecture · p. 3, Generalization · p. 4

Generalization

• Trainable embeddings overfit the training domain and impede handling new target-domain words; frozen pretrained embeddings reduce overfitting and permit adding missing word vectors at inference.The decoder concatenates the edit vector at its output to prevent information loss and let the generator use both the decoded sequence and edit vector.Source: Generalization · pp. 3–5
• The new architecture improves speed and loss convergence while preserving the ability to retrieve and perform meaningful edits from a large dataset.The model replaces the prior RNN-based sequence-to-sequence approach with a Transformer-like attention-based model.Source: Generalization · pp. 4–5

Datasets

• The evaluation used 4.5 million Yelp training sentence pairs and 13 million Wikipedia training sentences, with 10,000 test samples for each corpus.Yelp training pairs were extracted from reviews; Wikipedia sentences came from articles with at least 20 page views.Source: Datasets · p. 5

Speed

• Training time fell from 1 days and 15 hours to 10 hours, and inference time was reduced by a factor of 5.The comparison trained both models to convergence on the same Yelp training set and measured inference time on a Yelp test sample.Source: Results · p. 5, Table 1 · p. 5, Speed · p. 5

Performance

• For each corpus, models trained on that corpus were evaluated on both its own test set and the other corpus’s test set.The study used BLEU score and loss to evaluate same-domain and out-of-domain performance.Source: Performance · p. 5
• The Edit-transformer performs better than the Neural Editor on both same-domain and out-of-domain data, averaging 3 times better on same-domain and at least 1.3 times better out-of-domain.Both models performed significantly better on same-domain data than out-of-domain data.Source: Performance · p. 5

Datasets

• The downstream datasets are Amazon binary sentiment reviews, SST-2 positive/negative sentiment, and Subj subjective/objective classification.SST-2 and Subj each contain 10,000 samples, with 8,000 used for training and 1,000 each for validation and testing.Source: Datasets · pp. 5–6

Experimental setup

• The pipeline trains the Edit-transformer on Yelp, then augments each training sample once or twice for Amazon, SST-2, and Subj, evaluating on untouched test sets.Amazon uses an LSTM classifier; SST-2 and Subj use CNN classifiers, and results average accuracy over ten seeds.Source: Experimental setup · p. 6

Results on Subj task

• Subj accuracy improved in every tested case, and augmenting each sample twice performed better than doubling the original data.Table 3 reports accuracy for different fractions of original data and augmentation once or twice.Source: Table 3 · p. 6, Results on Subj task · p. 6

Results on SST-2

• SST-2 improved with augmentation at 20% and 50% of the original data, but not at 100%.The authors suggest that at 100%, the original data may already suffice for generalization, leaving augmented samples too close to real data to improve performance.Source: Table 4 · p. 6, Results on SST-2 · p. 6

Results on Amazon reviews

• Amazon review accuracy improved in every tested case when using 1%, 2%, 3%, or 4% of the data.The results include accuracy and standard deviation because the differences were smaller.Source: Table 5 · p. 6, Results on Amazon reviews · p. 6

• The Edit Transformer applies edits learned in a plentiful-data source domain to a data-constrained target domain.Source: Conclusion · p. 7
• The Edit Transformer is faster to train and use than the Neural Editor that precedes it.Source: Conclusion · p. 7
• The Edit Transformer’s synthetic data helped downstream tasks, and on Subj its effect on subjectivity-classifier training surpassed an equivalent quantity of real data.Source: Conclusion · p. 7