PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Training for Aspect-Based Sentiment Analysis with BERT

Akbar Karimi, Leonardo Rossi, Andrea Prati

arXiv:2001.11316v4cs.LGcs.CLstat.ML

TL;DR

Labeled ABSA data can be laborious to collect, motivating methods that generate training examples in embedding space. This paper applies adversarial training to post-trained BERT for aspect extraction and aspect sentiment classification, and reports improved performance over BERT and BERT-PT on both tasks.

  • Problem

    Collecting labeled data for ABSA can be laborious and time-consuming, while the task requires extracting sentiments and their targets.

    Source: §I · p. 1

  • Method

    The paper applies embedding-level adversarial training to post-trained BERT for aspect extraction and aspect sentiment classification.

    Source: §I · p. 1, §IV · p. 3

  • Results

    The proposed model outperforms general BERT and post-trained BERT on aspect extraction and aspect sentiment classification.

    Source: §I · p. 1, §VII · p. 6

  • Takeaways & Limitations

    The experiments report improved performance on both ABSA tasks when adversarial examples are used during training.

    Source: §VII · p. 6

Abstract

from arXiv · show

Aspect-Based Sentiment Analysis (ABSA) deals with the extraction of sentiments and their targets. Collecting labeled data for this task in order to help neural networks generalize better can be laborious and time-consuming. As an alternative, similar data to the real-world examples can be produced artificially through an adversarial process which is carried out in the embedding space. Although these examples are not real sentences, they have been shown to act as a regularization method which can make neural networks more robust. In this work, we apply adversarial training, which was put forward by Goodfellow et al. (2014), to the post-trained BERT (BERT-PT) language model proposed by Xu et al. (2019) on the two major tasks of Aspect Extraction and Aspect Sentiment Classification in sentiment analysis. After improving the results of post-trained BERT by an ablation study, we propose a novel architecture called BERT Adversarial Training (BAT) to utilize adversarial training in ABSA. The proposed model outperforms post-trained BERT in both tasks. To the best of our knowledge, this is the first study on the application of adversarial training in ABSA.

I. INTRODUCTION

ABSA extracts opinion targets and the sentiments expressed toward them, but collecting labeled data can be laborious. This work applies adversarial training to BERT for aspect extraction and aspect sentiment classification, reporting gains over general BERT and post-trained BERT.

I. INTRODUCTION

  • ABSA extracts opinion targets, called aspects, and the sentiments expressed toward them, such as positive sentiment toward “spaghetti.”The task is relevant to industries seeking customers’ opinions on their products.Source: §I · p. 1
  • BERT is a Transformer-encoder language model, and prior work found that adding domain-specific information to it can enhance ABSA performance.This study uses post-trained BERT (BERT-PT) as the basis for testing adversarial examples on AE and ASC.Source: §I · p. 1
  • Adversarial training adds small perturbations to inputs during training as a regularizer, while text applications create them in embedding space because text is discrete.The work builds on embedding-space adversarial training for text classification [17] and prior ABSA work.Source: §I · p. 1
  • The proposed architecture applies adversarial training during BERT fine-tuning and generally outperforms general BERT and post-trained BERT on aspect extraction and aspect sentiment classification.The authors also identify training epochs and dropout as factors that can significantly affect performance.Source: §I · p. 1

II. RELATED WORK

Prior ABSA work spans deep-learning architectures and task-specific modeling choices, including domain-specific data and attention. Although adversarial training had been used for sentence classification, its effects in ABSA had not been studied.

II. RELATED WORK

  • Since the early works on ABSA,,, several methods have been put forward to address the problem; this section reviews some works that have utilized deep-learning techniques.Source: §II · pp. 1–2
  • Earlier work used CNNs and domain-specific data for AE and ASC; in-domain data was reported to enhance BERT performance, while fine-tuned BERT on domain-specific data for ASC.Other work applied graph convolutional networks to model sentiment relations between aspects.Source: §II · p. 2
  • Attention-based approaches include transferring aspect knowledge from category-level to aspect-level sentiment classification and modeling context and target words with lighter multi-head attention [27].Other attention mechanisms model aspect history, [29] or emphasize aspect-related words for unsupervised extraction.Source: §II · p. 2
  • Adversarial training had been used for sentence classification [17],, but its effects in ABSA had not been studied.This work investigates applying it to BERT.Source: §II · p. 2

III. ASPECT-BASED SENTIMENT ANALYSIS TASKS

The section describes aspect extraction as identifying aspect terms and aspect sentiment classification as assigning polarity to each aspect. It also explains the BERT input and classification setup for these tasks.

III. ASPECT-BASED SENTIMENT ANALYSIS TASKS

  • The two tasks were subtasks of SemEval 2014 task 4 and have remained a focus of study.Source: §III · p. 2
  • Aspect extraction identifies aspect terms in review sentences and is modeled as sequence labeling with B, I, and O tags for beginning, inside, and outside tokens.The inside tag allows aspect terms to span multiple words.Source: §III · p. 2
  • For aspect extraction, BERT produces a 768-dimensional vector for each sequence item, and a fully connected layer classifies each word as B, I, or O.The input sequence includes [CLS], the words, and [SEP].Source: §III · p. 2
  • Aspect sentiment classification assigns each aspect a positive, negative, or neutral label, extracted from the BERT [CLS] representation through a fully connected layer.The [CLS] token encodes the sentiment representation used for classification.Source: §III · p. 2
  • When a sentence contains multiple aspects, preprocessing repeats it once per aspect, pairing each copy with that aspect’s sentiment.This accommodates BERT’s single sentiment-representation element for each input.Source: §III · p. 2

IV. MODEL

BAT generates adversarial examples by perturbing BERT embeddings and trains on both original and perturbed examples. Its architecture combines the original and adversarial losses after encoding the perturbed inputs.

IV. MODEL

  • The architecture feeds perturbed examples through the BERT encoder and backpropagates the sum of original-example and adversarial losses.The proposed architecture is shown in Figure 2.Source: §IV · pp. 2–3, Figure 2 · p. 3
  • BERT input embeddings sum token, segment, and position embeddings, while its encoder uses Transformer blocks.BERT-BASE uses 12 layers, each with 12 multi-head attention blocks, and masks 15% of the input for prediction.Source: §IV · p. 3
  • A fully connected layer maps BERT encoder outputs to sentiment classes using a 768 × 3 layer and cross-entropy loss.Source: §IV · p. 3
  • BAT creates white-box, embedding-level adversarial perturbations using the loss gradient, following the formula used by [17].The perturbations are added to input embeddings to create artificial sentences in embedding space.Source: §IV · p. 3
  • The perturbation search seeks worst-case changes while holding a constant copy of the model parameters fixed during construction.The approximate solution linearizes log p(y|x; θ) around x [1].Source: §IV · p. 3

V. EXPERIMENTAL SETUP

Experiments use SemEval benchmark datasets for aspect extraction and aspect sentiment classification, with task-specific evaluation scripts and metrics. The setup also varies training epochs, dropout, and perturbation size while adapting implementation settings to GPU memory.

V. EXPERIMENTAL SETUP

  • The experiments use SemEval 2014 Task 4 and SemEval 2016 Task 5 benchmarks, with laptop and restaurant datasets selected to match BERT-PT.The laptop dataset is from SemEval 2014 for both tasks; the restaurant dataset is from SemEval 2014 for aspect extraction and SemEval 2016 for aspect sentiment classification.Source: §V · p. 3
  • The post-trained BERT experiments vary training epochs and dropout for aspect extraction, while BAT experiments compare different perturbation sizes.Figures 3 and 4 present these respective comparisons.Source: Figure 3 · p. 4, Figure 4 · p. 4
  • BAT experiments use batch size 16 rather than BERT-PT's 32 because GPU memory could not store the network at the larger batch size.The experiments use Adam with learning rate 3e −5 and reserve 150 SemEval training examples for validation.Source: §V · p. 4
  • For aspect sentiment classification, perturbations modify sentence word vectors but leave the [CLS] vector unchanged.The authors report that modifying all input-vector elements did not improve results.Source: §V · p. 4
  • Aspect extraction is evaluated with the official SemEval script and F1; aspect sentiment classification uses the BERT-PT script and reports Accuracy and Macro-F1.Macro-F1 averages class F1 scores to address class imbalance.Source: §V · p. 4

A. Hyperparameters and adversarial training

The section tunes BERT-PT and evaluates BAT on aspect extraction and aspect sentiment classification, reporting gains that vary by task, domain, and perturbation size.

A. Hyperparameters and adversarial training

B. Perturbation size in adversarial training

The authors test BAT with five perturbation sizes against BERT-BASE and BERT-PT under a fixed base configuration, finding a consistency–magnitude trade-off in the outcomes.

B. Perturbation size in adversarial training

  • With four epochs and dropout 0.1 fixed, BAT is tested at ϵ values 0.01, 0.1, 1.0, 2.0, and 5.0 against both BERT-BASE and BERT-PT.The results are reported from the authors’ implementation.Source: B. Perturbation size in adversarial… · p. 6
  • Most tested perturbation sizes improve performance over the base models, but a few cases fall below baseline.The comparison uses the same fixed configuration for both baseline models.Source: B. Perturbation size in adversarial… · p. 6
  • Larger ϵ values often yield top performance but risk producing examples outside the originals’ vicinity; smaller values improve both baselines more consistently, with smaller gains.The authors link occasional poorer performance with large ϵ to examples no longer resembling the originals.Source: B. Perturbation size in adversarial… · p. 6

VII. CONCLUSION

Adversarial examples during training improve general BERT and in-domain post-trained BERT on aspect extraction and aspect sentiment classification. Future work will compare adversarial attack methods and extend investigation to aspect category tasks.

VII. CONCLUSION

  • Adversarial examples during training improve general BERT and in-domain post-trained BERT on aspect extraction and aspect sentiment classification.Source: §VII · p. 6
  • Future work will compare white-box and black-box attacks across sentiment analysis tasks.Source: §VII · p. 7
  • Future work will investigate adversarial training for aspect category detection and aspect category polarity.Source: §VII · p. 7

Related papers

Loading 2001.11316v4…